Advanced workflow batching
Fine-tuned control of the execution of whole workflows or sections of workflows can be achieved using metadata describing the relationships between particular samples and using control flow elements in a workflow design. Complex analysis goals that can be met in a straightforward manner include:
- Grouping the data into different subsets to be analyzed together in particular sections of a workflow. Groupings of data can be used in the following ways:
- Different groupings of data are used as inputs to different sections of the same workflow.
For details, see Batching part of a workflow and Multiple levels of batching.
- Different workflow inputs follow different paths through parts of a workflow. Based on metadata, samples can be distributed into groups to follow different analysis paths in some workflow sections, at the same time as processing them individually and identically through other sections of the same workflow.
Configuring Collect and Distribute elements is central to the design of this workflow. This is described in Control flow elements. Running such workflows is described in Running part of a workflow multiple times.
- Different groupings of data are used as inputs to different sections of the same workflow.
For details, see Batching part of a workflow and Multiple levels of batching.
- Matching particular workflow inputs for each workflow run. Where more than one input to a workflow changes per run, the particular input data to use for each run can be defined using metadata. The simplest case is as described in Batching workflows with more than one input changing per run. However, more complex scenarios, such as when intermediate results should be merged or parts of the workflow should be run multiple times, can also be catered for, as described in Matching up inputs with each other and analyzing them together later in the workflow.
Examples in this section make reference to CLC Genomics Workbench tools and data types commonly analyzed using that software. However, the principles apply equally to workflows created in the CLC Main Workbench.
Subsections