Configuring input and output elements


Configuring Workflow Input elements

Workflow Input elements are the main element type for bringing data into a workflow. At least one such element must be present in a workflow. By default, when a workflow is launched, the workflow wizard will prompt for data to be selected from the Navigation Area, or for data files to be imported on-the-fly using any compatible importer.

Workflow Input elements support the input of CLC format data, raw NGS data formats, such as fastq and fasta format files, and some other formats. When launching the workflow, data outside CLC locations is selected by choosing the "Select files for import" option. Doing this is referred to as on-the-fly import.

Like other workflow elements, Input elements can be configured to restrict the options available for configuration when launching the workflow. See Basic configuration of workflow elements for more on locking and unlocking element options.

Configuring import options

Selection of input data from the Navigation Area (already imported data) or import of raw data using on-the-fly import can be enabled or disabled in Input elements. (figure 14.39).

When on-the-fly import is enabled, you can choose whether to limit the importers available when the workflow is launched, and you can configure settings for importers that are selected. On-the-fly import options are:

Image configure-workflow-input-element-genomics
Figure 14.39: Workflow Input elements can be configured to limit where data can be selected from and what importers can be used for on-the-fly import.

Where reference data is needed as input to a tool, it can be configured directly in the relevant input channel, or an Input element can be connected to that input channel. Reference data can be preconfigured in a workflow element, so that when launching the workflow, that data is used by default.

Further details about reference data and workflows

Input channels where reference data is expected can have a data element explicitly selected or a "workflow role" can be specified (figure 14.40).

Specifying a workflow role can be useful in workflows requiring various reference data elements (e.g. a reference sequence, annotation tracks, variant tracks, etc.) and where that workflow will be run using different sets of reference data. Workflow roles prevent the need to explicitly specify each reference data element when launching the workflow using different reference data from the previous run. Workflow roles are used in combination with Reference Data Sets, which are managed using the Reference Data Manager (Reference Data Manager).

In a Reference Data Set, a workflow role is defined for each element in that Set (QIAGEN Sets). A workflow role can be assigned to each element of your own data imported to the Reference Data Manager, (Custom Sets).

You can specify both a reference data element and a role for a given input:

Image inputroleconfig
Figure 14.40: A workflow role has been configured in this workflow Input element. When launching this workflow, a Reference Data Set would be prompted for by the wizard. The data element with the specified role in that Reference Data Set would then be used as input.

Image launchingwithwfroles
Figure 14.41: When one or more workflow elements has been configured with a workflow role, you are prompted to select a Reference Set. The elements from that set with the relevent roles are used in the analysis. Here, the option to use default reference data - i.e. the specified elements, is also available. This reflects the fact that this workflow has at least one workflow element configured with both a workflow role and a data element, and there are no locked inputs relying only on a workflow role.

On-the-fly import versus using workflow elements for specific importers

For importing data as the first step of a workflow, on-the-fly importer, as described above, is the most flexible and commonly used option. However, workflow input elements for specific NGS importers are also available.

Examples using each of these options are shown in figure 14.42. How these translate when launching the workflow is shown in figure 14.43. The relative merits of each option are outlined in table 14.1.

Image workflow_input_elements_and_import
Figure 14.42: Raw data can imported as part of a workflow run in 2 ways. Left: Include an Input element. and use on-the-fly import. Right: Use a specific Import element. Here, the Illumina import element was included.

Image workflow_import_on_launch
Figure 14.43: Top: Launching a workflow with an Input element and choosing to select files to import on-the-fly. Bottom: Launching a workfow with a dedicated import element, in this case, an Illumina import element.


Table 14.1: Workflow import methods compared
Functionality Input element Dedicated import element
Running in batch mode Supported.
Check the Batch option in the launch wizard.		 Not supported.
(The Batch option is not visible in the launch wizard).
Iterate elements Supported. Supported.
Choosing an importer when launching Any available importer can be selected when launching. Use of already-imported data is also supported. Workflow authors can specify the importers available when launching. Only data formats relevant for the specific importer can be selected for use.
Configuring import options Options for all importers allowed by the workflow author can be configured, and set to be unlocked or locked. Import options for the specific importer can be configured,and set to be unlocked or locked.
Saving imported elements Not supported.
The elements created during import are not saved.		 Supported.
If an Output element is attached to the Import element, the elements created during import can be saved.


Notes:


Configuring Workflow Output and Export elements

Results generated by a workflow are only saved if the relevant output channel of a workflow element is connected to a Workflow Output element or an Export element. Data sent to output channels without an Output or Export element attached are not saved.

Terminal workflow elements with output channels must have at least one Workflow Output element or Export element connected.

The naming pattern for workflow outputs and exports can be specified by configuring Workflow Output elements and Export elements respectively. To do this, double click on a Workflow Output or Export element, or right-click and select the option Configure.... Naming patterns can be configured in the Custom output name field in the configuration dialog.

The rest of this section is about configuring the Custom output name field, with a focus on the use of placeholders. This information applies to both Workflow Output elements and Export elements. Other configuration settings for Export elements are the same as for export tools, described in Export tool parameters. Placeholders available for export tools, run directly (not via a workflow) are different and are described in export tools section of the manual.

Configuring custom output names

By default, a placeholders is used to specify the name of an output or exported file, as seen in figure 14.44. Placeholders specify a type of information to include in the output name, and are a convenient way to apply a consistent naming pattern. They are replaced by the relevant information when the output is created.

The placeholders available are listed below. Hover the mouse cursor over the Custom output name field in the configuration dialog to see a tooltip containing this list. Text-based forms of the placeholders are not case specific.

You can choose any combination of the placeholders and text, including punctuation, when configuring output or export names. For example, {input}({day}-{month}-{year}), or {2} variant track as shown in figure 14.45. In the latter case, if the first workflow input was named Sample 1, the name of the output generated would be "Sample 1 variant track".

Image workflow_output_name
Figure 14.44: The names that outputs are given can configured. The default naming uses the placeholder {1}, which is a synonym for the placeholder {name}.

Image workflowoutputvarianttrack
Figure 14.45: Providing a custom name for the result.

It is also possible to save workflow outputs and exports into subfolders by using a forward slash character / at the start of the output name definition. For example the custom output name /variants/{name} refers to a folder "variants" that would lie under the location selected for storing the workflow outputs. When defining subfolders for outputs or exports, all later forward slash characters in the configuration, except the last one, will be interpreted as further levels of subfolders. For example, a name like /variants/level2/level3/myoutput would put the data item called myoutput into a folder called level3 within a folder called level2, which itself is inside a folder called variants. The variants folder would be placed under the location selected for storing the workflow outputs. If the folders specified in the configuration do not already exist, they are created.

Note: In some circumstances, outputs from workflow element output channels without a Workflow Output element or an Export element connected may be generated during a workflow run. Such intermediate results are normally deleted automatically after the workflow run completes. If a problem arises such that the workflow does not complete normally, intermediate results may not be deleted and will be in a folder named after the workflow with the word "intermediate" in its name.


Footnotes

...sec:importNGS14.1
Paired read handling for workflows launched in batch mode, or workflows with Iterate elements, is the same as for the importer tools themselves: If the Paired option is checked, files are handled as described in the manual section on NGS importers. In CLC Genomics Workbench21.x, this was also the case in most circumstances. However, if batch units were based on data organization and all files were in the same folder, each file was treated as a separate batch unit irrespective of whether the Paired option was checked.