Selective download of results
When the amount of data generated by a workflow execution is so large that it is not practical to download all the results at once, individual results can be downloaded using the Workflow Result Metadata table, which is generated for each batch.
In the Cloud Job Search, download the Workflow Result Metadata table by pressing the Download Metadata button.
Each workflow output is shown in a row of the Workflow Result Metadata table. The path to the output in the AWS S3 location is indicated in the column titled "External path", as shown in figure 6.6. When one or more rows in the table are selected, the Find Associated Data button is enabled. Pressing this button opens a new table, where the external references (not yet downloaded) are listed, as well as any results already downloaded into the Navigation Area.
Figure 6.17: The "External path" column in the Workflow Result Metadata table shows the path to the result in AWS S3. Press the Find Associated Data button to open a second table, where any associated data, in the Navigation Area and in the cloud, will be shown.
Individual data elements can be downloaded by selecting the required row in the table, and pressing the Download button. Data elements are placed in the subfolder (if any) specified by the workflow design for the given output, and new folders are created as required. Therefore, we recommend selecting the same folder when downloading different results for the same workflow. This will ensure that the outputs are organized in the same folder structure that they would have been if the workflow had been executed in the CLC Workbench.
When an element is downloaded, it is automatically associated with the relevant row of the Workflow Result Metadata table, and can be found when pressing the Find Associated Data button again, as shown in 6.7. External references will not be removed from the Workflow Result Metadata table by downloading a result, and results can be downloaded any number of times.
Figure 6.18: When the results have been downloaded, they will be automatically associated with the given row of the Workflow Result Metadata table, and can be found by pressing the "Find Associated Data" button again.
A connection to the CLC Genomics Cloud Engine is required to find the Workflow Result Metadata table itself and download it in the Cloud Job Search. However, only an AWS S3 connection is required for downloading the results through the metadata table. That is, by saving the Workflow Result Metadata table, you will be able to download the results from AWS S3 at any later point, as long as the data is still available in AWS S3, even without a connection to the CLC Genomics Cloud Engine, and without using the Cloud Job Search.
Note: Exported data cannot be downloaded selectively. To download exported data, you must use the Download All Results button, see section Downloading all results.