Intermediate workflow result handling
Workflows contain a series of tasks, most of which produce data. Some data elements produced during a workflow execution are intermediate results: they are needed downstream analysis steps but are then deleted when the analysis successfully completes. Where these intermediate results should be saved is controlled using the Intermediate workflow result handling options (figure 6.13):
- In the location final analysis results are stored Intermediate results are stored in a dedicated subfolder of the folder that is selected to store outputs when the workflow is launched.
- In a temporary directory Intermediate results are stored in a temporary directory on the system the job is executed on, i.e. the single server, job node or grid node. This is usually faster than storing intermediate results under the output folder, particularly if the output folder is on a network drive. Note that if this option is selected, there must be enough temporary space to hold all the intermediate results on the systems that workflows are executed on.
Figure 6.1: Intermediate workflow result handling options.