Choosing a workflow queuing option
In general terms:
- With the Submit individual tasks to any available node option, each task is scheduled individually. Where large numbers of workflows are submitted and each workflow consists of several tasks, it can lead to a large scheduling overhead, with thousands of jobs being placed in the queue. However, on a multi-node system with spare capacity, this option provides the highest potential level of parallelization.
- With the Submit all tasks to a single node option, a single job submission is made for a workflow, regardless of how many tasks that workflow consists of, so scheduling overhead is minimized. However, only non-exclusive jobs within a workflow have the potential be executed in parallel.
- With the Submit tasks in each workflow block to a single node option, the scheduling overhead is low, with a single job submission per workflow block to be executed. Here, iterated workflow blocks have the potential to be executed in parallel on multi-node setups.
Further considerations relating to the choice of workflow queuing option include:
- Workflow blocks and full workflows run on a single node can leverage caching mechanisms, which aids performance.
On systems that frequently run at or close to capacity, the opportunity for gains through concurrent processing of parallel workflow tasks on multiple nodes is much lower than on a system with spare capacity. So here, the performance gains using the Submit all tasks to a single node or Submit tasks in each workflow block to a single node option can outweigh those of submitting tasks separately, even when they are computationally intensive tasks.
- Where resource allocation is a focus, such as where many users are sharing the resources, the Submit all tasks to a single node option may help with resource access by different users.
For example, consider a grid node setup with 20 nodes, where one user submits 15 workflows with 10 tasks in each workflow. If each task is submitted as an individual job, those 150 jobs can be submitted across all 20 nodes. When the next user submits a job, it would be queued behind these 150 jobs. With the Submit all tasks to a single node option, the 15 workflows would have been sent to a maximum of 15 nodes, leaving 5 nodes available for the other user's job. Where workflows do not contain iteration blocks, the Submit tasks in each workflow block to a single node option would have the same effect.
- Blocks or entire workflows are only submitted to nodes capable of running all the included tasks. Where node hardware is not homogeneous or where certain job nodes have been dedicated to running only certain types of analyses, this should be considered when selecting the Submit all tasks to a single node or Submit tasks in each workflow block to a single node options.
- For grid node setups only: Where demand for gridworker licenses exceeds the number available, jobs wait in the queue until a license becomes available. The number of gridworker licenses needed to execute all tasks in a given workflow depends on the workflow queuing option selected.
With the Submit individual tasks to any available node option, the number of gridworker licenses needed equals the number of tasks. When the Submit all tasks to a single node option is selected, only one gridworker license will be needed for the whole workflow. This is also the case wtih the Submit tasks in each workflow block to a single node option when running workflows without an iteration block. For a workflow with one or more iteration blocks, the number of licenses used will be equal to the number of iterations of the blocks within the workflow plus one for each additional, non-iterated block.