Multi-job processing on grid

Certain types of CLC Server jobs, known as "non-exclusive" jobs, can be scheduled to run concurrently on the same grid node when appropriate. Non-exclusive jobs are those that have reasonably low demands for system resources. A list of such jobs is provided in the Appendix, Non-exclusive Algorithms.

There are two ways non-exclusive jobs can be configured to run concurrently on a grid node:

  1. In the context of a workflow executed on a single grid node
    When the "Single entity" job queueing option is chosen, as described in Job queuing options, all tasks that are part of a given workflow run are executed on a single grid node. If a workflow design includes parallel non-exclusive tasks, these can run concurrently on the grid node. By default, up to 10 such non-exclusive jobs can be run concurrently. This value can be changed in the "Maximum number of concurrent jobs" field, available in the "Server setup" area of the Job Distribution tab. That field is visible when the server mode "Single server" has been selected, as shown in figure 6.7. The value configured is passed through to the CLC Server queue of the grid worker.

    Image singleserversettings
    Figure 6.7: The Maximum number of concurrent jobs setting is visible when the Single server mode is selected.

    A grid setup can be run with the Single server or Master server mode set. If you prefer to have the Master server option set, then to alter the "Maximum number of concurrent jobs" setting, change the mode to Single server, set the desired value, and then change the mode back to Single server. This this will, however, have the downside that the "Maximum number of concurrent jobs" value will not be obvious when reviewing the CLC Server configuration.

    Limitation: The maximum value that can be entered in the "Maximum number of concurrent jobs" field is the number of cores on the master server. Setting that maximum value is the equivalent of checking the unlimited box. If this value is smaller than the desired value, as might happen when grid nodes have many more cpu than the master server, then we recommend leaving this field blank so that the default value of 10 is used.

    Licensing note: When using the "Single entity" job queuing option, only a single CLC Grid Worker is launched for a given workflow run. Thus, irrespective of the number of concurrently running jobs in such a workflow run, only a single license is used. For further details on the "Single entity" job queuing option, see Job queuing options.

  2. In cases other than workflows executed on a single grid node
    To support the concurrent execution of non-exclusive jobs submitted to the CLC Server outside workflows, or within workflows when the "Classic" job queuing option has been selected, information about the CPU or thread requirements of these jobs must be passed to the grid scheduler.

    Non-exclusive algorithms expose their CPU or thread usage, and this information can be passed on to the grid scheduler via the COMMAND_THREAD_MIN and COMMAND_THREAD_MAX variables in the Shared native specification of a grid preset. The variable COMMAND_THREAD_MAX would be used alone as an argument when a single value should be specified, or both the variables COMMAND_THREAD_MIN and COMMAND_THREAD_MAX can be provided when a range is required. An example of specifying a range is shown in the image of a grid present in Configure and save grid presets.

    One can also use the functions take_lower_of and take_higher_of for settings relevant to configuring multiple job processing. For example, to specify 4 as the maximum number of cores to be used by a non-exclusive job, the following could be used as the argument to the relevant parameter in the Shared native specification of a grid preset: {#take_lower_of COMMAND_THREAD_MAX, 4}. As the non-exclusive job passes on its thread usage requirements via the COMMAND_THREAD_MAX variable, this evaluates to 4 if that requirement is higher than 4 or the value specified by the job if is lower than 4.

    Further details about grid preset configuration, including Shared native specifications, functions and variables, can be found in Configure grid presets.

    Licensing note: Each CLC Grid Worker launched, whether it is to run alone on a node or run alongside a job already running on a particular node, will attempt to get a license from the CLC License Server. Once the job is complete, the license will be returned.