Configure grid presets

The details of the communication between the master server and the grid when submitting a job is configured using grid presets. The users selects a preset when starting the job as explained in Starting grid jobs.

To configure the presets, log into the web-interface of the CLC Science Server on your master machine and navigate through these tabs in the web administrative interface:

        Admin (Image preferences) | Job distribution (Image jobnodes)

Choose the Grid Presets section and click the Create New Preset button.

Image gridpresets
Figure 6.15: Configuring presets.

For each preset, the following information can be set:

Preset name
The name of the preset as it will be presented in the Workbench when starting a job (see Starting grid jobs) and as you will be able to refer to it when using the Command Line Tools. Alphanumeric characters can be used, and hyphens are fine within, but not at the start of preset names.
Native library path
The full path to the grid-specific DRMAA library.
Shared work directory
The path to a directory that can be accessed by both the CLC Server and the Grid Workers. Temporary directories are created within this area during each job run. These temporary directories hold files used for communication between the CLC Server and Grid Worker.
Path to CLC Grid Worker
This field should contain the path to a directory on a shared file system that is readable from all execution hosts.

The CLC Grid Worker, along with associated settings files, is extracted from the installation area of the CLC Science Server software, and is then deployed to this location when you save your grid preset, or whenever you update plugins on your system.

If this directory does not exist, it will be created.

In versions of CLC Science Serverearlier than 5.0, this path needed to point at the clcgridworker script itself. To support backwards compatibility with existing setups, we ask that you do not use the name "clcgridworker" for a directory you wish your CLC Grid Worker to be deployed to.

Job category
The name of the job category - a parameter passed to the underlying grid system.

Grid mode
"Legacy" or "Resource Aware". Legacy mode is default when migrating an existing pre-6.5 server to 6.5 or later. Resource aware allows the grid preset to use the Shared native specification to submit jobs that do not utilize an entire execution node.

Native specification
List of native parameters to pass to the grid e.g. associations to specific grid queues or memory requirements (see below). Clicking on the f(x) next to Native Specification pops up a box allowing the insertion of particular variables into the Native Specification box. This is described further .

Shared native specification
Native specification used for jobs that can share the execution node with other jobs. This is described further .

Below are examples of OGE-specific arguments one might provide in the native specification field of a Grid Preset. Please see your grid scheduling documentation to determine what options are available for your scheduling system.

Example 1: To redirect standard output and error output, you might put the following in the Native specification field: 

    -o <path/to/standard_out> -e <path/to/error_out>
This corresponds to the following qsub command being generated: 
    qsub my_script -o <path/to/standard_out> -e <path/to/error_out>

Example 2: Use a specific OGE queue for all jobs. 

    -hard -l qname=<name_of_queue>
This corresponds to the following qsub command: 
    qsub my_script -q queue_name

f(x) - adding variables evaluated at run-time
Grid Presets are esentially static in nature, with most options being defined directly in the preset itself. In some cases though, it may be of interest to have variables that are evaluated at runtime. Currently, five such variables can be added to the Native Specification line:

USER_NAME
The name of the user who is logged into the server and is submitting the analysis request. All grid jobs are submitted by the user that runs the CLC Server process, so this variable might be added to, for example, log usage statistics for actual users of the system, or to send an email to the an email account of a form that includes the contents of this variable. For example, the type of text that follows could be put into the Native specification field: 
    -M {USER_NAME}@yourmailserver.com
COMMAND_NAME
The name of the CLC Science Server command to be executed on the grid by the clcgridworker executable.
COMMAND_ID
The ID of the CLC Science Server command to be executed on the grid.
COMMAND_THREAD_MIN
Evaluates to the minimum number of thread required to run the command being submitted. Only valid in Shared native specification.
COMMAND_THREAD_MAX
Evaluates to the maximum number of threads supported by the command being submitted. Only valid in Shared native specification.

These variables can be added by the administrator directly into the Native Specification or Shared Native Specification box, by surrounding the variable name with curly brackets. Alternatively, to ensure the proper syntax, you can click on the f(x) link and choose the variable to insert.

These variables can be used by the administrator in any way that fits with the native grid system, and that does not cause clashes with the way the CLC Server and Grid Workers communicate. For example, in cases where grid usage is closely monitored, it may be desirable to include the user name of the analysis job in metadata reports, so that computer resource time can be logged. Another example might be to set an option such as -q {COMMAND_NAME} if there were, for example, certain commands to be submitted to queues of the same name as the commands.

Besides the above-mentioned variables, two functions exist for use in native specifications. Functions are invoked with the following syntax: {#function arg1, arg2, [... argn]}:

take_lower_of
Evaluates to the lowest integer-value of its argument. The arguments are either a constant integer or a variable-name. If an argument is a a string, which is not a variable-name, or if the variable expands to a non-integer, the argument is ignored. For instance {#take_lower_of 8,4,FOO} evaluates to 4 and ignores the non-integer, non-variable "FOO" string.

take_higher_of
Evaluates to the highest integer-value of its argument. The arguments are either a constant integer or a variable-name. If an argument is a a string, which is not a variable-name, or if the variable expands to a non-integer, the argument is ignored. For instance {#take_higher_of 8,4,FOO} evaluates to 8 and ignores the non-integer, non-variable "FOO" string.