Configure grid presets
The details of the communication between the master server and the grid when submitting a job is configured using grid presets. The users selects a preset when starting the job as explained in Starting grid jobs.
To configure the presets, log into the web-interface of the CLC Genomics Server on your master machine and navigate through these tabs in the web administrative interface:
Admin (
) | Job distribution (
)
Choose the Grid Presets section and click the Create New Preset button.
Figure 6.15: Configuring presets.
For each preset, the following information can be set:
- Preset name
- The name of the preset as it will be presented in the Workbench when starting a job (see Starting grid jobs) and as you will be able to refer to it when using the Command Line Tools. Alphanumeric characters can be used, and hyphens are fine within, but not at the start of, preset names.
- Native library path
- The full path to the grid-specific DRMAA library.
- Shared work directory
- The path to a directory that can be accessed by both the CLC Server and the Grid Workers. Temporary directories are created within this area during each job run. These temporary directories hold files used for communication between the CLC Server and Grid Worker.
- Path to CLC Grid Worker
- This field should contain the path to a directory on a shared file system that is readable from all execution hosts.
The CLC Grid Worker, along with associated settings files, is extracted from the installation area of the CLC Genomics Server software, and is then deployed to this location when you save your grid preset, or whenever you update plugins on your system.
If this directory does not exist, it will be created.
In versions of the Genomics Server earlier than 5.0, this path needed to point at the clcgridworker script itself. To support backwards compatibility with existing setups, we ask that you do not use the name "clcgridworker" for a directory you wish your CLC Grid Worker to be deployed to.
- Job category
- The name of the job category - a parameter passed to the underlying grid system.
- Native specification
- List of native parameters to pass to the grid e.g. associations to specific grid queues or memory requirements (see below). Clicking on the f(x) next to Native Specification pops up a box allowing the insertion of particular variables into the Native Specification box. This is described further .
Below are examples of OGE-specific arguments one might provide in the native specification field of a Grid Preset. Please see your grid scheduling documentation to determine what options are available for your scheduling system.
Example 1: To redirect standard output and error output, you might put the following in the Native specification field:
-o <path/to/standard_out> -e <path/to/error_out>
This corresponds to the following qsub command being generated:
qsub my_script -o <path/to/standard_out> -e <path/to/error_out>
Example 2: Use a specific OGE queue for all jobs.
-hard -l qname=<name_of_queue>
This corresponds to the following qsub command:
qsub my_script -q queue_name
f(x) - adding variables evaluated at run-time
Grid Presets are esentially static in nature, with most options being defined directly in the preset itself. In some cases though, it may be of interest to have variables that are evaluated at runtime. Currently, three such variables can be added to the Native Specification line:
- USER_NAME
- The name of the user who is logged into the server and is submitting the analysis request. All grid jobs are submitted by the user that runs the CLC Server process, so this variable might be added to, for example, log usage statistics for actual users of the system, or to send an email to the an email account of a form that includes the contents of this variable. For example, the type of text that follows could be put into the Native specification field:
-M {USER_NAME}@yourmailserver.com - COMMAND_NAME
- The name of the CLC Genomics Server command to be executed on the grid by the clcgridworker executable.
- COMMAND_ID
- The ID of the CLC Genomics Server command to be executed on the grid.
These variables can be added by the administrator directly into the Native Specification box, by surrounding the variable name with curly brackets. Alternatively, to ensure the proper syntax, you can click on the f(x) link and choose the variable to insert.
These variables can be used by the administrator in any way that fits with the native grid system, and that does not cause clashes with the way the CLC Server and Grid Workers communicate. For example, in cases where grid usage is closely monitored, it may be desirable to include the user name of the analysis job in metadata reports, so that computer resource time can be logged. Another example might be to set an option such as -q {COMMAND_NAME} if there were, for example, certain commands to be submitted to queues of the same name as the commands.