Configuring your setup

The information in this section assumes that on the machine that will act as the master, the CLC Server software is installed, the license has been downloaded (Downloading a License), and the CLC Server has been restarted.

When logged in as a user in the admin group on the master node, go to:

        Configuration (Image configuration_16) | Job processing (Image jobnodes_16)

Open the Server settings section.

Under Server setup configure the following:

Save these configuration changes, and then, under Master and execution node settings configure the following:

Save the configuration by clicking on the Save Configuration button under the Fairness factor setting to register the changes made.

Image masternode_withaddjobnode
Figure 6.1: When configuring a job node setup, the master node is configured, and then job nodes are attached by clicking on the "Attach Node..." button to attach job nodes.

If the Attach Node button in the Job nodes section is greyed out, please ensure that the server mode selected is MASTER_NODE and that you have clicked on the Save Configuration button below the Fairness factor setting.

Before attaching job nodes to the master, the CLC Server software will need to be installed and running on the machines that will act as job nodes. Note: Do not install license files on the job nodes or configure settings on the job nodes. The licensing information and other settings are taken from the master.

Still on the master node, do the following for each job node:

Image selectcommands_onlyselected
Figure 6.3: Select Server Commands to run on the job node.

Repeat this process for each job node and click on the Save Configuration button under the Fairness factor setting when you are done.

A warning dialog is presented if there are types of jobs that are not enabled on any of the nodes.

Once set up, job nodes automatically inherit all configurations on the master node.

To test that access works for both job nodes and the master node, click on the "check setup" link in the upper right corner of the web page (see Check set-up).

Job processing order on job nodes

When a node has finished a job, it takes the first job in the queue of a type the node is configured to process. Depending on how the system is configured, the job that is first in the queue will not necessarily be the next job processed.

The Fairness factor value is the number of times that a job in the queue can be overtaken by other jobs before resources are reserved for it to run. With the default value of 10, a job could be overtaken by 10 others before resources are reserved for it that will allow it to run. A fairness factor of 0 means that the job at the head of the queue will not be overtaken by other jobs. See the information below about concurrent job processing, where the connection between job types and the fairness factor setting is described.

After updating the Fairness factor value, save the configuration by clicking on the Save Configuration button under that setting. This saves all changes made to settings in the "Master and execution node" section.

Concurrent job processing

There are three general categories of tools: non-exclusive, streaming and exclusive, described below. The Maximum number of concurrent jobs setting applies to tools in the non-exclusive and streaming categories only. Tools in the exclusive category are always run alone.

The maximum allowable value for the number of jobs that can be run concurrently is equal to the number of cores on the system. The default value is 10 or the number of cores on the system, whichever is lowest. If a CPU limit has been set, then the default is 10 or that CPU limit value, whichever is lowest.

The minimum value is 1, which equates to disabling multi-job processing.

Tool categories:

While non-exclusive algorithms are generally expected to have low demands on system resources, when working with very large genomes, setting a lower limit for the maximum number of concurrent jobs is worth considering.

See Appendix Non-exclusive Algorithms for a list of CLC Genomics Server algorithms that can be be run alongside others on a given machine.

Fairness factor and concurrent job processing In a situation where there are many non-exclusive jobs and some exclusive jobs being submitted, it is desirable to be able to clear the queue at some point to allow the exclusive job to have a system to itself so it can run. The fairness factor setting is used to determine how many jobs can move ahead of an exclusive job in the queue before the exclusive job will get priority and a system will be reserved for it. The same fairness factor applies to streaming jobs being overtaken in the queue by non-exclusive jobs.

Reminder: certain nodes can be reserved for use by only certain tools. See Controlling access to the server, server tasks and external data.

Troubleshooting job node setups