Configuring a job node setup
Before starting the configuration of a job node setup, please ensure that:
- The CLC Server software has been installed on the system that will act as the master and a license file has been downloaded and installed.
We recommend that adding CLC Server File System Locations, installing plugins, and other making other modifications on the master node, are done after job nodes are attached. In this way, the job nodes will inherit the configurations made on the master node.
- The CLC Server software has been installed and is running on systems that will be attached as job nodes.
Do not install license files on the job nodes or configure settings on the job nodes. The licensing information and other settings are taken from the master.
Configuring the master node
When logged in to the master node as a user in the admin group, go to:
Configuration (
) | Job processing (
)
Open the Server settings section.
Under Server setup configure the following:
- Server mode Select
Master node: Managing processingfrom the list of server modes. - CPU limit Enter the maximum number of CPU the CLC Server should use. This is set to unlimited by default, meaning that up to all cores of the system can be used.
- Fairness factor When a job node has finished a job, it takes the first job in the queue of a type the node is configured to process. Depending on how the system is configured, the job that is first in the queue will not necessarily be the next job processed.
The Fairness factor value is the number of times that a job in the queue can be overtaken by other jobs before resources are reserved for it to run. With the default value of 10, a job could be overtaken by 10 others before resources are reserved for it that will allow it to run. A fairness factor of 0 means that the job at the head of the queue will not be overtaken by other jobs. See the information below about concurrent job processing, where the connection between job types and the fairness factor setting is described.
To save changes to the Fairness factor value, click on the Save Configuration button under that setting.
Reminder: certain nodes can be reserved for use by only certain tools. See Controlling access to the server, server tasks and external data.
- Master node host - Enter the hostname that execution nodes should use to contact the master node. To see a list of host addresses and host names reported by the master node, click on the text "More suggestions".
- Master node port - Enter the port that execution nodes should use to contact the master node. This is usually 7777. The listening port of the master may be changed (see Changing the listening port). Communication between a master and execution nodes uses HTTP by default. Configuring communication using HTTPS is described in SSL and encryption.
The Maximum number of concurrent jobs is disabled for master nodes, as it is not relevant for master nodes.
Save the configuration by clicking on the Save Configuration button.
Attaching job nodes
Before attaching a job node, the CLC Server software must be installed and running on the node. No license should be downloaded to job nodes, and no server configuration is done directly on the nodes.
Configuration of job nodes is done purely via the master node. The rest of this section assumes you are logged into the web administrative interface on the master node as an administrative user.
Figure 7.1: When configuring a job node setup, the master node is configured first, using settings in the "Server mode" section. Job nodes can then be attached by clicking on the "Attach Node..." button in the "Execution node settings" section.
If the Attach Node button in the "Job nodes settings area" section is greyed out, ensure that the server mode selected in the "Server setup" area is Master node: Managing processing and that you have clicked on the Save Configuration button.
Attach and configure job nodes by doing the following in the "Job node settings" area under "Execution node settings":
- Click on the Attach Node button to attach and configure a new job node (figure 7.1).
- Enter information about the node in the dialog that is presented (figure 7.2):
- Host The host name of the job node.
- Port The port the job node can be contacted on. This is usually port 7777 for HTTP communication or port 8443 for HTTPS communication.
- Use HTTPS for master -> node communication Check this box to indicate that communication from the master node to the job node should be encrypted. This option, and the HTTPS configuration... and Check connection.... buttons are described further in Master-job node encrypted communication.
- Displayname The name to show at the top of the web client for that node.
- CPU limit The maximum number of CPUs the CLC Server should use. This is set to unlimited by default, meaning that up to all cores of the system can be used.
- Maximum number of concurrent jobs Limit the maximum number of jobs that are allowed to run concurrently on the single server. Further information about this setting is provided below.
- Manage Job Types... (Optional) Specify the types of jobs that can be run on the node. By default, all tools can run on the node. This equates to selecting the "Any installed Server Command" option in the dialog. When the "Only selected Server Commands" option is selected, a list of tools appears (figure 7.3). Select the tools that can be run on the job node. Enter text in the search field at the top to narrow down the list. Click on the Modify button to save this setting.
- Click on Create to save the job node settings.
Repeat this process for each job node. Once set up, job nodes automatically inherit all configurations on the master node.
Figure 7.3: Select Server Commands to run on the job node.
A warning dialog is presented if there are types of jobs that are not enabled on any of the nodes.
For information about encrypting communication between master and job nodes, see Master-job node encrypted communication.
Checking the job node setup
To test that access works for both job nodes and the master node, click on the "check setup" link in the upper right corner of the web page (see Check set-up).
Troubleshooting job node setups
- Disable root squashing Root squashing often needs to be disabled because it prevents the servers from writing and accessing the files as the same user. Read more about this at https://nfs.sourceforge.net/#faq_b11.
- Bringing job nodes back in sync with the master It is expected the job nodes will stay in sync with the master. However, if one of the job nodes gets out of sync, click on the Resync job nodes button, which can be seen in figure 7.1.
Before resyncing, ensure no jobs are running on any nodes. We recommend using Maintenance Mode for this, as this allows current jobs to complete but stops further jobs from being submitted. Once all the running jobs have completed, maintenance tasks can be safely carried out. Maintenance Mode is described further in section Server maintenance.
Resyncing nodes will detach and then reattach all job nodes, and includes a full reinstallation of all installed plugins from the master as well as reapplication of all settings on the master to each job node. When resyncing is complete, restart your setup. This can be done using the Restart option under Server maintenance area, as described in section Server maintenance.