Supported grid scheduling systems

Grid integration in CLC Genomics Server is done using DRMAA. Integration with a grid scheduling system that provides a working DRMAA library should be possible if it also supports limiting the number of grid workers in accordance with the number of licenses.

QIAGEN Bioinformatics tested the following third party scheduling systems UNIVA, PBS Pro and IBM Platform LSF. We have verified that our software runs with these specific versions:

To fully support running the CLC Genomics Server, the scheduling system must provide some means of limiting the number of CLC jobs launched for execution so that when this number exceeds the number of CLC Grid Worker licenses, excess tasks are held in the queue until licenses are released. In LSF and UNIVA for example, the number of simultaneous CLC jobs sent for execution on the cluster can be controlled by configuring a "Consumable Resource". This is described in more detail in Configuring licenses as a consumable resource.

TORQUE from Adaptive Computing is, however, an example of a system that works for submitting CLC jobs, but that cannot be supported because it does not provide a means of limiting the number of CLC jobs. As far as we know, there is no way to limit the number of CLC jobs sent simultaneously to the cluster to match the number of CLC Grid Worker licenses. So, with TORQUE, if you had three Grid Worker licenses, up to three jobs could be run simultaneously. However, if three jobs are already running and you launch a fourth job, then this fourth job will fail because there would be no license available for it. This limitation can be overcome, allowing you to work with systems such as TORQUE, if you control the job submission in some other way so the license number is not exceeded. One possible setup for this is if you have a one-node-runs-one-job setup. You could then set up a queue where jobs are only sent to a certain number of nodes, where that number matches the number of CLC Grid Worker licenses you have.