Understanding memory settings
Most work done by the CLC Server is done via its java process. However, some tools use external native binaries for the computational phases. These include those with a de novo assembly or mapping phases and those involving BLAST tools.
Java process
Each grid preset can have its own memory limit for the java process. This can be useful where separate queues are used for low overheads tasks, such as import jobs and trimming jobs, and high overhead tasks, such as de novo assemblies or read mappings.
Unless a memory limit is explicitly configured for a grid worker, auto-detection is used to determine the value to apply. This should be appropriate for most circumstances. The limit is determined as follows:
- If virtual memory is limited (
ulimit -v), a quarter of that value or 50GB, whichever is smaller. Otherwise: - If resident memory is limited (
ulimit -m), half of that value or 50GB, whichever is smaller. Otherwise: - Half of the amount of available physical memory or 50GB, whichever is smaller.
Order of priority for java process memory settings The first value found according to the list below will be applied:
- A memory limit specified in a
clcgridworker.vmoptionsfile. This method is not recommended. - A memory limit specified in a CLC Grid Preset. If values determined by auto-detection are not appropriate, this method can be used to specify a custom value. See Configure grid presets.
- Auto-detection, as described above.
External binaries
Some tools make use of external binaries during one or more phases. Memory usage by external binaries is not restricted by limits set for the java process. For this reason, we recommend caution if you plan to submit jobs of these types to nodes that are being used simultaneously for other work.