Understanding memory settings
Java process memory limits
Most CLC Server work is done via its java process. The default memory limit for the java process is set according to the following rules:
- If virtual memory is limited (
ulimit -v): set the limit to a quarter of that value or 50GB, whichever is smaller.Otherwise:
- If resident memory is limited (
ulimit -m), set the limit to half of that value or 50GB, whichever is smaller.Otherwise:
- Set the limit to half of the amount of available physical memory or 50GB, whichever is smaller.
The default limit should suit most circumstances. However, there are cases where it may not, for example, where particular queues are designated for low overheads tasks, such as import jobs and trimming jobs, and for high overhead tasks, such as de novo assemblies or read mappings. In such cases, a memory limit can be explicitly configured for each grid preset6.1. This is described in Configure grid presets.
External binaries
Some tools use external binaries for one or more computational phases. These include, but are not limited to, those with a de novo assembly or mapping phases and those involving BLAST tools. Memory usage by external binaries is not restricted by limits set for the java process. For this reason, we recommend caution if you plan to submit jobs of these types to nodes that are being used simultaneously for other work.
Footnotes
- ... preset6.1
- Directly specifying memory limits in a "clcgridworker.vmoptions" file using -Xmx options is not recommended. Such settings override those described on this page.