Adding more AWS Batch queues for CLC jobs
To add another AWS Batch queue for running CLC analyses using on-demand EC2 instances:
- Log into the AWS console as a user with privileges that will allow the infrastructure described above to be created.
- Set the region to the one the AWS resources should be established in.
- Copy the URL below:
https://qiagen-clc-genomics-cloud-formation.s3.eu-central-1.amazonaws.com/cgc-queue-ondemand.json
- Go to CloudFormation and click on Create stack.
- Configure the settings for the new queue. This includes:
- vCPUs per job We recommend setting this to the number of vCPUs available on the instance types you specify. If there are different numbers of vCPUs on the instance types specified, provide the lowest value.
- Total available memory per job We recommend that this is set to the available amount of memory on the instance types you specify, minus 1000 MB. So for instance types with 32 GB of memory, the recommended value would be 31000.
- Amount of memory allocated for Java per job We recommend that this is set to approximately half the total memory, up to a maximum of 50GB. So, with 31000 MB of total memory allocated, a reasonable value here would be 15500 MB. With 128000 MB total memory, a reasonable value would be 50000MB.
- max vCPUs This value controls the number of EC2 instances that can be started up in parallel in this queue. For example, if the vCPUs per job were set to 16, then the value of 160 here would mean a maximum of 10 EC2 instances could be running at the same time in this queue.
- Allowed instance types One or more EC2 instance types that can be started up when jobs are sent to this queue. Linux 64-bit (x86_64 / AMD64) instance types are supported. See the AWS documentation for details of the types available, https://aws.amazon.com/ec2/instance-types/ and their costs, https://aws.amazon.com/ec2/pricing/on-demand/.
- Disk size Specify the size of the disk to allocate to each EC2 instance.
AWS Batch queues made using this template can be used by the AWS IAM "SubmitterUser" created using the standard template (see Setting up AWS resources).
When an AWS Batch queue is created using the CloudFormation templates provided by QIAGEN, it is mapped to an individual compute environment configured with the latest Amazon ECS-optimized AMI available at that time. Thus, the AMI version used when jobs are sent to queues established at different times may be different. See AWS Batch queues, compute environments and AMIs for further details.