AWS Batch queues, compute environments, AMIs and images
AWS Batch queues, compute environments and AMIs
AWS Batch queues created using the CloudFormation templates provided by QIAGEN are each mapped to an individual compute environment. Each compute environment is configured to use the latest Amazon ECS-optimized AMI available at the time it is established. Thus, in a CLC Genomics Cloud setup, all jobs sent to a given AWS Batch queue will use the same AMI version.
AWS Batch does not upgrade the AMIs in existing compute environments. Thus, compute environments established at different times may contain different AMI versions. An example where this could happen in a CLC Genomics Cloud context would be where a new AWS Batch queue is created some time after the initial infrastructure was set up, and there had been an AMI update released in the meantime.
We generally do not expect performance issues using supported, older AMI versions. From a security perspective, AWS Batch compute environments established using the CloudFormation templates provided by QIAGEN are placed in a security group that rejects incoming traffic that was not initiated from the EC2 host.
The AMI used for a job
The image ID for the AMI used for a job is recorded near the top of the CloudWatch log (aka Execution Log). Information about that public image, such as the Amazon Linux AMI version and the deprecation time for that AMI can be found via the AWS Console:
Console | EC2 | Images | AMIs
See the changelog at AWS for details of each AMI release: https://github.com/aws/amazon-ecs-ami/blob/main/CHANGELOG.md
Accessing a newer AMI
To update the AMI being used, a new compute environment is needed. An easy way to achieve this is to create a new AWS Batch queue, as described in Adding more AWS Batch queues for CLC jobs. Queues that are no longer needed can be deleted, as described in Deleting or redeploying CLC Genomics Cloud infrastructure on AWS.
CLC software and plugin versions used for a job
CLC software version used: Execution of CLC jobs using AWS Batch involves pulling the Docker image from AWS ECR that contains the CLC Workflow Engine version corresponding to the version of the CLC product used to submit the cloud job. For example:
- A job submitted directly from CLC Genomics Workbench 23.0.4 will pull the image containing CLC Workflow Engine 23.0.4.
- A job submitted via CLC Genomics Server 23.0.5 will pull the image containing CLC Workflow Engine 23.0.5. This is the case irrespective of the client software version used to submit the job to the CLC Server.
The CLC Workflow Engine is released in conjunction with the CLC Main Workbench, CLC Genomics Workbench, CLC Genomics Server, and CLC Server Command Line Tools. Thus, there is a CLC Workflow Engine version corresponding to every released CLC Workbench and CLC Genomics Server version capable of submitting jobs to AWS Batch.
Plugin versions used: Each plugin contributing tools needed for the analysis is installed in the CLC Workflow Engine before the analysis is run. The version of each plugin installed on the CLC Workflow Engine is the same as the version installed on the CLC product used to submit the cloud job. E.g. When jobs are submitted via a CLC Genomics Server, it is the version of the plugin on the CLC Server that will be installed on the CLC Workflow Engine.