• Product manuals

Browse the manual

• Introduction to the CLC Genomics Cloud
  • Overview of CLC Genomics Cloud
  • CLC Genomics Cloud requirements
  • Contact information
  • Installing the CLC Cloud Module
    • Licensing modules
    • Uninstalling modules
  • Installing the Cloud Server Plugin
• Setting up CLC Genomics Cloud resources on AWS
  • Setting up AWS resources
  • Adding more AWS Batch queues for CLC jobs
  • Adding more AWS IAM users
  • Deleting or redeploying CLC Genomics Cloud infrastructure on AWS
• Using a CLC Workbench to work on the cloud
  • Prerequisites for working on the cloud using a Workbench
  • Configuring access to AWS in the Workbench
    • Public S3 buckets
  • Submitting analyses to AWS from a Workbench
  • Working with AWS S3 using the Remote Files tab
  • Accessing results from the Processes tab
• Using a CLC Server to work on the cloud
  • Benefits of submitting jobs via a CLC Server
  • AWS Connections in the CLC Server
  • Configuring cloud presets
  • Browse AWS S3 locations
  • Submitting workflows to the cloud via a CLC Server using a CLC Workbench
  • Submitting workflows to the cloud using the CLC Server Command Line Tools
• Integrating third party tools into CLC software
  • Installing external applications in a CLC Workbench
• General information about inputs and results
  • General information about input data for cloud analyses
    • Reference data for analyses on the cloud
    • Using locally held data for analyses on AWS S3
  • General information about outputs from cloud analyses
• Monitoring and reviewing CLC cloud jobs
  • Using a CLC Workbench to monitor and review cloud jobs
  • Using a CLC Server to monitor and review cloud jobs
  • Using the AWS Console to monitor and review cloud jobs
• Appendix
  • Standard AWS Batch queues for CLC Genomics Cloud
  • AWS Batch queues, compute environments, AMIs and images
  • System file retention policies
  • Making custom plugins available for cloud analyses

Adding more AWS Batch queues for CLC jobs

Additional AWS Batch queues for running CLC analyses using on-demand EC2 instances can be added using a CloudFormation template provided by QIAGEN, described below.

When created, each AWS Batch queues is mapped to an individual compute environment configured with the latest Amazon ECS-optimized AMI available at that time. As AWS Batch does not upgrade the AMIs in existing compute environments, adding queues can be used as a step in updating the AMIs available in a CLC Genomics Cloud setup. For further information, see AWS Batch queues, compute environments, AMIs and images.

To add an AWS Batch queue to an existing CLC Genomics Cloud setup:

• Log into the AWS console as a user with privileges that will allow the infrastructure described above to be created.
• Set the region to the one the AWS resources should be established in.
• Copy the URL below:

  https://qiagen-clc-genomics-cloud-formation.s3.eu-central-1.amazonaws.com/cgc-queue-ondemand.json

• Go to CloudFormation and click on Create stack. Choose the option "With new resources (standard)" from the drop down menu.

• In the Create stack step, keep "Choose an existing template" selected and in the "Specify template" step, keep the "Amazon S3 URL" option selected and paste the URL to the CloudFormation template you just copied into that field.

• Configure the settings for the new queue. This includes:
  • vCPUs per job We recommend setting this to the number of vCPUs available on the instance types you specify. If there are different numbers of vCPUs on the instance types specified, provide the lowest value.
  • Total available memory per job We recommend that this is set to the available amount of memory on the instance types you specify, minus 1000 MB. So for instance types with 32 GB of memory, the recommended value would be 31000.
  • Amount of memory allocated for Java per job We recommend that this is set to approximately half the total memory, up to a maximum of 50GB. So, with 31000 MB of total memory allocated, a reasonable value here would be 15500 MB. With 128000 MB total memory, a reasonable value would be 50000MB.
  • max vCPUs This value controls the number of EC2 instances that can be started up in parallel in this queue. For example, if the vCPUs per job were set to 16, then the value of 160 here would mean a maximum of 10 EC2 instances could be running at the same time in this queue.
  • Allowed instance types One or more EC2 instance types that can be started up when jobs are sent to this queue. Linux 64-bit (x86_64 / AMD64) instance types are supported. See the AWS documentation for details of the types available, https://aws.amazon.com/ec2/instance-types/ and their costs, https://aws.amazon.com/ec2/pricing/on-demand/.
  • Disk size Specify the size of the disk to allocate to each EC2 instance.

• When prompted, agree to the AWS conditions and click on the Submit button.

AWS Batch queues made using this template can be used by the AWS IAM "SubmitterUser" created using the standard template (see Setting up AWS resources).

---