Using a CLC Server to monitor and review cloud jobs

Information about jobs submitted and run on a CLC Genomics Cloud setup can be found in several areas under the Management tab of the CLC Server web client:

Links to documentation about these areas are provided at the end of this section.

Image clcserver-auditlog-cloudjob
Figure 7.3: An audit log entry for a completed batch job, with two batch units. The CGC links in the Status column indicate a CLC Genomics Cloud setup was used as the execution environment.

When a cloud job is in progress, clicking on the CGC link beside a job in the Queue or in the Status column of the Audit log opens a dialog. Before the job starts running, the dialog gives some basic information and a link to the results folder (figure 7.4). While a job is being executed, the AWS log viewer will be opened, allowing progress of the job to be monitored via the CloudWatch log. Clicking on the Reload log button reloads the log and scrolls to latest entry. Search functionality is available at the bottom left (figure 7.5).

Image cgcllink-for-queued-job
Figure 7.4: The CGC link for a job still being executed was clicked on, opening a dialog with basic information and a link to the results folder.

Image awslogviewer-server
Figure 7.5: Clicking on the CGC link for a running job opens the AWS log viewer. Contents can be searched by typing terms into the field at the bottom left. Clicking on the Reload log button reloads the log and scrolls to the latest entry.

When a job is complete, an entry in the audit log for that process is added with "Command done" in the Operation field. Clicking on the CGC link in the Status column of these entries opens a "CGC Workflow result" dialog with information about the job, as well as buttons for downloading results, for downloading a copy of the workflow that was run, and for downloading the AWS log, i.e. the Cloud Watch log stream for that job (figure 7.6).

Users can open a workflow result dialog from the Browse S3 locations area, under the Element info tab. When a workflow-result.json file is selected, a button labeled "View workflow result..." becomes available (figure 7.7). Clicking this opens a dialog with contents like that shown in figure 7.6.

Image cgc-workflow-result-dialog-server
Figure 7.6: Results of jobs run on a CLC Genomics Cloud setup can be accessed through a dialog available via the audit log or when browsing S3.

Image server-workflowlog-from-s3browser
Figure 7.7: Selecting a workflow-result.json file in the Browse S3 locations tab reveals functionality that can be taken, including the "View workflow result..." button that opens a dialog like that shown in figure 7.6.

Links to related documentation in the CLC Server manual: