"Seegrid will be due for a migration to confluence on the 1st of August. Any update on or after the 1st of August will NOT be migrated"

Tutorial on using TCRM in VHIRL


This tutorial provides step-by-step guidance on how to use VHIRL Portal to run a TCRM simulation in the cloud and retrieve the results through VHIRL Portal.

Step 0 – Preparation

To run through this tutorial, you need an internet connection and a web browser installed on your computer.

This tutorial was successfully tested with the following web browser: Firefox 31.0.

This tutorial has been prepared for VHIRL Demo : http://auscope-portal-dev.arrc.csiro.au/VHIRL-Portal/gmap.html

Step 1 – Dataset Selection

TCRM uses the iBTrACS data set, which has already been made available to the code in the cloud, so there is no need for the user to select a dataset in VHIRL.

Go straight to Step 2 - Job Construction.

Step 2 – Job Construction

The next step is to build a job to process the wind simulation by using the Job Wizard which can be accessed by using the Submit Jobs link from the top menu.

To construct and submit a job in VHIRL, you need to sign in with your OpenID credentials (see this link if you haven't got an OpenID account set up yet) or you can use your Gmail account if you have any.

Step 2.1 Select Job Series

The first step in creating a job is to assign it to a series:
  1. Select Create new series and enter the Series Name as TCRM_Tutorial.
  2. You can provide your own series description.

Step 2.1 Select Job Series

Step 2.2 Manage Job Input Files

Once the name and description of a series are provided, click Next to proceed. Click Yes in the dialog to confirm that you don't want to use a captured data set.

Step 2.2 Confirm no dataset

VHIRL will show your input files after you entered job details. Since this job doesn't require any, VHIRL will let you know that it doesn't have any input files allocated.

Step 2.2 No input files

Click Next to proceed.

Step 2.3 Define Your Job Script

In VHIRL, the user can define the python code the runs TCRM from scratch, copy and paste it from somewhere or import the code from existing script template.

For the purpose of this tutorial, we choose to work from existing script template.

  1. On the Define your job script page, expand the Understanding cyclone risk tree and double click on TCRM example node to import the TCRM script template into the script builder.
  2. For this example we will use the default values for all the parameters except Max threads, which should be set to 4 so the job completes in a reasonable time. Step 2.3 Define Your Job Script
  3. Click on the Apply Template button to move on.
  4. Once the "Apply Template" button is clicked, the TCRM python code will be inserted into the script builder as shown below. TCRM Python Code
  5. Click Next to proceed.

Step 2.4 Enter Job Details

Enter the job details:

  1. Enter Job Name : TCRM_Tutorial_Job1
  2. Enter Job Description : A TCRM Tutorial Job
  3. Select Compute Provider : National eResearch Collaboration Tools and Resources. At this point the Resources entry will be pre-filled with a vm type that is suitable for running the number of threads you selected in the previous step.
  4. Select Storage Provider : National eResearch Collaboration Tools and Resources (Melbourne)
  5. Select Toolbox : TCRM
  6. Select Resources : m1.large - 4 CPU(s), 16384 MB RAM
  7. Check Email Notification check box if you wish to receive email notification upon job completion. Step 2.4 Enter Job Details

This tutorial will use the Compute and Storage Providers from National eResearch Collaboration Tools and Resources (NeCTAR) to run the job and to store its input data files and also execution results. For more details about Compute and Storage Providers, refer to VGL Guide.

Note: At the time of preparing this tutorial, NeCTAR (Melbourne) is the most stable Storage Provider.

Step 2.5 Review Job Before Submission

The following review page will be shown before your job submission. It gives you opportunity to add further input files to your job and examine input files to be submitted for processing.

Step 2.5 Review Job Before Submission

Click Submit Job to start the simulation running.

If the job submission is successful, you will be redirected to “Monitor Jobs” page where you can then monitor the status of the submitted job and view/download the job’s input and output files. VHIRL will display an error message if it fails to submit the job to the cloud for execution.

Step 3 – Job Monitoring

At this point in time, you should have submitted a job named TCRM_Tutorial_Job1 for processing in the cloud.

A job belongs to a job series. To monitor the status of your submitted job, you must first select a series from the “Series List” pane. In our case, select a series named TCRM_Tutorial. If you have a large number of series records created, use the Query button to search for that series.

Once the TCRM_Tutorial series is selected, all jobs belonging to that series will be listed on the Jobs of selected series pane.

At any point in time, a job can be in one of the following four states:
No. Status Description
1. Saved A job will be in this state if it hasn’t been submitted for processing yet or it was cancelled by the user shortly after its submission. You can edit, submit or delete a Saved job but not cancel or duplicate it.
2. Pending A job will be in this state if it has already been successfully submitted to the cloud for processing and is waiting for a compute resource to process it. You can only cancel and duplicate a “Pending” job.
3. Active A job will be in this state if it is being processed by the compute resource. Like a Pending job, you can only cancel and duplicate an Active job.
4. Done A job will be in this state if it has completed its execution. A completed job does not guarantee the job is successfully executed. VHIRL doesn’t provide a straight forward way to indicate a job execution success or failure. The only way to figure out this is to look at the files it generated.
The number of files generated by an active or completed job is different depending on which toolbox you select to process your job in. Every successfully executed job will have a file called vegl.sh.log generated. This file keeps track of the job execution log and it can be used to troubleshoot why a job failed in its execution. You can only delete or duplicate a Done job.
In this tutorial, we are only interested in our previously submitted job named TCRM_Tutorial_Job1. To update its job status, use the Refresh button. A job normally (provided you don’t cancel it during its execution) goes through the following lifecycle: Saved -> Pending/Active -> Done.

The following screenshot demonstrates that the job named TCRM_Tutorial_Job1 is in Pending state:

Step 3 Monitor Jobs - Pending

Depending on the size of your input dataset, and the resources you chosen (number of CPU(s) and RAM size), the computational logic and processing load can take few minutes to few hours to finish a job at NeCTAR.

Every job that executes will generate a log file that captures the standard output of the executing job. That log file is written to a file called vegl.sh.log which can be downloaded or alternatively browsed through the inbuilt log viewer.

You can click on the Files tab on Details pane to browse through TCRM_Tutorial_Job1 job's input and output files:

Step 3 Monitor Jobs - Done

You can use the Logs tab and its sub-tabs on Details pane to inspect the TCRM_Tutorial_Job1 job's execution log:

-- GeoffreySquire - 14 Aug 2014
Topic attachments
I Attachment Action Size Date Who Comment
Step2_1.pngpng Step2_1.png manage 67.7 K 14 Aug 2014 - 14:11 GeoffreySquire  
Step2_2_confirm.pngpng Step2_2_confirm.png manage 70.7 K 14 Aug 2014 - 14:11 GeoffreySquire  
Step2_2_inputs.pngpng Step2_2_inputs.png manage 56.3 K 12 Nov 2014 - 08:47 GeoffreySquire  
Step2_3_params.pngpng Step2_3_params.png manage 94.5 K 12 Nov 2014 - 08:44 GeoffreySquire  
Step2_3_template.pngpng Step2_3_template.png manage 129.5 K 12 Nov 2014 - 08:44 GeoffreySquire  
Step2_4.pngpng Step2_4.png manage 81.6 K 12 Nov 2014 - 08:45 GeoffreySquire  
Step2_4_2.pngpng Step2_4_2.png manage 130.5 K 14 Aug 2014 - 14:39 GeoffreySquire  
Step2_5.pngpng Step2_5.png manage 55.4 K 14 Aug 2014 - 14:39 GeoffreySquire  
Step3_1.pngpng Step3_1.png manage 69.7 K 14 Aug 2014 - 14:41 GeoffreySquire  
Step3_2.pngpng Step3_2.png manage 123.2 K 15 Aug 2014 - 07:22 GeoffreySquire  
Step3_3.pngpng Step3_3.png manage 180.4 K 15 Aug 2014 - 07:23 GeoffreySquire  
Topic revision: r7 - 12 Nov 2014, GeoffreySquire

Current license: All material on this collaboration platform is licensed under a Creative Commons Attribution 3.0 Australia Licence (CC BY 3.0).