"Seegrid will be due for a migration to confluence on the 1st of August. Any update on or after the 1st of August will NOT be migrated"

Tutorial on using ANUGA in VHIRL

Overview

This tutorial provides step-by-step guidance on how to use VHIRL Portal to submit selected Busselton dataset to be processed by ANUGA in the cloud and retrieve the results through VHIRL Portal.

Step 0 – Preparation

To run through this tutorial, you need an internet connection and a web browser installed on your computer.

This tutorial was successfully tested with the following web browser: Firefox 33.1.

This tutorial has been prepared for VHIRL Demo : http://auscope-portal-dev.arrc.csiro.au/VHIRL-Portal/gmap.html

Step 1 – Dataset Selection

The first step in using ANUGA in VHIRL is the user must first capture a subset of Busselton data from data selection page:
  1. From the Featured Layers tab, select Busselton.
  2. Double click on the magnifying glass to zoom to the layer's spatial region on the map.
  3. Click on Add to Map button to add the selected layer on the map.
  4. Click on the Select Data button on the right side of the map and draw boundary box to select the spatial region around Busselton.
    A small window will pop up to confirm the dataset you wish to select as below:

    Step1_1.png

  5. To proceed on with the data capturing, click to select the check box left of the required dataset.
  6. Click the Capture Data button.
  7. Click on create a job link if you wish to create a job for submission.

    Step1_2.png

Step 2 – Job Construction

Once the data is captured, the next step is to build a job to process the captured dataset by using the Job Wizard which can be accessed by using the create a job link right after selecting a spatial regioun or Submit Jobs link from the top menu.

To construct and submit a job in VHIRL, you need to sign in with your OpenID credentials (see this link if you haven't got an OpenID account set up yet) or you can use your Gmail account if you have any.

Step 2.1 Select Job Series

The first step in creating a job is to assign it to a series:
  1. Select Create new series and enter the Series Name as ANUGA_Tutorial.
  2. You can provide your own series description.

    Step 2.1 Select Job Series

Step 2.2 Manage Job Input Files

VHIRL will show your input files after you entered job details. You should see the dataset you captured in Step 1 displayed on the “Input files” panel.

Step2_2.png

At this step, you can add more inputs to the job. For this tutorial, no additional files are required.

Step 2.3 Define Your Job Script

In VHIRL, the user can define an ANUGA python code from scratch, copy and paste it from somewhere or import the code from existing script template.

For the purpose of this tutorial, we choose to work from existing script template.
  1. On the Define your job script page, expand the Regional Indundation modelling tree and double click on the ANUGA Busselton example node to import the ANUGA script template into the script builder.
  2. When prompted to provide the path to an input file, select /tmp/elevation (i.e. the dataset captured in Step 1) from the Dataset dropdown list.
  3. There are a few input parameters on the Enter Parameters prompt, for the purpose of this tutorial, we will use the default values of those input parameters and click on "Apply Template" button.

    Step2_3_1.png

  4. Once the "Apply Template" button is clicked, the ANUGA python code will be inserted into the script builder as shown below.

    Step2_3_2.png

Step 2.4 Enter Job Details

Once the name and description of a series are provided, click Next to proceed and enter the job details:
  1. Enter Job Name : ANUGA_Tutorial_Job1
  2. Enter Job Description : An ANUGA Tutorial Job
  3. Select Compute Provider : National eResearch Collaboration Tools and Resources
  4. Select Storage Provider : National eResearch Collaboration Tools and Resources (Melbourne)
  5. Select Toolbox : ANUGA
  6. Select Resources : m1.medium - 2CPU(s), 8192 MB RAM
  7. Check Email Notification check box if you wish to receive email notification upon job completion.

    Step2_4.png
This tutorial will use the Compute and Storage Providers from National eResearch Collaboration Tools and Resources (NeCTAR) to run the job and to store its input data files and also execution results.
For more details about Compute and Storage Providers, refer to VGL Guide.

Note: At the time of preparing this tutorial, NeCTAR (Melbourne) is the most stable Storage Providers.

Step 2.5 Review Job Before Submission

The following review page will be shown before your job submission.
It gives you opportunity to add further input files to your job and examine input files to be submitted for processing.

Step 2.5 Review Job Before Submission

If the job submission is successful, you will be redirected to “Monitor Jobs” page where you can then monitor the status of the submitted job and view/download the job’s input and output files. VHIRL will display an error message if it fails to submit the job to the cloud for execution.

Step 3 – Job Monitoring

At this point in time, you should have submitted a job named NUGA_Tutorial_Job1 for processing in the cloud.

A job belongs to a job series. To monitor the status of your submitted job, you must first select a series from the “Series List” pane. In our case, select a series named ANUGA_Tutorial. If you have a large number of series records created, use the Query button to search for that series.

Once the ANUGA_Tutorial series is selected, all jobs belonging to that series will be listed on the Jobs of selected series pane.

At any point in time, a job can be in one of the following four states:
No. Status Description
1. Saved A job will be in this state if it hasn’t been submitted for processing yet or it was cancelled by the user shortly after its submission. You can edit, submit or delete a Saved job but not cancel or duplicate it.
2. Pending A job will be in this state if it has already been successfully submitted to the cloud for processing and is waiting for a compute resource to process it. You can only cancel and duplicate a “Pending” job.
3. Active A job will be in this state if it is being processed by the compute resource. Like a Pending job, you can only cancel and duplicate an Active job.
4. Done A job will be in this state if it has completed its execution. A completed job does not guarantee the job is successfully executed. VHIRL doesn’t provide a straight forward way to indicate a job execution success or failure. The only way to figure out this is to look at the files it generated.
The number of files generated by an active or completed job is different depending on which toolbox you select to process your job in. Every successfully executed job will have a file called vegl.sh.log generated. This file keeps track of the job execution log and it can be used to troubleshoot why a job failed in its execution. You can only delete or duplicate a Done job.
In this tutorial, we are only interested in our previously submitted job named ANUGA_Tutorial_Job1. To update its job status, use the Refresh button. A job normally (provided you don’t cancel it during its execution) goes through the following lifecycle: Saved -> Pending/Active -> Done.

The following screenshot demonstrates that the job named ANUGA_Tutorial_Job1 is in Pending state:

Step 3 Monitor Jobs - Pending

Depending on the size of your input dataset, and the resources you chosen (number of CPU(s) and RAM size), the computational logic and processing load can take few minutes to few hours to finish a job at NeCTAR.

Every job that executes will generate a log file that captures the standard output of the executing job. That log file is written to a file called vl.sh.log which can be downloaded or alternatively browsed through the inbuilt log viewer.

You can click on the Files tab on Details pane to browse through ANUGA_Tutorial_Job1 job's input and output files:

Step 3 Monitor Jobs - Done

You can use the Logs tab and its sub-tabs on Details pane to inspect the ANUGA_Tutorial_Job1 job's execution log:

Topic attachments
I Attachment Action Size Date Who Comment
Step1_1.pngpng Step1_1.png manage 434.1 K 14 Nov 2014 - 13:44 AlexKrumpholz Step 1.1 Select the dataset
Step1_2.pngpng Step1_2.png manage 448.1 K 14 Nov 2014 - 13:46 AlexKrumpholz Step 1.2 Create a job
Step2_1.pngpng Step2_1.png manage 111.7 K 14 Nov 2014 - 15:00 AlexKrumpholz Step 2.1 Select Job Series
Step2_2.pngpng Step2_2.png manage 92.4 K 14 Nov 2014 - 15:07 AlexKrumpholz Step 2.2 Manage Job Input Files
Step2_3_1.pngpng Step2_3_1.png manage 98.2 K 16 Nov 2014 - 17:57 AlexKrumpholz Step 2.3 Define Your Job Script
Step2_3_2.pngpng Step2_3_2.png manage 176.1 K 16 Nov 2014 - 17:58 AlexKrumpholz Step 2.3 ANUGA Python Code
Step2_4.pngpng Step2_4.png manage 120.0 K 16 Nov 2014 - 18:04 AlexKrumpholz Step 2.4 Enter Job Details
Step2_5.pngpng Step2_5.png manage 104.1 K 16 Nov 2014 - 18:22 AlexKrumpholz Step 2.5 Review Job Before Submission
Step3_1.pngpng Step3_1.png manage 112.1 K 16 Nov 2014 - 18:36 AlexKrumpholz Step 3.1 Monitor Jobs
Step3_2.pngpng Step3_2.png manage 144.1 K 16 Nov 2014 - 18:37 AlexKrumpholz Step 3.2 Completed Job - Files
Step3_3.pngpng Step3_3.png manage 243.5 K 16 Nov 2014 - 18:50 AlexKrumpholz Step 3.3 Completed Job - Logs
bunbury_1km_extend.csvcsv bunbury_1km_extend.csv manage 0.6 K 20 Aug 2014 - 11:18 RyanFraser  
busselton_1km.csvcsv busselton_1km.csv manage 1.1 K 20 Aug 2014 - 11:19 RyanFraser  
busselton_20m.csvcsv busselton_20m.csv manage 0.5 K 20 Aug 2014 - 11:19 RyanFraser  
busselton_extent_edit.csvcsv busselton_extent_edit.csv manage 0.2 K 20 Aug 2014 - 11:19 RyanFraser  
Topic revision: r11 - 16 Nov 2014, AlexKrumpholz
 

Current license: All material on this collaboration platform is licensed under a Creative Commons Attribution 3.0 Australia Licence (CC BY 3.0).