"Seegrid will be due for a migration to confluence on the 1st of August. Any update on or after the 1st of August will NOT be migrated"

Tutorial on using AEM in VGL

Overview

An Airborne Electromagnetic (AEM) system is carried by an aircraft, often a helicopter, and measures the changes in conductivity of the ground to a depth of several hundred metres. The conductivity response in the ground is commonly caused by the presence of electrically conductive materials such as salt or saline water, graphite, clays and sulfide minerals. In VGL, you provide GA AEM program with AEM data that spans a geographic area, a few parameters and submit it to the cloud for processing. Once the processing is completed, you will get inversion output header and values files in ASCII text format. These output files can then be used by scientist to generate maps and cross-sections showing how ground conductivity varies with depth. This information may also help understand, for example, the connection between deeper aquifers and the alluvial aquifer system. This tutorial will provide a step-by-step guidance on how to perform the above mentioned tasks with a pre-defined dataset and an AEM phython script template. The objectives are to get you familiar with VGL and eventually run your own AEM code in VGL.

Step 0 – Preparation

To run through this tutorial, you need an internet connection and a web browser installed on your computer.

This tutorial was successfully tested with the following web browser: Internet Explorer 10 and Firefox 27.x.

This tutorial has been prepared for VGL 1.3.0 Release.

The following shows a list of VGL deployment environments:

Step 1 – Dataset Selection

The first step in using AEM in VGL is the user must first capture a subset of AEM data from data selection page (see Part I of VGL User Guide for detailed instructions).

This tutorial will make use of the following dataset:

WMS/WFS Layer: Musgrave Airbone Electromagnetic Survey (ga:musgrave_aem)
Selection Size: Approximately 7585 data points in total.
Click on Capture Data button to capture the data.

Step 2 – Job Construction

Once the data is captured, the next step is to build a job to process the captured dataset by using the “Job Wizard” which can be accessed by using the 'create a job' or ‘Submit Jobs’ link. The 'create a job' link can be found in the Request Saved notification pop-up window whereas the 'Submit Jobs' link can be found on the top right hand corner of VGL website.

To construct and submit a job in VGL, you need to sign in with your OpenID credentials (see this link if you haven't got an OpenID account set up yet) or you can use your Gmail account if you have any.

Select Job Series

The first step in creating a job is to assign it to a series. Let us proceed to create a new series and name it “AEM_Tutorial”. You can provide your own series description.

Enter Job Details

Once the name and description of a series are provided, click Next to proceed and enter the job details. To run AEM python code, the user must select ‘AEM-Inversion’ software from the Toolbox drop-down list.

This tutorial will use the Compute and Storage Providers from National eResearch Collaboration Tools and Resources (NeCTAR) - Melbourne - to run the job and to store its input data files and also execution results (see Compute and Storage Providers in VGL Guide for further details).

Note: At the time of preparing this tutorial, NeCTAR (Melbourne) is the most stable Compute and Storage providers.

Manage Job Input Files

VGL will show your input files after you entered job details. You should see the dataset you captured in Step 1 displayed on the “Input files” panel.

At this step, you can add more inputs to the job (see Part I of VGL User Guide for further details).

This tutorial does not require you to provide any further input.

Define Your Job Script

In VGL, the user can define an AEM python code from scratch, copy and paste it from somewhere or import the code from existing script template.

For the purpose of this tutorial, we choose to work from existing script template. On “Define your job script” page, expand the “AEM Inversion Examples” tree and double click on “AEM Inversion" node to import the AEM inversion’s script template into the script builder.

When prompted to provide the path to an input file, select “/tmp/subset-request” (i.e. the dataset captured in Step 1) from the Dataset dropdown list. There are many other input parameters and column mappings on the "Enter Parameters" prompt, for the purpose of this tutorial, we will use the default values of those input parameters and click on "Apply Template" button.

Once the "Apply Template" button is clicked, the AEM python code will be inserted into the script builder as shown below.

Review Job before Submission

The following review page will be shown before your job submission. It gives you opportunity to add further input files to your job and examine input files to be submitted for processing.

As this tutorial does not require any further input, click “Submit Job” to proceed.

If the job submission is successful, you will be redirected to “Monitor Jobs” page where you can then monitor the status of the submitted job and view/download the job’s input and output files. VGL will display an error message if it fails to submit the job to the cloud for execution.

Step 3 – Job Monitoring

At this point in time, you should have submitted a job named “AEM_Tutorial_Job1” for processing in the cloud.

A job belongs to a job series. To monitor the status of your submitted job, you must first select a series from the “Series List” pane. In our case, select a series named “AEM_Tutorial”. If you have a large number of series records created, use the “Query” button to search for that series.

Once the “AEM_Tutorial” series is selected, all jobs belonging to that series will be listed on the “Jobs of selected series” pane.

At any point in time, a job can be in one of the following four states:
No. Status Description
1. Saved A job will be in this state if it hasn’t been submitted for processing yet or it was cancelled by the user shortly after its submission. You can edit, submit or delete a “Saved” job but not cancel or duplicate it.
2. Pending A job will be in this state if it has already been successfully submitted to the cloud for processing and is waiting for a compute resource to process it. You can only cancel and duplicate a “Pending” job.
3. Active A job will be in this state if it is being processed by the compute resource. Like a “Pending” job, you can only cancel and duplicate an “Active” job.
4. Done A job will be in this state if it has completed its execution. A completed job does not guarantee the job is successfully executed. VGL v1.1 doesn’t provide a straight forward way to indicate a job execution success or failure. The only way to figure out this is to look at the files it generated.
The number of files generated by an active or completed job is different depending on which toolbox you select to process your job in. Every successfully executed job will have a file called “vegl.sh.log” generated. This file keeps track of the job execution log and it can be used to troubleshoot why a job failed in its execution. You can only delete or duplicate a “Done” job.
In this tutorial, we are only interested in our previously submitted job named “AEM_Tutorial_Job1”. To update its job status, use the “Refresh” button. A job normally (provided you don’t cancel it during its execution) goes through the following lifecycle: Saved -> Pending/Active -> Done.

The following screenshot demonstrates that the job named "AEM_Tutorial_Job1" is in "Pending" state:

Depending on the size of your input dataset, and the resources you chosen (number of CPU(s) and RAM size), the computational logic and processing load can take few minutes to few hours to finish a job at NeCTAR. For this tutorial, we have selected 7587 records and a small resource, the job took around 26 minutes to finish on NeCTAR's research cloud.

Every job that executes will generate a log file that captures the standard output of the executing job. That log file is written to a file called "vegl.sh.log" which can be downloaded or alternatively browsed through the inbuilt log viewer.

You can click on the "Files" tab on "Details" pane to browse through "AEM_Tutorial_Job1" job's input and output files:

You can use the "Logs" tab and its sub-tabs on "Details" pane to inspect the "AEM_Tutorial_Job1" job's execution log:

-- FlorenceTan - 24 Mar 2014
Topic attachments
I Attachment Action Size Date Who Comment
Step1.pngpng Step1.png manage 400.3 K 24 Mar 2014 - 16:13 FlorenceTan Step 1 - Data Selection
Step2_1.pngpng Step2_1.png manage 118.8 K 24 Mar 2014 - 16:14 FlorenceTan Step 2.1 - Select Job Series
Step2_2.pngpng Step2_2.png manage 131.3 K 24 Mar 2014 - 16:14 FlorenceTan Step 2.2 - Enter Job Details
Step2_3.pngpng Step2_3.png manage 81.5 K 24 Mar 2014 - 16:14 FlorenceTan Step 2.3 - Manage Job Input Files
Step2_4.pngpng Step2_4.png manage 165.8 K 24 Mar 2014 - 16:14 FlorenceTan Step 2.4 - Define Your Job Script
Step2_5.pngpng Step2_5.png manage 174.9 K 24 Mar 2014 - 16:14 FlorenceTan Step 2.5 - AEM Python Template
Step2_6.pngpng Step2_6.png manage 93.6 K 24 Mar 2014 - 16:15 FlorenceTan Step 2.6 - Review Job before Submission
Step3_1.pngpng Step3_1.png manage 105.4 K 24 Mar 2014 - 16:15 FlorenceTan Step 3.1 - Pending Job
Step3_2.pngpng Step3_2.png manage 143.6 K 25 Mar 2014 - 09:30 FlorenceTan Step 3.2 - Files Tab
Step3_3.pngpng Step3_3.png manage 88.5 K 25 Mar 2014 - 09:30 FlorenceTan Step 3.3 - Logs Tab
Topic revision: r6 - 01 Sep 2014, TerryRankine
 

Current license: All material on this collaboration platform is licensed under a Creative Commons Attribution 3.0 Australia Licence (CC BY 3.0).