"Seegrid will be due for a migration to confluence on the 1st of August. Any update on or after the 1st of August will NOT be migrated"

Architecture

ssc-system.png

Catalogue Browser

An HTML interface is provided for users to browse the entries in the catalogue. It provides listings of entries by type, navigation between related entries and text search in entry names and descriptions.

Catalogue Manager

There is a basic HTML interface that allows creating, updating and deleting entries.

User/Version Management

There is support in the information model for tracking authorship and versioning of entries in the catalogue. User authentication and version management still need to be implemented in the application.

Information Model

Information model

The catalogue stores the same data for every entry: a unique id, a name and description, and metadata. In order to include links and some formatting information, the description field contains markdown text that is rendered by the browser interface, and passed on to clients of the API.

Problem

A description of a scientific problem. May be associated with one or more solutions in the catalogue.

Solution

A workflow that uses a scientific code to solve a specific problem. There should be enough information in the catalogue about a solution for another user or system (e.g. the VHIRL portal) to instantiate it. The solution entry contains a reference to the problem is solves, and the scientific code it uses (toolbox). The workflow itself is captured as a python script template and a description of the variables required to run it. A solution may also describe extra dependencies that it requires over and above what the toolbox provides.

The python script should include placeholders for input (and later output) variables. Each placeholder is a variable name wrapped in ${}, and the variables field contains descriptions of each variable in the template.

Variables

Each variable description has the following fields.

Field Description Required? Valid values
name Variable name and placeholder string in template  
type Datatype of the variable's value string, int, double, random-int, file
label User friendly label    
description Longer description about this variable    
default Default initial value    
values List of valid values    
min Minimum value for a numeric variable    
max Maximum value for a numeric variable    
step Step between min/max numeric values    

Toolbox

Describes a scientific code and the dependencies it requires to function. The source is a reference to a source control repository (git or svn), including the URL where the source code can be checked out, plus any other information required to specify the desired version (git branch etc). Homepage and license provide links to more information for users. Images contains a list of cloud images that are known to include the necessary software and dependencies. The toolbox should also contain a list of dependencies required for the code to run.

Dependencies

Dependencies should provide enough information to build an environment that will run the scientific code or solution. A dependency can be a system package, a python package or a python/pip requirements.txt file. A package must specify the type and name, and optionally a version. If no version is supplied then the latest will be assumed. A toolbox can specify a requirements file as a path relative to the root of the checked out source code.

Deployment

API

There are methods to request a listing of all entries of a specific type (problem, solution, toolbox), or to request full details about a specific entry. The listing methods include a subset of the entry data, common to all entry types, for each entry in the list. The requests to the API can return HTML or JSON, as requested. The examples below show the results when requesting application/json.

GET /problems

Return all available problems from the catalogue.
{
    "problems": [
        {
            "@id": "http://vhirl-dev.csiro.au/scm/problems/1",
            "author": "Fred",
            "created_at": "Thu, 06 Nov 2014 16:23:47 GMT",
            "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.",
            "id": 1,
            "name": "Regional Inundation modelling (storm-surge or tsunamis)",
            "type": "problem",
            "uri": "http://vhirl-dev.csiro.au/scm/problems/1",
            "version": 1
        },
        {
            "@id": "http://vhirl-dev.csiro.au/scm/problems/2",
            "author": "Fred",
            "created_at": "Thu, 06 Nov 2014 16:23:47 GMT",
            "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.",
            "id": 2,
            "name": "Understanding cyclone risk",
            "type": "problem",
            "uri": "http://vhirl-dev.csiro.au/scm/problems/2",
            "version": 1
        }
    ]
}

GET /problems/:id

Return a detailed description of the problem with id.
{
    "@id": "http://vhirl-dev.csiro.au/scm/problems/1",
    "author": "Fred",
    "created_at": "Thu, 06 Nov 2014 16:23:47 GMT",
    "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.",
    "id": 1,
    "name": "Regional Inundation modelling (storm-surge or tsunamis)",
    "uri": "http://vhirl-dev.csiro.au/scm/problems/1",
    "version": 1
}

GET /solutions[?problem=:problem_id]

Return a listing of all solutions in the catalogue. Each solution entry includes links to, and brief descriptions of, the related problem and toolbox. If an optional problem id is specified, then only return solutions that solve the specified problem.
{
    "solutions": [
        {
            "@id": "http://vhirl-dev.csiro.au/scm/solutions/1",
            "author": "Fred",
            "created_at": "Thu, 06 Nov 2014 16:23:47 GMT",
            "description": "This template contains a pre-canned event (a wave) entering on the WEST of the grid, heading east, designed for coastal simulations of the Busselton-Bunbury area.\r\n\r\nThe implemented solver (Parallel finite volume method for hydrodynamic inundation modelling) is [described here](http://journal.austms.org.au/ojs/index.php/ANZIAMJ/article/view/153/)\r\n\r\nAn overview of the solver is [available here](http://www.ga.gov.au/corporate_data/69370/Rec2009_036.pdf).\r\n\r\nInputs:\r\n\r\nThe template can be customised in a variety of ways \r\n\r\n\tXxxx\r\n\r\n\tYyyy\r\n\r\n\tZzzz\r\n\r\nAnd is paired with a DEM of the area in question.\r\n\r\n\r\n\r\nOutputs:\r\n\r\n\tScreenshots of xxx\r\n\r\n\tCustom ANUGA SSW (database) of the simulation",
            "id": 1,
            "name": "ANUGA Busselton example",
            "problem": {
                "@id": "http://vhirl-dev.csiro.au/scm/problems/1",
                "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.",
                "id": 1,
                "name": "Regional Inundation modelling (storm-surge or tsunamis)",
                "uri": "http://vhirl-dev.csiro.au/scm/problems/1"
            },
            "toolbox": {
                "@id": "http://vhirl-dev.csiro.au/scm/toolboxes/1",
                "description": "ANUGA is a tool which can simulate events and their effects as they \u2018progress\u2019 or travel through a scenario in a model. You can model the \u2018wave\u2019 or storm surge to measure the impact and risk for known locations.",
                "id": 1,
                "name": "ANUGA",
                "uri": "http://vhirl-dev.csiro.au/scm/toolboxes/1"
            },
            "type": "solution",
            "uri": "http://vhirl-dev.csiro.au/scm/solutions/1",
            "version": 1
        },
        ...
    ]
}  

GET /solution/:id

Return full details about a specific solution. Includes links to, and desriptions of, the related problem and toolbox.

{
    "@id": "http://vhirl-dev.csiro.au/scm/solutions/1",
    "author": "Fred",
    "created_at": "Thu, 06 Nov 2014 16:23:47 GMT",
    "description": "This template contains a pre-canned event (a wave) entering on the WEST of the grid, heading east, designed for coastal simulations of the Busselton-Bunbury area.\r\n\r\nThe implemented solver (Parallel finite volume method for hydrodynamic inundation modelling) is [described here](http://journal.austms.org.au/ojs/index.php/ANZIAMJ/article/view/153/)\r\n\r\nAn overview of the solver is [available here](http://www.ga.gov.au/corporate_data/69370/Rec2009_036.pdf).\r\n\r\nInputs:\r\n\r\nThe template can be customised in a variety of ways \r\n\r\n\tXxxx\r\n\r\n\tYyyy\r\n\r\n\tZzzz\r\n\r\nAnd is paired with a DEM of the area in question.\r\n\r\n\r\n\r\nOutputs:\r\n\r\n\tScreenshots of xxx\r\n\r\n\tCustom ANUGA SSW (database) of the simulation",
    "id": 1,
    "name": "ANUGA Busselton example",
    "problem": {
        "@id": "http://vhirl-dev.csiro.au/scm/problems/1",
        "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.",
        "id": 1,
        "name": "Regional Inundation modelling (storm-surge or tsunamis)",
        "uri": "http://vhirl-dev.csiro.au/scm/problems/1"
    },
    "template": "\"\"\"Script for running a tsunami inundation scenario for Busselton, WA Australia.\r\n\r\nSource data such as elevation and boundary data is assumed to be available in\r\ndirectories specified by project.py.....",
    "toolbox": {
        "@id": "http://vhirl-dev.csiro.au/scm/toolboxes/1",
        "description": "ANUGA is a tool which can simulate events and their effects as they \u2018progress\u2019 or travel through a scenario in a model. You can model the \u2018wave\u2019 or storm surge to measure the impact and risk for known locations.",
        "id": 1,
        "name": "ANUGA",
        "uri": "http://vhirl-dev.csiro.au/scm/toolboxes/1"
    },
    "uri": "http://vhirl-dev.csiro.au/scm/solutions/1",
    "variables": [
        {
            "default": "400000",
            "label": "Base Scale",
            "min": 1.0,
            "name": "base_scale",
            "optional": false,
            "type": "int"
        },
        {
            "default": "0.0",
            "label": "Tide",
            "min": 0.0,
            "name": "tide",
            "optional": false,
            "step": 0.1,
            "type": "double"
        },
        {
            "default": "busselton",
            "label": "File Name",
            "name": "name_stem",
            "optional": false,
            "type": "string"
        },
        {
            "description": "This should have a more informative description of what the input should be.",
            "label": "Input dataset (NetCDF)",
            "name": "input_dataset",
            "optional": false,
            "type": "file"
        }
    ],
    "version": 1
}

GET /toolboxes

Return a listing of all the toolboxes in the catalogue.

{
    "toolboxes": [
        {
            "@id": "http://vhirl-dev.csiro.au/scm/toolboxes/1",
            "author": "Fred",
            "created_at": "Thu, 06 Nov 2014 16:23:47 GMT",
            "description": "ANUGA is a tool which can simulate events and their effects as they \u2018progress\u2019 or travel through a scenario in a model. You can model the \u2018wave\u2019 or storm surge to measure the impact and risk for known locations.",
            "id": 1,
            "name": "ANUGA",
            "type": "toolbox",
            "uri": "http://vhirl-dev.csiro.au/scm/toolboxes/1",
            "version": 1
        },
        {
            "@id": "http://vhirl-dev.csiro.au/scm/toolboxes/2",
            "author": "Fred",
            "created_at": "Thu, 06 Nov 2014 16:23:47 GMT",
            "description": "The Tropical Cyclone Risk Model is a stochastic tropical cyclone model developed by [Geoscience Australia](http://www.ga.gov.au) for estimating the wind hazard from tropical cyclones.",
            "id": 2,
            "name": "Tropical Cyclone Risk Model",
            "type": "toolbox",
            "uri": "http://vhirl-dev.csiro.au/scm/toolboxes/2",
            "version": 1
        }
    ]
}

GET /toolboxes/:id

Return a detailed description of the toolbox with id.

{
    "@id": "http://vhirl-dev.csiro.au/scm/toolboxes/1",
    "author": "Fred",
    "created_at": "Thu, 06 Nov 2014 16:23:47 GMT",
    "description": "ANUGA is a tool which can simulate events and their effects as they \u2018progress\u2019 or travel through a scenario in a model. You can model the \u2018wave\u2019 or storm surge to measure the impact and risk for known locations.",
    "homepage": "https://anuga.anu.edu.au",
    "id": 1,
    "images": [
        {
            "image_id": "Melbourne/7fa567fb-24bb-412b-8c15-5002370ca199",
            "provider": "necta:wr-openstack-compute",
            "sc_path": "/opt/anuga"
        },
        {
            "image_id": "RegionOne/3f98cc0f-8b27-44ce-8979-df1f57c00558",
            "provider": "nci-openstack-compute",
            "sc_path": "/opt/anuga"
        }
    ],
    "license": {
        "url": "https://anuga.anu.edu.au/svn/anuga/trunk/anuga_core/source/anuga/LICENSE.txt"
    },
    "name": "ANUGA",
    "source": {
        "type": "svn",
        "url": "https://anuga.anu.edu.au/svn/anuga/trunk/anuga_core/"
    },
    "uri": "http://vhirl-dev.csiro.au/scm/toolboxes/1",
    "version": 1
}

-- GeoffreySquire - 24 Nov 2014
Topic attachments
I Attachment Action Size Date Who Comment
ssc-model.graphmlgraphml ssc-model.graphml manage 10.0 K 24 Nov 2014 - 11:42 GeoffreySquire  
ssc-model.pngpng ssc-model.png manage 11.2 K 24 Nov 2014 - 11:42 GeoffreySquire  
ssc-model.svgsvg ssc-model.svg manage 11.1 K 24 Nov 2014 - 11:42 GeoffreySquire  
ssc-system.graphmlgraphml ssc-system.graphml manage 51.1 K 24 Nov 2014 - 10:23 GeoffreySquire  
ssc-system.pngpng ssc-system.png manage 30.3 K 24 Nov 2014 - 10:23 GeoffreySquire  
ssc-system.svgsvg ssc-system.svg manage 44.8 K 24 Nov 2014 - 10:23 GeoffreySquire  
Topic revision: r2 - 13 Feb 2015, RyanFraser
 

Current license: All material on this collaboration platform is licensed under a Creative Commons Attribution 3.0 Australia Licence (CC BY 3.0).