"Seegrid will be due for a migration to confluence on the 1st of August. Any update on or after the 1st of August will NOT be migrated"

Publish BOM Observational Data via 52°North Sensor Observation Services (SOS)


Sensor Observation Services (SOS) provide the mechanisms to manage deployed sensors and obtain “observational” data from them in an interoperable manner. SOS is currently not part of SISS; it would be beneficial that SOS is integrated into the stack.The ‘52°North’ project provides widely-deployed implementations of both SOS and WPS, and this activity will investigate the feasibility of integrating SOS within SISS, and undertake the integration activity to the extent possible.

Workflow (Target Date : End of Jun 2012)

  • Step 1 : Sensor Observation Service (SOS) Review
  • Step 2 : Identify and Obtain Suitable Timeseries Data
  • Step 3 : Create Database Mapping and Database Script
  • Step 4 : Develop ETL using Talend Open Studio
  • Setp 5 : Review 52°North SOS With BOM Timeseries Data
  • Step 6 : Deployment Guide

Step 1 : Sensor Observation Service (SOS) Review

  • Understand SOS and the difference between SOS1.0.0 and SOS2.0.0. Summarize of SOS specifications:
  • 52°North SOS has been mentioned in the project plan but it is good to evaluate other open source SOS for best solution. Review has been carried out on 52°North SOS, GeoCENS, MapServer SOS and Deegree SOS. The finding has been recorded at https://www.seegrid.csiro.au/wiki/Siss/SOSReview.
  • Base on the review, 52°North SOS is the best option as it is the only one that support SOS2.0 (with minimum required filters), with good documentation and responsive community support. However, it is not good enough to be integrated into SISS because of the tight coupling to it's own database, limited to only SWE Common schema for the timeseries result mapping and small change request in the mapping require changes in the source code. Further research is still on going for best solution but for this project, 52°North SOS will be use.

Step 2 : Identify and Obtain Suitable Timeseries Data

  • Andrew Woolf, Architecture Section Manager from BoM has indicated that AWRIS dataset will be used in this project.

Step 3 : Create Database Mapping and Database Script

  • 52°North SOS has tight coupling to it's own database (PostgreSQL), thus require good understanding of the database schema for database mapping and good knowledge of sql or other feeder tool (e.g. Talend ETL, script, etc) for feeding data to the database.
  • ETL has been chosen for feeding data from AWRIS database into SOS database.
  • Approach taken for the mapping :
    1. Create database mapping that map AWRIS database to SOS database. Completed mapping can be downloaded from : https://www.seegrid.csiro.au/subversion/ASRDC/BOM/SOS/databaseMapping/SOSDBMapping.xls
    2. Create views in AWRIS database with similar columns as the tables in SOS database, this enable easier ETL mapping.
    3. Completed database script for creating the views can be downloaded from : https://www.seegrid.csiro.au/subversion/ASRDC/BOM/SOS/databaseMapping/createViewScript.sql
  • Note : for demo purpose, the feature of interest id has been created similar to the feature of interest id in the observational data that published thru GeoServer WFS.

Step 4 : Develop ETL using Talend Open Studio

  • Talend Open Studio (TOS) has been used in the previous BoM work for mapping Services Information Management System (SIMS) data into GeoNetwork database. A guide has been written by Rini Angreani and it is a good start to learn ETL : https://www.seegrid.csiro.au/wiki/ASRDC/SIMSTalendETLGuide.
  • The ETL mapping for this project is simpler than the earlier project and some of the customised routine has been reused in this project (e.g. EmailGenerator and LogGenerator).
  • ETL job design for mapping data from AWRIS database to SOS database in summary :
    1. Initialize contexts parameters from an external config file (e.g. database connection details, email details, log file directory, etc).
    2. Create database connections to both AWRIS and SOS database. Terminate job and send email notification if connection fail.
    3. Truncate tables in SOS database before inserting records from AWRIS table to SOS table one by one. Terminate job, roll back database job and send email notification if encounter error.
    4. Final step is to create the Sensors' XML files.
    5. Logging is on most of the steps to log the sucess and failure of each job.
  • Completed ETL source can be downloaded from : https://www.seegrid.csiro.au/subversion/ASRDC/BOM/SOS/ETL/. The exported job can be downloaded from : https://twiki.auscope.org/projects/bom/mapping_awrisdb_to_sosdb_0.1.zip.

Step 5 : Review 52°North SOS With BOM Timeseries Data

Step 5 : Step 6 : Deployment Guide

Topic revision: r6 - 25 May 2012, FlorenceTan