"Seegrid will be due for a migration to confluence on the 1st of August. Any update on or after the 1st of August will NOT be migrated"

SISS Release Guide

Overview

This guide is for AuScope members who wish to actually package and release a new version of the Spatial Information Services Stack (SISS). For the high level release strategy guide please see SpatialInformationServicesStackReleases

There are four options for SISS releases depending on the needs of the user. These are:

1.SISS Installer

2.SISS-In-A-Box

3.Manual process

4.SISS Tool

Finalising Components

SISS itself is a collection of different software packages and a SISS release will involve updates to some/all of these components.

The first step in preparing a SISS release is to contact each of the persons responsible for the below components and discuss the following:
  • What is the latest stable release and should SISS be upgrading to it?
  • What is the current status of development?
    • Can a release/stable revision be made/discovered for the upcoming SISS release?
This is as much a discussion as it is a negotiation; There will normally be various drivers behind a SISS release and it's important that the component leads understand these driving forces.

As 'stable' versions for the components are identified you will need to ensure that the testing procedure for these components (assuming they have changed from the previous release) are run and pass.

SISS Components as of SISS 3.0
Component Implementations Testing Procedure
Registry Services GeoNetwork GeoNetwork - Basic test
Web (Feature/ Map) Services GeoServer Geoserver - SISSReleaseTestingGeoserver
Vocabulary Service Vocabulary Service Vocabulary Service - TODO
Coverage Services Coverage Development Coverages - TODO
Persistent Identify Service PID Service PID Service - TODO

Building SISS Installer

Once a list of components has been identified and confirmed as being 'stable' you will want to package these components into the SISS Installer.

Please note that the installer can only be developed AND executed under Windows XP. We have a build machine ready to go at cgisrv8 that can be accessed via remote desktop. If you use an existing build machine you can skip the 'Preparing Build Machine' section.

Preparing Build Machine

These steps are only required if you are setting up a build machine from scratch

Installing NSIS

The SISS Installer requires the 'Nullsoft Scriptable Install System' (NSIS) which can be downloaded from http://nsis.sourceforge.net/Download (if using a non Windows machine you will need to compile the source yourself).

After installing NSIS, you will need to install the following plugins
  • Image.nsh
    • To install just unzip plugin into root NSIS installation directory
  • TextReplace.nsh
    • To install just unzip anywhere and run the included install.exe
  • AccessControl.dll
    • To install just unzip plugin into root NSIS installation directory
  • Dialogs.nsh
    • Installation for this plugin is a little trickier...
    • Unzip ZIP into NSISDirectory/Contrib/Dialogs
    • Move NSISDirectory/Contrib/Dialogs/defines.nsh to NSISDirectory/include/defines.nsh
Note: You will need to add the root directory of the installation to your PATH environment variable so that 'makensis.exe' is accessable from the command line.

Getting started

Start by checking out the latest revision of the SISS Installer

Subversion - https://cgsrv1.arrc.csiro.au/subversion/siss-installer/trunk

Next begin by familiarising yourself with a few 'files of interest' in the SVN repository you just checked out.
File Overview
./SISS Installer Developer Guide.txt A readme describing where the opengeo suite and dashboard has been sourced from.
./installer/windows/SissInstaller.nsi A plaintext NSIS install script dictating a number of install steps.If you need to change the way a SISS component is installed, this is likely the place to make that change.
./apache-tomcat-7.0.42/data_dir The Geoserver data directory that gets installed into the current user's profile.
./apache-tomcat-7.0.42/webapps Where all SISS component WAR files should be unpacked.

Upgrading Geoserver

Test the version of GeoServer on https://cgsrv8.arrc.csiro.au/jenkins/view/Deployment-Test/

Run the jobs in the following order
  • GSTest-Deployment-ERML
  • GSTest-Deployment-NVCL

This process will set up

We can then run tests on https://twiki.auscope.org/wiki/Grid/AuScopePortalFilterManual to ensure the new Geoserver build doesn't break on setup.

The above set up will help assist test to ensure all software suite comply with each other.

Note: Both GSTest runs with optional build date parameter to allow us the choose the date of the build we want to build for GeoServer .

Go to HowSISSDeploymentTestWork for more information.

Upgrading THREDDS

TODO

Upgrading GeoNetwork

Undeploy previous test deployments on :

http://sisstest.arrc.csiro.au:8080/manager/html

Run the build in https://cgsrv8.arrc.csiro.au/jenkins/job/GeoNetwork-Deployment/ that will deploy the latest with RIF-CS in sisstest.arrc.csiro.au:8080/geonetwork and http://siss.csiro.au/siss/geonetwork/. Test the basic setup of GeoNetwork.

Following that, use GeoNetwork to harvest wfs from either GeoServer ERML or NVCL instance setup from earlier GeoServer test

Run some filter test on GeoNetwork such as getRecordById, Boundingbox Test and box intersect test on:

http://sisstest.arrc.csiro.au:8080/geonetwork/srv/en/test.csw

Upgrading SISSVoc

In https://cgsrv8.arrc.csiro.au/jenkins/ we have ELDA, ELDA config, ELDA Head and ELDA head config. Elda points to stables branches and EDLA head points to trunk therefore unstable.

For releasing SISSVoc,
  • Firstly speak to the component owner to determine if a new release is required.
  • Check the elda version that is build by the jenkin's job ELDA and ensure that it is the right version for the release.
  • If everything is set, build the release by running elda-config job. (select the right location and version)
All information regarding builds and version can be found in the job console output.

Upgrading PID Service

The Jenkins' job for PID Servce can be found here: https://cgsrv8.arrc.csiro.au/jenkins/job/PidService/

For releasing PID Service,
  • Firstly speak to the component owner (Pavel Golodoniuc) to determine if a new release is required.
  • If a new release is required, the component owner will tag the latest or stable version of PID Service for SISS release, proceed with the build, and update PID Service documentation with the new build's release note.

Building 'SISS in a Box' VM (manual process)

Notice: Starting from SISS v3.0 release, we will have a whole new way of building and upgrading SISS in a box VM (see the next section on Building SISS in a BOX VM using SISS Tool for further details).

The general strategy for upgrading the SISS in a box VM involves downloading the previous release and upgrading it's various components to match the release you are about to make.

Getting Started

  1. Download the previous VM release: https://www.seegrid.csiro.au/wiki/Siss/SISSReleases
  2. Download and install VMWare Player:http://www.vmware.com/products/player/
  3. Start the previous VM:

Upgrading VM

To upgrade the VM please follow the manual instructions for deploying your component (that should be included as part of each component's release).

Note - There are 2 geoserver instances geoserver-trunk and geoserver. The geoserver instance should not be touched during a Geoserver deployment as it is a requirement of Geonetwork, geoserver-trunk represents the SISS 'blessed' version of Geoserver

Packaging

Once you have verified that all components have been successfully deployed:
  1. Shutdown the VM shutdown -h now
  2. Package all VM files (skipping any logs) into a ZIP archive
  3. Upload the ZIP archive to downloads.auscope.org to make it available to the public
    • Do this by uploading your archive to =cgsrv8 /mnt/data/webdata/downloads.auscope.org-www/archive/siss-releases/=
  4. Moving forward we will upload the vm to file.ivec.org due to the size of the vm, we may run out of space on downloads.auscope.org.

Building 'SISS in a Box' VM (using SISS Tool)

Starting from SISS v3.0 release, the SISS in a box VM will be set up using a new tool called SISS Tool. It is a self-contained tool and a puppet module for installing SISS components and their dependencies on debian-based operating systems with minimal effort.

This SISS in a box VM will also be used in subsequent releases of SISS as a means to test SISS components upgrade using the SISS Tool.

Getting Started

  1. Request a VMware image with Ubuntu Server 12.04 installed from CSIRO's IM&T ServiceCentre. Alternatively, you can also create one yourself with VMware Player or download the image from VMware image providers such as VMware Solution Exchange, ThoughtPolice, Traffic Tool, etc.
  2. Once you got the image ready, open and start it up in VMware Player.
  3. At this point in time, you can work on the VM directly in VMware Player or use any SSH client such as PuTTY to connect to the VM (this will require SSH server installed on your VM). Regardless of the way you'll be interacting with the VM, follow the setup instructions and a complete example given in SISS Tool documentation to get the SISS in a box VM setup.

Upgrading VM

The same SISS Tool should be used to perform the upgrade for SISS in a box VM we set up since SISS v3.0 release. The plan or future roadmap is that SISS Tool will be enhanced for each SISS release and the enhanced SISS Tool should be used to test the VM upgrade before its release with other SISS artifacts.

The first release of SISS Tool (i.e. v1.0.1) hasn't been fully tested yet for SISS in a box VM upgrade. It has a class resource type which allows the version of SISS (supported by SISS Tool) to be specified. The idea is that the tool will then install or upgrade SISS components matching the specified SISS version number.

Packaging

This process is similar to the way a manually built SISS in a box VM is packaged (see the above section for further details).

Building SISS Tool

SISS Tool is a new tool that will be made available to users who are system admins starting from SISS v3.0 onwards. It is a self-contained tool and a puppet module for installing SISS components and their dependencies on debian-based operating systems with minimal effort. Its user guide can be found here https://www.seegrid.csiro.au/wiki/Siss/SISSTool and its implementation reference here https://cgsrv8.arrc.csiro.au/jenkins/job/siss-tool/ws/apidocs/index.html.

Getting Started

The first thing to get started with building SISS Tool is to get yourself familiarised with the tool by reading its user and reference guides provided earlier. In addition to that, you'll also need to know the basics of using Puppet and writing Puppet manifests (see this link http://docs.puppetlabs.com/ to learn more about Puppet).

Once you've familiarised yourself with the tool and are ready for SISS Tool development, check out the tool's master branch from Stash or CSIRO Git repository: https://stash.csiro.au/projects/SISS/repos/siss-tool, and familiarise yourself with its contents. In a nutshell, SISS Tool is essentially comprised of a number of interdependent open source third-party and in-house built puppet modules. We created our own 'siss' puppet module mainly to hide the complexity of working with other third-party puppet modules and also to provide an easy and managable way for future SISS release components upgrade.

Before moving on to talk more about those puppet modules, let's first look into 3 main files of interest in the SISS Tool's root and manifests directories.

File Overview
papply.sh This is the shell script that runs the tool's site manifest file (i.e. site.pp) which located under its own 'manifests' directory.
manifests/site.pp The tool's site manifest file that contains a statement to load the nodes manifest file. Note: Although it is possible to put all your Puppet manifests into this site manifest file, however, it is recommended that you keep this file small and use it only to load other manifest files, such as nodes.pp.
manifests/nodes.pp This manifest file contains a set of pre-defined resources ready for production use or for educational purpose. Note: If you create a new resource type for or change an existing resource type in 'siss' puppet module, it is advised that you also update this file to include the new resource type or to reflect the change .
Now, in regard to third-party and 'siss' puppet modules, they are all stored in SISS Tool's modules directory. Table below shows the interdependency between 'siss' puppet resource types and third-party puppet modules. Do take note that each thirty-party puppet module has other dependencies too and it is impractical for us here to list all those dependencies unless they have custom changes made by us that are worthy of mentioning. It is not normally the case for us to make changes to third-party puppet modules. However, sometimes, this is unavoidable when working with open source communities. If that happens, it is advised that you take note or make explicit of the changes made here and collaborate with respective open source user communities to get those changes incorporated into future release of their puppet modules.

Third-party Used In Comment
puppetlabs/apache 0.9.0 siss::ext::apache_web  
puppetlabs/postgresql 3.2.0 siss::ext::pidsvc_database  
maestrodev/wget 1.2.3 siss::component
siss::ext::openrdf
siss::ext::workshop
Added fetchzip.pp and fetchtarball.pp files from Sharon Tickell (Sharon.Tickell@csiro.au) to this module.
proteon/tomcat 0.1.15 siss::instance
siss::component
siss::component::database
siss::ext::openrdf

Modified the following manifest files (connector.pp, instance.pp, connector/ajp.pp, connector/http.pp, and connector/https.pp) so that multiple connectors can be enabled at the same time. The original behaviour allows only one connector to be enabled at a time.

Modified instance.pp file to accept 'http_port' parameter rather than defaulting to 8080.

Modified webapp.pp file to stop the cleaning up of tomcat instance resource type from running. It seems like that resource type has caused issue for web application deployment. TO DO: Further investigation is needed to properly under its original behaviour.

ripienaar/concat 0.2.0 siss
siss::instance
siss::component
siss::ext::openrdf
 
Having mentioned about third-party puppet modules, let's turn our focus now to SISS Tool's own puppet module. The initial development of SISS Tool is driven by a use case that wanting to make the installation and upgrading of SISS components seamless on linux based operating systems. This goal has been partially achieved (discounting the upgrading part) in the first release of SISS Tool. Please refer to the Features section of SISS Tool documentation to see what the current 'siss' puppet module can do.

Going forward the tool will need to be improved for performing SISS components upgrade and other use cases based on users' feedback and also SISS roadmap. The current design of 'siss' puppet module has divided its resource types into 2 categories namely core resource types and extension resource types (see the Reference section of SISS Tool documentation for further explanation). So, use that design and abstraction principle to guide you in designing your next 'siss' resource types. In addition to that, always remember to write inline documentation for 'siss' resource types using RDoc markup (http://docs.puppetlabs.com/guides/style_guide.html#puppet-doc) because online documentation can then be easily generated using the puppet doc command. Final notes on the design of your next 'siss' resource types are use your creativity to come out something intuitive for the users and talk to each SISS component owner for any common configurations that can be automated.

Last but not least, take note that SISS Tool does not make use of common default modulepaths (such as /etc/puppet/modules, /usr/share/puppet/modules, etc.) rather it uses the modulepath defined in its 'papply.sh' shell script which points to its own modules directory. This is a deliberate choice to ensure stability of the tool by isolating the puppet modules installed in common default modulepaths from the ones used by SISS Tool. As such, any upgrade to the puppet modules installed in the common default modulepaths won't be causing unexpected behaviours to SISS Tool.

Upgrading SISS Tool

Upgrading of SISS Tool requires you to understand the tool (see Getting Started section to learn more about the tool), and of course the enhancements or changes that you're going to make to the tool. It is impossible to list down detailed instructions on how to upgrade SISS Tool. Instead we'll suggest a number of things to consider when doing the upgrade as follows:
  1. If you're upgrading the tool's third-party puppet module, ensure you upgrade its dependencies too and most importantly you MUST test to ensure the tool still works as expected after the upgrade.
  2. Take note of the 'component_location' variable in siss class (i.e. init.pp file) as it holds the key/value pairs of SISS component names and their respective locations. You won't have to change this variable unless there is a change in a particular component name/location or there is a new component added to the new SISS release.
  3. For each SISS Tool upgrade, ensure its documentation (https://www.seegrid.csiro.au/wiki/Siss/SISSTool) is kept up-to-date with any changes that you've made to the tool.

Packaging SISS Tool

A Jenkins job has been set up to get the tool built (including its reference/puppet doc) and to copy it to a designated SISS release directory on iVEC Petabyte Data Store (PB Store). As of the first release of SISS Tool, the packaging of SISS is not fully automated yet and will require a number of up-front tasks to be carried out before each SISS new release. These up-front tasks are as follows:
  1. Once the tool's master branch has reached a stable and release stage, tag it with the right version (following AuScope /SISS versioning convention), and push it to remote origin repository. The first stable version was tagged with 'v1.0.0'.
  2. Next, you'll need to configure the tool's Jenkins job at https://cgsrv8.arrc.csiro.au/jenkins/job/siss-tool/ by (a) add in the new tag version to BRANCH parameter, and (b) add in the new SISS release number for SISS_VERSION parameter (this is used to determine the location in PB Store where the build should be copying to; see the job's Post-build actions section for further details). Remember to save your changes once it is done.
  3. The last thing here is you'll need to build the tool by using the 'Build with Parameters' link. On the 'Build with Parameters' page, choose the right branch to build and select the SISS version where the successful build should belong. Also, check the job's console output or status to ensure the job has successfully run.

Documenting Release

Documenting a SISS release is an ongoing issue and should NOT be an afterthought.
Pages to Maintain Notes
This page! This page contains a number of processes that may become outdated as SISS components change/evolve. It's important this guide stays up to date to ensure people follow it.
SISS Releases Page

A SISS release must be fully documented with:

  • Manual installation guide for SISS (ie a guide for every component)
  • Where the SISS installer can be found
  • Where the SISS in a box VM can be found
  • Where the SISS Tool can be found
  • Release notes

Announcing Release

Updating SISS Releases

The first step in announcing a release is to update the SISS releases page with all of your release artifacts. - https://www.seegrid.csiro.au/wiki/Siss/SISSReleases. It's easiest to copy an existing release as a template.

Updating SISS Website

Refer to siss_website_how-to.docx for documentation on the SISS website. To announce a SISS release, we will have to update the announcement page on the SISS website. The following files will need to be edited:
  • SISS Web\eventheader\text.xml Edit this for the content of the announcement page
  • SISS Web\download\text.xml Edit this to point to our latest release artifacts
The template used for the announcement on the SISS website is the same as the one used for the mailing list.


Announcing via Mailing List

The two mailing list that have been created for SISS are as follow: Do note that the mailing list consist of subscribers from external sources therefore keep the content professional.

The template for the announcement:

MAKE SURE YOU CHECK ALL OUTGOING LINKS IN THE EMAIL

Announcing SISS 2.0

We are pleased to announce the immediate availability of SISS 2.0, the second major release of the Spatial Information Services Stack.

We invite you to download the components, try them out and provide feedback.

This release contains the following components:
  1. Web Coverage Services - Thredds
  2. Modelling Tools - FullMoon
  3. Registry - GeoNetwork
  4. Persistent Identifier Service - Uniform Resource Name (URN) Resolver
  5. Vocabulary Services - SISSVoc
  6. Web Feature Services (WFS) / Web Map Services (WMS) - GeoServer
Highlights At A Glance:
  • Geoserver has increased performance, GML 3.2 support,a great number of incremental improvements and bug fixes
  • GeoNetwork has profile and RIF-CS support
  • SISSVoc has improved and refactored its code base
  • ..... and many more!
You may view our release notes for further details

Getting SISS 2.0

SISS 2.0 has tailored different options depending on your needs:
  1. Manual Installation - Production environment option 1
  2. "SISS-In-A-Box" - Virtual machine with included sample data
  3. SISS Installer - Windows ONLY demonstrational tool
  4. SISS Tool - Production environment option 2
Visit our SISS Releases page for more information and help us shape our future releases.

Support :

If you have questions concerning any of the services, please feel free to contact the SISS mailing list ( siss@lists.csiro.au)

Further information:

SISS website - http://siss.auscope.org/

SEEGrid Wiki - https://www.seegrid.csiro.au/wiki/Siss/SISSReleases

-- JoshVote - 19 Oct 2011
Topic revision: r33 - 22 Oct 2014, RiniAngreani
 

Current license: All material on this collaboration platform is licensed under a Creative Commons Attribution 3.0 Australia Licence (CC BY 3.0).