"Seegrid will be due for a migration to confluence on the 1st of August. Any update on or after the 1st of August will NOT be migrated"

SISS Tool

Introduction

SISS Tool is both a self contained tool and also a puppet module for installing SISS components and their dependencies on Debian-derived Operating Systems with minimal effort. Its target user is system admin and its intent is for use with newly provisioned server or virtual machine. In addition to ease of installation and basic configuration of SISS components, it also aims to provide an easy, managed and seamless path to SISS components' upgrade provided those components are previously installed using SISS Tool.

Precaution: Take note that if you have an existing infrastructure (with Apache Web Server, Apache Tomcat and PostgreSQL Database Server) already set up, this tool is not for you and you shouldn't be using it to install SISS components on any existing infrastructure as this may not work or in worst case scenario it will even corrupt your existing system configurations. To install SISS components on an existing infrastructure, please follow the manual installation procedure.

Features

  • Install/uninstall SISS components (geoserver, geonetwork, sissvoc, thredds and pidsvc).
  • Install/uninstall SISS workshop bundle (in zip format).
  • Install Tomcat 7 and create an instance to host SISS component.
  • Create JNDI data source for a given tomcat instance.
  • Install/uninstall Apache2 Web Server and configure AJP reverse proxy and rewrite rules.
  • Install/uninstall PostgreSQL database server and set up PID service database schema and configuration.
  • Install/Uninstall OpenRDF sesame and workbench web applications.
  • Provide basic configuration of SISS components.

Dependencies

In addition to core modules provided by Puppet Labs, SISS Tool also uses other third-party puppet modules. The following list shows third-party puppet modules in which SISS Tool depends on:
'puppetlabs/apache', '0.9.0' (Apache License, Version 2.0)
'puppetlabs/postgresql', '3.2.0' (Apache License, Version 2.0)
'maestrodev/wget', '1.2.3' (Apache License, Version 2.0)
'maestrodev/maven', '1.1.8' (Apache License, Version 2.0)
'proteon/tomcat', '0.1.15'
'proteon/java', '0.1.6'

Setup

There are 2 ways to set up SISS Tools

Pre-requisite: Installing Puppet
1. wget http://apt.puppetlabs.com/puppetlabs-release-precise.deb
2. sudo dpkg -i puppetlabs-release-precise.deb
3. sudo apt-get update
4. sudo apt-get -y install puppet
5. puppet --version

Note: For those who prefer the nitty-gritty of puppet installation, please refer to the official puppet installation documentation on http://docs.puppetlabs.com/guides/installation.html

(a) manual (SISS Tool as Tool)

1. wget http://siss.csiro.au/siss/releases/3.0/siss-tool-v1.0.1.zip
2. sudo apt-get install unzip (skip if you've already installed it)
3. unzip siss-tool-v1.0.1.zip -d siss-tool
4. cd siss-tool
5. edit manifests/nodes.pp (change the name of the node to your hostname and resource declarations to meet your requirements)
6. sudo ./papply

(b) puppetlabs forge (SISS Tool as Puppet Module) - planned but not available yet as we need to understand more about the licensing for publishing SISS Tool puppet module to Puppetlabs forge...
1. sudo puppet module install mlrgoh/siss
2. sudo puppet module list // check to see if all dependencies successfully installed (optional)
3. create your own puppet manifest file
4. sudo puppet apply <your puppet manifest file>

Usage

This section provides instructions and explanation on how SISS Tool's puppet module can be used to install and configure SISS components.

To set up Apache Tomcat and to create a Tomcat instance

To import SISS Tool's puppet module:
include siss

or

class { 'siss':
  version => 3.0,
}

The above statement initialises variables required for other SISS resource types.

To create a tomcat instance with the default parameters:
siss::instance { 'tomcat_1': }

The above resource declaration will install Apache Tomcat 7 (an open source Java Servlet and JavaServer Pages web container), create a Tomcat instance that runs with OpenJDK 1.7 on port 8080 and enable the tomcat service to start on boot.

To install a SISS component to a given Tomcat instance

siss::component { 'geonetwork@tomcat_1':
   app_name => 'geonetwork',
   ensure   => 'present',
   instance => 'tomcat_1',
   require => Siss::Instance['tomcat_1'],
}

The above resource declaration will deploy one of the SISS components, that is, GeoNetwork to a given Tomcat instance. The 'require' metaparameter specifies that the 'tomcat_1' instance needs to be installed before the SISS component can be deployed.

To uninstall a SISS component from a given Tomcat instance

siss::component { 'geonetwork@tomcat_1':
  app_name => 'geonetwork',
  ensure   => 'absent',
  instance => 'tomcat_1',
  require  => Siss::Instance['tomcat_1'],
}

The above resource declaration will undeploy GetNetwork from 'tomcat_1' instance. Take note that removing SISS component will also remove any custom configuration you've manually made to the component. So, make sure you backup your custom configuration before uninstalling the SISS component.

To add/remove JNDI data source to a given Tomcat instance

For SISS component that uses JNDI data source, the following resource type can be used to add and configure JNDI data source entry for a given Tomcat instance:
siss::component::database { 'jdbc/pidsvc':
  instance      => 'tomcat_1',
  jdbc_driver   => 'org.postgresql.Driver',
  jdbc_url      => 'jdbc:postgresql://localhost:5432/pidsvc',
  jdbc_username => 'pidsvc-admin',
  jdbc_password => 'admin#siss3',
}

To remove the above JNDI data source entry from a given Tomcat instance, simply delete the above resource declaration from your manifest file.

To set up Apache2's AJP reverse proxy and rewrite rule for SISS components

It is recommended that Apache2 Web Server be used to do reverse proxy for SISS components so that Apache Tomcat can be run behind an organisation's firewall. For this reason, SISS Tool's puppet module provided an extension resource type namely siss::ext::apache_web to install Apache2 Web Server and to configure reverse proxy and rewrite rule for SISS components.

$proxy_pass = [
  { 'path' => '/geoserver', 'url' => 'ajp://localhost:8009/geoserver' },
  { 'path' => '/geonetwork', 'url' => 'ajp://localhost:8009/geonetwork' },
  { 'path' => '/sissvoc', 'url' => 'ajp://localhost:8009/sissvoc' },
  { 'path' => '/thredds', 'url' => 'ajp://localhost:8009/thredds' },
  { 'path' => '/pidsvc', 'url' => 'ajp://localhost:8009/pidsvc' },
]

class { 'siss::ext::apache_web':
  ensure     => 'present',
  proxy_pass => $proxy_pass,
  rewrite_cond => "%{REQUEST_URI} !^/geoserver(?:\$|/)|geonetwork(?:\$|/)|sissvoc(?:\$|/)|thredds(?:\$|/)|pidsvc(?:\$|/)|.*?\\.html|favicon\\.ico|.*?\\.css|.*?\\.zip|.*?\\.jpg|.*pydio.*|.*?\\.png|.*?\\.js|robots\\.txt [NC]",
  rewrite_rule => "^/(.*) http://localhost/pidsvc/dispatcher?\$1 [NC,B,QSA,P,L]",
}

siss::instance { 'tomcat_1':
  ajp_connector => 'enabled',
  ajp_port      => 8009,
}

The above set of declarations will install Apache2 Web Server, create a default virtual host that listens on port 80, configure reverse proxy for SISS components on default virtual host, configure simple rewrite rule for PID Service (see PID Service Installation Guide for further details) on default virtual host, and enable AJP connector to listen on port 8009 on 'tomcat_1' instance.

To set up PID Service database schema and configuration

Some SISS components such as PID Service require the setup of its own database schema and configuration. SISS Tool's puppet module provides an extension resource type namely siss::ext::pidsvc_database to get this component specific database schema and configuration setup.

class { 'siss::ext::pidsvc_database':
  ensure             => 'present',
  postgres_password  => 'postgres',
  pidsvc_db_name     => 'pidsvc',
  pidsvc_db_user     => 'pidsvc-admin',
  pidsvc_db_password => 'admin#siss3',
}

The above resource type installs PostgreSQL database, and create PID Service database schema and configuration.

To install SISS workshop bundle unto a SISS-in-a-box's VM

class { 'siss::ext::workshop':
  ensure          => 'present',
  source          => 'https://cgsrv1.arrc.csiro.au/swrepo/siss-repos/3.0/workshop.zip',
  destination_dir => '/var/www',
  download_file   => 'workshop.zip',
}

The above resource type downloads SISS workshop bundle i.e. workshop.zip and unzips its content to user specified destination directory.

To uninstall SISS workshop bundle from a SISS-in-a-box's VM

class { 'siss::ext::workshop': 
  ensure          => 'absent',
  destination_dir => '/var/www',
}

This will remove SISS workshop bundle from user specified destination directory and restore the default index.html file.

Reference

There are two main resource types in SISS puppet module namely core and extension resource types.

The core resource types are the essential part - consists of classes and definitions - of the tool. They are used mainly to set up recommended SISS' runtime environments (i.e. Apache Tomcat and OpenJDK 1.7), and to install and configure SISS components. They'll be supported and enhanced in the future releases.

The extension resource types are provided to complement its core resource types. This means they have very basic configuration options sufficient enough to get their respective designated tasks done no more and no less. It may or may not get supported in the future release. Therefore, it is recommended that you write your own puppet manifest/module or re-use other third-party puppet module/s if the extension resource types do not meet your organisation's configuration and security requirements/use cases.

Our Implementation Rererence Guide can be found here: SISSTool3PuppetReference (deprecating) or here (puppet doc).

Limitations

SISS Tool has only been tested on Ubuntu 12.04 LTS (Precise Pangolin).

Future roadmap

The following items in our future roadmap list are only tentative. They may or may not make it to future SISS Tool releases.
  • To support advanced configuration for individual SISS components.
  • To support other Linux based OSs such as CentOS, Fedora, etc.
  • Choose an open source license and publish 'siss' puppet module to puppet forge.

Contributions

For those who are interested in SISS Tool puppet module code or manifests, you can clone them from the repository below. You are most welcome to make further contributions.

https://stash.csiro.au/scm/siss/siss-tool.git

For feedback and further discussion, you can send your message to configuration-management@lists.csiro.au. General information about the "Configuration-management" mailing list is at http://lists.csiro.au/mailman/listinfo/configuration-management.

A complete example

Let's briefly look into one scenario on how we could use SISS Tool to install SISS components on a new machine/server node. SISS in a box v3.0 (Virtual Machine) is set up based on the instructions provided in this example.

We'll use the tool to deploy SISS components based on the following logical UML deployment diagram:

siss-demo1-deployment.PNG

Take note that the above SISS deployment architecture (all in one box) is only for demonstration purpose and it is not our recommended deployment model.

Example directory structure

Before installing SISS components, we need to get SISS Tool setup (please refer to the above Setup section for further details).

The following tree diagram illustrates SISS Tool's directory structure once it is set up based on manual installation approach:

ubuntu@demo1:~$ cd siss-tool/
ubuntu@demo1:~/siss-tool$ tree -L 2
.
├── manifests
│ ├── nodes.pp
│ └── site.pp
├── modules
│ ├── apache
│ ├── apt
│ ├── concat
│ ├── firewall
│ ├── java
│ ├── maven
│ ├── postgresql
│ ├── profile_d
│ ├── siss
│ ├── stdlib
│ ├── tomcat
│ └── wget
└── papply.sh

14 directories, 3 files

nodes.pp

The purpose of this file is to enable you to have different configurations for each machine or node. For instance, in production environment, it is a common practice to have one node that runs a web server, another node that runs an application server and a third node that runs a database server. In this instance, you'll use a node declaration to tell Puppet which configuration belongs to each server node. The default nodes.pp file that comes with the tool contains only sample configuration of a single node.

The following nodes.pp file demonstrates the configuration we used to set up our SISS in a box v3.0 (Virtual Machine):
node 'demo1' {
  include siss

  # Install Apache2 web server and configure AJP reverse proxy
  # and rewrite rule with conditions for SISS components. The 
  # rewrite_cond contains an exclusion list of request paths
  # specified in proxy_pass variable. Any requested paths
  # that are not included in the exclusion list will be proxied
  # to PID service dispatcher.

  $proxy_pass = [
    { 'path' => '/geoserver', 'url' => 'ajp://localhost:8009/geoserver' },
    { 'path' => '/geonetwork', 'url' => 'ajp://localhost:8009/geonetwork' },
    { 'path' => '/sissvoc', 'url' => 'ajp://localhost:8009/sissvoc' },
    { 'path' => '/thredds', 'url' => 'ajp://localhost:8009/thredds' },
    { 'path' => '/pidsvc', 'url' => 'ajp://localhost:8009/pidsvc' },
    { 'path' => '/manager', 'url' => 'ajp://localhost:8009/manager' },
    { 'path' => '/openrdf-workbench', 'url' => 'ajp://localhost:8009/openrdf-workbench' },
  ]

  class { 'siss::ext::apache_web':
    ensure       => 'present',
    proxy_pass   => $proxy_pass,
    rewrite_cond => "%{REQUEST_URI} !^/geoserver(?:\$|/)|geonetwork(?:\$|/)|sissvoc(?:\$|/)|thredds(?:\$|/)|pidsvc(?:\$|/)|manager(?:\$|/)|openrdf-workbench(?:\$|/)|.*?\\.html|favicon\\.ico|.*?\\.css|.*?\\.zip|.*?\\.jpg|.*pydio.*|.*?\\.png|.*?\\.js|robots\\.txt [NC]",
    rewrite_rule => "^/(.*) http://localhost/pidsvc/dispatcher?\$1 [NC,B,QSA,P,L]",
  }
  
  # Create tomcat instance, enable AJP connector on port 8009, and override 
  # GeoServer's data directory and Thredd's configuration directory.

  siss::instance { 'tomcat_1':
    ajp_connector      => 'enabled',
    ajp_port           => 8009,
    geoserver_data_dir => '/tmp/geoserver',
    thredds_conf_dir   => '/tmp/thredds',
  }

  # Install SISS components

  siss::component { 'geoserver@tomcat_1':
    app_name => 'geoserver',
    ensure   => 'present',
    instance => 'tomcat_1',
    require  => Siss::Instance['tomcat_1'],
  }

  siss::component { 'thredds@tomcat_1':
    app_name => 'thredds',
    ensure   => 'absent',
    instance => 'tomcat_1',
    require  => Siss::Instance['tomcat_1'],
  }

  siss::component { 'geonetwork@tomcat_1':
    app_name => 'geonetwork',
    ensure   => 'present',
    instance => 'tomcat_1',
    require  => Siss::Instance['tomcat_1'],
  }

  siss::component { 'sissvoc@tomcat_1':
    app_name => 'sissvoc',
    ensure   => 'present',
    instance => 'tomcat_1',
    require  => Siss::Instane['tomcat_1'],
  }

  siss::component { 'pidsvc@tomcat_1':
    app_name => 'pidsvc',
    ensure   => 'present',
    instance => 'tomcat_1',
    require  => [ Siss::Instance['tomcat_1'], 
                  Siss::Component::Database['jdbc/pidsvc'], 
                  Class['siss::ext::pidsvc_database'] ],
  }

  # Set up PID service JNDI data source in tomcat_1 instance.

  siss::component::database { 'jdbc/pidsvc':
    instance      => 'tomcat_1',
    jdbc_driver   => 'org.postgresql.Driver',
    jdbc_url      => 'jdbc:postgresql://localhost:5432/pidsvc',
    jdbc_username => 'pidsvc-admin',
    jdbc_password => 'admin#siss3',
  }

  # Install PostgreSQL and set up PID service database 
  # schema and configuration.
  
  class { 'siss::ext::pidsvc_database':
    ensure             => 'present',
    postgres_password  => 'postgres',
    pidsvc_db_name     => 'pidsvc',
    pidsvc_db_user     => 'pidsvc-admin',
    pidsvc_db_password => 'admin#siss3',
  }  

  # Install OpenRDF's sesame and workbench web apps
  
  siss::ext::openrdf { 'sesame@tomcat_1':
    ensure   => 'absent',
    instance => 'tomcat_1',
    require  => Siss::Instance['tomcat_1'],
  }

  # Install SISS workshop bundle

  #class { 'siss::ext::workshop':
  #  ensure          => 'present',
  #  source          => 'http://siss.csiro.au/siss/releases/3.0/workshop.zip',
  #  destination_dir => '/var/www',
  #  download_file   => 'workshop.zip',
  #  require         => Class['siss::ext::apache_web'],
  #}

  # You can use Proteon/Tomcat module to add
  # Tomcat manager web app for tomcat_1 instance and
  # to configure a realm (authentication) for tomcat_1 instance
  # in order for user to access Tomcat manager web app.

  tomcat::webapp::manager { ['tomcat_1']:
    require => Siss::Instance['tomcat_1'],
  }

  tomcat::realm::userdatabase { 'tomcat_1_userdatabase_realm':
    instance => 'tomcat_1',
  }

  tomcat::realm::userdatabase::role { ['manager-gui']:
    instance => 'tomcat_1',
  }

  tomcat::realm::userdatabase::user { 'admin':
    instance => 'tomcat_1',
    password => 'admin',
    roles    => 'manager-gui',
  }
}

site.pp

The is the main manifest file for SISS Tool's executable shell script (papply.sh). Although it is possible to put all your Puppet manifests into this site manifest file, however, it is recommended that you keep this file small and use it only to load other manifest files, such as nodes.pp.
import 'nodes.pp'

To run the example site manifest:

cd siss-tool
sudo ./papply

Once the above site manifest file has successfully finished its execution, you'll be informed of which SISS components are installed and the basic information on how these components can be accessed as follows:
Notice: /Stage[main]/Siss/Concat[/var/log/siss-tool.log]/Exec[concat_/var/log/siss-tool.log]: Triggered 'refresh' from 2 events
Notice: /Stage[main]/Siss/Concat[/var/log/siss-tool.log]/File[/var/log/siss-tool.log]/content: content changed '{md5}49df5c8160e33cb73c174
Notice: Finished catalog run in 85.94 seconds

The following provides basic information for accessing SISS components
installed by SISS Tool on this machine. You may be able to access these
components via their respective reverse proxy paths (see nodes.pp
manifest file for further details).

Base URL:
  http://<host_ip_address>:8080

SISS Component URIs:
  /geonetwork
  /pidsvc
  /sissvoc
  /geoserver

Take note that SISS Tool is only capable of knowing the components managed by its own puppet module. Component that is not installed using SISS puppet resource type won't be included in the above reporting.

Troubleshooting

SISS Tool successfully run but tomcat instance failed to start up

It seems like the tomcat executable installed by SISS Tool puppet module requires X Window System related dependencies in order to properly run. This issue has been addressed by running tomcat executable with headless mode set to true. Install these dependencies if you run into X11 related issue (check your tomcat instance's catalina.out file).
sudo apt-get install libxi6 libxtst6 libxrender1

Failed to secure shell into my Ubuntu virtual machine or system

It is possible that you don't have an SSH server set up on your Ubuntu system or virtual machine. By default, SISS Tool doesn't install and configure ssh server. Use command below to install the SSH server if you don't have it installed on your system. If after installing the SSH server and you still can't secure shell into your Ubuntu virtual machine/system, please contact your system admin.

sudo apt-get install ssh openssh-server

-- RichardGoh - 26 Nov 2013
Topic revision: r38 - 13 Aug 2014, BenCaradocDavies
 

Current license: All material on this collaboration platform is licensed under a Creative Commons Attribution 3.0 Australia Licence (CC BY 3.0).