"Seegrid will be due for a migration to confluence on the 1st of August. Any update on or after the 1st of August will NOT be migrated"

Jenkins GeoServer Master Technical Notes

This page gives the technical details of the configuration of JenkinsGeoserverMaster. Note that "master" in this sense refers to the git development branch, not the Jenkins master.

Jenkins master

This view lists the jobs on Jenkins-master that build and publish GeoServer master:

Tools

OpenJDK 7: Debian package openjdk-7-jdk:amd64 installed on slave and configured on master
Name openjdk-7-debian-local
JAVA_HOME /usr/lib/jvm/java-7-openjdk-amd64
Install automatically unchecked

Maven 3: configured on master and automatically installed on slave
Name apache-maven-3.0
Install automatically checked
Install from Apache Version 3.0.4

This selection of tools results in the following "mvn -version" on the slave:
Apache Maven 3.0.4 (r1232337; 2012-01-17 16:44:56+0800)
Maven home: /home/jenkins/tools/Maven/apache-maven-3.0
Java version: 1.7.0_55, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-7-openjdk-amd64/jre
Default locale: en_AU, platform encoding: UTF-8
OS name: "linux", version: "3.2.0-4-amd64", arch: "amd64", family: "unix"

Slave node

The slave node must be configured on the master.

Under Manage Jenkins / Manage Nodes:
Name apacsrv4
# of executors 1
Remote FS Root /home/jenkins
Usage Leave this machine for tied jobs only
Launch method Launch slave agents via Java Web Start

The master has no means of connecting to the slave. Instead, the slave connects to the master via JNLP and launches itself. Once connected, the master is in control.

Jobs

  • Jobs are configured on the master.
  • Jobs are tied to a slave with a single executor to prevent inadvertent crossover of artifacts between builds via the slave's local Maven repository and to prevent multiple jobs attempting to concurrently use the same JDBC test fixture.
  • Because the only snapshot dependencies in GeoTools and GeoWebCache are internal and the only snapshot dependencies in GeoServer are internal and on GeoTools and GeoWebCache, the unwanted download of remote GeoTools, GeoWebCache, and GeoServer snapshots is prevented with the -nsu (no snaphot updates) command line option, new in Maven 3.0.4.
  • This configuration uses the "Advanced Project Option" "Block build when downstream project is building" for every job with a downstream to ensure that all downstream jobs (including transitive downstreams) are triggered and completed, so downstream jobs only use artifacts for which tests have passed on upstreams, and all jobs use the same code base. Without this option, a newly triggered upstream job might overwrite some artifacts before a downstream that consumes those artifacts has run; for example, a geotools-master might update artifacts which are then published by geserver-master-assembly before being tested by geoserver-master. This consistency guarantee is very important for published build integrity.
  • All jobs are built in a path with spaces, which catches many File<->URL conversion defects that are not otherwise detected.
  • For each job triggered by SCM changes, "Source Code Management" / "Branches to build" / "Advanced" / "Checkout/merge to local branch (optional)" is set to the name of the branch to ensure branch names are included in VERSION.txt. Without this option, Jenkins defaults to checking out a detached head, and VERSION.txt will not contain branch names.

geotools-master

  • https://cgsrv8.arrc.csiro.au/jenkins/view/geoserver-master/job/geotools-master/
  • Builds GeoTools master, including running JDBC tests for gt-jdbc-postgis and gt-jdbc-oracle.
  • Custom workspace: workspace/geotools-master with spaces
  • To ensure that artifacts used downstream are from the same commit, this build uses a private maven repository; after a successful build, org/geotools artifacts are copied to ~/.m2/repository.
  • Triggers geowebcache-master on success.
  • Debian wheezy build slaves with libgdal1 installed are affected by GEOT-4733:
    • Workaround is to either remove libgdal1 or set the environment property GDAL_LIBRARY_NAME to something nonexistent to prevent detection of the library. For example, in Jenkins with the EnvInject plugin, add an entry in Build Environment / Inject environment variables to the build process / Properties Content:
      GDAL_LIBRARY_NAME=disabled_see_geot-4733 
      

geowebcache-master

geoserver-master

geoserver-master-app-schema-online-test

geoserver-master-assembly

  • https://cgsrv8.arrc.csiro.au/jenkins/view/geoserver-master/job/geoserver-master-assembly/
  • Uses the same workspace as the previous job: workspace/geoserver-master with spaces
  • Assembles build artifacts.
    • The src.zip is removed because a bug in maven-assembly-plugin:2.1 causes the inclusion of unwanted target directories, making the zip file huge. See: GEOS-6168.
    • The javadoc-zip is removed because we do not build javadocs and so it it empty.
    • A new war-plugins-included.zip is added. This is a copy of the war.zip repackaged to include selected plugins (currently app-schema, oracle, sqlserver, mysql, and arcsde) with their jar files included in WEB-INF/lib for easy deployment. Other plugin files are in the plugins directory.
  • Every successful build is copied to the cgsrv1 swrepo:
  • The first successful build of each day (AWST) is archived:
    • http://siss.csiro.au/geoserver/geoserver-master/
    • Archive directories have ISO 8601 date stamps [Australian Western Standard Time (UTC+8)].
    • Because these archived builds are retained indefinitely, they are suitable for manual user-acceptance testing and deployment.
    • These builds have not undergone any manual testing.

Jenkins slave

The Jenkins build slave apacsrv4 is currently running Debian 6.0.5 amd64 with some testing packages.

User

The jenkins user ensures the resources the slave consumes are isolated and identifieable:
adduser --system --group --shell /bin/bash --gecos "Jenkins Slave" jenkins

Service

The jenkins system service ensures that the Jenkins slave process is started and stopped when the slave system is started or stopped. Note that this initscript attempts to update all its downloaded jars each time it is started.

The service configuration file /etc/init.d/jenkins:
#!/bin/sh
### BEGIN INIT INFO
# Provides:          jenkins
# Required-Start:    $network $syslog
# Required-Stop:     $network $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Jenkins Slave
# Description:       Slave for Jenkins instance on cgsrv8
### END INIT INFO

JENKINS_USER="jenkins"
JENKINS_HOME="/home/jenkins"
JENKINS_BASE_URL="https://cgsrv8.arrc.csiro.au/jenkins"
JENKINS_NODE="apacsrv4"

DESC="Jenkins Slave"
NAME="jenkins"
PIDFILE="/var/run/$NAME.pid"
COMMAND="/usr/bin/java -- -jar slave.jar -jnlpUrl $JENKINS_BASE_URL/computer/$JENKINS_NODE/slave-agent.jnlp"

. /lib/lsb/init-functions

start_jenkins_slave() {
    cd $JENKINS_HOME
    wget -q "$JENKINS_BASE_URL/jnlpJars/slave.jar" -O slave.jar.new
    if [ $? -eq 0 -a -f slave.jar.new ]; then
        rm *.jar
        mv slave.jar.new slave.jar
        chown ${JENKINS_USER}:${JENKINS_USER} slave.jar
    fi
    start-stop-daemon --start -d $JENKINS_HOME --quiet --background --make-pidfile --pidfile $PIDFILE --chuid $JENKINS_USER --exec $COMMAND
}

stop_jenkins_slave() {
    start-stop-daemon --stop -d $JENKINS_HOME --quiet --pidfile $PIDFILE
    if [ -e $PIDFILE ]; then
        rm $PIDFILE
    fi
}

case $1 in
    start)
        echo -n "Starting $DESC: $NAME"
        start_jenkins_slave
        echo
        ;;
    stop)
        echo -n "Stopping $DESC: $NAME"
        stop_jenkins_slave
        echo
        ;;
    restart)
        echo -n "Restarting $DESC: $NAME"
        stop_jenkins_slave
        sleep 1
        start_jenkins_slave
        echo
        ;;
    *)
        echo "Usage: $NAME {start|stop|restart}"
        exit 1
        ;;
esac

exit 0

Initscripts must be executable:
chmod 755 /etc/init.d/jenkins

Enable the service (insserv dependency-based initscript management replaces the use of update-rc.d in Debian releases before 6):
insserv jenkins

Start the service:
service jenkins start

You should now be able to see jar files downloaded into /home/jenkins.

Git

The slave uses native git for checkout; this must be installed manually:
apt-get install git

PostGIS

Package Version
postgresql-8.4 8.4.12-0squeeze1
postgresql-8.4-postgis 1.5.1-5

PostGIS timezone fix for Western Australia

  • GeoTools gt-jdbc-postgis tests fail in Western Australia because the timezone abbreviation "WST" is not recognised by PostgreSQL. The symptoms are error messages like:
    org.postgresql.util.PSQLException: ERROR: invalid input syntax for type timestamp: "Sun Feb 01 00:00:00 WST 2009"
    
  • See: GEOT-3905
  • Furthermore, once Australian timezones are configured, the presence of South Australian Time breaks time parsing: from the Postgresql manual: "Time zone abbreviations defined in the configuration file override non-timezone meanings built into PostgreSQL. For example, the Australia configuration file defines SAT (for South Australian Standard Time). When this file is active, SAT will not be recognized as an abbreviation for Saturday."
  • Defining an SAT timezone makes gt-jdbc-postgis PostgisTemporalFilterTest.testBeforeInterval fail (yes, one of the limits is indeed a Saturday: "Sat Aug 01 00:00:00 WST 2009"). Both SAT and WST present in one timestamp.
  • Fix this with:
    perl -ni -e 'print unless /^SAT/' /usr/share/postgresql/8.4/timezonesets/Australia
    
  • This will likely need to be repeated whenever postresql packages are updated.

PostGIS database creation

su - postgres
psql
create database jenkins;
\c jenkins
create language plpgsql;
\i /usr/share/postgresql/8.4/contrib/postgis-1.5/postgis.sql
\i /usr/share/postgresql/8.4/contrib/postgis-1.5/spatial_ref_sys.sql
create role jenkins with login password 'POSTGIS_PASSWORD';
alter database jenkins owner to jenkins;
alter table geometry_columns owner to jenkins;
alter table spatial_ref_sys owner to jenkins; 
alter view geography_columns owner to jenkins;
-- timezone fix for Western Australia
alter database jenkins set timezone_abbreviations to 'Australia';

PostgreSQL 9 also requires bytea_output = 'escape'. See See GEOT-3905:

alter database jenkins set bytea_output = 'escape';

GeoServer app-schema online 3D tests require PostGIS 2.0 and EPSG:4979:

INSERT into spatial_ref_sys (srid, auth_name, auth_srid, proj4text, srtext) values ( 4979, 'epsg', 4979, '+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs ', 'GEOGCS["WGS 84",DATUM["World Geodetic System 1984",SPHEROID["WGS 84",6378137.0,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0.0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.017453292519943295],AXIS["Geodetic latitude",NORTH],AXIS["Geodetic longitude",EAST],AXIS["Ellipsoidal height",UP],AUTHORITY["EPSG","4979"]]');

If PostGIS 2.0 or later is used, legacy.sql must be loaded:

\i /usr/share/postgresql/8.4/contrib/postgis-2.0/legacy.sql

If PostGIS 2.1 or later is used, the following functions used in the refdataset SQL (gs-app-schema-test online tests) must be created as they were removed in PostGIS 2.1:

create or replace function public.ndims(g geometry)
returns smallint as $$
begin
    return public.st_ndims(g);
end;
$$ language plpgsql immutable strict;

create or replace function public.srid(g geometry)
returns integer as $$
begin
    return public.st_srid(g);
end;
$$ language plpgsql immutable strict;

PostGIS test fixture configuration

The PostGIS fixture configuration file appears twice:
  1. /home/jenkins/.geotools/postgis.properties - to configure JDBC tests in GeoTools gt-jdbc-postgis
  2. /home/jenkins/.geoserver/postgis.properties - to configure JDBC tests in GeoServer app-schema-test and app-schema-postgis test
Note that the password is supplied twice as some fixtures use "passwd" and others use "password".

postgis.properties:
dbtype = postgisng
driver = org.postgresql.Driver
url = jdbc:postgresql://localhost/jenkins
host = localhost
port = 5432
database = jenkins
user = jenkins
passwd = POSTGIS_PASSWORD
password = POSTGIS_PASSWORD

PostGIS driver installation

The PostGIS driver is open source and is automatically installed by Maven.

Oracle

Oracle testing is performed against "Oracle Database 11g Enterprise Edition Release 11.1.0.7.0". The test user was created to have its own test schema. The test user does not have DBA permissions but does have most other permissions.

Oracle test fixture configuration

The Oracle fixture configuration file appears twice:
  1. /home/jenkins/.geotools/oracle.properties - to configure JDBC tests in GeoTools gt-jdbc-oracle
  2. /home/jenkins/.geoserver/oracle.properties - to configure JDBC tests in GeoServer app-schema-test and app-schema-oracle test
Note that the password is supplied twice as some fixtures use "passwd" and others use "password".

oracle.properties:
dbtype = Oracle
driver = oracle.jdbc.driver.OracleDriver
url = jdbc:oracle:thin:@ORACLE_FULLY_QUALIFIED_HOST_NAME:1521:ORACLE_SID
host = ORACLE_FULLY_QUALIFIED_HOST_NAME
port = 1521
database = ORACLE_DATABASE
user = ORACLE_USER
passwd = ORACLE_PASSWORD
password = ORACLE_PASSWORD

Oracle driver installation

To connect to the Oracle database, the proprietary Oracle driver ojdbc14.jar must be installed. For the GeoServer build, licensing restrictions prevent the distribution of this driver via Maven, so it must be obtained and installed manually.

Copy ojdbc14.jar from geoserver-*-oracle-plugin.zip and install it using:
su - jenkins
export JAVA_HOME=/home/jenkins/tools/JDK/sun-java-6
export M2_HOME=/home/jenkins/tools/Maven/apache-maven-3.0
export PATH=$JAVA_HOME/bin:$M2_HOME/bin:$PATH
mvn install:install-file -Dfile=/path/to/your/ojdbc14.jar -DgroupId=com.oracle -DartifactId=ojdbc14 -Dversion=10.2.0.3.0 -Dpackaging=jar

The identifiers -DgroupId=com.oracle -DartifactId=ojdbc14 -Dversion=10.2.0.3.0 must be used or dependency resolution will fail.

Future work

Image tests

perceptualdiff is installed on the slave but has not yet been enabled for GeoTools image tests. The GeoServer app-schema poms do not pass the required arguments to the surefire JVM argLine; fixing this will allow the app-schema WMS tests to be enabled.

Parallel builds

Maven 3 supports parallel builds with the -T command-line option. This was associated with some intermittent build failures so has been disabled. It also seemed to cause Maven/Jenkins integration to do strange things to console output. How can Jenkins track which output belongs to which module?

Private temporary directory

GeoTools uses the system temporary directory (/tmp on Linux) to store some temporary files, and the EPSG database in particular. This can cause strange build failures if multiple users or multiple builds attempt to create these resources simultaneously. It might be advantageous for each build to use a private temporary directory in its workspace. However, maven-surefire-plugin fails to run when -Djava.io.tmp is set to a path with spaces: incorrect handling of file paths causes surefire to not find the temporary files it uses to store arguments and the classpath.

Artifact archiving

Jenkins artifact archiving is not used. Builds communicate through the shared Maven local repository and (for the GeoServer jobs) a shared workspace. This could be rexamined.

Learnings

Maven dependency plugin

Using Maven dependency:go-offline and dependency:sources to perform all remote artifact resolution does not work because some builds use phases that introduce dependencies not resolved by the dependency plugin, causing builds to fail. A more robust solution that works in this case is the -nsu option, new in Maven 3.0.4. See above.

Jenkins Priority Sorter Plugin

Making a fixed-order build pipeline with the Jenkins Priority Sorter Plugin does not work because queued builds are started before triggers are added to the queue. See JENKINS-13710. Instead of priorities, this configuration uses the "Advanced Project Option" "Block build when downstream project is building" for every job with a downstream.

Jenkins Multiple SCM Plugin with Git

When the Jenkins Multiple SCM Plugin is used with multiple git repositories, each with a different Advanced "Local subdirectory for repo (optional)" setting, it is critically important that the Advanced option "Unique SCM name (optional)" is set or Bad Things Will Happen (the plugin gets confused an builds are constantly triggered). This build pipeline no longer uses the Multiple SCM Plugin.
Topic revision: r21 - 09 Oct 2014, BenCaradocDavies
 

Current license: All material on this collaboration platform is licensed under a Creative Commons Attribution 3.0 Australia Licence (CC BY 3.0).