"Seegrid will be due for a migration to confluence on the 1st of August. Any update on or after the 1st of August will NOT be migrated"

USGS TestBed2 - WFS

WMS Service

Information on setting up the WMS service is found here.

WFS Service

I have now set up Feature services for all of my testbed2 data layers.

The WFS service address for World Geology is here: http://mrmaps.usgs.gov/cocoon/iugs/wfsg?.

The WFS service address for conterminous U.S. Geology is here: http://mrmaps.usgs.gov/cocoon/iugs/wfs-us?.

The WFS service address for Alaska Geology is here: http://mrmaps.usgs.gov/cocoon/iugs/wfs-ak?.

The WFS service address for conterminous U.S. faults, etc. is here: http://mrmaps.usgs.gov/cocoon/iugs/wfs-usf?.

ALERT! Warning Please note that this is a development system; it is usually stable, but services may change or be interrupted at any time.

Capability documents are here for the World Geology data layer: http://mrmaps.usgs.gov/cocoon/iugs/wfsg?REQUEST=GetCapabilities

Capability documents are here for the conterminous U.S. Geology data layer: http://mrmaps.usgs.gov/cocoon/iugs/wfs-us?REQUEST=GetCapabilities

Capability documents are here for the Alaska Geology data layer: http://mrmaps.usgs.gov/cocoon/iugs/wfs-ak?REQUEST=GetCapabilities

Capability documents are here for the U.S. faults, etc. data layer: http://mrmaps.usgs.gov/cocoon/iugs/wfsusf?REQUEST=GetCapabilities

Step-by-step instructions for setting up Cocoon on an ArcIMS server are on Eric's page, here : CocoonWmsSteps

Processing for the WFS service is quite similar to that for the WMS service: requests are received by Cocoon, translated, if necessary, to an ArcIMS request and sent on to ArcIMS, which hosts the datasets.

Responses from ArcIMS are returned to Cocoon, where the response is translated into GeoSciML before being returned to the client.

The major difference is that while WMS requests are encoded in the requesting URL, WFS requests come in from an OGC compliant client as HTML GET requests. For testing purposes, I have also set up a pipeline to handle WFS requests which come in as a POST request from an HTML Form. See section below on sitemap for further details.

Many thanks to EricBoisvert. I could not have set this server up without his enthusiastic support!

-- BruceJohnson - 22 May 2006

Viewers

Most of the testing of the WFS service has been done using GAIA 2.0.5 as the OGC client. I highly recommend setting up a Gaia client for viewing this service.

Testbed data layers and maps can also be viewed using Phoenix. Please contact BoyanBrodaric if you want to use Phoenix, in order to obtain the required priviledges to access the testbed workspace.

For those who are ESRI users, the data layers can also be incorporated into ArcMAP projects by adding data layers from the following GIS Servers:

http://mrmaps.usgs.gov/cocoon/iugs/wfsg

http://mrmaps.usgs.gov/cocoon/iugs/wfs-us

http://mrmaps.usgs.gov/cocoon/iugs/wfs-ak

http://mrmaps.usgs.gov/cocoon/iugs/wfs-usf

-- BruceJohnson - 17 Apr 2006

Modifications

I'll not repeat all of Eric's explanations of how everything works, but would like to make available modifications I have made to his system for USGS use. For each document listed below, I'll try to include some brief comments on what was changed and give a link to the most recent version. If you have questions about the changes that I've made, don't hesitate to ask. That's how we all learn. If you have improvements to offer, please post them.

Sitemap

http://mrmaps.usgs.gov/cocoon/iugs/source

Here's the appropriate section from the sitemap.xmap file:

<!-- special version for WFS/GetFeature queries comming from a form -->
<map:pipeline>
  <map:match pattern="wfs*-form/GetFeature">
    <map:generate type="request"/>
    <map:transform src="style/extractGetFeature.xslt"/>
    <map:transform src="style/ogc2axl{1}.xslt"/>
    <map:select type="parameter">
      <map:parameter name="parameter-selector-test" value="{1}"/>
      <map:when test="g">
        <map:transform type="ArcIms">
          <map:parameter name="server" value="http://mrmaps.usgs.gov/servlet/com.esri.esrimap.Esrimap?ServiceName=IUGSW&CustomService=Query"/>
        </map:transform>
      </map:when>
      <map:otherwise>
        <map:transform type="ArcIms">
          <map:parameter name="server" value="http://mrmaps.usgs.gov/servlet/com.esri.esrimap.Esrimap?ServiceName=IUGSI&CustomService=Query"/>
        </map:transform>
      </map:otherwise>
    </map:select>
    <map:transform src="style/arcimsResponse.xslt"/>
    <map:transform type="stx" src="style/wgsmlFeature_axl.stx">
      <map:parameter name="layer" value="{1}"/>
    </map:transform>
    <map:serialize type="xml"/>
  </map:match>
</map:pipeline>

<!--  OGC WFS Requests  -->
<map:pipeline type="noncaching">
  <map:parameter name="outputBufferSize" value="8192"/>
  <map:match pattern="wfs*">
<!--  Check request parameter  -->
    <map:select type="parameter">
      <map:parameter name="parameter-selector-test" value="{request-param:REQUEST}"/>
      <map:when test="DescribeFeatureType">
        <map:generate src="schema/top.xsd"/>
      </map:when>
      <map:otherwise>
        <map:generate src="capabilities/IUGS_wfs{1}.cap"/>
      </map:otherwise>
    </map:select>
    <map:serialize type="xml"/>
  </map:match>

  <map:match pattern="wfs*/GetFeature">
    <map:generate type="xmlStream"/>
    <map:transform src="style/ogc2axl{1}.xslt"/>
    <map:select type="parameter">
      <map:parameter name="parameter-selector-test" value="{1}"/>
      <map:when test="g">
        <map:transform type="ArcIms">
          <map:parameter name="server" value="http://mrmaps.usgs.gov/servlet/com.esri.esrimap.Esrimap?ServiceName=IUGSW&CustomService=Query"/>
        </map:transform>
      </map:when>
      <map:otherwise>
        <map:transform type="ArcIms">
          <map:parameter name="server" value="http://mrmaps.usgs.gov/servlet/com.esri.esrimap.Esrimap?ServiceName=IUGSI&CustomService=Query"/>
        </map:transform>
      </map:otherwise>
    </map:select>
    <map:transform src="style/arcimsResponse.xslt"/>
    <map:transform type="stx" src="style/wgsmlFeature_axl.stx">
      <map:parameter name="layer" value="{1}"/>
    </map:transform>
    <map:serialize type="xml"/>
  </map:match>
</map:pipeline>

There are three main sections to this pipeline. The second <map:match> section of the pipeline is used to return a portion of the GeoSciML schema when a DescribeFeatureType request is received, or return a capabilities document if any other request is received for this service. The first and third <map:match> portions of the sitemap pipeline deal with incoming GetFeature requests. So, at this point, the service only responds to two specific WFS requests, DescribeFeatureType and GetFeature. All other requests are returned a capabilities document. I'll try to cover the highlights of each step in the process in the individual sections, below.

-- BruceJohnson - 17 Apr 2006

revised -- BruceJohnson - 22 May 2006

GetFeature

This is the heart of the WFS service. The GetFeature request may include a bounding box (in map coordinates), a query string to restrict the results to a collection of features that meet some data criteria, and other options dealing with symbolization, etc. At this stage, the only option that is handled is the bounding box. The relevant portion of the sitemap is here:

  <map:match pattern="wfs*/GetFeature">
    <map:generate type="xmlStream"/>
    <map:transform src="style/ogc2axl{1}.xslt"/>
    <map:select type="parameter">
      <map:parameter name="parameter-selector-test" value="{1}"/>
      <map:when test="g">
        <map:transform type="ArcIms">
          <map:parameter name="server" value="http://mrmaps.usgs.gov/servlet/com.esri.esrimap.Esrimap?ServiceName=IUGSW&CustomService=Query"/>
        </map:transform>
      </map:when>
      <map:otherwise>
        <map:transform type="ArcIms">
          <map:parameter name="server" value="http://mrmaps.usgs.gov/servlet/com.esri.esrimap.Esrimap?ServiceName=IUGSI&CustomService=Query"/>
        </map:transform>
      </map:otherwise>
    </map:select>
    <map:transform src="style/arcimsResponse.xslt"/>
    <map:transform type="stx" src="style/wgsmlFeature_axl.stx">
      <map:parameter name="layer" value="{1}"/>
    </map:transform>
    <map:serialize type="xml"/>
  </map:match>

The only new portions of this process (different from the WMS processing) are the two transforms, ogc2axl{1}.xslt and wgsmlFeature_axl.stx, and the compression of two pipelines (in the case of the WMS services) into one. The first, ogc2axl{1}.xslt, is an XSL Transform that converts the incoming XML request into AXL to then feed to the ArcIMS server. The second, wgsmlFeature_axl.stx, is an STX transform that converts the AXL response from ArcIMS to GeoSciML for return to the client.

The compression of two (or here, 4) pipelines into one is accomplished by using a wildcard in the <map:match pattern>. Currently, wfs* matches wfsg, wfs-us, wfs-usf, and wfs-ak. The suffix (matched by the *) is substituted everywhere in the pipeline where {1} occurs. Therefore, ogc2axl{1}.xslt actually calls one of the following translators, depending on which service has been requested:

http://mrmaps.usgs.gov/cocoon/iugs/source/style/ogc2axlg.xslt

http://mrmaps.usgs.gov/cocoon/iugs/source/style/ogc2axl-us.xslt

http://mrmaps.usgs.gov/cocoon/iugs/source/style/ogc2axl-usf.xslt

http://mrmaps.usgs.gov/cocoon/iugs/source/style/ogc2axl-ak.xslt

Each of these files transforms the incoming OGC compliant GetFeature request into an ArcIMS GET_FEATURES request. I have used 4 separate files for this translation because the layer id and actual database file name are required in the AXL request. I could have used one file with a parameter passed to it, but chose this path as somewhat easier, at present. I may change my mind later. There are two slightly tricky parts of this translation. The first is obtaining the coordinates of the bounding box and placing them into the appropriate position in the AXL request. Apparently, the client can use at least two different methods of specifying the bounding box, either with gml:Envelope or gml:Box. Gaia uses the gml:Box construction; I'm not sure which construction other clients use. In any case, the two constructions are handled by two templates at the end of the document with section headings that look like this:

  <xsl:template match="gml:Envelope">
   . . . . .
  <xsl:template match="gml:Box">

In the first case, one extracts the required values from gml:upperCorner and gml:lowerCorner elements; in the second case, the values are extracted from a gml:coordinates element.

The second tricky part of this transform is the use of the ArcIMS SPATIALQUERY 'accuracy' attribute. This attribute causes ArcIMS to generalize the spatial data for the response.

The code starts with the:
    <xsl:choose>

within the SPATIALQUERY element. There is more detail, on the Performance page, as to why I wanted to do this, but the mechanism used here is to extract xmin and xmax from either the upper and lower corner elements or the coordinates element, subtract to get the X map extent, and divide by an integer (currently 1000) to get an approximation of the size of a pixel in map units. Given that information, ArcIMS is able to reduce the volume of spatial data returned to something appropriate to the resolution of the map. The code here is very cludgy; it was thrown together to see if the concept would work. It does work, but the code should be replaced with something a bit cleaner, probably an XML Server Page instead of an XSL Transform.

-- BruceJohnson - 17 Apr 2006

major revision - BruceJohnson - 22 May 2006

http://mrmaps.usgs.gov/cocoon/iugs/source/style/wgsmlFeature_axl.stx

This file started off as an attempt to translate the wgsmlunit.xslt translator from the WMS service into STX for faster throughput. The difference between XSLT and STX as a transator is that XSLT uses a DOM parser and STX uses a SAX parser. DOM parsers read the entire incoming document into memory before translation begins; SAX parsers read the first XML element and it's child elements and then begin translating, reading in more elements as processing proceeds. In small document processing, there's little speed difference and the DOM parser offers the advantage of having access to the entire document, in some cases making the translation simpler. With large documents, the requirement of putting the whole document in memory before processing can begin slows the processing considerably. Thus, the desire to transform the output with STX.

Most of the .stx file is a pretty simple translation of wgsmlunit.xslt into STX format. Much of the syntax of STX is identical to XSL, so the transformation goes fairly quickly. There are, however, a few gotchas based on the need to process the document sequentially.

At the beginning of the document, you'll see a list of global variables that are declared. These are used to hold the incoming field data as it is read in. The service that has been called is used to set the layerId variable to something appropriate. This variable is then used to create Feature ids. The variables are then used to form the appropriate output. Then the group named 'featureContent' (near the bottom of the file) is applied to the incoming stream. This is a series of templates which extract the field data for the incoming feature.

Once all of the fields have been stored in variables (at the point where a POLYGON or POLYLINE tag is recognized in the input) output of the feature begins. Most output is simply re-formatting and outputting the input data. As in previous versions, a test is made as to the type of feature so that only the appropriate fields are output.

For WFS services, I am currently outputting a collection of gsml:MappedFeatures. For this reason, you'll see the spatial data in the output before the specification attribute which leads to the GeologicFeature attributes. A decision was made in Orleans (4/06) that WFS GetFeature services should return a collection of GeologicFeatures. Because the backend GIS server is only capable of serving the equivalent of MappedFeatures, to serve GeologicFeatures for the WFS service requires either some complicated processing (involving sorting the entire response before translating it) or considerable redundancy in the response (repeating the GeologicFeature attributes for each MappedFeature). At present, I have not solved the processing problem, and am unhappy with the redundancy. I'd be delighted if someone else has a simpler solution.

The only complicated portions of the translator are those that deal with the spatial data. Early in the translation file, you'll see a section that looks like this:

    <gsml:shape>
      <stx:result-buffer name="rings" clear="yes">
        <gml:Polygon srsName="EPSG:4269">
          <stx:process-children group="shape"/>
        </gml:Polygon>
      </stx:result-buffer>
      <stx:if test="$collapse">
        <gml:Null>collapsed</gml:Null>
        <stx:assign name="collapse" select="false()"/>
      </stx:if>
      <stx:else>
        <stx:process-buffer name="rings"/>
      </stx:else>
    </gsml:shape>

It sets up a temporary buffer, named 'rings', and fills the buffer with the output generated by the group 'shape'. Group 'shape' looks like this:

    <stx:group name="shape">
      <stx:template match="RING[1]/COORDS">
        <stx:if test="string-length(translate(.,';-0123456789. ','|'))>2">
          <gml:exterior>
            <gml:LinearRing>
              <gml:coordinates>
                <stx:value-of select="translate(.,' ;', ', ')"/>
              </gml:coordinates>
            </gml:LinearRing>
          </gml:exterior>
        </stx:if>
        <stx:else>
          <stx:assign name="collapse" select="true()"/>
        </stx:else>
      </stx:template>
      <stx:template match="RING[position()>1]/COORDS">
        <stx:if test="string-length(translate(.,';-0123456789. ','|'))>2">
          <gml:interior>
            <gml:LinearRing>
              <gml:coordinates>
                <stx:value-of select="translate(.,' ;', ', ')"/>
              </gml:coordinates>
            </gml:LinearRing>
          </gml:interior>
        </stx:if>
      </stx:template>
    </stx:group>

The first template matches the first <RING> in the input file; the second template matches any subsequent <RING>s. ArcIMS does not differentiate between exterior and interior rings in the polygon, other than by position in the file, but gml does, so they have to be treated differently. Within each template, you'll see a complicated string-length() calculation. This is required because of the spatial data generalization I'm doing. Apparently, it's possible for ArcIMS to generalize a <RING> with only 3 points, first, last, and first again; in other words, a polygon that's degenerate to a line segment. I don't know about other clients, but Gaia chokes on these. If the degenerate polygon occurs as an interior ring, I just drop the interior ring; if it occurs as an exterior ring, I set a variable named, 'collapse'.

Back to the upper piece of code that called these templates to fill the buffer. I now test for the variable, 'collapse'. If it's false, I output the contents of the buffer; if it's true, I substitute:

       <gml:Null>collapsed</gml:Null>

for the entire polygon. Gaia seems to be happy with this format. The buffer is required because I have to read all of the spatial data before I can choose whether to output the gml:Polygon or gml:Null. The data must be read, and output, between the <gml:Polygon> and </gml:Polygon> tags. Without the buffer, by the time I read the spatial data, it would be too late to substitute <gml:Null> for <gml:Polygon>.

-- BruceJohnson - 17 Apr 2006

major revision - BruceJohnson - 22 May 2006


Additional changes to wgsmlFeature_axl.stx

It was pointed out in Orleans that the gml:id attribute was not actually useable for anything other than as a place marker for the document. I was using it to record the actual (persistent) id of individual features in my database. To preserve that data and make it available to the client, I have duplicated the information in an additional <gml:name> property for each feature.

I re-arranged the data elements to coinside more closely with GeoSciML ver. 1.1. This included:
  • Moving the IUGS and USGS lithologic classification information from the <classifier> property to the new union added to the <compositionPart> property. For now, the proportion property is hardwired into the translator; eventually, that information will come directly from the database.
  • Removing required nil values from all properties that were changed from 1..* cardinality to 0..* (required to optional).
  • Minor property name changes.

I have also modified the code (and my databases) to treat geologic ages as multi-values. Where there is only one value for age, the code continues to return a single <gsml:CGI_TermValue> for the value property. Where there are two age values in the database, the code now returns a <gsml:CGI_TermRange> with appropriate upper and lower properties.

The final addition to the translator is two sections to deal with line features (as opposed to polygon features). After the section that processes the data for polygons, there is a new section for processing line data that starts with:

<stx:template match="POLYLINE">

This section is used to create the GeoSciML output for structural features (Faults and Contacts). It works in much the same way as the previous section for GeologicUnits, but includes the appropriate properties for GeologicStructures.

At the end of the file is another new section for processing the spatial data for POLYLINES. This is much simpler than the POLYGON processing section because one does not have to deal with the complication of internal and external rings.

-- BruceJohnson - 22 May 2006

STX issues

After more than a week of struggling with getting STX to work correctly on this translation, two things became apparent. The first is that the version of STX that we're using (Joost) is not a complete STX implementation. Information about what is included in the implementation is found here. In particular, the tokenize() function is not implemented. This explains some of the strange string function work-arounds found in my translators. STX documentation is found here.

-- BruceJohnson - 17 Apr 2006

A second, more troubling, problem was the use of buffers. I could not see a way to implement the translation needed without the use of buffers, as mentioned above. However, I could not get a buffer to work. Any buffer, in any portion of the file, at any time. This lead to way too many days struggling with the poor documentation (IMHO) and lack of available expertise. I finally stumbled on the answer. Look carefully at the group declarations in the file discussed above. You'll see the attribute:

pass-through="all"

Without this attribute, buffers do not work! As far as I can tell this behavior is not documented, and I consider it a bug in the Joost implementation. The pass-through attribute causes all input elements that are not matched by a template to be passed on to the output. Usually, this is not the desired result, and the default is to have pass-through set to 'none'. But, pass-through must be set to 'all' for the group containing the buffer for the buffer to be readable. This is why you'll see several templates in the translator that match elements, but do nothing. It's to prevent those elements from passing through automatically.

-- BruceJohnson - 17 Apr 2006

WFS Performance

Documentation of some WFS performance issues and test results are found here.

Questions and Comments


Topic revision: r8 - 15 Oct 2010, UnknownUser
 

Current license: All material on this collaboration platform is licensed under a Creative Commons Attribution 3.0 Australia Licence (CC BY 3.0).