"Seegrid will be due for a migration to confluence on the 1st of August. Any update on or after the 1st of August will NOT be migrated"

OGC Compliance Test Language - comments on proposed Best Practice

Review by Laurent Lefort

Background: OGC Vote

This is prompted by a request from Simon Cox acting as CSIRO's Technical Committee Voting Member for this matter to comment on whether

- 06-124r4 “Compliance Test Language” should be approved as an official OGC Best Practice document for public release.

The document can be downloaded from here: https://portal.opengeospatial.org/files/?artifact_id=42043

Level of expertise

CTL Implementation

The specification points to the Team Engine as the reference implmentation http://sourceforge.net/projects/teamengine/ but this distribution is not complete (lack of the examples and the scripts avaialble for OGC standards are not included in it because they are available on a different part of the OGC web site) and the examples used to test the scripts?)

This page http://cite.opengeospatial.org/test_engine give some info on the CTL scripts already developed but it is hard to judge of their quality (information on individual CTL script is hard to find (bad sign)) and also on the ease of integration of the them in a larger framework.

teamengine is bundled in this geoserver distro (with 6 scripts) http://gridlock.openplans.org:8080/hudson/job/cite-wcs-1.0/ws/tools/engine/build.xml/*view*/

Use in OGC

- IOOS https://geo-ide.noaa.gov/wiki/index.php?title=IOOS_DIF_DATA_PROVIDER_GUIDE#Web_Services_Testing
(extract) CTL reports “Errors due to the unnecessary strictness of the OGC test (truncated seconds in timeDate value, non-OGC URNs, HTTP instead of HTTPS in URLs, etc.)“

Two questions:
  • are the CTL scripts made available by OGC too strict?
  • is the tool flexible enough to specify/program the right level of test but too inflexible to let users choose the tests they want?


Web Service Testing

Checking what is available “on the market” for the test of any type of web services:

Review: is CTL Best Practices?

Hard to say – One of my concerns is that it may not be complete and does not let users do all the tests they should do (and I think this is bad for a BP document). My gut feeling is also that the specification as it stands does not allow “mashups” of other people’s CTL scripts. .

And I don’t think it is OGC’s business to invent such things even if it is OGC’s business to develop and offer validators which users can run online and/or offline.

Detailed review:
  • one “strong reject” : CTL does not have a mechanism to define tests for “modular specs” – let users to disable the “too strict” tests they don’t like)
  • one “weak reject” : not many CTL scripts available and no evidence that CTL is easy to use to develop online validation services
  • one “weak accept”: TU Delft student saying it is useful in INSPIRE context
  • one “weak accept”: evidence that it is used in an OGC-relevant project
Result: “weak reject” on the basis of what I learned (even considering your reluctance to “get in the way”) because I see CTL as an approach which would generate some extra friction and prevent the trend towards the creation of more modular specs in OGC which for me is essential in the current standard landscape.

I hope that these CTL limitations are known and acknowledged in the OGC community (and even more so in the INSPIRE community as it might be the place where the next generation of validation tools for OGC standards will be developed).

-- LaurentLefort - 01 Jun 2011

Comment from an anonymous implementer of OGC standards

The specification is not addressing the real problems:

  • CITE tests are hard to write, hard to understand, hard to run. This is not changing since they are still using XML and XSLT.

  • the tool reporting facilities are still rather poor, however I don't see this spec addressing the tool issues themselves

  • the process to build and maintain the official CITE test suites is closed and opaque, given that CITE are public tests they should be governed in a public way as well (open mailing, open jira, open discussion)

  • the test suite is built along with the reference implementation once the spec and its errors is beyond salvation, because it's already published
The document addresses none of those issues, merely specifies better how a test suite is to be made and I guess provides new testing abilities.
Topic revision: r6 - 02 Jun 2011, SimonCox

Current license: All material on this collaboration platform is licensed under a Creative Commons Attribution 3.0 Australia Licence (CC BY 3.0).