"Seegrid will be due for a migration to confluence on the 1st of August. Any update on or after the 1st of August will NOT be migrated"

PID Service User Guide

Contents

User Guide

Core Principle

The PID Service intercepts each incoming HTTP request and passes it through to the dispatcher servlet. This compares the incoming request (URI) with patterns configured in the PID Service. When matched, a set of finer grained conditions is tested, triggering different actions depending on particulars of the incoming request. The behaviour might vary depending on the requested content type(s), query string parameters of the request, file extension, etc. When a matching condition is found the service performs a set of actions defined in the PID service configuration, such as HTTP header manipulation, redirects, proxying requests, delegating resolution to another service, etc. If there is no matching condition the service will perform a default action - usually a user-defined HTTP response or proxying request through to another location.

Core principle activity diagram

Once a mapping is created it cannot be deleted from the system, but it may be suppressed. The service keeps track of all modifications performed on a mapping rule, including suppression of rules - turning them into a state also known as "tombstoned". Tombstoned rules are ignored by the dispatcher but are still searchable (via "Deprecated mappings" search) and viewable by the PID Service Management Console. Tombstoned rules may be reinstated and brought back to life.

PID Service UI Screenshots

Comparison with PURL

The PID Service has many similarities with the well-known PURL service - both provide a general-purpose URI redirection service configured through a web-form interface. PID service varies from PURL in two key areas.

Rules

PID Service configuration is a superset of PURL, extending it in the following ways:
  1. mapping rules may be expressed as regular expressions
  2. a variety of additional conditions can be configured, supplementing the basic URI match
  3. rules may be arranged in a hierarchy, so that the most accurate match is preferred to a more coarse-grained match
  4. tombstoned rules may be revived
  5. proxying is supported as well as HTTP redirects

Access control

PURL services segment access according to the first level of directory/path below the service root, and in this way a single public PURL service can support configuration by a large number of users.

PID service has no specific authentication and access control mechanism. The mapping configuration interface is more complex, so there is not the same expectation of public deployment. Delegated access control capability may be achieved through deployment of a separate PID Service for each primary path, each with separate authentication rules matching the delegation.

Mapping Types

The PID Service supports a few different mapping types, which dictate the way a user defines a URI pattern recognition logic. Currently two mapping types are implemented - one-to-one mappings (1:1) and regular expression (Regex) based patterns. Each type is processed differently by the service and comes with its own pros and cons described below.

One-to-One

One-to-One mapping type is the most basic and efficient type that provides an exact mapping for an URI. High efficiency is achieved by the use of database indices used for searching. This mapping type is recommended when there's no need for extra flexibility (i.e. pattern generalization) in the mapping, such as the use of alpha-numeric identifiers in the URI.

[NOTE: this is the main kind of mapping implemented by PURL.]

Regex

Regular expression based URI mappings provide greater flexibility to define a URI pattern at the price of lower efficiency. It requires the system to retrieve all the Regex-based URI mappings from the database and iterate through all of them until match is found. Basically, the more Regex-based mappings are registered in the system the more time it takes to find a match. However, if the number of Regex-based mappings is kept to a reasonable figure the benefits will overweight the downsides.

For example, if you have to handle URIs for EPSG codes (e.g. http://spatialreference.org/ref/epsg/4326/), where there's a finite number of them, it might be better to use One-to-One mappings. Whereas for URIs like:
  • http://services.auscope.org/resource/feature/pirsa/miningfeatureoccurrence/mine/21
  • http://services.auscope.org/resource/feature/pirsa/mineraloccurrence/21deposit
where it is hard or impractical to create a separate mapping for each possible URI.

Order of Precedence

The service implemented in such a way that it tries to find an exact match at first for the whole URI. It gives a significant performance boost when system is handling thousands of URI mappings as it allows to put all the burden on the database engine and use its indices efficiently. If an exact match isn't found the system will retrieve all the Regex-based mappings from the database and iterate through them to find a match following the principle and inheritance relationships described in Rule inheritance section.

Rule inheritance

The rule inheritance concept is based on the one-way tree-like hierarchy of mapping rules, where the top parent is a built-in “Catch all” rule. All mappings that are not explicitly defined as being a subset of a more generic rule will automatically inherit from the “Catch all” rule. The inheritance mechanism is built on a single inheritance principle, where any particular rule may be a subset of one and only one parent rule.

The hierarchical structure allows visual representation of the mapping dependency tree for better understanding of rules from a user perspective. The tree is organised in such a way that all rules eventually end up inheriting from the “Catch all” mapping rule that is also being used when there no matching rules were found. The selection of a matching condition (and a set of actions associated with one or another condition) follows the principle below:

  1. Find an exact match (assert one-to-one mappings);
  2. Find a regex-based match starting assertions from the deepest level up the tree until it matches a user-defined condition or reaches “Catch all” mapping.

The rule inheritance concept provides mechanisms to control the order in which regex-based mapping rules are asserted by the system. Due to the nature of regular expressions there's a possibility to have multiple regex-based mappings matching the same URI and residing at the same level in the tree hierarchy. This forms an exceptional use case where there's no way to enforce the order in which URI mapping rules are processed and order of precedence is effectively undefined. This, however, should not be an issue as there should be no overlap between different regular expressions at the same level within any particular logical branch.

The following rules are in place within rule inheritance architecture:

  1. By default, all rules are inheriting from the top level “Catch all” rule unless it is redefined by the user. No orphaned mapping rules are allowed.
  2. One-to-one mapping rules may only inherit from regex-based mapping rules, assuming the URI matches the regular expression of the parent mapping rule;
  3. Regex-based mapping rules may only inherit from another regex-based mapping rules with the following conditions:
    • The validity of inheritance is not checked at the time of mapping rule creation due to complexities associated with the regular expressions;
    • However, when the condition search logic progresses up the hierarchy to the parent regex-based mapping it will be checked that URI matches the regular expression of the parent rule. If it doesn't the regular expression of the parent rule, then it falls back to the top level “Catch all” mapping rule that matches any URI by design.
  4. Single inheritance principle means that any one rule may only inherit from one parent only.

Handling Work-Flow

The following diagram provides a high level perspective on the workflow used for dispatching incoming HTTP requests. Once a matching mapping rule is found either as exact one-to-one mapping or a matching regex-based mapping rule it will proceed to selection of a matching condition. At this stage the PID Service will iterate through the ordered list of conditions defined for the matching mapping rule and, if none is matching the incoming request, it will proceed to the parent mapping rule's set of conditions. It will keep going through parents' conditions until either one the conditions matched the incoming HTTP request or it reaches the "Catch all" mapping rule.

In none of the conditions is matching the request the PID Service will trigger the first defined default action starting from the initial matching mapping rule and progressing through the chain of parents up to "Catch all" mapping rule as shown on the activity diagram below.

Handling workflow activity diagram

Conceptual View of the Inheritance Principle

The following diagram represents conceptual view of the PID Service architecture that implements single mapping rule inheritance principle. Examples below will give an idea of the workflow that is being used to identify a matching mapping rule condition and eventually a set of actions associated with that rule.

Further in the document, mapping level represents the depth of the rule in the hierarchical tree that is shown by the red vertical bar on the left hand side of the diagram. Blue boxes on the diagram below denote regex-based mapping rules and green boxes denote one-to-one mapping rules.

Example 1: /myPersistentID

  1. Find a matching one-to-one mapping irrespective of mapping level;
  2. Matching one-to-one mapping rule is found (at level 1);
  3. Rule-specific conditions are asserted (from /myPersistentIDmapping);
    • If matching condition is found its actions are executed;
    • If matching condition is NOTfound then parent (i.e. “Catch all”) conditions are asserted;
      • If matching condition is found in “Catch all” mapping its actions are executed;
      • If matching condition is NOT found then /myPersistentIDdefault action is executed;
        • If no default action is defined for /myPersistentID rule then “Catch all” default action is executed.

Example 2: /siset/specific/something

  1. Find a matching one-to-one mapping irrespective of mapping level;
  2. No exact match found;
  3. Regex-based mapping rules are retrieved from the database and asserted in the following order:
    • From deepest mapping level to the top (see the order on the diagram above marked with numbers in blue circles);
    • The order in which regex-based mapping rules are asserted within a single level is not defined (see the concept why this is no longer an issue). Thus rules marked with a blue circle (2*) can be asserted in an undefined order;
  4. Matching regex-based mapping rule is found at level 2 – /siset/specific/.*
  5. Rule-specific conditions are asserted (from /siset/specific/.*mapping rule);
    • If matching condition is found its actions are executed;
    • If matching condition is NOT found then parent conditions are asserted progressively starting from immediate parent (i.e. /siset/.*) and up to the “Catch all” mapping rule. If none of defined conditions are matching request then a user-defined default action is selected starting from deepest level up to the very top. The order in which conditions are asserted and respective action is chosen is marked by numbers in orange circles on the diagram above.

Example 3: /siset/specific/id2

  1. Find a matching one-to-one mapping irrespective of mapping level;
  2. Matching one-to-one mapping rule is found (at level 3);
  3. Rule-specific conditions are asserted (from /siset/specific/id2mapping);
    • If matching condition is found its actions are executed;
    • If matching condition is NOT found then parent conditions are asserted progressively starting from immediate parent (i.e. /siset/specific/.*) and up to the “Catch all” mapping rule. If none of defined conditions are matching request then a user-defined default action is selected starting from deepest level up to the very top as in previous example.

Cookbook

For more real-world examples please refer to PID Service Cookbook.

Mapping Configuration

Mapping rule inheritance is configured via Inheritance section on the Mapping Configuration page. By default, all new mapping rules are inheriting from "Catch all" built-in rule. To change the parent click on the link and it'll bring a search text box that might be used to lookup for the parent mapping. Please note that when assigning parent for one-to-one mapping rule the service will automatically check that URI pattern matches the parent parent.

Below "Next level mapping" line there's a visual representation of the inheritance tree showing an immediate parent up to two levels up and the number of dependent mapping rules.

To search for parent start typing in the text box that appeared when you clicked on the parent link:




the list of matching mapping rules will pop up in the drop-down box:

Once the mapping is set the dependency tree underneath will be updated automatically.

Mapping Chart

Mapping chart (see link provided on the left hand menu in the PID Service Management Web Console) provides an interactive chart that visually represents all relationships between mapping rules. The chart is implemented as a radial graph with Catch-all mapping rule (depicted by a red star icon) in the centre with multiple orbits around it, where each orbits depicts level of nesting of each particular rule. Regular expression based rules are represented by blue circles whereas one-to-one mapping rules are marked by green squares. Clicking on a rule on the chart will reposition chart in such a way that clicked rule will be placed in the centre, highlighting immediate parents and children. On the right hand side menu there's an infobox that provides basic information about the mapping rules, such as the title, pattern, description, etc., as well as the number of dependent rules and a link to "Mapping Configuration" page.



On the "Mapping Configuration" page there's an icon in the toolbar that would bring the user back to the Mapping Chart highlighting the mapping rule that used has come from:

Default Action

Default action associated with a rule is designed to be executed when a URI match is found and none of specific conditions are met (it is also possible to have no conditions at all associated with the mapping). Default action may be set to one of 30x-type HTTP redirects, 4xx-type HTTP responses or can be proxied to another location. Default action is optional and may be omitted.

For details refer to explanation of each action below.

Conditions

The concept of conditions in the PID Service provides a mechanism for finer grained URI resolution based on various properties on an incoming request, such as content type, extension, or query string parameters appended to original request. Conditions are processed sequentially in the same order as they are defined. Thus, since there's a possibility that there're multiple matching conditions for the same request, only the first matching condition will be processed. To change the order move conditions up and down using arrows on the user interface.

Conditions have a type and a value that provides a matching condition.

Condition Types

Comparator

Comparator condition allows to perform a simple comparison between two operands. Multiple comparisons may be performed at once in a single condition where the result is a logical AND between all comparisons.

value <left operand>=<right operand>[&<left operand>=<right operand>[...]]
Backslash ("\") is used as an escape character for $, & and =.

Function calls are allowed for construction of operands. For example:
${QS:test}=2\=2

compares "test" querystring parameter with "2=2".
$2=some.rdf&${QS:test}=a\&a

compares second URI matching capture group ($2) with "some.rdf" AND "test" querystring parameter with "a&a".

ComparatorI (case-insensitive)

Same as Comparator above but performs case-insensitive string comparison.

ConditionSet

Points to a predefined set of conditions serialized as a condition set. The only configuration option required is the name of a condition set to use.

value Name of a condition set.

ContentType

Detects a specific content type requested by the user (via Accept HTTP header) by comparing its value with the condition defined by a regular expression.

value A regular expression used to match the Accept HTTP header.

Example:

GET http://example.org/ HTTP/1.1
Accept: application/json, text/javascript, */*

will be matched by application/json or application/(json|xml), but won't be matched by ^application/json$.


The ContentType conditions are processed in a special order driven by the sequential order of MIME types and their respective Q-factors in the Accept header. In other words for the following request:

GET http://example.org/ HTTP/1.1
Accept: application/json, text/javascript, */*

with the following conditions in place:
ContentType application/javascript
ContentType application/json

application/json condition will get processed as it comes first in the Accept header.

If no ContentType conditions are met at all and / is present in the original Accept header then the first appearing ContentType will get processed.

Taking into account the nature of incoming request driven processing of ContentType conditions, URI mappings that contain multiple ContentType conditions are processed in a slightly different order to the one in which they're defined. All ContentType conditions will be processed together one by one right after the first appearance of ContentType condition. For example, if there's a mapping with the following conditions defined (left column) then they will be processed in the order shown in the right column.

Conditons (as defined)
Extension
QueryString
ContentType
Extension
ContentType
ContentType

Conditons (actual processing order)
Extension
QueryString
ContentType
ContentType
ContentType
Extension

Extension

The PID Service has a capability to recognise extensions at the end of URIs (i.e. the part of an URI that comes immediately after the last dot in the URI path component). Extensions are generally excluded from an URI when searching for matching mapping rule and are treated separately as intrinsic properties of URIs. The only exception is made to URI mappings where that "extension" actually represents part of URI. For further details see "Handling extension-like URIs"section below.

"Extension" condition type allows to detect a specific extension appended at the end of URI (e.g. http://example.org/resource.xml) by comparing its value with the condition defined by a regular expression.

value A regular expression used to match extension.

Technically being a regular expression you can use the following condition to match both htmand html extensions.

Extension ^html?$
as well as a simplified version that will match htm, htmland xhtml extensions:

Extension htm
or very restricted version that only matches html extension:

Extension ^html$
Handling extension-like URIs

As file extensions are treated separately from the URI pattern they don't need to be captured by the URI pattern. However, there's a possibility that it will be required to create a mapping for URIs like http://example.org/docs/geosciml/2.1 where "1" (i.e. the part of an URI that comes immediately after the last dot in the URI path component) is treated as an extension. To avoid conflicts the service will initially make an attempt to find an exact match for the whole URI /docs/geosciml/2.1(with a null extension) and if such a mapping isn't found it will make another attempt to find a match for /docs/geosciml/2 with an "Extension" condition defined that matches "1".

Note that for for pattern based (i.e. Regex) mapping there's no such concept as an extension. All parts of a URI are treated as part of that URI whether it has "." in it or not. Therefore, "Extension" condition type should NOT be used with pattern based mappings and some functions, such as, RESOURCE_NAME_EXT will return the same result as their extension-less counterparts (i.e. RESOURCE_NAME).

QueryString

The QueryString condition type allows to recognise multiple query string parameters in a single condition. Taking into account position agnostic nature of query string parameters, condition value has a special syntax used to recognise multiple parameters using regular expressions. The syntax is very similar to query string notation itself:

value <query string parameter name>=<regular expression>[&<query string parameter name>=<regular expression>[...]]
where <query string parameter name> may be followed by a question mark (?) indicating that parameter is optional. However, if query parameter is present it must match the regular expression. Thus, to make a query string parameter mandatory without actually caring about its content, you may use the following pattern:

mandatory_param=.*

For example, let's assume you have the following two conditions defined with one Proxy action each:

QueryString _view=(\d)?(.+)&_format=(x?html?)&_lang?.*=
  Proxy -> http://www.google.com/?q=View_${C:_view:1}-${C:_view:2}%20Language_${C:_lang:0}%20Format_${C:_format:1}
QueryString _view=(one)&_format=(\d+)
  Proxy -> http://www.google.com/?q=View_${C:_view:1}%20Format_${C:_format:1}
The first condition expects a _view parameter that contains one digit followed by one of more random characters; html, xhtml, xhtm or htm in the _format parameter and anything in the _lang parameter, where the latter one is optional and may be omitted. The second condition expects one in the _viewparameter and a number in the _format parameter.

Please note that QueryString condition type is using a special syntax to address condition match capture groups. Each query string parameter is associated with its own Matcher object that can be addressed using the following syntax:

${C:<query string parameter name>[:<capture group index>]}

Where [:<capture group index>] part is optional. If capture group index is missing then zero-group is used for substitution (i.e. the whole query string parameter).

For more information on using captured group in action items see Using parameters captured from URI and condition matching.

HttpHeader

The HttpHeader condition type is implemented in exact same way as QueryStringtype but operates on HTTP headers collection instead of query string parameters.

Actions

Once a matching rule and condition are found the service will run a series of actions defined by the user. In most cases the action will set one or another type of redirect/proxy action to point to the physical resource location. Actions are executed sequentially in the same order as they are defined. To change the order move actions up and down using arrows on the user interface.

If no HTTP response code setting action (30x, 4xx, Proxy) is defined then an empty response will be sent to the client.

Actions are defined by a type and a name-value pair that provide additional runtime parameters for the action handler. Some action types require no parameters at all, some are expecting only certain values for the name parameter and some are accepting any arbitrary values. Explanation of each action type is provided below.

Action Types

301 Moved Permanenly

Sets HTTP response code to 301 and redirects user to another location. See explanation on http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html.

The only supported parameter is location that points to the target resource.

302 Found - Simple Redirection

Sets HTTP response code to 302 and redirects user to another location. See explanation on http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html.

The only supported parameter is location that points to the target resource.

303 See Other

Sets HTTP response code to 303 and redirects user to another location. See explanation on http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html.

The only supported parameter is location that points to the target resource.

307 Temporary Redirection

Sets HTTP response code to 307 and redirects user to another location. See explanation on http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html.

The only supported parameter is location that points to the target resource.

404 Not Found

Sets HTTP response code to 404 indicating that resource is not found. See explanation on http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html.

No action parameters supported.

410 Permanently Gone

Sets HTTP response code to 410 indicating that resource is not found. See explanation on http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html.

No action parameters supported.

415 Unsupported Media Type

Sets HTTP response code to 415 indicating that requested media type is not supported. See explanation on http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html.

No action parameters supported.

Proxy

Proxies HTTP request to another location preserving original HTTP headers and passing retrieved HTTP headers back to the client.

The only supported parameter is location that points to the target resource. For Proxy actions only, the complete target URI must be supplied. For actions with 3** redirects, a partial URI can be supplied for targets on the same server.

Security Consideration

The use of Proxy action causes the server to initiate an HTTP request on user's behalf to the destination URL preserving HTTP Accept headers from the original request. Service maintainer should be responsible for ensuring that firewall rules allow outbound requests to be initiated from the server. However, if for whatever reason the use of Proxy action must be prohibited, service custodian may disable this action by setting the following setting from /WEB-INF/pidsvc.properties file to false.

allowProxyAction = true

When Proxy action is disabled any attempt to use it in the URI mapping will be superseded by 302 Simple Redirection action.

AddHttpHeader

When proxying requests to another location AddHttpHeader action type allows to add additional HTTP headers to outgoing HTTP request in addition to original HTTP headers.

Action parameters:

Name Value
<header name> HTTP header value

RemoveHttpHeader

When proxying requests to another location RemoveHttpHeader action type allows to remove HTTP headers from outgoing HTTP request that are passed through from the original request.

Action parameters:

Name Value
<header name to remove> [none]

ClearHttpHeaders

When proxying requests to another location ClearHttpHeaders action type allows to remove all HTTP headers from outgoing HTTP request.

No action parameters supported.

Using parameters captured from URI and condition matching

The PID Service provides an interface that allows to use captured groups from regular expressions used for URI pattern recognition (Regex-based URIs mappings) and condition matching where regular expressions are supported. Captured groups can used in any action items (URI mapping default action and condition-specific actions) using special placeholders ($0 for the complete match, or $1 for the first capture group, etc).

For example, if you have a Regex-based URI mapping that captures two groups:

^/id/isotc211/(.+/)([^/\.]+)$

they can be used in an action using $1 and $2 placeholders:

302 Simple redirection location http://def.seegrid.csiro.au/isotc211/$1$2.rdf

To refer to a group captured from condition value the following syntax is used:

${C:<index>}

where <index> is a group number.

Please note that some condition types may have a special syntax to address capture groups, e.g. QueryString.


For example, if you have a ContentType condition that used a regular expression to capture some data:

ContentType text/(x?html)
then captured group can be referred to in the following manner:

302 Simple redirection location http://example.org/file.${C:1}
In one-to-one mappings only default substitution group is supported that is mapped to the original request URI (without scheme and hostname components).

Merging Rules

The PID Service provides functionality to merge mapping rules definitions. When merging one mapping rule with another the target mapping rule will get conditions from the source mapping rule, either appending them or replacing existing conditions of the same type and definition. This operations will not affect the default action or URI pattern of the target mapping rule, which will remain intact. Merge is an atomic operation and may cause rules to behave differently once modification has occurred. However, it will create a new record in the change history making it possible to rollback the changes.

Merge operation is available both via user interface and via an API. To merge mapping rules via UI open mapping configuration and click "Merge mapping" button on the toolbar:

This will bring a popup screen with a simple file upload form:

You need to upload a mapping rule in any supported file format (binary or XML-based backup) that contains a singlemapping rule with one or more conditions.

The following options are available to control the behavior of the merge operation:

  • Replace conditions wherever possible- when this option is selected the service will replace fully matching conditions (by type and definition) instead of appending new ones. This option will also eliminate fully matching conditions. By default, new conditions from uploaded configuration file will be appended to the bottom of the list of conditions.

Please note that merge operation may change the behavior of the mapping rule, so always check the conditions to ensure correct behavior. Take into account that when appending conditions some conditions may become unreachable if there're other conditions of the same type and definition are already defined for the mapping rule, which will take precedence over the new ones. Once merge operation has finished successfully affected conditions will be marked with a red marker as shown below:

Same operation may be performed programatically via an API described here.

Condition Sets

A new concept of condition sets has been introduced from version 1.1.131. Condition set is a pre-bundled set of rules without a pattern or URI associated with it, which can be re-used by separate rules that partly implement the same behavior. This concept is quite different from the concept of rule inheritance. Whereas inheritance concept makes the rule behaving like its parent(s) while at the same time allowing to redefine or cancel the behavior of selected conditions from the parent rules, the concept of condition sets is an inclusionmechanism. This concept barely links a rule to a set of pre-defined conditions that exist independently in the system.

Condition sets are managed independently via "Condition sets" menu in the Administration section. The interface is simple and is implemented in the same way conditions are managed in a rule editing screen with the only difference that it has a name and is not associated with any URI or URI pattern, i.e. exists on its own. In order to engage a condition set in a rule a newly introduced condition type - ConditionSet- needs to be used. The only configuration options required is the name of the condition set as shown below:

A separate section in the cookbook provides additional information and worked examples on how to use condition sets efficiently - see cookbook examples.

Lookup Maps

The PID Service provides a functionality to look up and resolve values at runtime when constructing a rewrite URI. This is done via the use of LOOKUP function.

The PID Service defines two types of lookup maps - static and dynamic (HttpResolver) maps. The former one allows to define a static set of mappings from one value to another, while the latter type allows to invoke a web service call and retrieve data dynamically.

Each lookup map is defined by an unique namespace and a type that sets the resolution behaviour for the map. An attempt to resolve a lookup key from an undefined lookup map or a value that is not mapped to anything will result in an empty string.

Default behaviour defines an action that is performed is lookup value cannot be resolved. It may be specified as a constant value or pass-through that simply returns the original value.

Static

Straightforward mapping from one value to another.

Dynamic - HttpResolver

HttpResolver lookup map provides a mechanism to invoke a web service call at runtime and retrieve a resolution value from obtained HTTP response.

An attempt to resolve a lookup key via dynamic lookup map will result in a web service call set by HTTP endpoint property, which is dynamically constructed using the lookup key. By default, lookup key will be appended to the end of HTTP endpoint URL. Alternatively you can use $0 placeholder to indicate where lookup key needs to be placed. See highlighted value in the HTTP endpoint field in the screenshot provided above.

Once the HTTP response is obtained one of two extractors is used to obtain the lookup value - Regex or XPath.

Regex extractor is using a regular expression to extract value from HTTP response. If provided regular expression is matching the response and has at least one capture group defined then the first capture group is returned, otherwise the whole matching string is returned.

XPath extractor is used to extract a certain value from XML-based response. An example is given at the screenshot above.

Export / Import

Full Backup

To backup all lookup maps in the PID Service go to the Lookup Maps administration page and click "Export all lookup maps":

There's also a posibility to include all lookup maps into full database backup. See Backup a whole databasefor details.

Export Individual Lookup Map

To export an individual lookup map open it in the user interface and click "Export lookup map" button on the toolbar:

Import Lookup Map Configurations

To import a lookup map (or a lookup map backup file) go to the Import tab on the Lookup Maps administration page and upload a file using a form provided:

Functions

When rewriting URI there's a range of functions supported by the PID Service that may be used in action items. The generic function syntax is:

${FN_NAME:<parameter 1>:<parameter 2>}

Special characters in all function arguments/parameters must be escaped (i.e. prefixed with a backslash):
  • colon
  • backslash
  • dollar sign
  • curly brackets
  • equal sign
  • ampersand

Function calls may be nested, for example:

${ENV:${QS:variable}}

All unrecognised/unresolved functions return empty string.

Function: URI

URI function provides an equivalent to $1 placeholders used in Regex-based mappings to access regular expression capture groups.

Syntax:

${URI:<index>}

where <index> is a regular expression capture group index. "0" capture group is always available even for one-to-one mappings an refers to the whole URI (e.g. /id/test.ext?arg=1), which is equivalent to ${ENV:ORIGINAL_URI}.

Function: RAW

By default all function call results are URI encoded to ensure the validity of constructed redirect action. In some circumstances it might be required to keep the string unencoded. This can be done via the use of RAW function.

Syntax:

${RAW:<input>}

where <input> is an input parameter that needs to be preserved as an unencoded URI.

Function: C

Provides access to regular expression capture groups and other data captured by condition matching logic as described earlier in Using parameters captured from URI and condition matching.

Syntax:

${C:<index>}

where <index> is a regular expression capture group index.

Function: ENV

Provides access to various runtime variables.

Syntax:

${ENV:<variable>}

where <variable> is one of the variables from the table below.

Variable Description Example
REQUEST_URI The base part of request URI without hostname, extension and querystring. /id/test
REQUEST_URI_EXT As REQUEST_URI with extension. /id/test.ext
REQUEST_URI_QS
ORIGINAL_URI
As REQUEST_URI_EXT with querystring. /id/test.ext?arg=1
FULL_REQUEST_URI As REQUEST_URI with hostname. http://example.org:8080/id/test
FULL_REQUEST_URI_BASE Return the base part of an URI with hostname. Returns http://example.org:8080/id/ for http://example.org:8080/id/test
FULL_REQUEST_URI_EXT As FULL_REQUEST_URI with extension. http://example.org:8080/id/test.ext
FULL_REQUEST_URI_QS As FULL_REQUEST_URI_EXT with querystring. http://example.org:8080/id/test.ext?arg=1
URI_REGISTER Return the base part of an URI with hostname without trailing slash. Returns http://example.org:8080/id for http://example.org:8080/id/test
QUERY_STRING Request querystring. arg=1
FILENAME
RESOURCE_NAME
Request filename without extension. Returns test for http://example.org/id/test.ext
FILENAME_EXT
RESOURCE_NAME_EXT
Request filename with extension. Returns test.ext for http://example.org/id/test.ext
EXTENSION
EXT
Extension. ext
DOT_EXTENSION
DOT_EXT
Extension with a leading dot. .ext
SERVER_NAME Server hostname. example.org
SERVER_ADDR Server address. http://example.org:8080

Function: LOOKUP

Resolve the function argument via user-defined lookup map.

Syntax:

${LOOKUP:<namespace>:<parameter>}

where <namespace>is a namespace of lookup map and <parameter> is a value to resolve.

Function: QS

Provides access to original request querystring parameters.

Syntax:

${QS[:<parameter>]}

where <parameter> is a name of querystring parameter.

If <parameter> is missing full querystring is returned.

Function: HTTP_HEADER

Provides access to original request HTTP headers collection.

Syntax:

${HTTP_HEADER:<parameter>}

where <parameter> is a name of HTTP header.

Function: IF_THEN_ELSE

Basic logical statement.

Syntax:

${IF_THEN_ELSE:<logical_expression>:<statement>:<elsestatement>}

logical expression supports the same syntax as Comparator condition type and is case-sensitive.

Function: ISNULL

Replaces NULL (empty string) with the specified replacement value.

Syntax:

${ISNULL:<check_expression>:<replacement_value>}

Function: NULLIF

Returns a NULL value (empty string) if the two specified expressions are equivalent.

Syntax:

${NULLIF:<expression>:<expression>}

Function: COALESCE

Returns the first non-null expression among its arguments.

Syntax:

${COALESCE:<expression_1>:<expression_2>[:...n]}

Function: LOWERCASE

Returns a character expression after converting uppercase character data to lowercase.

Syntax:

${LOWERCASE:<expression>}

Function: UPPERCASE

Returns a character expression after converting lowercase character data to uppercase.

Syntax:

${UPPERCASE:<expression>}

Rule Database Maintenance

Backup / Export

Backup a whole database

Database backup functionality is available both via the user interface and via API. To backup the database go to the Backup/restore screen accessible via the left hand side menu:

Two types of database backup are available:

  • Partial backup - saves only the latest state of each mapping allowing to replicate the current active state of the PID Service. The change history will be lost.
  • Full backup- saves all the mapping information including change history for each mapping.

Two options allow to control the content of the database that gets into the backup file.

  • Include deprecated mappings - backup file will contain deprecated (tomb-stoned/suspended) mappings, which will be restored as such during restore.
  • Include lookup maps- this options is selected by default so that backup file also includes lookup maps configuration as well.

Export individual rules

To export an individual mapping rule open its configuration in the user interface and find Export mappingbutton on the mapping toolbar (see screenshot below).

The same two types of mapping export are available as for the backup:

  • Partial backup - exports a single state of the mapping only. You might export ANY state of the mapping not only the latest but the one that is currently displayed in the user interface. The change history will be lost.
  • Full backup- saves all the mapping information including change history for the mapping.

Please note that the PID Service doesn't keep track dependencies on other artifacts, such as lookup maps that might be used by the mapping rule. Therefore, it is user's responsibility to export those lookup maps separately.

Restore / Import

The PID Service provides batch import facility that is similar to backup/restore function of the service and allows users to import multiple mapping rules at once using an XML file. The batch file may be prepared manually (e.g. using XML editing tools like XMLSpy) or using third-party applications.

The batch file must conform to the XML Schema provided by the service that is accessible from one of the following locations:

Visualized version of backup.xsd XML schema can be obtained from backup_xml_schema.png

Sample batch files may be obtained from:

There're two ways to import a batch file into the PID Service - either using the PID Service Web Console or using the API.

Import using the PID Service Web Console

To import a batch file in an interactive mode open your PID Service web console (normally at http://your_instance.org/pidsvc/), click Backup/restore on the left hand side menu, switch to Restoretab and upload your batch file.

Batch file import

Import using API

To import a batch file using the Application Programming Interface follow the instruction provided in the API Reference.

Debugging

When an error is encountered at the server side and an exception is thrown you will either get an HTTP Response 500 with error description and full stack trace or you will get a message box indicating an error (when using UI):

To make debugging easier the full stack trace will be returned back to the user in X-PID-Service-Message HTTP header that can be captured using any HTTP Debugging Tools (e.g. Fiddler 2, Firebug, etc.).

Settings

Base / Authoritative URI

In certain circumstances the PID Service may be deployed on the worker node that has a different hostname from the one where dispatcher service is working. This configuration setting allows to set up a base / authoritative URI that is different from the actual host. However, dispatcher hooks still need to be correctly set in the web server configuration. If this setting is not set the service assumes the URI dispatcher is running on the same host.

Tracing

Dispatcher tracing mode if enabled allows to trace execution of the resolution process. To enable the dispatcher tracing mode go to the "Setting" section and check "Dispatcher tracing mode" box. Tracing mode can only be enabled at the service level.

Once enabled, an attempt to resolve an URI will result in a plain text response like the one provided below:

00:00:00.000   Dispatch /id/first_test.qs?test=arg
00:00:00.000   Find exact match: /id/first_test [.qs]
00:00:00.000   Match found: true
00:00:00.000      Default action ID: 1097
00:00:00.000      Condition: Type=ConditionExtension; ID=707; Match=^qs$; Aux=java.util.regex.Matcher[pattern=^qs$ region=0,2 lastmatch=qs];
00:00:00.000      AuxiliaryData: true
00:00:00.000   Run actions.
00:00:00.010   Type=ActionProxy; Name=location; Value=http://www.google.com/?q=${QS:test}%20${ENV:QUERY_STRING};
00:00:00.010   Expand   http://www.google.com/?q=${QS:test}%20${ENV:QUERY_STRING}
00:00:00.010      http://www.google.com/?q=arg%20${ENV:QUERY_STRING}
00:00:00.010      http://www.google.com/?q=arg%20test%3Darg
00:00:00.010   GET http://www.google.com/?q=arg%20test%3Darg HTTP/1.1
00:00:00.010      connection: Keep-Alive
00:00:00.010      accept-language: en-AU
00:00:00.010      x-forwarded-for: 127.0.0.1
00:00:00.010      accept: text/html, application/xhtml+xml, */*
00:00:00.010      x-forwarded-host: localhost
00:00:00.010      user-agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
00:00:00.010      x-forwarded-server: marching-kf.nexus.csiro.au
00:00:00.010      X-Original-URI: http://localhost/id/first_test.qs?test=arg
00:00:00.310   HTTP/1.1 200 OK
00:00:00.310      Content-Type: text/html; charset=UTF-8
00:00:00.460      Content-Length: 108982
00:00:00.460      Date: Fri, 30 Nov 2012 04:37:16 GMT
00:00:00.460      P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
00:00:00.460      Server: gws
00:00:00.460      X-XSS-Protection: 1; mode=block
00:00:00.460      X-Frame-Options: SAMEORIGIN

This feature can also be triggered for an individual HTTP request by including _pidsvctracequerystring parameter to the reqest.

Application Programming Interface (API)

See PID Service Application Programming Interface (API).

-- PavelGolodoniuc - 11 Oct 2012
Topic attachments
I Attachment Action Size Date Who Comment
2015-11-18_115742.pngpng 2015-11-18_115742.png manage 90.2 K 18 Nov 2015 - 12:01 PavelGolodoniuc PID Service UI screenshot
2015-11-18_115756.pngpng 2015-11-18_115756.png manage 81.0 K 18 Nov 2015 - 12:01 PavelGolodoniuc PID Service UI screenshot
2015-11-18_115820.pngpng 2015-11-18_115820.png manage 86.0 K 18 Nov 2015 - 12:01 PavelGolodoniuc PID Service UI screenshot
2015-11-18_115827.pngpng 2015-11-18_115827.png manage 53.6 K 18 Nov 2015 - 12:01 PavelGolodoniuc PID Service UI screenshot
2015-11-18_115833.pngpng 2015-11-18_115833.png manage 57.2 K 18 Nov 2015 - 12:02 PavelGolodoniuc PID Service UI screenshot
2015-11-18_115837.pngpng 2015-11-18_115837.png manage 68.9 K 18 Nov 2015 - 12:02 PavelGolodoniuc PID Service UI screenshot
500_Internal_Server_Error.pngpng 500_Internal_Server_Error.png manage 4.2 K 16 Oct 2012 - 11:54 PavelGolodoniuc  
ConditionSet.pngpng ConditionSet.png manage 1.7 K 26 Jun 2014 - 15:40 PavelGolodoniuc Use of ConditionSet
Core_Principle_Activity_Diagram.pngpng Core_Principle_Activity_Diagram.png manage 19.4 K 23 Jul 2013 - 10:42 PavelGolodoniuc Core principle activity diagram
Handling_Workflow_Activity_Diagram.pngpng Handling_Workflow_Activity_Diagram.png manage 34.5 K 22 Jul 2013 - 12:00 PavelGolodoniuc Handling workflow activity diagram
backup_ui.pngpng backup_ui.png manage 32.7 K 25 Oct 2013 - 13:11 PavelGolodoniuc Backup UI
backup_xml_schema.pngpng backup_xml_schema.png manage 62.7 K 24 Jul 2013 - 11:38 PavelGolodoniuc backup.xsd schema visualisation
base_uri_ui.pngpng base_uri_ui.png manage 7.4 K 18 Oct 2013 - 16:48 PavelGolodoniuc Base URI UI
export_mapping_ui.pngpng export_mapping_ui.png manage 31.9 K 25 Oct 2013 - 13:12 PavelGolodoniuc Export individual mapping UI
lookup_backup_ui.pngpng lookup_backup_ui.png manage 5.5 K 25 Oct 2013 - 13:00 PavelGolodoniuc Lookup maps - backup all maps
lookup_dynamic.pngpng lookup_dynamic.png manage 17.5 K 07 Mar 2013 - 16:28 PavelGolodoniuc Dynamic lookup configuration
lookup_export_ui.pngpng lookup_export_ui.png manage 26.9 K 16 Oct 2013 - 11:53 PavelGolodoniuc Lookup maps - export map
lookup_import_ui.pngpng lookup_import_ui.png manage 9.4 K 16 Oct 2013 - 11:49 PavelGolodoniuc Lookup maps - import map
lookup_static.pngpng lookup_static.png manage 12.9 K 07 Mar 2013 - 16:28 PavelGolodoniuc Static lookup configuration
mapping_cfg_inheritance_section.pngpng mapping_cfg_inheritance_section.png manage 6.5 K 22 Jul 2013 - 15:54 PavelGolodoniuc Mapping Configuration - Inheritance Section
mapping_cfg_parent_lookup.pngpng mapping_cfg_parent_lookup.png manage 11.8 K 22 Jul 2013 - 15:54 PavelGolodoniuc Mapping Configuration - Parent Lookup
mapping_cfg_select_parent.pngpng mapping_cfg_select_parent.png manage 7.2 K 22 Jul 2013 - 15:54 PavelGolodoniuc Mapping Configuration - Parent Selection
mapping_chart.pngpng mapping_chart.png manage 74.2 K 22 Jul 2013 - 15:55 PavelGolodoniuc Mapping Chart
mapping_chart_toolbar.pngpng mapping_chart_toolbar.png manage 11.4 K 23 Jul 2013 - 10:48 PavelGolodoniuc Mapping Configuration - Toolbar
merge_affected_conditions.pngpng merge_affected_conditions.png manage 2.8 K 18 Oct 2013 - 16:01 PavelGolodoniuc Merge rules - markers
merge_toolbar_ui.pngpng merge_toolbar_ui.png manage 6.6 K 18 Oct 2013 - 16:01 PavelGolodoniuc Merge rules - toolbar
merge_ui.pngpng merge_ui.png manage 21.8 K 18 Oct 2013 - 16:01 PavelGolodoniuc Merge rules - UI
regex_test.xmlxml regex_test.xml manage 0.7 K 29 Jul 2013 - 10:40 PavelGolodoniuc Sample batch file (Regex mapping)
restore_backup.pngpng restore_backup.png manage 64.2 K 24 Jul 2013 - 11:23 PavelGolodoniuc Backup restore
rule_inheritance_conceptual_view.pngpng rule_inheritance_conceptual_view.png manage 29.1 K 22 Jul 2013 - 14:15 PavelGolodoniuc Rule inheritance - Conceptual view diagram
test.xmlxml test.xml manage 1.1 K 29 Jul 2013 - 10:40 PavelGolodoniuc Sample batch file (one-to-one mappings)
tracing_mode.pngpng tracing_mode.png manage 3.2 K 30 Nov 2012 - 12:33 PavelGolodoniuc Tracing mode
ui_actions.pngpng ui_actions.png manage 2.5 K 12 Oct 2012 - 12:24 PavelGolodoniuc UI Actions
ui_condition.pngpng ui_condition.png manage 1.6 K 12 Oct 2012 - 12:24 PavelGolodoniuc UI Conditions
Topic revision: r35 - 18 Nov 2015, PavelGolodoniuc
 

Current license: All material on this collaboration platform is licensed under a Creative Commons Attribution 3.0 Australia Licence (CC BY 3.0).