"Seegrid will be due for a migration to confluence on the 1st of August. Any update on or after the 1st of August will NOT be migrated"
Back to AdxDiscussions

Scope discussion


The pre-existing commercial requirements dictate the use of the above detail (see NEWMONT assay data report layout attached to LegacyAssayFormats). The Newmont company were one of the first companies to request additional laboratory processing data to be made available. The SIF and other commonly used flat file formats were unable to be extended in a manner to allow the additional data. The Newmont data specification currently requires this detail as do several other minor assay data formats.

ADX data streams MUST as a minimum, provide the same if not more data than current requirements as presently utilised by industry.

The ADXProjectBrief highlighted this important issue and the initial ADX draft was constructed with this knowledge and comprehensive first hand experience in a commercial laboratory environment.

The above detail must be provided to allow minimum compatibility with existing data requirements.

XML formats are intrinsically extensible. If a particular company has additional requirements then these can always be added to the data types using the XML Schema extension mechanism.

It appears that there is still a question of scope here. Is ADX intended to be a assay reporting system, or is it required to reproduce a LIMS. The view of our representative of a requestor organisation, who is responsible for receiving assay data reports, is that the amount of information in ADX 1.0 is too heavyweight and detailed for routine reporting. Thus
  1. there is a risk that it will not actually be adopted by the laboratories
  2. encapsulates information that is not necessary in normal operation.

Note that we are not saying that the internal laboratory management information is never of interest, but rather that much of it is additional to that appropriate within a standard assay report. It should be reserved for subsequent queries if required by the client. Probably this additional information should be factored out into a separate module, which can be worked up in a separate project.

Reference to project brief clause 3.5.

-- SimonCox - 20 Aug 2003

It appears that some of the management, contract, etc information may be beyond the scope of laboratory reporting.

The requirement is that
  • the Requestor can tie the results back to their database
  • the Provider can tie the results back to the records in their LIMS system.

It does NOT mean that all the information from both should be reproduced in the transferred data.

However, in the 2003-05-13 ADX project summary it says

  • "All available metadata can be sent with each batch of data ..."

In the 2003-01-22 project brief the emphasis is on "chain of custody". Does this refer to the custody of the Specimen, or the Result? Maybe use a "living document" approach with an activity list (in reverse-chron order, at the end of the document).

The emphasis in the 2003-05-13 project brief is QA/QC which I think we have handled with purpose and relatedSpecimen/Analysis tags.

-- SimonCox - 11 Aug 2003

A.

1 The section above highlights a major problem with the industry at present where the requestor and provider provide disparate data encodings with no ability to automatically tie the data back to the remote data system.

It is imperitive to enable ALL permitted available laboratory metadata to be sent with each batch of data. This resolves several issues which were identified in the ADXProjectBrief document.

What does "ALL permitted available laboratory metadata" mean?

Let me try to restate the problem to make sure I understand it.

Specimens and measurements have labels or identifiers assigned by both requestor and provider. Requestor and provider have an interest in different and only partly overlapping suites of information about these objects. For example, the requestor knows where a specimen came from, what project it is associated with, etc, which are of no interest to the provider. On the other hand, the provider knows which instrument was used, which rack the tube sat in, etc which are of no immediate interest to the requestor. There is a subset of the information that needs to be transferred from the provider to the requestor - this is the scope of the reporting operation.

reportVlims.png

The key operational requirement is that the identifier used by the requestor can be tied by the provider to the identifier used by the provider for the same object (specimen, measurement, etc). Thus, when the requestor says " tell me more about measurement XYZ123" the provider knows that this is their measurement fg678D, and they can recover the extra information from their LIMS system.

Simply carting ALL the information around ALL the time seems like a heavyweight solution to what is really identity synchronisation problem.

Of course this scenario does assume that the lab LIMS retains the additional information for a reasonable period of time. Is this part of the standard contracts?

-- SimonCox - 20 Aug 2003

2. Historical data archiving - Permits all data considered surplus at present times to become useful at some later date. An attendee at a recent conference hosted by Metech gave a presentation discussing 100yrs of data sitting in archival sheds, ranging from paper to modern data tapes cartridges All mixed format and they were attempting to reprocess it to gain additional value for project areas they were intending to re-visit. The re-processing and re-collating process was expensive and time consuming and frought with expired hardware and data formats.

Did the reprocessing require that the rack and tube number was available? More likely the specimen desciption (including location), analytical procedure and result. How much detail about each? -- SimonCox - 20 Aug 2003

3. Please consider all information is vital if due-dilligence or litegation work is being carried out requiring historic data to be reprocessed to evaluate exploration/mining projects.

Is it the responsibility of the requestor to archive all the detailed LIMS information in case of future litigation? Or is this something that can reasonably (contractually?) assigned to the provider? For what period? -- SimonCox - 20 Aug 2003

4. The exchange of maximum data allows the recipient to pick and choose what they require from the data, ie selectively import data to build data sets that match thier current data processing capabilities and needs.

Most OTS (Off The Shelf) LIMS systems are unable to provide the full extent data specified by the initial draft version of ADX1.0. It is unfortunate, at the present time that most ADX data should be considered optional until the industry and systems mature to the point where all data is provided regardless. This should not be seen as a caveat, but an indication of the inadequate data exchange between the assay community.

If by "maximum" you mean all the information the LIMS holds, then we are talking a very different beast here. This should not be designed by hand, but should be autogenerated from the LIMS schemas (or equivalent). My assumption was that we wanted a standardised "view" of the information that was needed for a client to ingest into their system for the next phase of their analysis. This "view" could optimally be a subset of the full information set, but will not be that in round 1 at least. -- SimonCox - 20 Aug 2003

5. Where data is lacking at the recipient end, the ADX supplimental data, specifically Technique/Procedures area can be used to build data structures and codings in a non-LIMS systems that match the laboratory or dispatchers system. This will allow the next communication of data to match data more effiently. Ie Field geologist will request and recieve a ADX data stream to build the a data set to match the laboratory LIMS codes. When specimens are dispatched they are coded and processed according to the specific requirements of each individual laboratory LIMS data system requirments. LIMS codes vary significantly between varioius companies and even laboratories within the same group.

ie Field client recieves an initial ADX(Buildtechniquecodes) from XYZ laboratory, when making submissions, the codes then match or can be done electronically from the remote field office.

The field client also sends check samples to alternate laboratories. ABC Lab Co also dispatch their own initial ADX data containing the required detail to allow the field to build technique and analysis codes to allow complete integration and enhance data communication.

The field client then can submit their specimens to the lab of choice using their own offsite (multi-LIMS-subset data) to manually/electronically registering specimens and analysis requirements.

Q. In the 2003-01-22 project brief the emphasis is on "chain of custody". Does this refer to the custody of the Specimen, or the Result?

Maybe use a "living document" approach with an activity list (in reverse-chron order, at the end of the document).

Chain of Custody

A. The Chain Of Custody (COC) applies mainly to specimen movement or transitional event history.

The COC is essential for the mandatory EPA compliance requiements in some states of the USA/UK and other developed countries.

The intention is to (optionally) allow the full recording of dispatch and receival of specimens at every EVENT or point of contact where contamination,loss, interferrance or degredation to specimen quality may occur during transit or processing. This includes the timing between specimen relocation or stages of processing where the resonance, colouration, rotting, radiation or capacitance discharge of specimens can change the reliability of any measured value. (water, pathology, envrionmental,soil, electrical etc). eg. Soil specimens exposed to direct sunlight being tested for Asenic. or Water samples being tested for colouration changes.

The COC is divided into two distinct zones:- FIELD - is outside lab control - usually during transportation. LABORATORY - which is within laboratory control.

The FIELD is specimen and result related as extended timing events in the field can be detrimental to the spemimens. ie Extended storage or transporation when exposed to rain or wind can degrade or contaminate specimens and effect measured values.

The LABORATORY is specimen and result related as extended timing events in the laboratory can be detrimentalto the specimens and measured values. IE extended drying (roasting) extended digestion (evaporation), extended delays between processing of batches can be detrimental to specimens. eg a rack of prepared test tubes with slurry samples may degrade rapidly while other batches are being processed with in the laboratory.

Tracking specimen movements and processing events within the laboratory can pick up the gaps of time between relocation and analysis which can effectively negate valid measured values.

These COC is now a form of external audit check for reliability on sensitive specimen measured values.

-- DavidHester - 12 Aug 2003

Great - I think COC can be handled quite nicely using a standard approach. The idea would be for a specimen to have [0..*] "events" associated with it. These events would include both transportation and preparation events. Each event will have an "operator" (person or organisation) and time period associated with it, along with a description and a link to a more explicit description of a procedure (when available). I believe this is a scalable pattern.

-- SimonCox - 20 Aug 2003

The chain of custody event may apply to a bulk container of specimens during shipping. e.g. Pallet, qty of drums, shipping container, trolleys, racks etc. It is very impractical to try to identify which specimen is in the shipping container until it arrives. The COC needs to be generalised for bulk specimen relocation and possibly? specimen specific which was the rational behind the ShippingLog. If COC was applied as an event to each individual geochem specimen the volume of duplicated data would be enourmous and I believe unnecessary.

The application of an event per specimen would be ideal for small quantitiies of specimens per batch, using the current ShippingLog concept can be applied effectively without the duplication.

The Transport/Shipping section covering COC should be considered a "event history log",which is specimen and/or specimen-group specific.

The ADX does not provide a full replacement of LIMS data, although it is possible to replicate similar features using this data.

ADX 1.0 is designed to provide a data transport mechanism to allow the recipient to rebuild a full data set to match their (probably reduced?) data processing capabilities for either Laboratory/LIMS or Field/Data management system. There are many LIMS features NOT included in the ADX specification.

* The risk of poor acceptance by the community was outlined in the ADXprojectbrief. The acceptance of ADX data will be dependant on the limitations imposed under the selected/chosen schema used to validate ADX data. *

The major risk that needs to be considered is the current proposal to reduce data availability (reducing ADX scope). If this occurs ADX will be perceived by the community as "just another assay format" which will then be compared against the cost/benifit to change software/systems/practices. If ADX is reduced it will prevent the uptake of ADX from current intrenched assay exchange formats.

ADX data can and should be readily validated by obsolete DTD's, "optional element schemas" or "fully specified schemas" that is GML/OWS compliant. The contained ADX data is the important factor along with its ease of use by the community and various systems.

Commercial laboratories are required to retain data for a number of years under certain contracts, they also maintain data for a certain period regardless of contract obligations. This however is a grey area.... It has been proven in a number of cases that the field customer often requests the laboratory to re-issue data that has been missplaced or lost, this can also occur after a company has been taken over to re-construct data that is been modified by the previous owners or to perform due dilligence.

Re-reporting data is usually performed over a period of time by the laboratory with a fee to recover data from various systems and archive formats and media. In some cases data has to be re-keyed manually from fading (carbon copy) hard copy reports stored in archives.

Data recovery is not always possible due to the limitations of expired hardware and data formats and systems which have been replaced over a period of several years.

Hence the need for a archive format for both laboratory and field use.

- DavidHester - 21 Aug 2003

Modular structure vs. all-in-one

It has been my experience that simpler formats (more consistent and modular ones, not necessarily less information) get greater acceptance because the programming costs to support it are so much lower. The refactoring we are doing to ADX are pushing it in this direction. The debate over how much information to include may simply be the result of discovering a useful subset and associated use-case for it - something afforded by the extensibility of XML.

It may be that we end up with several styles of top-level documents which are constructed out of the same basic set of ADX components. In fact, we will probably end up with a few more once we start finding a few different use-cases we hadn't thought of. There is no need to try and get it all in one document (with the consequential programming pain that generates) with the modularity we have.

-- RobertWoodcock - 21 Aug 2003

I can perceive a number of problems supporting both archival and reporting use cases. The primary concern being that of which use-case is physically stored for long term use. It will be critical and not easily determined until its too late. ie mixture of archival and report-only use-case ADX data. Both contain the measured values that can be extracted but other critical information on how those values were determined may be omitted from one or more data files.

Please consider the consequences of, dare I mention it here; BRE-EX/BUSANG. This event in geo-chem history was very damaging to various businesses involved, investors and investigated by numerous global bodies who demanded ALL data to be supplied and analysed on a number of seperate occasions.

The archival and legal use-cases aside, there are a number of use-cases outlined in the ADX 1.0 document in sections 3 which highlight the possible and required commercial usage of ADX data in its present form. A "report only" use-case would be a good start for limited (academic) purposes, but may impede on general commercial acceptance as it is too similar in content to existing (simple) data formats.

The report-use case diagram (requestor/providor?) above is a very limited view of commercial assay data usage.

ie.The diagram above indicates that the specimen location is not important to the laboratory, I believe this and many other assumptions are incorrect.

The laboratory often draw upon field project details to identify which specimen locations require additional analysis attention. This may include high sulphides from one pit vs native copper in another. Each specimen source may be identified by the Field/customer as being from the same mine. The Laboratory requires additional processing and techniques to ensure the measured values are not interferred with by naturally occuring masking chemical elements. The techniques is then critically important to the Field stat's person to allow external checks,stats and QC/QA to be performed to produce and report meaningful graphs and correlations.

-- DavidHester - 25 Aug 2003

Final comments from Pindan Software Pty Ltd.

Overlooked in this postmortem commentary is the fact that LIMS and Laboratory members of the ADX community require more detail immediately than currently provided in ADX2. The required detail was covered in ADX1.0 and is suitable for archiving, however the schema did require more work. The schema may need to be uniquely defined per providor depending on their data capabilities.

ADX1.0 documentation was developed to show the maximum data and potential use-cases of assay related information in a very short period of development time. This was to maximise the starting point of the project and allow continued tag additions to the data structures and schema validation by other members of the assay community. The additions and change requests to the ADX data format was to be managed by CSIRO without "dilution" of the current "base" ADX1.0 document data requirements.

ADX1.0 development occurred during pre-release stages of GML1.0 thro GML3.0. The GML concepts and definitions were and still are very much in flux and continue to under-go extensive changes to the schema components. At the time of ADX1.0 handover GML1.0 was being formalised and prepared for discussion.

Consideration of all sponsors requirements were not always considered important until pressure from certain committe members brought to attention current commercial apects of various existing assay data formats; which were originally considered in ADX1.0

The extensive 20yrs knowledge of a commercial LIMS developer for laboratory and assay related use-case information and advice was not always considered suitable by ADX2.0 developers.

I believe that ADX2.0 falls short of original requirements. I therefore request that the ADX2.0 format should NOT be released to the public until ADX:ARCHIVE schema is made available. Concerns of mixed ADX data content, various validation schemas that will be implemented over a short period of time in the commercial world complicate the use, storage and acceptance of any ADX standardised format.

The multi use-case requirements outlined in ADX1.0 documentation can be derived from a single ADX:archive schema. These use-cases are "needed" commercially immediately by members of ADX community.

I request sponsors and other parties involved who identify with the above or have concerns provide their input/comments now to show assay community support by adding their name and company to the below list. Thankyou.

Company Name Position Date Business
Pindan Software David Hester Managing Director 30-Oct-2003 LIMS

-- DavidHester - 30 Oct 2003
Topic revision: r3 - 15 Oct 2010, UnknownUser
 

Current license: All material on this collaboration platform is licensed under a Creative Commons Attribution 3.0 Australia Licence (CC BY 3.0).