Difference between revisions of "ADN Standards"

From Earth Science Information Partners (ESIP)
m
(Undo revision 40996 by 157.55.33.44 (talk))
 
(12 intermediate revisions by 2 users not shown)
Line 4: Line 4:
  
  
=== [[Standards_and_Conventions_for_WCS_Server | Standards and Conventions to enable the AQ Data Network]] ===
+
=== <big>[[Standards_and_Conventions_for_WCS_Server | Standards and Conventions to enable the AQ Data Network]]</big> ===
  
 
=== e-Reporting guidelines ===
 
=== e-Reporting guidelines ===
  
* '''[http://www.eea.europa.eu/publications/reporting-and-exchanging-air-quality EEA Technical report 5/2012]''' on Reporting and exchanging air quality information using e-Reporting
+
* '''[http://www.eea.europa.eu/publications/reporting-and-exchanging-air-quality EEA Technical report 5/2012]''' on Reporting and exchanging air quality information in Europe using e-Reporting
 +
 
 +
* '''[http://inspire.jrc.ec.europa.eu/index.cfm/pageid/48 INSPIRE directive]''' aims to create a European Union (EU) spatial data infrastructure. It came into force on 15 May 2007 and will be implemented in various stages, with full implementation required by 2019. ([http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2007:108:0001:0014:EN:PDF Text of the directive]); draft text of [[media:INSPIRE_DataSpecification_AC-MF_v3.0rc2_draft.pdf | data specifications for atmospheric conditions and meteorological features]]
 +
 
 +
* There are two fora in Europe aiming to establish standards in air quality monitoring and modelling: '''[http://ies.jrc.ec.europa.eu/aquila-homepage.html AQUILA] and [http://fairmode.ew.eea.europa.eu/ FAIRMODE]
  
 
=== Data formats and data exchange protocols ===
 
=== Data formats and data exchange protocols ===
Line 22: Line 26:
 
These standards are mainly describing ways to specify the necessary information to find and access a specific data set, including for example information on the responsible contact person. Technically, this is implemented with XML documents that can be validated against XML schema files. The international standard for describing geospatial metadata is ISO 19115. The implementation of ISO 19115 is described in ISO 19139. In Europe, the INSPIRE directive (see [http://inspire.jrc.ec.europa.eu/]) is establishing another standard, which is largely compatible with ISO19115, but in some areas less and in other areas more stringent (concerning required metadata). Unfortunately, ISO documents are not freely available. However, it is possible to find a good part of information about it on the internet.  
 
These standards are mainly describing ways to specify the necessary information to find and access a specific data set, including for example information on the responsible contact person. Technically, this is implemented with XML documents that can be validated against XML schema files. The international standard for describing geospatial metadata is ISO 19115. The implementation of ISO 19115 is described in ISO 19139. In Europe, the INSPIRE directive (see [http://inspire.jrc.ec.europa.eu/]) is establishing another standard, which is largely compatible with ISO19115, but in some areas less and in other areas more stringent (concerning required metadata). Unfortunately, ISO documents are not freely available. However, it is possible to find a good part of information about it on the internet.  
  
* [http://www.iso.org/iso/catalogue_detail.htm?csnumber=26020 | ISO 19115]: See also '''[[Standards_and_Conventions_for_WCS_Server#ISO_19115l|metadata requirements]]'''
+
* [https://geo-ide.noaa.gov/wiki/index.php?title=Discovery-level_metadata_content_standards Discovery-Level Metadata Content Standards]: covers FGDC, ISO 19115, WMO core, DIF, CF, and THREDDS
 +
 
 +
* [http://www.iso.org/iso/catalogue_detail.htm?csnumber=26020 ISO 19115]: See also '''[[Standards_and_Conventions_for_WCS_Server#ISO_19115l|metadata requirements]]'''
  
 
* [http://standards.iso.org/ittf/PubliclyAvailableStandards/ISO_19139_Schemas/resources/Codelist ISO 19139 codelists]: These codelists contain the controlled vocabulary defined by ISO 19139
 
* [http://standards.iso.org/ittf/PubliclyAvailableStandards/ISO_19139_Schemas/resources/Codelist ISO 19139 codelists]: These codelists contain the controlled vocabulary defined by ISO 19139
Line 32: Line 38:
 
* [http://inspire.jrc.ec.europa.eu/index.cfm/pageid/47 INSPIRE Implementing Rules] provides access to INSPIRE documents including metadata legislation and guidance documents, data specifications, etc.
 
* [http://inspire.jrc.ec.europa.eu/index.cfm/pageid/47 INSPIRE Implementing Rules] provides access to INSPIRE documents including metadata legislation and guidance documents, data specifications, etc.
  
* [[http://cf-pcmdi.llnl.gov/ | Climate and Forecasting (CF)]] '''[[Standards_and_Conventions_for_WCS_Server#CF_Conventions|metadata conventions]]''': These metadata conventions are very different from ISO 19115 and INSPIRE in that they establish a standard for describing the information content '''in''' a data file (whereas ISO and INSPIRE describe information '''about''' a file or service). Nevertheless, there is a close relation between the two: firstly, the information should be consistent, secondly, information contained in the file can be harvested to (automatically) create XML metadata files which should then follow the ISO (and INSPIRE) rules.
+
* [http://cf-pcmdi.llnl.gov/ Climate and Forecasting (CF)] '''[[Standards_and_Conventions_for_WCS_Server#CF_Conventions|metadata conventions]]''': These metadata conventions are very different from ISO 19115 and INSPIRE in that they establish a standard for describing the information content '''in''' a data file (whereas ISO and INSPIRE describe information '''about''' a file or service). Nevertheless, there is a close relation between the two: firstly, the information should be consistent, secondly, information contained in the file can be harvested to (automatically) create XML metadata files which should then follow the ISO (and INSPIRE) rules.
  
 
* AQ Community metadata "facets" discussion on [[AQ_Community_Metadata_Discussion | Solta]]: In order to enable the air quality data network, the initial approach has been to define so-called metadata facets based on the content requirements that were seen as essential for the dataset discovery (see for example the [http://webapps.datafed.net/CORE.uFIND Core.uFind catalogue] at WUSTL or the [http://actris.nilu.no/ ACTRIS catalogue] at NILU. These metadata facets are not necessarily compliant with the ISO or INSPIRE standards (and they may be incomplete). Discussions to extend the thematic information that should be available in metadata records and how thi sinformation can be mapped onto the existing ISO and INSPIRE standards are at the heart of the [[Air_Quality_Metadata_Workshop_Dublin_2012| GEo AQ CoP Dublin workshop 2012]].
 
* AQ Community metadata "facets" discussion on [[AQ_Community_Metadata_Discussion | Solta]]: In order to enable the air quality data network, the initial approach has been to define so-called metadata facets based on the content requirements that were seen as essential for the dataset discovery (see for example the [http://webapps.datafed.net/CORE.uFIND Core.uFind catalogue] at WUSTL or the [http://actris.nilu.no/ ACTRIS catalogue] at NILU. These metadata facets are not necessarily compliant with the ISO or INSPIRE standards (and they may be incomplete). Discussions to extend the thematic information that should be available in metadata records and how thi sinformation can be mapped onto the existing ISO and INSPIRE standards are at the heart of the [[Air_Quality_Metadata_Workshop_Dublin_2012| GEo AQ CoP Dublin workshop 2012]].
 +
 +
* [http://gaw.empa.ch/et-wdc/docs/2012/doc_5.2_GAWSIS_metadata_profile-20120322.pdf GAWSIS metadata definition]: Description of discovery metadata information for Global Atmosphere Watch (GAW) (station) data sets [provided by J. Klausen, MeteoSwiss]
 +
 +
* [http://www.seadatanet.org/Standards-Software/Metadata-formats SeaDataNet] (an EU project) established a community metadata standard which could potentially serve as a role-model for the AQ/AC community.
 +
 +
* [[media:PASODOBLE_MyAir_D_CC-QUALITY_2-2-2.pdf | PASODOBLE MyAir]] metadata standard [provided by A. de Rudder]
 +
 +
=== Describing Data Quality ===
 +
 +
There are various attempts to define data quality indicators and good practices for quality assurance and quality control. Here, we just refer to the [http://qa4eo.org/index.html QA4EO] (Quality Assurance for Earth Observations) initiative and their [http://qa4eo.org/docs/QA4EO_Principles_v4.0.pdf QA4EO Guiding Principles] document.
 +
 +
=== Mapping ISO, netCDF and HDF metadata ===
 +
 +
As the distinction between discovery metadata (the realm of ISO and INSPIRE) and descriptive metadata (primary focus of netcdf-cf and HDF conventions, such as GEOMS) is blurred, it is important to map one standard onto the other. Ted Haberman has done some pioneering work on this: [http://wiki.esipfed.org/index.php/NetCDF,_HDF,_and_ISO_Metadata].
  
 
=== Tools for handling (ISO and INSPIRE) metadata ===
 
=== Tools for handling (ISO and INSPIRE) metadata ===
Line 46: Line 66:
 
=== Data set type specific issues ===
 
=== Data set type specific issues ===
  
''Model data'''
+
'''Model data'''
  
 
* atmospheric models generally use a spherical lat/lon coordinate system. There is a need to define the appropriate coordinate reference system (CRS) in the associated metadata. Terms that are often seen in this context are "WGS84" and "urn:ogc:def:crs:OGC:2:84" or "urn:ogc:def:crs:EPSG::4326". An [[CRS_Specification| email exchange with EPSG]] on this provides some further input to this discussion. Also check out the discussion on [http://www.unidata.ucar.edu/mailing_lists/archives/netcdf-java/2011/msg00020.html CF-ESRI mail archive].
 
* atmospheric models generally use a spherical lat/lon coordinate system. There is a need to define the appropriate coordinate reference system (CRS) in the associated metadata. Terms that are often seen in this context are "WGS84" and "urn:ogc:def:crs:OGC:2:84" or "urn:ogc:def:crs:EPSG::4326". An [[CRS_Specification| email exchange with EPSG]] on this provides some further input to this discussion. Also check out the discussion on [http://www.unidata.ucar.edu/mailing_lists/archives/netcdf-java/2011/msg00020.html CF-ESRI mail archive].
 +
 +
 +
'''Continuous data'''
 +
 +
* (discovery) metadata records for continuous data need to specify the begin and end of a time series. In the case of continuous data (i.e. data that are continuously updated until "now"), this can be accomplished via specific gml tags, namely ''indetermibatePosition''. The example below was provided by Paul Hasenohr of EEA, Kopenhagen:
 +
 +
<pre>
 +
<gmd:extent>
 +
  <gmd:EX_Extent>
 +
      <gmd:temporalElement>
 +
        <gmd:EX_TemporalExtent>
 +
            <gmd:extent>
 +
              <gml:TimePeriod gml:id="d28e322a1049886">
 +
                  <gml:begin>
 +
                    <gml:TimeInstant gml:id="tstart">
 +
                        <gml:timePosition>2011-04-01T00:00:00</gml:timePosition>
 +
                    </gml:TimeInstant>
 +
                  </gml:begin>
 +
                  <gml:end>
 +
                      <gml:TimeInstant gml:id="tend">
 +
                        <gml:timePosition indeterminatePosition="now"/>
 +
                      </gml:TimeInstant>
 +
                  </gml:end>
 +
                </gml:TimePeriod>
 +
            </gmd:extent>
 +
          </gmd:EX_TemporalExtent>
 +
      </gmd:temporalElement>
 +
    </gmd:EX_Extent>
 +
</gmd:extent>
 +
</pre>
 +
 +
'''Trajectory data'''
 +
 +
Copied from [https://geo-ide.noaa.gov/wiki/index.php?title=ISO_Extents#Extents_for_Lines]
 +
<pre>
 +
<gmd:extent>
 +
<gmd:EX_Extent>
 +
  <gmd:description>
 +
  <gco:CharacterString>coordinates of ship track</gco:CharacterString>
 +
  </gmd:description>
 +
  <gmd:geographicElement>
 +
  <gmd:EX_BoundingPolygon>
 +
    <gmd:polygon>
 +
    <gml:LineString gml:id="coordsId1" srsName="urn:ogc:def:crs:EPSG::4326">
 +
      <gml:posList>
 +
120.27748,22.61839 120.18761,22.63769 120.17214,22.53991 ... 120.28458,22.61497
 +
      </gml:posList>
 +
      </gml:LineString>
 +
    </gmd:polygon>
 +
    </gmd:EX_BoundingPolygon>
 +
  </gmd:geographicElement>
 +
  </gmd:EX_Extent>
 +
</gmd:extent>
 +
</pre>
 +
 +
 +
----
 +
 +
Back to [[http://wiki.esipfed.org/index.php/Air_Quality_Metadata_Workshop_Dublin_2012 | Dublin workshop]] page

Latest revision as of 10:58, October 3, 2012

< Back to AQ CoP.png | Workshops | Air Quality Data Network



Standards and Conventions to enable the AQ Data Network

e-Reporting guidelines

  • There are two fora in Europe aiming to establish standards in air quality monitoring and modelling: AQUILA and FAIRMODE

Data formats and data exchange protocols

  • | OGC-WCS: The Open Geospatial Consortium develops and maintains the interface standard for interoperable protocols such as WCS (web coverage service), WFS (web feature service), and WMS (web map service). The current version of WCS is 1.1 (apparently, version 1.1.2 was deprecated recently due to the lack of a standard testing procedure). Version 2.0 is in preparation. The WCS 2.0 core specification is available, but this contains very limitied specific information on how the standard can actually be implemented. The relevant extensions (for example for netcdf encoding) are still awaiting a committee decision [July 2012]. A summary description how WCS works can be found here: Web Coverage Service (WCS)

Metadata standards

General metadata specifications for geospatial information

These standards are mainly describing ways to specify the necessary information to find and access a specific data set, including for example information on the responsible contact person. Technically, this is implemented with XML documents that can be validated against XML schema files. The international standard for describing geospatial metadata is ISO 19115. The implementation of ISO 19115 is described in ISO 19139. In Europe, the INSPIRE directive (see [1]) is establishing another standard, which is largely compatible with ISO19115, but in some areas less and in other areas more stringent (concerning required metadata). Unfortunately, ISO documents are not freely available. However, it is possible to find a good part of information about it on the internet.

  • GEOMS metadata standard for ground-based data and satellite validation
  • INSPIRE Implementing Rules provides access to INSPIRE documents including metadata legislation and guidance documents, data specifications, etc.
  • Climate and Forecasting (CF) metadata conventions: These metadata conventions are very different from ISO 19115 and INSPIRE in that they establish a standard for describing the information content in a data file (whereas ISO and INSPIRE describe information about a file or service). Nevertheless, there is a close relation between the two: firstly, the information should be consistent, secondly, information contained in the file can be harvested to (automatically) create XML metadata files which should then follow the ISO (and INSPIRE) rules.
  • AQ Community metadata "facets" discussion on Solta: In order to enable the air quality data network, the initial approach has been to define so-called metadata facets based on the content requirements that were seen as essential for the dataset discovery (see for example the Core.uFind catalogue at WUSTL or the ACTRIS catalogue at NILU. These metadata facets are not necessarily compliant with the ISO or INSPIRE standards (and they may be incomplete). Discussions to extend the thematic information that should be available in metadata records and how thi sinformation can be mapped onto the existing ISO and INSPIRE standards are at the heart of the GEo AQ CoP Dublin workshop 2012.
  • GAWSIS metadata definition: Description of discovery metadata information for Global Atmosphere Watch (GAW) (station) data sets [provided by J. Klausen, MeteoSwiss]
  • SeaDataNet (an EU project) established a community metadata standard which could potentially serve as a role-model for the AQ/AC community.

Describing Data Quality

There are various attempts to define data quality indicators and good practices for quality assurance and quality control. Here, we just refer to the QA4EO (Quality Assurance for Earth Observations) initiative and their QA4EO Guiding Principles document.

Mapping ISO, netCDF and HDF metadata

As the distinction between discovery metadata (the realm of ISO and INSPIRE) and descriptive metadata (primary focus of netcdf-cf and HDF conventions, such as GEOMS) is blurred, it is important to map one standard onto the other. Ted Haberman has done some pioneering work on this: [2].

Tools for handling (ISO and INSPIRE) metadata

There are commercial and free metadata editors available which include schema specifications for ISO19115 and INSPIRE. The INSPIRE Geoportal also makes a metadata editor and validator available.

  • CatMDEdit is an open source metadata editor which is farily straightforward to use and can at least help to obatin a better understanding of the ISO 19115 and INSPIRE metadata structures (see example at ISO_INSPIRE_Metadata_Structure).

Other tools that are mentioned on the NOAA GEO-IDE wiki are Oxygen (commercial; manual available at [3] and XMLSpy (also commercial).

Data set type specific issues

Model data


Continuous data

  • (discovery) metadata records for continuous data need to specify the begin and end of a time series. In the case of continuous data (i.e. data that are continuously updated until "now"), this can be accomplished via specific gml tags, namely indetermibatePosition. The example below was provided by Paul Hasenohr of EEA, Kopenhagen:
<gmd:extent>
   <gmd:EX_Extent>
      <gmd:temporalElement>
         <gmd:EX_TemporalExtent>
            <gmd:extent>
               <gml:TimePeriod gml:id="d28e322a1049886">
                  <gml:begin>
                     <gml:TimeInstant gml:id="tstart">
                        <gml:timePosition>2011-04-01T00:00:00</gml:timePosition>
                     </gml:TimeInstant>
                   </gml:begin>
                   <gml:end>
                      <gml:TimeInstant gml:id="tend">
                         <gml:timePosition indeterminatePosition="now"/>
                      </gml:TimeInstant>
                   </gml:end>
                </gml:TimePeriod>
             </gmd:extent>
          </gmd:EX_TemporalExtent>
       </gmd:temporalElement>
    </gmd:EX_Extent>
</gmd:extent>

Trajectory data

Copied from [4]

<gmd:extent>
 <gmd:EX_Extent>
  <gmd:description>
   <gco:CharacterString>coordinates of ship track</gco:CharacterString>
  </gmd:description>
  <gmd:geographicElement>
   <gmd:EX_BoundingPolygon>
    <gmd:polygon>
     <gml:LineString gml:id="coordsId1" srsName="urn:ogc:def:crs:EPSG::4326">
      <gml:posList>
120.27748,22.61839 120.18761,22.63769 120.17214,22.53991 ... 120.28458,22.61497
      </gml:posList>
      </gml:LineString>
     </gmd:polygon>
    </gmd:EX_BoundingPolygon>
   </gmd:geographicElement>
  </gmd:EX_Extent>
</gmd:extent>



Back to [| Dublin workshop] page