Difference between revisions of "ADN Standards"

From Earth Science Information Partners (ESIP)
(Reverted edits by MartinSchultz (talk) to last revision by Rmenon)
Line 2: Line 2:
[[Standards and Conventions for WCS Server|Standards and Conventions]]<br>
:<small>[[Standards_and_Conventions_for_WCS_Server#WCS_Standard|WCS]], [[Standards_and_Conventions_for_WCS_Server#netCDF_Data_Model|netCDF]], [[Standards_and_Conventions_for_WCS_Server#CF_Conventions|CF]], [[Standards_and_Conventions_for_WCS_Server#ISO_19115l|ISO]]</small>
[[WCS Server Software|WCS Server Software]]<br>
:<small>[[WCS_Server_Software#Hosting_the_Open-Source_WCS_Server_Project_at_SourceForge|Download]] | Install server | Libraries</small><br>
:<small>[[WCS_Server_Software#WCS_Server_for_CF-netCDF_Grid_Data_Type| Grid data]] | Create CF netCDF files | [[Example_CF-NetCDF_for_Satellite|Example]] </small><br>
:<small>[[WCS_Server_Software#WCS_Server_for_Station-Point_Data_Type|Point data]] '''|''' SQL Server Interface | [[Example_SQL_for_Stations|Example]]</small><br>
[[WCS_Tools_and_Methods|WCS Tools and Methods]]<br>
:<small>CF Checker | ncML | ncDUMP | nc lib</small>
=== [[Standards_and_Conventions_for_WCS_Server | Standards and Conventions to enable the AQ Data Network]] ===
=== <big>[[Standards_and_Conventions_for_WCS_Server | Standards and Conventions to enable the AQ Data Network]]</big> ===
* [[http://www.opengeospatial.org | OGC]] '''[[Standards_and_Conventions_for_WCS_Server#WCS_Standard|Web Coverage Service (WCS)]]'''
* [[http://www.unidata.ucar.edu/software/netcdf/ | Network Common Data Format (netCDF)]] '''[[Standards_and_Conventions_for_WCS_Server#netCDF_Data_Model|data model]]'''
=== e-Reporting guidelines ===
* [[http://cf-pcmdi.llnl.gov/ | Climate and Forecasting (CF)]] '''[[Standards_and_Conventions_for_WCS_Server#CF_Conventions|metadata conventions]]'''
* [[http://www.iso.org/iso/catalogue_detail.htm?csnumber=26020 | ISO 19115]] '''[[Standards_and_Conventions_for_WCS_Server#ISO_19115l|metadata requirements]]'''
* '''[http://www.eea.europa.eu/publications/reporting-and-exchanging-air-quality EEA Technical report 5/2012]''' on Reporting and exchanging air quality information in Europe using e-Reporting
* '''[http://inspire.jrc.ec.europa.eu/index.cfm/pageid/48 INSPIRE directive]''' aims to create a European Union (EU) spatial data infrastructure. It came into force on 15 May 2007 and will be implemented in various stages, with full implementation required by 2019. ([http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2007:108:0001:0014:EN:PDF Text of the directive]); draft text of [[media:INSPIRE_DataSpecification_AC-MF_v3.0rc2_draft.pdf | data specifications for atmospheric conditions and meteorological features]]
* There are two fora in Europe aiming to establish standards in air quality monitoring and modelling: '''[http://ies.jrc.ec.europa.eu/aquila-homepage.html AQUILA] and [http://fairmode.ew.eea.europa.eu/ FAIRMODE]
=== Data formats and data exchange protocols ===
* '''[http://www.opengeospatial.org/standards/wcs | OGC-WCS]''': The Open Geospatial Consortium develops and maintains the interface standard for interoperable protocols such as WCS (web coverage service), WFS (web feature service), and WMS (web map service). The current version of WCS is 1.1 (apparently, version 1.1.2 was deprecated recently due to the lack of a standard testing procedure). Version 2.0 is in preparation. The WCS 2.0 core specification is available, but this contains very limitied specific information on how the standard can actually be implemented. The relevant extensions (for example for netcdf encoding) are still awaiting a committee decision [July 2012]. A summary description how WCS works can be found here: '''[[Standards_and_Conventions_for_WCS_Server#WCS_Standard|Web Coverage Service (WCS)]]'''
* '''[http://www.unidata.ucar.edu/software/netcdf/ Network Common Data Format (netCDF)]''': Netcdf is a binary file encoding format that allows to store attributes (i.e. metadata information) along with the data variables. For further information check out '''[[Standards_and_Conventions_for_WCS_Server#netCDF_Data_Model|data model]]'''
=== Metadata standards ===
'''General metadata specifications for geospatial information'''
These standards are mainly describing ways to specify the necessary information to find and access a specific data set, including for example information on the responsible contact person. Technically, this is implemented with XML documents that can be validated against XML schema files. The international standard for describing geospatial metadata is ISO 19115. The implementation of ISO 19115 is described in ISO 19139. In Europe, the INSPIRE directive (see [http://inspire.jrc.ec.europa.eu/]) is establishing another standard, which is largely compatible with ISO19115, but in some areas less and in other areas more stringent (concerning required metadata). Unfortunately, ISO documents are not freely available. However, it is possible to find a good part of information about it on the internet.
* [https://geo-ide.noaa.gov/wiki/index.php?title=Discovery-level_metadata_content_standards Discovery-Level Metadata Content Standards]: covers FGDC, ISO 19115, WMO core, DIF, CF, and THREDDS
* [http://www.iso.org/iso/catalogue_detail.htm?csnumber=26020 ISO 19115]: See also '''[[Standards_and_Conventions_for_WCS_Server#ISO_19115l|metadata requirements]]'''
* [http://standards.iso.org/ittf/PubliclyAvailableStandards/ISO_19139_Schemas/resources/Codelist ISO 19139 codelists]: These codelists contain the controlled vocabulary defined by ISO 19139
*  [https://geo-ide.noaa.gov/wiki/index.php?title=Coverages_and_ISO_Metadata NOAA GEO-IDE Metadata and Coverages] and [https://geo-ide.noaa.gov/wiki/index.php?title=NetCDF_Attribute_Convention_for_Dataset_Discovery netcdf attribute convention for dataset discovery] including its conformance test at [https://geo-ide.noaa.gov/wiki/index.php?title=NetCDF_Attribute_Convention_for_Dataset_Discovery_Conformance_Test | C-Test]. A helpful introduction to ISO metadata is available at '''[https://geo-ide.noaa.gov/wiki/index.php?title=Category:ISO_Building_Blocks ISO Building Blocks]'''
* [http://avdc.gsfc.nasa.gov/index.php?site=1178067684 GEOMS] metadata standard for ground-based data and satellite validation
* [http://inspire.jrc.ec.europa.eu/index.cfm/pageid/47 INSPIRE Implementing Rules] provides access to INSPIRE documents including metadata legislation and guidance documents, data specifications, etc.
* [http://cf-pcmdi.llnl.gov/ Climate and Forecasting (CF)] '''[[Standards_and_Conventions_for_WCS_Server#CF_Conventions|metadata conventions]]''': These metadata conventions are very different from ISO 19115 and INSPIRE in that they establish a standard for describing the information content '''in''' a data file (whereas ISO and INSPIRE describe information '''about''' a file or service). Nevertheless, there is a close relation between the two: firstly, the information should be consistent, secondly, information contained in the file can be harvested to (automatically) create XML metadata files which should then follow the ISO (and INSPIRE) rules.
* AQ Community metadata "facets" discussion on [[AQ_Community_Metadata_Discussion | Solta]]: In order to enable the air quality data network, the initial approach has been to define so-called metadata facets based on the content requirements that were seen as essential for the dataset discovery (see for example the [http://webapps.datafed.net/CORE.uFIND Core.uFind catalogue] at WUSTL or the [http://actris.nilu.no/ ACTRIS catalogue] at NILU. These metadata facets are not necessarily compliant with the ISO or INSPIRE standards (and they may be incomplete). Discussions to extend the thematic information that should be available in metadata records and how thi sinformation can be mapped onto the existing ISO and INSPIRE standards are at the heart of the [[Air_Quality_Metadata_Workshop_Dublin_2012| GEo AQ CoP Dublin workshop 2012]].
* [http://gaw.empa.ch/et-wdc/docs/2012/doc_5.2_GAWSIS_metadata_profile-20120322.pdf GAWSIS metadata definition]: Description of discovery metadata information for Global Atmosphere Watch (GAW) (station) data sets [provided by J. Klausen, MeteoSwiss]
* [http://www.seadatanet.org/Standards-Software/Metadata-formats SeaDataNet] (an EU project) established a community metadata standard which could potentially serve as a role-model for the AQ/AC community.
* [[media:PASODOBLE_MyAir_D_CC-QUALITY_2-2-2.pdf | PASODOBLE MyAir]] metadata standard [provided by A. de Rudder]
=== Describing Data Quality ===
There are various attempts to define data quality indicators and good practices for quality assurance and quality control. Here, we just refer to the [http://qa4eo.org/index.html QA4EO] (Quality Assurance for Earth Observations) initiative and their [http://qa4eo.org/docs/QA4EO_Principles_v4.0.pdf QA4EO Guiding Principles] document.
=== Mapping ISO, netCDF and HDF metadata ===
As the distinction between discovery metadata (the realm of ISO and INSPIRE) and descriptive metadata (primary focus of netcdf-cf and HDF conventions, such as GEOMS) is blurred, it is important to map one standard onto the other. Ted Haberman has done some pioneering work on this: [http://wiki.esipfed.org/index.php/NetCDF,_HDF,_and_ISO_Metadata].
=== Tools for handling (ISO and INSPIRE) metadata ===
There are commercial and free metadata editors available which include schema specifications for ISO19115 and INSPIRE. The [http://inspire-geoportal.ec.europa.eu/ INSPIRE Geoportal] also makes a metadata editor and validator available.
* '''[http://catmdedit.sourceforge.net/ CatMDEdit]''' is an open source metadata editor which is farily straightforward to use and can at least help to obatin a better understanding of the ISO 19115 and INSPIRE metadata structures (see example at [[ISO_INSPIRE_Metadata_Structure]]).
Other tools that are mentioned on the NOAA GEO-IDE wiki are [http://www.oxygenxml.com/ Oxygen] (commercial; manual available at [http://www.oxygenxml.com/doc/ug-editor/] and [http://www.altova.com/de/ XMLSpy] (also commercial).
=== Data set type specific issues ===
'''Model data'''
* atmospheric models generally use a spherical lat/lon coordinate system. There is a need to define the appropriate coordinate reference system (CRS) in the associated metadata. Terms that are often seen in this context are "WGS84" and "urn:ogc:def:crs:OGC:2:84" or "urn:ogc:def:crs:EPSG::4326". An [[CRS_Specification| email exchange with EPSG]] on this provides some further input to this discussion. Also check out the discussion on [http://www.unidata.ucar.edu/mailing_lists/archives/netcdf-java/2011/msg00020.html CF-ESRI mail archive].
'''Continuous data'''
* (discovery) metadata records for continuous data need to specify the begin and end of a time series. In the case of continuous data (i.e. data that are continuously updated until "now"), this can be accomplished via specific gml tags, namely ''indetermibatePosition''. The example below was provided by Paul Hasenohr of EEA, Kopenhagen:
              <gml:TimePeriod gml:id="d28e322a1049886">
                    <gml:TimeInstant gml:id="tstart">
                      <gml:TimeInstant gml:id="tend">
                        <gml:timePosition indeterminatePosition="now"/>
'''Trajectory data'''
Copied from [https://geo-ide.noaa.gov/wiki/index.php?title=ISO_Extents#Extents_for_Lines]
  <gco:CharacterString>coordinates of ship track</gco:CharacterString>
    <gml:LineString gml:id="coordsId1" srsName="urn:ogc:def:crs:EPSG::4326">
120.27748,22.61839 120.18761,22.63769 120.17214,22.53991 ... 120.28458,22.61497
Back to [[http://wiki.esipfed.org/index.php/Air_Quality_Metadata_Workshop_Dublin_2012 | Dublin workshop]] page

Revision as of 09:39, September 19, 2012