Difference between revisions of "2010-08-27: Virtual Workshop WCS Implementation"

From Earth Science Information Partners (ESIP)
 
(26 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 +
{{AQ Backlinks}} <small> >> [[Interoperability of Air Quality Data Systems]] | [[Standards Implementation Virtual Workshops]]<br></small>
 +
==Telecon/Screenshare Info ==
 +
1:30pm EDT <br> 
 +
Phone: 866-489-0573 (US, Canada, VOIP); 205-354-0149 (international)<br>
 +
Meeting Code: *2279431*<br>
 +
 +
Screenshare details:<br>
 +
Please join my meeting, https://www1.gotomeeting.com/join/954083041
 +
 +
 
==Introduction ==
 
==Introduction ==
 
Who is this workshop for? Anyone interested in:  
 
Who is this workshop for? Anyone interested in:  
Line 11: Line 21:
 
Solution: Create a standard interface to the data using OGC Web Coverage Service  
 
Solution: Create a standard interface to the data using OGC Web Coverage Service  
  
* [http://wiki.esipfed.org/index.php/WCS_Access_to_netCDF_Files#Data_access_through_OGC_WCS_protocol Data Access through OGC WCS Protocol]
+
* Introductory Screencast to WCS (5 min): [http://capita.wustl.edu/capita/capitareports/061024_DataFed_WCS/Screencast/DataFed_WCS/DataFed_WCS.html Data Access through OGC WCS Protocol]
  
 
==In-depth Details (programmers?) ==
 
==In-depth Details (programmers?) ==
 
Case studies of how Data Systems Implemented WCS:  
 
Case studies of how Data Systems Implemented WCS:  
* DataFed - Kari Hoijarvi, Rudy Husar  
+
* [http://webapps.datafed.net/datafed_catalog.aspx DataFed] - Kari Hoijarvi, Rudy Husar  
 
* DLR
 
* DLR
 
* Giovanni/NASA - Wenli Yang/Greg Leptoukh
 
* Giovanni/NASA - Wenli Yang/Greg Leptoukh
* RSIG - Todd Plessel
+
* [http://badger.epa.gov/rsig/webscripts.html RSIG] - Todd Plessel
 
* Airnow- Steve Ludewig/Tim Dye
 
* Airnow- Steve Ludewig/Tim Dye
* Univ. of New Mexico - Karl Benedict
+
* [[Two WCS Implementations at the University of New Mexico, Earth Data Analysis Center]] - Karl Benedict
 
* Unidata - Ben Domenico
 
* Unidata - Ben Domenico
  
Line 30: Line 40:
 
Questions from Steve Ludewig, Sonoma Tech  
 
Questions from Steve Ludewig, Sonoma Tech  
 
* The WCS protocol – how can/should it be extended for air quality
 
* The WCS protocol – how can/should it be extended for air quality
 +
** Units are the most glaring omission.
 +
<nowiki><units>ug/m3</units>
 +
</nowiki>RSIG includes units in the label:<nowiki>
 +
            <label>pm25_daily_average(ug/m3)</label></nowiki> -- Todd Plessel
 +
** Meeting notes: Connect to the WCS revision working group
 +
** Meeting notes: New major changes coming with [http://www.opengeospatial.org/standards/requests/61 WCS 2.0] - Comment period ended April 14, 2010.
 +
*** Structure revamped
 +
*** Concise document (core and extensions - 1 extension for making CF-netCDF)
 +
*** Units are mentioned
 +
*** GALEON paused until 2.0 came out, anticipating major changes
 
* How should we handle different or even multiple parameter requests? (separate arguments? Lists as arguments?)
 
* How should we handle different or even multiple parameter requests? (separate arguments? Lists as arguments?)
 +
** At the WCS URL level, V1.0.0 only allows one variable per request while V1.1.x allows multiple variables.  If a client or portal is provided (e.g., the DataFed browser), the client or portal should be able to handle multiple variables case even in V1.0.0 by sending multiple requests. -- Wenli Yang
 +
** Comma-separated list. E.g.,COVERAGE=aod,std,cld,cls,sfc,ch1,sig,mos RSIG example: http://badger.epa.gov/rsig/rsigserver?SERVICE=wcs&VERSION=1.0.0&REQUEST=GetCoverage&COVERAGE=gasp.aod,gasp.std,gasp.cld,gasp.cls,gasp.sfc,gasp.ch1,gasp.sig,gasp.mos&AOD_RANGE=0,1&STD_RANGE=0,1&CLD_RANGE=1,1&CLS_RANGE=5,25&SFC_RANGE=0,0.5&CH1_RANGE=0,0.3&SIG_RANGE=0.01,0.5&MOS_RANGE=0,0.5&TIME=2008-06-21T22:00:00Z/2008-06-22T12:00:00Z&BBOX=-76.1,38,-75.5,39.6&FORMAT=ascii -- Todd Plessel\
 +
*** WCS 1.0.0 explicitly does not accept this, but WCS 1.1.x allows RangeSubset=Field1;Field2, Coverage can have any amount of fields.  --[[User:Hoijarvi|Hoijarvi]].
 +
** Meeting notes: Use standard specified ;
 
* Duration codes?
 
* Duration codes?
 +
** WCS spec is a UTC time range, e.g.,: TIME=2006-07-05T00:00:00Z/2006-07-09T23:59:59Z so all WCS implementation should support this.
 +
** Meeting notes:  PT1h, PT1d ... ISO Std.
 
* Parameter occurrence codes (POC)?
 
* Parameter occurrence codes (POC)?
 +
** More trouble than they're worth for most users who would rather just think about named variables over time and spatial ranges. (But feel free to extend your own implementation if needed to support your users.) -- Todd Plessel
 
* Time_Sequence: Ours currently is a single date/time -- should it accommodate a time range (performance is a concern)?  What time standard? (UTC?)
 
* Time_Sequence: Ours currently is a single date/time -- should it accommodate a time range (performance is a concern)?  What time standard? (UTC?)
 +
** The GSFC WCS servers support time range and will pack multiple time slices (e.g., daily, monthly) data into one single file, thus increasing the dimensionality of the output, e.g., 2D spatial => 3D spatial and temporal.  There might be, however, a performance issue.  If a client requests a long time period, e.g., an entire year, for a daily product, the server may not be able to fulfill the request.  Chris Lynnes suggested that the server only do one time step, which means that the client shall send multiple requests for multiple times and assemble the after receiving the data. --Wenli Yang
 +
** Yes. It must to be WCS-compliant. Absolutely otherwise it cannot be readily combined with other global data. WCS spec is a UTC time range, e.g.,: TIME=2006-07-05T00:00:00Z/2006-07-09T23:59:59Z
 +
so all WCS implementation should support this. -- Todd Plessel
 
* Use of an API key for authentication.
 
* Use of an API key for authentication.
 +
** Authentication should not be required for a pubic WCS. (But customize yours if you must.) -- Todd Plessel
 +
* Meeting Notes: Discussion - Two perspectives 1. To implement standard compliant and then wrap the services to fit a particular project needs with a smart client 2. Serve the project first and if possible then add standard interface.
 +
* Meeting Notes: Discussion - No way to specify the limit that a client can request in the describe coverage (i.e. only data that can be returned in 1 min), so some providers aggregate and subset ahead of the web service.
 
* Output (payload) formats
 
* Output (payload) formats
 +
** This is a crucial aspect for compatibility and performance that is not addressed by the WCS spec (presumably since it is not a data model). E.g., FORMAT=netcdf does not specify the conventions.RSIG supports the following output formats: http://badger.epa.gov/rsig/webscripts.html#output
 +
** GSFC servers currently support HDF-EOS and netCDF-CF1, but there are issues for some coverages in CF1 compatibility. --Wenli Yang
 
* Field naming
 
* Field naming
 +
** CF Conventions may eventually become widespread, but they need to be extended for AQ variables (e.g., see CMAQ variable names). -- Todd Plessel
 +
* Meeting Notes: Discussion Need a data model
 +
* Meeting Notes: Discussion netCDF-CF conventions are proposed to OGC as an independent, binary encoding with core and extensions
 +
* CF Conventions need to be extended with AQ names [http://wiki.esipfed.org/index.php/CF_Standard_Names_-_CF_Standard_Names_-_Submitted_Atmospheric_Chemistry_and_Aerosol_Terms CF Standard Names already submitted for Atmospheric Chem/Aerosol community]
  
 
=== Implementation of WCS ===  
 
=== Implementation of WCS ===  
Line 45: Line 84:
 
==== Testing Services ====
 
==== Testing Services ====
 
* [http://services-test.eoportal.org/wfssw/pages/request/getWCSRequest.jsp WCS Service Tester] - Tests DescribeCoverage and GetCapabilities  
 
* [http://services-test.eoportal.org/wfssw/pages/request/getWCSRequest.jsp WCS Service Tester] - Tests DescribeCoverage and GetCapabilities  
 +
** does not understand 1.1.2 [[User:Hoijarvi|Hoijarvi]]
 
* OGC Test Engine
 
* OGC Test Engine
** [http://cite.opengeospatial.org/test_engine/wcs/1.0.0/exec_summary WCS Test Executive Summary] |  [http://cite.opengeospatial.org/test_engine/wcs/1.0.0/assertions#I.-Test%20Requirements WCS Assertions and Test Requirements]
+
** [http://cite.opengeospatial.org/test_engine/wcs/1.0.0/exec_summary WCS Test Executive Summary] |  [http://cite.opengeospatial.org/test_engine/wcs/1.0.0/assertions#I.-Test%20Requirements WCS Assertions and Test Requirements] | [http://cite.opengeospatial.org/te2/ WCS 1.0 compliance test]
 
* [http://wiki.esipfed.org/index.php/Service_Interoperability_Tests Previous Service Interop Tests]
 
* [http://wiki.esipfed.org/index.php/Service_Interoperability_Tests Previous Service Interop Tests]
  
Line 52: Line 92:
 
List your server here if you have a WCS implementation. Test with the Service Checker for GetCapabilites, DescribeCoverage and GetCoverage with the [http://services-test.eoportal.org/wfssw/pages/request/getWCSRequest.jsp WCS Service Tester]  
 
List your server here if you have a WCS implementation. Test with the Service Checker for GetCapabilites, DescribeCoverage and GetCoverage with the [http://services-test.eoportal.org/wfssw/pages/request/getWCSRequest.jsp WCS Service Tester]  
  
* [http://wiki.esipfed.org/index.php/Available_Services Available services]
 
 
* DataFed - Passed GetCap/DescribeCoverage
 
* DataFed - Passed GetCap/DescribeCoverage
 
** Base URL http://webapps.datafed.net/AERONETd.ogc
 
** Base URL http://webapps.datafed.net/AERONETd.ogc
Line 68: Line 107:
 
** Base URL http://motherlode.ucar.edu:8080/thredds/wcs/fmrc/NCEP/GFS/Global_0p5deg/runs/NCEP-GFS-Global_0p5deg_RUN_2010-08-24T12:00:00Z
 
** Base URL http://motherlode.ucar.edu:8080/thredds/wcs/fmrc/NCEP/GFS/Global_0p5deg/runs/NCEP-GFS-Global_0p5deg_RUN_2010-08-24T12:00:00Z
 
** Tested Coverage: ICAO_Standard_Atmosphere_Reference_Height_tropopause
 
** Tested Coverage: ICAO_Standard_Atmosphere_Reference_Height_tropopause
 +
* [http://wiki.esipfed.org/index.php/Available_Services Other Available services]
 +
* EDC, Univ. NM
 +
** Base URL: http://rgisbeta.unm.edu/datasets/90593/services/ogc/wcs
 +
** Tested Coverage:36106-B3_DEM__27334
 +
* GSFC WCS for OMI products- Passed GetCap/DescribeCoverage
 +
** Base URL http://acdisc.sci.gsfc.nasa.gov/daac-bin/wcsL3
 +
** Tested Coverage: OMNO2e:OMI%20Column%20Amount%20NO2:ColumnAmountNO2CS30
 +
* GSFC WCS for AIRS products - Passed GetCap/DescribeCoverage
 +
** Base URL http://acdisc.sci.gsfc.nasa.gov/daac-bin/wcsAIRSL3
 +
** Tested Coverage: AIRX3STM:descending:CO_total_column_D
 +
* GSFC Giovanni - Passed GetCap/DescribeCoverage
 +
** Base URL http://giovanniplus-ts1.sci.gsfc.nasa.gov/daac-bin/WCS/wcsG3TS1
 +
** Tested Coverage: OMTO3e.003::ColumnAmountO3
  
 
== Participants ==  
 
== Participants ==  
Next to your name add if you are looking for guidance or have implemented a successful WCS and could provide guidance  
+
Next to your name add if you are looking for guidance or have implemented a successful WCS and could provide guidance
 
+
* Ben Domenico
 +
* Wenli Yang
 +
* Steve Ludewig
 +
* Rudy Husar
 +
* Steve Browdy
 +
* Stefan Falke
 +
* Erin Robinson
 +
* Carol Meyer
 +
* Karl Benedict
 +
* ...
  
 
== Next Workshop ==  
 
== Next Workshop ==  
 
We will continue the discussion on WCS in telecon/webex form in 6 weeks. In hopes that the information gained from this session can be implemented/tested and we can move forward with a set of interoperable web services.
 
We will continue the discussion on WCS in telecon/webex form in 6 weeks. In hopes that the information gained from this session can be implemented/tested and we can move forward with a set of interoperable web services.

Latest revision as of 09:27, September 8, 2010

Main ESIP page >> Main AQ Work Group page >> Interoperability of Air Quality Data Systems | Standards Implementation Virtual Workshops

Telecon/Screenshare Info

1:30pm EDT
Phone: 866-489-0573 (US, Canada, VOIP); 205-354-0149 (international)
Meeting Code: *2279431*

Screenshare details:
Please join my meeting, https://www1.gotomeeting.com/join/954083041


Introduction

Who is this workshop for? Anyone interested in:

  • Making their data more widely accessible
  • Using standard tools and catalogs instead of custom-made
  • Need help adding a standard interface to data

Participants in this workshop may have varied backgrounds on web services (from novice to expert). The varied experience will allow participants to form mentor relationships.

Problem: Data is distributed, in different formats and can't be accessed by common tools within the AQ Community

Solution: Create a standard interface to the data using OGC Web Coverage Service

In-depth Details (programmers?)

Case studies of how Data Systems Implemented WCS:

How-To

Questions about WCS

What WCS questions/issues have you encountered? If you know the answer to these questions please respond below the question with ** and add your name.


Questions from Steve Ludewig, Sonoma Tech

  • The WCS protocol – how can/should it be extended for air quality
    • Units are the most glaring omission.

<units>ug/m3</units> RSIG includes units in the label: <label>pm25_daily_average(ug/m3)</label> -- Todd Plessel

    • Meeting notes: Connect to the WCS revision working group
    • Meeting notes: New major changes coming with WCS 2.0 - Comment period ended April 14, 2010.
      • Structure revamped
      • Concise document (core and extensions - 1 extension for making CF-netCDF)
      • Units are mentioned
      • GALEON paused until 2.0 came out, anticipating major changes
  • How should we handle different or even multiple parameter requests? (separate arguments? Lists as arguments?)
  • Duration codes?
    • WCS spec is a UTC time range, e.g.,: TIME=2006-07-05T00:00:00Z/2006-07-09T23:59:59Z so all WCS implementation should support this.
    • Meeting notes: PT1h, PT1d ... ISO Std.
  • Parameter occurrence codes (POC)?
    • More trouble than they're worth for most users who would rather just think about named variables over time and spatial ranges. (But feel free to extend your own implementation if needed to support your users.) -- Todd Plessel
  • Time_Sequence: Ours currently is a single date/time -- should it accommodate a time range (performance is a concern)? What time standard? (UTC?)
    • The GSFC WCS servers support time range and will pack multiple time slices (e.g., daily, monthly) data into one single file, thus increasing the dimensionality of the output, e.g., 2D spatial => 3D spatial and temporal. There might be, however, a performance issue. If a client requests a long time period, e.g., an entire year, for a daily product, the server may not be able to fulfill the request. Chris Lynnes suggested that the server only do one time step, which means that the client shall send multiple requests for multiple times and assemble the after receiving the data. --Wenli Yang
    • Yes. It must to be WCS-compliant. Absolutely otherwise it cannot be readily combined with other global data. WCS spec is a UTC time range, e.g.,: TIME=2006-07-05T00:00:00Z/2006-07-09T23:59:59Z

so all WCS implementation should support this. -- Todd Plessel

  • Use of an API key for authentication.
    • Authentication should not be required for a pubic WCS. (But customize yours if you must.) -- Todd Plessel
  • Meeting Notes: Discussion - Two perspectives 1. To implement standard compliant and then wrap the services to fit a particular project needs with a smart client 2. Serve the project first and if possible then add standard interface.
  • Meeting Notes: Discussion - No way to specify the limit that a client can request in the describe coverage (i.e. only data that can be returned in 1 min), so some providers aggregate and subset ahead of the web service.
  • Output (payload) formats
    • This is a crucial aspect for compatibility and performance that is not addressed by the WCS spec (presumably since it is not a data model). E.g., FORMAT=netcdf does not specify the conventions.RSIG supports the following output formats: http://badger.epa.gov/rsig/webscripts.html#output
    • GSFC servers currently support HDF-EOS and netCDF-CF1, but there are issues for some coverages in CF1 compatibility. --Wenli Yang
  • Field naming
    • CF Conventions may eventually become widespread, but they need to be extended for AQ variables (e.g., see CMAQ variable names). -- Todd Plessel
  • Meeting Notes: Discussion Need a data model
  • Meeting Notes: Discussion netCDF-CF conventions are proposed to OGC as an independent, binary encoding with core and extensions
  • CF Conventions need to be extended with AQ names CF Standard Names already submitted for Atmospheric Chem/Aerosol community

Implementation of WCS

Testing Services

Outcome

List your server here if you have a WCS implementation. Test with the Service Checker for GetCapabilites, DescribeCoverage and GetCoverage with the WCS Service Tester

Participants

Next to your name add if you are looking for guidance or have implemented a successful WCS and could provide guidance

  • Ben Domenico
  • Wenli Yang
  • Steve Ludewig
  • Rudy Husar
  • Steve Browdy
  • Stefan Falke
  • Erin Robinson
  • Carol Meyer
  • Karl Benedict
  • ...

Next Workshop

We will continue the discussion on WCS in telecon/webex form in 6 weeks. In hopes that the information gained from this session can be implemented/tested and we can move forward with a set of interoperable web services.