Difference between revisions of "SOAPifying WCS"
(51 intermediate revisions by 3 users not shown) | |||
Line 1: | Line 1: | ||
+ | {{backlink}} | ||
== SOAPifying WCS: Approach and Issues == | == SOAPifying WCS: Approach and Issues == | ||
Kari Hoijarvi, CAPITA, hoijarvi@me.wustl.edu | Kari Hoijarvi, CAPITA, hoijarvi@me.wustl.edu | ||
− | + | ===WCS Schema Components=== | |
+ | The WCS schema is assembled from multiple components. The official schemas are copied into into the DataFed workspace http://datafed.net/xs/OGC/wcs/1.0.0/ | ||
+ | describeCoverage.xsd | ||
+ | getCoverage.xsd | ||
+ | gml4wcs.xsd | ||
+ | owsBase.xsd | ||
+ | values.xsd | ||
+ | wcsCapabilities.xsd | ||
+ | wcsfix.xsd | ||
+ | Schema components gml4wcs.xsd, values.xsd and xlinks.xsd (from ../../gml/3.1.1/xlink/xlinks.xsd) were used as is, but the other schemas were not legal and had to be fixed. | ||
===Fixing the WCS Schema=== | ===Fixing the WCS Schema=== | ||
+ | Schemas use "import" instead of "include" from the same namespace. The rule is, that schemas with different namespaces must import and a namespace that is partitioned must be included. Therefore http://datafed.net/xs/OGC/wcs/1.0.0/wcsfix.xsd contains all the schemas from namespace http://www.opengis.net/wcs. In my opinion, xml schema include is a mistake and should not be used. | ||
− | + | In wcsfix.xsd and owsBase.xsd had several things that .NET 1.1, .NET 2.0 and MSXML 6.0 did | |
+ | not consider legal. Due to complexity of the schema language, I do not know if the schemas were illegal or if they exposed bugs from Microsoft's validators. I have previously experienced both. These are tagged with comment 'FIX' in wcsfix.xsd | ||
− | ' | + | *wcsCapabilities had a problematic type AbstractDescriptionBaseType which was edited heavily. Again, I don't know if this was illegal or if the MS validator was unable to process it. |
− | + | *wcsDescribecoverage was inserted as the were | |
+ | *wcsGetCoverage has our extension FilterExtensionType which currently has only one element: loc_code to query point data for time view from a location. This cannot be implemented with bbox since locations may have the same lat+lon coordinates. WFS has a standard way to put in such a filter, but WCS does not. Here is an exaple of [http://webapps.datafed.net/ogc_translator.wsfl?SERVICE=wfs&REQUEST=GetFeature&VERSION=1.0.0&CRS=EPSG:4326&TypeName=AIRNOW¶m_abbr=pmfine&outputFormat=CSV&loc_code=000010102 WFS service using loc_code filter] and this is how our [http://webapps.datafed.net/ogc_translator.wsfl?SERVICE=wcs&REQUEST=GetCoverage&VERSION=1.0.0&CRS=EPSG:4326&COVERAGE=AIRNOW.pmfine&FORMAT=CSV&BBOX=-180,-90,180,90,0,0&WIDTH=900&HEIGHT=450&DEPTH=0&loc_code=000010102 WCS query with loc_code looks]. It would be a good idea to allow servers implement their own filter and aggregator extensions, just like WMS allows server specific parameters in the url. | ||
− | + | ===WCS WSDL=== | |
− | + | The WCS web service is defined by its [http://webapps.datafed.net/WCS.asmx?WSDL WSDL]. The WSDL portType parts define service calls, message parts define their input and output types, and embedded schemas provide strong typing for input and output data. | |
− | |||
− | |||
− | |||
− | |||
− | + | All the schema components were combined into a single, [http://datafed.net/xs/OGC/wcs/1.0.0/wcsfix.xsd 'fixed' WCS schema wcsfix.xsd]. The fixing involved making the schemas compliant with schema verifiers. Next, the full wcsfix.xsd, as well as all the referenced schemas, were inserted into the [http://webapps.datafed.net/WCS.asmx?WSDL WCS service WSDL]. An outline of the WCS WSDL is pictured below. | |
− | + | ||
− | |||
− | + | [[Image:WCS WSDL.gif |200px]] | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | Looking into nodes <message name="GetCapabilitiesIn">, you'll see that | + | Looking into nodes <message name="GetCapabilitiesIn">, you'll see that the input messages as well as the XML-based output messages are straight from wcs schema: input, output |
− | the input messages as well as the XML-based output messages are straight | ||
− | from wcs schema: | ||
− | input, output | ||
wcs:GetCapabilities, wcs:WCS_Capabilities | wcs:GetCapabilities, wcs:WCS_Capabilities | ||
wcs:DescribeCoverage, wcs:CoverageDescription | wcs:DescribeCoverage, wcs:CoverageDescription | ||
wcs:GetCoverage, N/A | wcs:GetCoverage, N/A | ||
− | The GetCoverage call is problematic, because WCS returns often binary | + | ===Create Output Type=== |
− | types like NetCDF, GeoTIFF etc. | + | The GetCoverage call is problematic, because WCS returns often binary types like NetCDF, GeoTIFF etc. We added the element dfc:GetCoverageResponse for output: |
− | |||
− | We | ||
<xs:element name="GetCoverageResponse"> | <xs:element name="GetCoverageResponse"> | ||
<xs:complexType> | <xs:complexType> | ||
Line 57: | Line 54: | ||
</xs:element> | </xs:element> | ||
− | Although it is possible to embed binary data to soap output using mime | + | Although it is possible to embed binary data to soap output using mime types or base64 encoding, we did not go that way. The output contains an url to data, so that it can be retrieved using HTTP-GET request. This also works around the SOAP limitation of max foure megabyte message size, which is too small for big queries. Http-get has no built-in upper limit. In |
− | types or base64 encoding, we did not go that way. The output contains an | ||
− | url to data, so that it can be retrieved using HTTP-GET request. In | ||
addition to this pointer to data, the envelope has: | addition to this pointer to data, the envelope has: | ||
− | content type: for example application/x-netcdf | + | *content type: for example application/x-netcdf, just like in the HTTP-GET response for the final data. |
− | format: the format that was used to query this data, for example NetCDF | + | *format: the format parameter that was used to query this data, for example NetCDF |
− | domainSubset: space and time boundaries for data | + | *domainSubset: space and time boundaries for data |
− | ncml:netcdf: ncML description of data, if the data is in NetCDF cube | + | *ncml:netcdf: ncML description of data, if the data is in NetCDF cube |
− | Statistics: min, max, avg, sum, total_count, null_count | + | *Statistics: min, max, avg, sum, total_count, null_count |
− | LineageList: Soap input Envelope that was used for the query, optionally | + | *LineageList: Soap input Envelope that was used for the query, optionally |
− | other comments like the actual SQL clause that was performed, the actual | + | *other comments like the actual SQL clause that was performed, the actual |
− | URL that was retrieved, etc. | + | *URL that was retrieved, etc. |
− | |||
===SOAP Envelope=== | ===SOAP Envelope=== | ||
− | + | To create SOAP out of xml (which xml?) is just matter of wrapping it into an envelope: | |
− | |||
− | |||
− | |||
− | To create SOAP out of xml is just matter of wrapping it into an envelope: | ||
<Envelope xmlns="http://schemas.xmlsoap.org/soap/envelope/"> | <Envelope xmlns="http://schemas.xmlsoap.org/soap/envelope/"> | ||
Line 85: | Line 75: | ||
</Envelope> | </Envelope> | ||
− | + | [http://webapps.datafed.net/ogc_translator.wsfl?SERVICE=wcs&REQUEST=GetCoverage&VERSION=1.0.0&CRS=EPSG:4326&COVERAGE=NCDC_AVG_WIND.wind&FORMAT=CSV&BBOX=70,0,150,60,10,10&TIME=2005-06-01T00:00:00Z&WIDTH=1200&HEIGHT=800&DEPTH=99 Example SOAP Envelope] | |
+ | |||
+ | ===Summary=== | ||
+ | So in order to SOAPify WCS required: | ||
+ | |||
+ | * fixing some schemas | ||
+ | * including the combined fixed schemas in the WSDL | ||
+ | * creating output type in WSDL | ||
+ | * wrapping xml into SOAP envelope | ||
+ | |||
+ | ==Examples Using Python== | ||
+ | ===WCS Data Access Service Example=== | ||
+ | This example uses popular python scripting language to call Datafed WCS via SOAP. Calling WCS data access service for grid data and returning CSV table: [[WCS GetCoverage for Grid returning CSV Example]] | ||
+ | ===Service Chain Example: Data Access, Render, Save=== | ||
+ | This example uses popular python scripting language to call Datafed services. The service chain calls WCS data access service for point data, renders the points on a map image and saves the image on a disk: [[DataFed Service Chain Example]] | ||
+ | |||
+ | ==Related Links== | ||
+ | Link to an earlier WCS SOAP/WSDL page by [http://datafedwiki.wustl.edu/index.php?title=WCS_WSDL Kari] | ||
+ | |||
+ | Recommendation to GALEON by [http://www.unidata.ucar.edu/projects/THREDDS/GALEON/Reports/OGC+specs+analysis+_IMAA-UNIFI_+0.3.pdf S. Nativi, L. Bigagli and P.Mazzetti], Feb 2006 | ||
+ | * WCS bindings to HTTP GET/POST and SOAP should be consistent (this is a wellknown issue addressed in WCS 1.1). | ||
+ | * A standard description of the expected WCS SOAP interface (e.g a WCS wsdl) should be defined, to improve interoperability of SOAP-based WCS clients and servers. We would recommend SOAP service with literal encoding, due to parameters complexity using either RPC or Document style. | ||
+ | |||
+ | Early implementationof WCS SOAP wrapping at [http://athena.pin.unifi.it:8080/galeon/services UFlorence/IMAA] | ||
+ | |||
+ | [[Draft Requirements|Earth Information Exchange Requirements 2006]] | ||
+ | * The Exchange shall support access to 4-D science data products in scientific data formats (e.g., netCDF and HDF-EOS) by enabling users to: i) visualize candidate data products; ii) download data; iii) follow links to data sources. | ||
+ | |||
+ | *The Exchange shall enable access to data through commonly used protocols such as: i) WMS/WCS/WFS; ii) OPeNDAP | ||
+ | |||
+ | *GALEON Interoperability Experiment:[http://capita.wustl.edu/capita/researchareas/GALEON/Reports/060206/060206_GALEON_IE_DataFed.htm WCS Data Exchange at the DataFed Server/Client] | ||
+ | |||
+ | *[http://galeon-wcs.jot.com/pointers WCS SOAP Reports on OGC] | ||
+ | |||
+ | *[[WCS Slides]] | ||
+ | [[Category:WCS]][[Category:SOAP]][[Category:Interoperability]] |
Latest revision as of 03:15, June 7, 2006
What links here: SOAPifying WCS
SOAPifying WCS: Approach and Issues
Kari Hoijarvi, CAPITA, hoijarvi@me.wustl.edu
WCS Schema Components
The WCS schema is assembled from multiple components. The official schemas are copied into into the DataFed workspace http://datafed.net/xs/OGC/wcs/1.0.0/
describeCoverage.xsd getCoverage.xsd gml4wcs.xsd owsBase.xsd values.xsd wcsCapabilities.xsd wcsfix.xsd
Schema components gml4wcs.xsd, values.xsd and xlinks.xsd (from ../../gml/3.1.1/xlink/xlinks.xsd) were used as is, but the other schemas were not legal and had to be fixed.
Fixing the WCS Schema
Schemas use "import" instead of "include" from the same namespace. The rule is, that schemas with different namespaces must import and a namespace that is partitioned must be included. Therefore http://datafed.net/xs/OGC/wcs/1.0.0/wcsfix.xsd contains all the schemas from namespace http://www.opengis.net/wcs. In my opinion, xml schema include is a mistake and should not be used.
In wcsfix.xsd and owsBase.xsd had several things that .NET 1.1, .NET 2.0 and MSXML 6.0 did not consider legal. Due to complexity of the schema language, I do not know if the schemas were illegal or if they exposed bugs from Microsoft's validators. I have previously experienced both. These are tagged with comment 'FIX' in wcsfix.xsd
- wcsCapabilities had a problematic type AbstractDescriptionBaseType which was edited heavily. Again, I don't know if this was illegal or if the MS validator was unable to process it.
- wcsDescribecoverage was inserted as the were
- wcsGetCoverage has our extension FilterExtensionType which currently has only one element: loc_code to query point data for time view from a location. This cannot be implemented with bbox since locations may have the same lat+lon coordinates. WFS has a standard way to put in such a filter, but WCS does not. Here is an exaple of WFS service using loc_code filter and this is how our WCS query with loc_code looks. It would be a good idea to allow servers implement their own filter and aggregator extensions, just like WMS allows server specific parameters in the url.
WCS WSDL
The WCS web service is defined by its WSDL. The WSDL portType parts define service calls, message parts define their input and output types, and embedded schemas provide strong typing for input and output data.
All the schema components were combined into a single, 'fixed' WCS schema wcsfix.xsd. The fixing involved making the schemas compliant with schema verifiers. Next, the full wcsfix.xsd, as well as all the referenced schemas, were inserted into the WCS service WSDL. An outline of the WCS WSDL is pictured below.
Looking into nodes <message name="GetCapabilitiesIn">, you'll see that the input messages as well as the XML-based output messages are straight from wcs schema: input, output
wcs:GetCapabilities, wcs:WCS_Capabilities
wcs:DescribeCoverage, wcs:CoverageDescription
wcs:GetCoverage, N/A
Create Output Type
The GetCoverage call is problematic, because WCS returns often binary types like NetCDF, GeoTIFF etc. We added the element dfc:GetCoverageResponse for output:
<xs:element name="GetCoverageResponse"> <xs:complexType> <xs:all> <xs:element name="data" type="xs:anyURI"/> <xs:element name="content_type" type="xs:string"/> <xs:element name="format" type="xs:string"/> <xs:element name="domainSubset" type="wcs:DomainSubsetType"/> <xs:element ref="ncml:netcdf" minOccurs="0"/> <xs:element ref="lin:Statistics" minOccurs="0"/> <xs:element ref="lin:LineageList" minOccurs="0"/> </xs:all> </xs:complexType> </xs:element>
Although it is possible to embed binary data to soap output using mime types or base64 encoding, we did not go that way. The output contains an url to data, so that it can be retrieved using HTTP-GET request. This also works around the SOAP limitation of max foure megabyte message size, which is too small for big queries. Http-get has no built-in upper limit. In addition to this pointer to data, the envelope has:
- content type: for example application/x-netcdf, just like in the HTTP-GET response for the final data.
- format: the format parameter that was used to query this data, for example NetCDF
- domainSubset: space and time boundaries for data
- ncml:netcdf: ncML description of data, if the data is in NetCDF cube
- Statistics: min, max, avg, sum, total_count, null_count
- LineageList: Soap input Envelope that was used for the query, optionally
- other comments like the actual SQL clause that was performed, the actual
- URL that was retrieved, etc.
SOAP Envelope
To create SOAP out of xml (which xml?) is just matter of wrapping it into an envelope:
<Envelope xmlns="http://schemas.xmlsoap.org/soap/envelope/"> <Body> <GetCoverage ... </Body> </Envelope>
Summary
So in order to SOAPify WCS required:
- fixing some schemas
- including the combined fixed schemas in the WSDL
- creating output type in WSDL
- wrapping xml into SOAP envelope
Examples Using Python
WCS Data Access Service Example
This example uses popular python scripting language to call Datafed WCS via SOAP. Calling WCS data access service for grid data and returning CSV table: WCS GetCoverage for Grid returning CSV Example
Service Chain Example: Data Access, Render, Save
This example uses popular python scripting language to call Datafed services. The service chain calls WCS data access service for point data, renders the points on a map image and saves the image on a disk: DataFed Service Chain Example
Related Links
Link to an earlier WCS SOAP/WSDL page by Kari
Recommendation to GALEON by S. Nativi, L. Bigagli and P.Mazzetti, Feb 2006
- WCS bindings to HTTP GET/POST and SOAP should be consistent (this is a wellknown issue addressed in WCS 1.1).
- A standard description of the expected WCS SOAP interface (e.g a WCS wsdl) should be defined, to improve interoperability of SOAP-based WCS clients and servers. We would recommend SOAP service with literal encoding, due to parameters complexity using either RPC or Document style.
Early implementationof WCS SOAP wrapping at UFlorence/IMAA
Earth Information Exchange Requirements 2006
- The Exchange shall support access to 4-D science data products in scientific data formats (e.g., netCDF and HDF-EOS) by enabling users to: i) visualize candidate data products; ii) download data; iii) follow links to data sources.
- The Exchange shall enable access to data through commonly used protocols such as: i) WMS/WCS/WFS; ii) OPeNDAP
- GALEON Interoperability Experiment:WCS Data Exchange at the DataFed Server/Client