Difference between revisions of "Glossary"

From Earth Science Information Partners (ESIP)
(44 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 
[[WCS_Access_to_netCDF_Files| Back to WCS Wrapper]]   
 
[[WCS_Access_to_netCDF_Files| Back to WCS Wrapper]]   
  
Glossary for Common Terms and Standard Names in Datafed WCS Wrapper Framework
+
__TOC__
This is the 'WCS Glossary' form. To add a page with this form, enter the page name below; if a page with that name already exists, you will be sent to a form to edit that page.
+
[[WCS General Intro]]
{{#forminput:WCS Glossary}}
 
<ask limit="500" order=ascending>[[TermDesc:=*]][[ExampleURL:=*]][[Category:Glossary]][[Category:WCS]]</ask>
 
== AQ_uFIND ==
 
  
A front end to GEOSS Clearinghouse. Currently can be used to find WMS services.
+
[[WCS Server StationPoint Data]], [[:Category:StationPoint]]
  
Example use: [http://webapps.datafed.net/AQ_uFIND.aspx?datatype=point AQ_uFIND.aspx?datatype=point]
+
[[WCS Server Cube Data]] [[:Category:Cube]]
  
== Capabilities Processor ==
+
[[WCS Mediators and Clients]] [[:Category:Mediator]] or [[:Category:Client]]
  
This component creates the standard XML documents for WCS and WFS services.
+
[[AQ Infrastructure]]
  
It operates on Metadata and data configuration. From Metadata the capabilities document gets Title, Abstract, Keywords, Contact person etc...
+
Glossary for Common Terms and Standard Names in Datafed WCS Wrapper Framework
 
+
This is the 'WCS Glossary' form. To add a page with this form, enter the page name below; if a page with that name already exists, you will be sent to a form to edit that page.
From data configuration the processor gets full information of each coverage.
+
{{#forminput:Glossary}}
 
 
[http://wiki.esipfed.org/index.php/WCS_Wrapper_Configuration_for_Cubes WCS Wrapper Configuration for Cubes]
 
 
 
[http://wiki.esipfed.org/index.php/WCS_Wrapper_Configuration_for_Point_Data WCS Wrapper Configuration for Point Data]
 
 
 
== Coverage Processor for Cubes ==
 
 
 
The Coverage Processor is a component that performs three different activities:
 
 
 
* WCS Query Parser. The syntax is checked and output is binary object of all the query elements.
 
* Subsetter. This component finds the desired coverage and applies filters to read a subset of the data:
 
** Fields: A client querying wind data may be interested in speed and direction, but reject air pressure.
 
** Bounding Box: restrict response to certain geographical area.
 
** Time: Default time, One time, list of times, periodic range of times.
 
** Grid size and interpolation: High resolution data can be interpolated to lower resolution.
 
** By dimension: Select only one or some wavelengths,elevations
 
* Formatter. The binary data is returned in desired format. Currently supported are NetCDF-CF for cubes and CSV, Comma Separated Values for points.
 
 
 
The NetCDF-CF based processor is completely generic for any compatible netCDF-CF file.
 
 
 
== Coverage Processor for Points ==
 
 
 
The Coverage Processor for points is a component that performs three different activities:
 
 
 
* WCS Query Parser. The syntax is checked and output is binary object of all the query elements.
 
* SQL Query Compiler.
 
** Fields:
 
*** Select required fields
 
*** join in location table
 
*** join in tables for fields
 
** Bounding Box:
 
*** Add where filter by location lat and lon.
 
*** Time: Add filter by data datetime.
 
*** location: if requested, add filter by location
 
* Formatter. The table data is returned in desired format. Currently the only supported is CSV, Comma Separated Values.
 
 
 
SQL processors can be either configured to supported DB schema types, or custom written for esoteric databases.
 
 
 
By writing a custom processor, anything can be used as a data source.
 
 
 
== Custom Coverage Processor for Cubes ==
 
 
 
A custom module for data storage that is not supported directly.
 
 
 
Example: [http://128.252.202.19:8080/static/HTAP/HTAP_wcs.py HTAP_wcs.py custom WCS module]
 
 
 
The HTAP demo service stores data in daily netCDF-CF files. The custom module must:
 
 
 
* Check, that only one datetime in the TimeSequence is used, because getting timeseries from separate files is not supported.
 
* Locate the correct file. Normally, the file is just the coverage identifier and '.nc' extension. In this case the template is '''GEMAQ-v1p0_SR1_sfc_%(year)s_%(doy)s.nc''' year and doy, Julian day, gets replaced.
 
 
 
The HTAP_wcs.py implements this by inheriting the default netCDF-CF processor and overriding the _input_file method.
 
 
 
    def _input_file(self, query):
 
 
 
Check that the query has exactly one datetime.
 
 
 
        if len(query.time) != 1:
 
            raise owsutil.OwsError(
 
                'CustomError',
 
                'time parameter must match exactly one datetime')
 
 
 
Get the single datetime
 
       
 
        datetimes = query.time.all_datetimes()
 
        dt = datetimes[0]
 
 
 
Now get the file template from the configuration
 
 
 
        config = self._load_config(query)[query.identifier]
 
        files = metadata['files']
 
        ncfile = files['template'] % {'year':str(dt.year), 'doy': str(iso_time.day_of_year(dt)).zfill(3)}
 
 
 
Now we have the filename, get the folder and return the full file path name.
 
 
 
        src_root, last = __file__, ''
 
        while last != 'web':
 
            src_root, last = os.path.split(src_root)
 
        return os.path.join(src_root, files['path'], ncfile)
 
 
 
By writing a custom processor, anything can be used as a data source.
 
 
 
== Custom Coverage Processor for Points ==
 
 
 
The service [http://128.252.202.19:8080/static/CIRA/CIRA_wcs.py CIRA/VIEWS] cannot access the database directly due to firewall. The SQL query can still be compiled by with configuration, but it has to be executed via a proxy server. The custom processor overrides ''getcoverage'' method and does just that.
 
 
 
By writing a custom processor, anything can be used as a data source.
 
 
 
== Data Configuration for Cubes ==
 
 
 
For standard netCDF-CF files, the configuration is automatic.
 
 
 
* Each file becomes a coverage.
 
* Each variable becomes a field.
 
 
 
This is by far the easiest way to create a WCS service. Examples are [http://128.252.202.19:8080/static/testprovider/index.html testprovider] which comes with the installation package, and
 
[http://128.252.202.19:8080/static/NASA/index.html NASA] which serves some datasets downloaded from NASA.
 
 
 
For daily netCDF-CF files it is possible to create a service without compiling them into single file. See [http://wiki.esipfed.org/index.php/WCS_Wrapper_Configuration_for_Cubes#Serving_data_from_periodic_collection_of_NetCDF_files Serving data from periodic collection of NetCDF files] as an example.
 
 
 
By creating a custom handler, it is possible to use anything as a data source.
 
 
 
== Data Configuration for Points ==
 
 
 
Programmed instructions for the framework how to access data.
 
 
 
There should be a netCDF-CF like convention for SQL databases. That would allow the point coverage processor just connect to the DB and serve, without any other configuration. But since such convention does not exist, manual configuration is needed.
 
 
 
From file [http://128.252.202.19:8080/static/point/point_config.py point_config.py]
 
 
 
Coverage information and it's descriptions:
 
 
 
    point_info = {
 
        'SURF_MET':
 
            {
 
                'Title':'Surface Meteorological Observations',
 
                'Abstract':'Dummy test data.',
 
 
 
The covered area and time. The Time dimension is a true dimension here, but contrary to grid data, the X-Y dimensions for point data are not dimensions, but attributes of the location dimension. Time dimension format is [http://en.wikipedia.org/wiki/ISO_8601 ISO 8601] (start-inclusive)/(end-inclusive)/periodicity. PT1H means Periodicity Time 1 Hour, P1D would mean Periodicity Time 1 Day
 
 
 
                'axes':{
 
                    'X':(-180, 179.75),
 
                    'Y':(-90, 89.383),
 
                    'T':iso_time.parse('2009-09-01T12:00:00/2009-09-03T12:00:00/PT1H'),
 
                    },
 
 
 
Then comes the description of the fields.
 
 
 
                'fields':{
 
                    'TEMP':{
 
                        'Title':'Temperature',
 
                        'datatype': 'float',
 
                        'units':'deg F',
 
 
 
The location table is a real dimension, Latitude and Longitude are attributes along location axis, not dimensions themselves. So a typical point dataset with locations and regular time intervals is a 2-dimensional dataset. In this case, the location table is shared, so we use the previously declared variable 'location_info' If the location tables are parameter specific, they need to be specified individually.
 
 
 
                        'axes':location_info,
 
 
 
The access instructions. This configuration is using 'complete_view', so the administrator has created the view that joins together the location table and the temperature data table. The SQL query will typically look like '''select loc_code, lat, lon, datetime, temp, flag from TEMP_V where datetime = '2009-09-01 and (lat between 34 and 44) and (lon between -90 and -80)'''. This is by far the easiest way to configure the WCS.
 
 
 
                        'complete_view':{
 
                            'view_alias':'TEMP_V',
 
                            'columns':['loc_code','lat','lon','datetime','temp', 'flag'],
 
                            },
 
                        },
 
 
 
By creating a custom handler, it is possible to store data anywhere. You still need to declare the configuration in a python dictionary, just like above.
 
 
 
== Datafed Browser ==
 
 
 
TODO: describe classic browser
 
 
 
TODO: describe browsing WCS without df catalog http://webapps.datafed.net/datafed.aspx?wcs=http://128.252.202.19:8080/CIRA&coverage=VIEWS&param_abbr=SO4f
 
 
 
TODO: Describe GE Plugin browser
 
 
 
== Feature Processor for Points ==
 
 
 
Web Feature Service, WFS, is good in publishing geographic information that does not change by time.
 
 
 
With datafed WCS it is used to publish the location table for point data, because WCS DescribeCoverage Document does not support such rich dimensions well and location tables are static geographic information.
 
 
 
The component that performs three different activities:
 
 
 
* WFS Query Parser. The syntax is checked and output is binary object of all the query elements.
 
 
 
* Subsetter.
 
** Each field may have different location table. If a the data is sparse, some fields have data only in a few locations, it makes sense to return only those locations.
 
** Locations may also be filtered by geographic bounding box.
 
** Other WFS filters are not implemented.
 
 
 
* Formatter. The data is returned in desired format. Currently the only supported is CSV, Comma Separated Values
 
 
 
== GEOSS Clearinghouse ==
 
 
 
The Clearinghouse is a component in the [http://www.earthobservations.org/gci_gci.shtml GEOSS Common Infrastructure]. One of it's functions is the [http://www.earthobservations.org/gci_cr.shtml GEOSS Components and Services Registry]
 
 
 
== Google Earth ==
 
 
 
Google Earth is a standalone program, that can display data on 3D globe.
 
 
 
Google Earth Plugin is a browser extension, that puts interactive Google Earth on a web page.
 
 
 
http://earth.google.com/
 
 
 
== ISO 19115 Metadata ==
 
 
 
Description of a service, with strictly defined XML presentation. Contains service URL's and metadata about the service.
 
 
 
== ISO 19115 Maker ==
 
 
 
A public service to create an ISO 19115 record from a WCS or WMS service.
 
 
 
If the Capabilities document contains necessary keywords, the document can be created automatically:
 
[http://webapps.datafed.net/geoss.wsfl?from_wms=http:%2f%2fwebapps.datafed.net%2fAIRNOW.ogc%3fservice%3dWMS%26version%3d1.1.1%26request%3dGetCapabilities&layer=pmfine ISO 19115 for AIRNOW pmfine WMS].
 
 
 
Without keywords in the URL, the metadata can be passed via URL parameters.
 
 
 
== KML Keyhole Markup Language ==
 
 
 
KML is the way to describe content in Google Earth and Google Maps. [http://code.google.com/apis/kml/documentation/index.html KML documentation] is hosted by google.
 
 
 
== KML Maker ==
 
 
 
Datafed tools produce KML directly out of data, which can be produced with WCS or WMS services.
 
 
 
[http://webapps.datafed.net/ge.aspx?dataset_abbr=cov_54456&view_id=map&field=SO4f&param_abbr=SO4f&scale_max=5&view_scale=0.25&kml_header=http%3a%2f%2fwebapps.datafed.net%2fviews%2fconsoles%2fpnt_hdr.xml&pr=static&time_min=2006-08-03&time_max=2006-09-01&time_step=P3D KML from a CIRA/VIEWS showing SO4f] and [http://webapps.datafed.net/loop.wsfl?init=cgi.wsfl%3fview_state%3dconsoles%252fpnt_hdr%26dataset_abbr%3dcov_54456%26param_abbr%3dSO4f%26name%3dcov_54456.SO4f%252b2006-08-03%252b-%252b2006-09-01%26export_filename%3dcov_54456.SO4f.kmz%26export_format%3dkmz&calc=cgi.wsfl%3fdataset_abbr%3dcov_54456%26view_id%3dmap%26export_type%3dpoint%26ignore_cache%3dtrue&accu=kml_merge_timeseries.wsfl&done=mark_kml_header.wsfl%3fbacklink%3dhttp%253a%252f%252fwebapps.datafed.net%252fge.aspx%253fdataset_abbr%253dcov_54456%2526view_id%253dmap%2526field%253dSO4f%2526param_abbr%253dSO4f%2526scale_max%253d5%2526view_scale%253d0.25%2526kml_header%253dhttp%25253a%25252f%25252fwebapps.datafed.net%25252fviews%25252fconsoles%25252fpnt_hdr.xml%2526pr%253dstatic%2526time_min%253d2006-08-03%2526time_max%253d2006-09-01%2526time_step%253dP3D%2526kmz_file%253d%26agg_limit%3d50%26agg_oper%3davg%26cursor_visible%3dfalse%26dataset_abbr%3dcov_54456%26days_of_week_filter%3dMon%2bTue%2bWed%2bThu%2bFri%2bSat%2bSun%26field%3dSO4f%26height%3d500%26hours_of_day_filter%3d0%2b1%2b2%2b3%2b4%2b5%2b6%2b7%2b8%2b9%2b10%2b11%2b12%2b13%2b14%2b15%2b16%2b17%2b18%2b19%2b20%2b21%2b22%2b23%26julian_after%3d5%26julian_before%3d-5%26julian_on%3doff%26lat%3d38.5%26lat_cursor_size%3d2%26lat_max%3d52%26lat_min%3d25%26loc_code%3d010730023%26lon%3d-96.0%26lon_cursor_size%3d2%26lon_max%3d-65%26lon_min%3d-127%26margins%3dfalse%26months_filter%3dTTT-TTT-TTT-TTT%26num_levels%3d6%26param_abbr%3dSO4f%26scale_max%3d5%26scale_min%3d0%26scale_mode%3dlinear%26symbol_height%3d15%26symbol_width%3d15%26view_scale%3d0.25%26width%3d900&pr=static&time_range=2006-08-03/2006-09-01/P3D&dataset_abbr=cov_54456&view_id=map&field=SO4f&height=500&lat_max=52&lat_min=25&lon_max=-65&lon_min=-127&param_abbr=SO4f&scale_max=5&view_scale=0.25&width=900&name=cov_54456.SO4f+2006-08-03+-+2006-09-01&ovl_name=cov_54456.SO4f:%20$(datetime) direct link]
 
 
 
[http://webapps.datafed.net/wms_ge.aspx?server=http://gdata1.sci.gsfc.nasa.gov/daac-bin/G3/giovanni-wms.cgi&time_min=2010-05-01&time_max=2010-05-05 KML from NASA giovanni WMS] and [http://webapps.datafed.net/loop.wsfl?init=cgi.wsfl%3fview_state%3dARC%2fKMZ_timeseries_header%26name%3dAE_DyOcn.002%253a%253aHigh_res_cloud%2b2010-05-01%2b-%2b2010-05-05%26export_filename%3dAE_DyOcn.002%253a%253aHigh_res_cloud.kmz&calc=http%3a%2f%2fgdata1.sci.gsfc.nasa.gov%2fdaac-bin%2fG3%2fgiovanni-wms.cgi%3fservice%3dWMS%26request%3dGetMap%26version%3d1.1.1%26srs%3dEPSG%3a4326%26layers%3dAE_DyOcn.002%3a%3aHigh_res_cloud%26time%3d%24(datetime)%26bbox%3d-180%2c-90%2c180%2c90%26width%3d800%26height%3d600%26bgcolor%3d0xFFFFFF%26transparent%3dFALSE%26exceptions%3dapplication%2fvnd.ogc.se_xml%26styles%3d%26format%3dimage%2fgif&accu=kml_merge_timeseries.wsfl&done=mark_kml_header.wsfl%3fbacklink%3dhttp%253a%252f%252fwebapps.datafed.net%252fwms_ge.aspx%253fkml_header%253dARC%25252fKMZ_timeseries_header.xml%2526wms%253dhttp%25253a%25252f%25252fgdata1.sci.gsfc.nasa.gov%25252fdaac-bin%25252fG3%25252fgiovanni-wms.cgi%25253fservice%25253dWMS%252526request%25253dGetMap%252526version%25253d1.1.1%252526srs%25253dEPSG%25253a4326%252526layers%25253dAE_DyOcn.002%25253a%25253aHigh_res_cloud%252526time%25253d2010-05-01%252526bbox%25253d-180%25252c-90%25252c180%25252c90%252526width%25253d800%252526height%25253d600%252526bgcolor%25253d0xFFFFFF%252526transparent%25253dFALSE%252526exceptions%25253dapplication%25252fvnd.ogc.se_xml%252526styles%25253d%252526format%25253dimage%25252fgif%2526time_min%253d2010-05-01%2526time_max%253d2010-05-05%2526time_step%253dP1D%2526kmz_file%253d&time_range=2010-05-01/2010-05-05/P1D&time_min=2010-05-01&time_max=2010-05-05&lat_min=-90&lat_max=90&lon_min=-180&lon_max=180&bgcolor=0xFFFFFF&name=AE_DyOcn.002%3a%3aHigh_res_cloud+2010-05-01+-+2010-05-05&ovl_name=AE_DyOcn.002%3a%3aHigh_res_cloud:%20$(datetime) direct link]
 
 
 
Fast Precompiled Examples:
 
 
 
[http://webapps.datafed.net/views/test/point_demo_20100827.kmz Point Demo]
 
 
 
[http://webapps.datafed.net/views/test/image_demo_20100827.kmz Gridded Demo]
 
 
 
== Location Table for Points ==
 
 
 
The location table describes the location dimension for point data.
 
 
 
The fields that datafed uses are:
 
 
 
* Mandatory fields:
 
 
 
** loc_code: A unique text field, used to identify a location.
 
** lat: Latitude of the location in degrees_north
 
** Lon: Elevation of the location in degrees_east
 
 
 
* Optional datafed fields:
 
 
 
** loc_name: Reasonably short text describing location.
 
** elev: elevation in meters.
 
 
 
* data specific fields:
 
** Any field with any name
 
 
 
Good loc_codes are short abbreviations like ACAD and YOSE for Acadia and Yosemite National Parks. Completely numeric loc codes are possible, but more difficult to recognize and since leading zeros are significant, tools like excel may think they're numbers and cut them off.
 
 
 
If the loc_codes are long, like 9 characters, it's useful to generate a numeric 16-bit primary key for the location table and use it for joining the data tables with the location table. This may help in indexing and speed things up quite a bit.
 
 
 
Example: [http://128.252.202.19:8080/CIRA?service=WFS&version=1.0.0&request=GetFeature&typename=SO4f&outputformat=text/csv CIRA/VIEWS location table]
 
 
 
Part of the CSV response:
 
 
 
    loc_code, loc_name, lat, lon, elev
 
    WHRI1, White River NF, 39.1536, -106.8209, 3413.5
 
    WICA1, Wind Cave, 43.5576, -103.4838, 1296
 
    WIMO1, Wichita Mountains, 34.7323, -98.713, 509
 
    YELL2, Yellowstone NP 2, 44.5653, -110.4002, 2425
 
 
 
== Metadata ==
 
 
 
Abstract, Contact Information, Keywords and any other such documentation that is needed in classifying or finding the service. The metadata is accessible for the user via capabilities and coverage description documents.
 
 
 
Every provider should have '''wcs_capabilities.conf''' that lists keywords and contact information. The format is simple, copy one from the [http://128.252.202.19:8080/static/testprovider/wcs_capabilities.conf testprovider] and edit it.
 
 
 
    # this file provides some information about the provider
 
    # and is incorporated into the respective WCS responses.
 
    # all currently available field identifiers are listed below.
 
    # please define every identifier only once.
 
    # other identifiers will be ignored, input is case sensitive.
 
    # the format is always <identifier>: <value>.
 
    # whitespaces before and after <value> will be stripped.
 
    # KEYWORDS can take a comma separated list that will then be
 
    # included in the respective keyword tags
 
    # empty lines and lines starting with "#" will be ignored.
 
    PROVIDER_TITLE: National Climate Data Center
 
    PROVIDER_ABSTRACT: National Climate Data Center is the worlds largest archive of climate data.
 
    KEYWORDS: Domain:Aerosol, Platform:Network, Instrument:Unknown, DataType:Point, Distributor:DataFed, Originator:NCDC, TimeRes:Minute, Vertical:Surface, TopicCategory:climatologyMeteorologyAtmosphere
 
    FEES: NONE
 
    CONSTRAINTS: NONE
 
    PROVIDER_SITE: http://lwf.ncdc.noaa.gov/oa/ncdc.html
 
    CONTACT_INDIVIDUAL: Climate Contact, Climate Services Branch, National Climatic Data Center
 
    CONTACT_PHONE: 828-271-4800
 
    CONTACT_STREET: 151 Patton Avenue Room 468
 
    CONTACT_CITY: Asheville
 
    CONTACT_ADM_AREA: North Carolina
 
    CONTACT_POSTCODE: 28801-5001
 
    CONTACT_COUNTRY: USA
 
    CONTACT_EMAIL: ncdc.info@noaa.gov
 
 
 
Here is the real live NCDC [http://128.252.202.19:8080/static/NCDC/wcs_capabilities.conf wcs_capabilities.conf]
 
 
 
== NetCDF-CF ==
 
 
 
NetCDF file format contains four kinds of information:
 
 
 
* Global attributes
 
** Simple name=value pairs
 
 
 
* Dimensions
 
** Only declares the length of the dimension
 
** Contains no dimension data.
 
 
 
* Variables
 
** Array data with any number of dimensions.
 
** Zero dimensions meaning scalar data.
 
 
 
* Variable Attributes:
 
** Simple name=value pairs associated to a variable.
 
 
 
While these are enough to describe any data, it's not easy to interpret what the data actually means. What is self-evident for humans is difficult for a computer program to reason. If you have a NetCDF viewer, it should be possible just open the file and display the data on a geographic map. But making a program that can automatically get the geographic dimensions from a NC file, is very difficult.
 
 
 
Conventions come to rescue. CF-1.0 Standardizes many things:
 
 
 
* Standard name: what is the measure data about
 
* Units
 
* How to tell, that a variable is one of the following:
 
** Data Variable, containing real data.
 
** Dimension Coordinate Variable, containing dimension coordinates.
 
** Dimension Bounds Variable, containing lower and upper bounds of a dimension coordinate.
 
* Projection
 
 
 
With this part standardized, your program can list the data variables for you and tell you exactly what you can filter by.
 
 
 
Links:
 
 
 
CF 1.0 - [http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.4/cf-conventions.html 1.4] contain conventions for cube data.
 
 
 
[https://cf-pcmdi.llnl.gov/trac/wiki/PointObservationConventions Unofficial CF-1.5] contains point data encoding. '''Expired certificate''', add security exception.
 
 
 
[http://www.cfconventions.org/ CF Conventions]
 
 
 
[http://www.unidata.ucar.edu/software/netcdf/conventions.html NetCDF Conventions]
 
 
 
[http://www.unidata.ucar.edu/software/netcdf/ Unidata NetCDF documentation]
 
 
 
[http://wiki.esipfed.org/index.php/Creating_NetCDF_CF_Files Creating NetCDF CF Files]
 
 
 
 
 
 
 
 
 
 
 
== Point Data Example ==
 
 
 
All of the configuration is done using python dictionaries and lists. The syntax is simple, This is a list:
 
 
 
    ['loc_code', 'lat', 'lon']
 
 
 
and this is a dictionary:
 
 
 
    {'key1':'value1', 'key2': 'value2' }
 
 
 
The test provider [http://128.252.202.19:8080/point point] is an example how to configure this service to use a SQL database to serve point data.
 
 
 
The data for the demonstration is stored for into sqlite, which is distributed with python by default. The project has following files:
 
 
 
* [http://128.252.202.19:8080/static/point/pntdata.py pntdata.py]: This script creates the test database and fills it with dummy data.
 
* '''pntdata.db''': The sqlite database file created by pntdata.py
 
* [http://128.252.202.19:8080/static/point/point_config.py point_config.py]:
 
** Declares the location table in the SQL database.
 
** Mapping the coverages and fields to SQL tables.
 
* [http://128.252.202.19:8080/static/point/point_WCS.py point_WCS.py] is the custom WCS handler
 
* [http://128.252.202.19:8080/static/point/point_WFS.py point_WFS.py] is the custom WFS handler that delivers the location table.
 
 
 
== Location Configuration for Points ==
 
 
 
From file [http://128.252.202.19:8080/static/point/point_config.py point_config.py]
 
 
 
In it's simplest case, SQL views are used to create required location table, so no aliasing is needed.
 
 
 
    location_info = {
 
        'location':{
 
            'service':'WFS',
 
            'version':'1.0.0',
 
            },
 
        }
 
 
 
These are the standard names that datafed uses:
 
 
 
* The dimension name is "location".
 
* No aliasing is needed, since the DB table/view and column names are standard.
 
* The view/table name in the DB is "location".
 
* The columns are lat", "lon" and "loc_code" and loc_code is a text type, not an integer.
 
 
 
 
 
In the CIRA/VIEWS case, table and fields are alised:
 
 
 
    VIEWS_location_info = {
 
        'location':{
 
            'service':'WFS',
 
            'version':'1.0.0',
 
            'table_alias':'Site',
 
            'columns':{
 
                'loc_code':{'column_alias':'SiteCode'},
 
                'lat':{'column_alias':'Latitude'},
 
                'lon':{'column_alias':'Longitude'},
 
                }
 
            },
 
        }
 
* The dimension name is still "location"
 
* The location table is called "Site"
 
* "SiteCode", "Latitude" and "Longitude" are aliased to "loc_code", "lat" and "lon".
 
 
 
== SQL Database for Points ==
 
 
 
Currently the datafed WCS for points supports one kind of point data: Fixed locations and regular intervals.
 
 
 
Point data is often stored in SQL databases. There's no standard schema, like CF-1.0 convention for NetCDF files, so it is not possible to just connect and start serving. You have to create the configuration description.
 
 
 
One of the most powerful ideas in relational database design is the concept of a '''view'''. You don't need to change the existing data tables, creating a view that makes your DB to look like the one needed is enough.
 
 
 
The Simplest Case: Configure with SQL Views
 
 
 
Location Table/View
 
 
 
The common thing between different databases is, that they need to have a location table. The current implementation is based on time series from stationary locations.
 
 
 
    table/view location
 
    +----------+-------+---------+-------+
 
    | loc_code | lat  | lon    | elev  |
 
    +----------+-------+---------+-------+
 
    | KMOD    | 37.63 | -120.95 |  30.0 |
 
    | KSTL    | 38.75 |  -90.37 | 172.0 |
 
    | KUGN    | 42.42 |  -87.87 | 222.0 |
 
    |...      |      |        |      |
 
    +----------+-------+---------+-------+
 
 
 
Here loc_code is the primary key and lat,lon is the location. Optional fields can be added. Your database may have a location table with different name and different field names, but that does not matter. CIRA VIEWS database has a location table, but it's called '''Site''' and it spells full '''longitude'''. The datafed browser uses standard names loc_code, loc_name, lat and lon for browsing; to get plug-and-play compatibility we these names are needed. In the CIRA VIEWS database, the view creation would be:
 
 
 
    create view location as
 
    select
 
        SiteCode as loc_code,
 
        Latitude as lat,
 
        Longitude as lon
 
    from Site
 
 
 
The primary key is loc_code, being unique for all the locations.
 
 
 
Because WCS does not have a good place to describe a location table, we use WFS, Web Feature Service to do the same. [http://128.252.202.19:8080/CIRA?service=WFS&Version=1.0.0&Request=GetFeature&typename=VIEWS&outputFormat=text/csv Sample WFS Call.]
 
 
 
Data Views
 
 
 
Each data variable needs a view. For example:
 
 
 
    create view TEMP_V as
 
    select
 
        location.loc_code,
 
        location.lat,
 
        location.lon,
 
        TEMP_base.datetime,
 
        TEMP_base.temp,
 
        TEMP_base.flag
 
    from location
 
    inner join TEMP_base on TEMP_base.loc_code = location.loc_code
 
 
 
Each parameter has its own data view that looks like
 
 
 
    view TEMP_V
 
    +----------+-------+---------+------------+------+------+
 
    | loc_code | lat  | lon    | datetime  | temp | flag |
 
    +----------+-------+---------+------------+------+------+
 
    | KMOD    | 37.63 | -120.95 | 2009-06-01 | 87.8 | X    |
 
    | KMOD    | 37.63 | -120.95 | 2009-06-02 | 82.3 |      |
 
    | KSTL    | 38.75 |  -90.37 | 2009-06-01 | 78.6 |      |
 
    | ...      |      |        |            |      |      |
 
    +----------+-------+---------+------------+------+------+
 
 
 
    view DEWP_V
 
    +----------+-----------------+------------+------+
 
    | loc_code | lat  | lon    | datetime  | dewp |
 
    +----------+-----------------+------------+------+
 
    | KMOD    | 37.63 | -120.95 | 2009-06-01 | 51.4 |
 
    | KMOD    | 37.63 | -120.95 | 2009-06-02 | 51.4 |
 
    | KSTL    | 38.75 |  -90.37 | 2009-06-01 | 34.9 |
 
    | ...      |      |        |            |      |
 
    +----------+-----------------+------------+------+
 
 
 
== WCS Capabilities Document ==
 
 
 
The document contains all the high level information about a service.
 
 
 
* Metadata
 
* List of Coverages.
 
 
 
From [http://128.252.202.19:8080/NRL?service=WCS&acceptversions=1.1.2&Request=GetCapabilities NRL service at datafed]
 
* Description of the Service
 
** Title=''Naval Research Laboratory''
 
** Abstract=''Naval Research Laboratory Marine Meteorology Division''
 
* Keywords ''Domain:Aerosol'' etc...
 
* Contact information
 
* HTTP access information
 
* List of coverages in the service
 
** Identifier=''NAAPS''
 
** Title=''NOGAPS and NAAPS fields in press coordinate''
 
** Abstract=none
 
** Latitude and Longitude bounds = whole world
 
 
 
== WCS Describe Coverage Document For Cubes ==
 
 
 
The document describes the coverage in detail, so that the user knows what the data is and what are the dimensions of the data.
 
 
 
NRL NAAPS for [http://128.252.202.19:8080/NRL?service=WCS&version=1.1.2&Request=DescribeCoverage&identifiers=NAAPS DescribeCoverage]
 
 
 
    <CoverageDescription>
 
 
 
* Title=''NOGAPS and NAAPS fields in press coordinate''
 
* Abstract=none
 
* Identifier=NAAPS
 
 
 
    <ows11:Title>NOGAPS and NAAPS fields in press coordinate</ows11:Title>
 
    <ows11:Abstract></ows11:Abstract>
 
    <Identifier>NAAPS</Identifier>
 
 
 
* Latitude and Longitude bounds in EPSG:4326 and WGS 84 projections. This is regardless what the data projection is.
 
 
 
    <ows11:WGS84BoundingBox crs="urn:ogc:def:crs:OGC:2:84">
 
        <ows11:LowerCorner>-179.0000 -89.5000</ows11:LowerCorner>
 
        <ows11:UpperCorner>+180.0000 +89.5000</ows11:UpperCorner>
 
    </ows11:WGS84BoundingBox>
 
 
 
* Grid bounds and size in the WGS 84 projection, it's the only currently supported projection.
 
 
 
    <GridCRS>
 
        <GridBaseCRS>urn:ogc:def:crs:OGC:2:84</GridBaseCRS>
 
        <GridType>urn:ogc:def:method:WCS:1.1:2dSimpleGrid</GridType>
 
        <GridOrigin>-179.0000 -89.5000</GridOrigin>
 
        <GridOffsets>+1.0000 +1.0000</GridOffsets>
 
        <GridCS>urn:ogc:def:cs:OGC:0.0:Grid2dSquareCS</GridCS>
 
    </GridCRS>
 
 
 
* Time dimension
 
 
 
    <TimePeriod>
 
        <BeginPosition>2007-05-09T00:00:00Z</BeginPosition>
 
        <EndPosition>2007-05-20T00:00:00Z</EndPosition>
 
        <TimeResolution>PT6H</TimeResolution>
 
    </TimePeriod>
 
 
 
* Fields of Coverage. The example is the second on the list.
 
 
 
    <Field>
 
 
 
** Human Readable Title
 
 
 
        <ows11:Title>specific humidity - nogaps</ows11:Title>
 
 
 
** Machine Readable Identifier
 
 
 
        <Identifier>sphu</Identifier>
 
 
 
** Data Definition. Value can be any number, datatype is IEEE 32 bit float and Units Of Measure ia g/g ratio
 
 
 
 
 
        <Definition>
 
            <ows11:AnyValue/>
 
            <ows11:DataType>float</ows11:DataType>
 
            <ows11:UOM>(g/g)</ows11:UOM>
 
        </Definition>
 
 
 
** IEEE standard for not a number.
 
 
 
        <NullValue>nan</NullValue>
 
 
 
** Currently no interpolation is allowed. [http://schemas.opengis.net/wcs/1.1.0/interpolationMethods.xml List of Interpolations]
 
 
 
        <InterpolationMethods>
 
            <Default>none</Default>
 
        </InterpolationMethods>
 
 
 
** Elevation dimension, values from 1000 to 10 mbar.
 
 
 
        <Axis identifier="elev">
 
            <ows11:Title>elevation</ows11:Title>
 
            <AvailableKeys>
 
                <Key>1000.0</Key>
 
                <Key>975.0</Key>
 
                <Key>950.0</Key>
 
                ....
 
                <Key>20.0</Key>
 
                <Key>10.0</Key>
 
            </AvailableKeys>
 
 
 
** Human readable meaning of the axis. The netCDF-CF has axiscode "Z" for elevation and depth dimension, so we use it.
 
 
 
            <ows11:Meaning>Z</ows11:Meaning>
 
 
 
** Type and Units Of Measure
 
 
 
            <ows11:DataType>double</ows11:DataType>
 
            <ows11:UOM>mbar</ows11:UOM>
 
        </Axis>
 
    </Field>
 
 
 
At the very end:
 
 
 
* Supported Coordinate Systems, at the very end, linear lat-lon urn:ogc:def:crs:EPSG::4326 and lon-lat urn:ogc:def:crs:OGC:2:84
 
 
 
    <SupportedCRS>urn:ogc:def:crs:EPSG::4326</SupportedCRS>
 
    <SupportedCRS>urn:ogc:def:crs:OGC:2:84</SupportedCRS>
 
 
 
* Supported Formats: image/netcdf and application/x-netcdf are the same, netCDF-CF. There is a need to standardize this format.
 
 
 
    <SupportedFormat>image/netcdf</SupportedFormat>
 
    <SupportedFormat>application/x-netcdf</SupportedFormat>
 
 
 
== WCS Describe Coverage Document For Points ==
 
 
 
The document describes the coverage in detail, so that the user knows what the data is and what are the dimensions of the data.
 
 
 
National Climate Data Center ASOS [http://128.252.202.19:8080/NCDC?Service=WCS&Version=1.1.2&Request=DescribeCoverage&identifiers=ASOS DescribeCoverage]
 
 
 
    <CoverageDescription>
 
 
 
* Title=''ASOS 1 minute data''
 
* Abstract=none
 
* Identifier=ASOS
 
 
 
    <ows11:Title>ASOS 1 minute data</ows11:Title>
 
    <Identifier>ASOS</Identifier>
 
 
 
* Latitude and Longitude bounds in EPSG:4326 and WGS 84 projections. This is regardless what the data projection is.
 
 
 
    <ows11:WGS84BoundingBox crs="urn:ogc:def:crs:OGC:2:84">
 
        <ows11:LowerCorner>-180.0000 -90.0000</ows11:LowerCorner>
 
        <ows11:UpperCorner>+179.7500 +89.3830</ows11:UpperCorner>
 
    </ows11:WGS84BoundingBox>
 
 
 
* Time dimension
 
 
 
    <TimePeriod>
 
        <BeginPosition>2009-01-01T00:00:00Z</BeginPosition>
 
        <EndPosition>2010-06-30T00:00:00Z</EndPosition>
 
        <TimeResolution>PT1M</TimeResolution>
 
    </TimePeriod>
 
 
 
* Fields of Coverage. The example is the first on the list.
 
 
 
    <Field>
 
 
 
** Human Readable Title
 
 
 
        <ows11:Title>Dew Point</ows11:Title>
 
 
 
** Machine Readable Identifier
 
 
 
        <Identifier>DEWP</Identifier>
 
 
 
** Data Definition. Value can be any number, datatype is IEEE 32 bit float and Units Of Measure is Fahrenheit ratio
 
 
 
 
 
        <Definition>
 
            <ows11:AnyValue/>
 
            <ows11:DataType>float</ows11:DataType>
 
            <ows11:UOM>F</ows11:UOM>
 
        </Definition>
 
 
 
** IEEE standard for not a number.
 
 
 
        <NullValue>nan</NullValue>
 
 
 
** Currently no interpolation is allowed. [http://schemas.opengis.net/wcs/1.1.0/interpolationMethods.xml List of Interpolations]
 
 
 
        <InterpolationMethods>
 
            <Default>none</Default>
 
        </InterpolationMethods>
 
 
 
** Location dimension
 
 
 
        <Axis identifier="location">
 
 
 
** Datafed convention. The key is a WFS call to get the location table.
 
            <AvailableKeys>
 
                <Key>http://128.252.202.19:8080/NCDC?service=WFS&amp;Version=1.0.0&amp;Request=GetFeature&amp;typename=ASOS&amp;filter=field:DEWP&amp;outputFormat=text/csv</Key>
 
            </AvailableKeys>
 
 
 
** Human readable explanation for the dimension
 
 
 
            <ows11:Meaning>location</ows11:Meaning>
 
        </Axis>
 
    </Field>
 
 
 
At the very end:
 
 
 
* Supported Coordinate Systems, at the very end, linear lat-lon urn:ogc:def:crs:EPSG::4326 and lon-lat urn:ogc:def:crs:OGC:2:84
 
 
 
    <SupportedCRS>urn:ogc:def:crs:EPSG::4326</SupportedCRS>
 
    <SupportedCRS>urn:ogc:def:crs:OGC:2:84</SupportedCRS>
 
 
 
* Supported Formats: The more natural [http://en.wikipedia.org/wiki/Comma-separated_values text/csv] is recommended and application/octet-stream is widely used legacy mime type for CSV.
 
 
 
    <SupportedFormat>text/csv</SupportedFormat>
 
    <SupportedFormat>application/octet-stream</SupportedFormat>
 
 
 
    </CoverageDescription>
 
 
 
== WCS GetCoverage Query ==
 
 
 
Query Structure:
 
 
 
Address, port and provider
 
 
 
    http://128.252.202.19:8080/CIRA?
 
 
 
Identifying service, version and request
 
 
 
    Service=WCS
 
    Version=1.1.2
 
    Request=GetCoverage
 
 
 
Identifying the coverage. Multiple coverages are not allowed.
 
 
 
    Identifier=VIEWS
 
 
 
Restricting boundingbox. Longitude range is -100..-90 and latitude range is 30..40
 
 
 
    BoundingBox=-100,30,-90,40,urn:ogc:def:crs:OGC:2:84
 
 
 
Selecting time. from 2008-07-02 to 2008-07-08 every three days: 2008-07-02, 2008-07-05, 2008-07-08
 
 
 
    TimeSequence=2008-07-02/2008-07-08/P3D
 
 
 
 
 
Selecting field SO4f only.
 
 
 
    RangeSubset=SO4f
 
 
 
Want text/csv
 
 
 
    Format=text/csv
 
 
 
Store the CSV file into the server and return only XML envelope with a pointer to it.
 
 
 
    Store=true
 
 
 
Live Query: [http://128.252.202.19:8080/CIRA?Service=WCS&Version=1.1.2&Request=GetCoverage&Identifier=VIEWS&BoundingBox=-100,30,-90,40,urn:ogc:def:crs:OGC:2:84&TimeSequence=2008-07-02/2008-07-08/P3D&RangeSubset=SO4f&Format=text/csv&Store=true VIEWS Map slice for SO4f]
 
 
 
Examples:
 
 
 
[http://128.252.202.19:8080/CIRA?Service=WCS&Version=1.1.2&Request=GetCoverage&Identifier=VIEWS&Format=text/csv&Store=false&TimeSequence=2008-07-02&RangeSubset=SO4f&BoundingBox=-100,30,-90,40,urn:ogc:def:crs:OGC:2:84 CIRA/VIEWS small geo range for one datetime &store=true]
 
 
 
 
 
[http://wiki.esipfed.org/index.php/WCS_Wrapper_Configuration_for_Point_Data#GetCoverage GetCoverage for points]
 
 
 
[http://128.252.202.19:8080/CIRA?Service=WCS&Version=1.1.2&Request=GetCoverage&Identifier=VIEWS&Format=text/csv&Store=true&TimeSequence=2008-07-02&RangeSubset=SO4f&BoundingBox=-100,30,-90,40,urn:ogc:def:crs:OGC:2:84 CIRA/VIEWS small geo range for one datetime &store=false]
 
 
 
[http://128.252.202.19:8080/CIRA?Service=WCS&Version=1.1.2&Request=GetCoverage&Identifier=VIEWS&Format=text/csv&Store=false&TimeSequence=2008-07-02&RangeSubset=SO4f&BoundingBox=-100,30,-90,40,urn:ogc:def:crs:OGC:2:84 CIRA/VIEWS small geo range for one datetime &store=true]
 
 
 
 
 
Timeseries for one location. RangeSubset both selects fields and filters them by dimensions.
 
 
 
RangeSubset='''BEXT'''<nowiki>[</nowiki>'''location'''<nowiki>[</nowiki>'''13935'''<nowiki>]]</nowiki> selects field '''BEXT''' filtering by dimension '''location''' by loc_code='''13935'''.
 
 
 
The parameter '''store=true''' makes the return to be an xml document containing some meta information and a url to the CSV result. You need to copy the url from the XML envelope and paste it to address bar to retrieve it yourself.
 
 
 
<html><a href="http://128.252.202.19:8080/NCDC?service=WCS&version=1.1.2&Request=GetCoverage&format=text/csv&store=true&identifier=ASOS&RangeSubset=BEXT[location[13935]]&TimeSequence=2010-05-01T00:00:00Z/2010-06-20T00:00:00Z/PT1M">http://128.252.202.19:8080/NCDC?service=WCS&version=1.1.2&Request=GetCoverage&format=text/csv&store=true&identifier=ASOS&RangeSubset=BEXT[location[13935]]&TimeSequence=2010-05-01T00:00:00Z/2010-06-20T00:00:00Z/PT1M</a></html>
 
 
 
Same query, but The parameter '''store=false''' returns both the XML envelope and the CSV file in the same request.
 
 
 
So far '''only Mozilla Firefox''' can open it automatically. Internet Explorer, Safari and Chrome are not recommended here.
 
 
 
<html><a href="http://128.252.202.19:8080/NCDC?service=WCS&version=1.1.2&Request=GetCoverage&format=text/csv&store=false&identifier=ASOS&RangeSubset=BEXT[location[13935]]&TimeSequence=2010-05-01T00:00:00Z/2010-06-20T00:00:00Z/PT1M">http://128.252.202.19:8080/NCDC?service=WCS&version=1.1.2&Request=GetCoverage&format=text/csv&store=false&identifier=ASOS&RangeSubset=BEXT[location[13935]]&TimeSequence=2010-05-01T00:00:00Z/2010-06-20T00:00:00Z/PT1M</a></html>
 
 
 
== WFS Capabilities Document ==
 
 
 
The document contains all the high level information about a service
 
 
 
TODO: implement
 
 
 
TODO: samples
 
 
 
== WFS DescribeFeatureType ==
 
 
 
 
 
TODO: implement
 
 
 
TODO: samples
 
 
 
== WFS GetFeature Query ==
 
 
 
The main query to get data from a WFS
 
  
[http://128.252.202.19:8080/CIRA?Service=WFS&version=1.0.0&request=GetFeature&typename=VIEWS&outputformat=text/csv CIRA/VIEWS location table]
+
{| border="1" width="800px" bordercolor="#FFFFFF" "
 +
|- valign="center" bgcolor="#F5F5F5"
 +
|width="200px" | Term
 +
|width="450px"|
 +
Description
 +
|Tags
 +
|}
 +
<ask  format=template template=GlossaryShortMacro limit="500" order=ascending>[[TermDesc:=*]][[TermDesc:=+]]][[Glossary Domain:=*]][[Category:Glossary]]</ask>

Revision as of 21:34, April 11, 2014

Back to WCS Wrapper


WCS General Intro

WCS Server StationPoint Data, Category:StationPoint

WCS Server Cube Data Category:Cube

WCS Mediators and Clients Category:Mediator or Category:Client

AQ Infrastructure

Glossary for Common Terms and Standard Names in Datafed WCS Wrapper Framework This is the 'WCS Glossary' form. To add a page with this form, enter the page name below; if a page with that name already exists, you will be sent to a form to edit that page.


Term

Description

Tags

<ask format=template template=GlossaryShortMacro limit="500" order=ascending>

+]*"*" is not in the list (WCS, HTAP, AQInfrastructure) of allowed values for the "Glossary Domain" property.</ask>