Difference between revisions of "Talk:Candidate Technical Topics"
From Earth Science Information Partners (ESIP)
| Line 1: | Line 1: | ||
== -- [[User:MDecker|MDecker]] 10:03, 23 August 2011 (MDT) == | == -- [[User:MDecker|MDecker]] 10:03, 23 August 2011 (MDT) == | ||
| − | CF-API | + | =CF-API= |
* need to read and write CF-compliant files easily | * need to read and write CF-compliant files easily | ||
** add a python interface to ucar libcf? http://www.unidata.ucar.edu/software/libcf/ | ** add a python interface to ucar libcf? http://www.unidata.ucar.edu/software/libcf/ | ||
| − | Performance/Virtual Datasets | + | =Performance/Virtual Datasets= |
* non-compressed data preferred | * non-compressed data preferred | ||
* many files vs. single file for queries | * many files vs. single file for queries | ||
| Line 13: | Line 13: | ||
*** need to limit query size on server side (datafed browser: currently client side management) | *** need to limit query size on server side (datafed browser: currently client side management) | ||
| − | Common NetCDF Python Interface, NetCDF4 | + | =Common NetCDF Python Interface, NetCDF4= |
* Kari cloned PyNIO interface for Windows, so no problem right now for cross platform development | * Kari cloned PyNIO interface for Windows, so no problem right now for cross platform development | ||
* solve other problems first, keep an eye open | * solve other problems first, keep an eye open | ||
| − | * NetCDF4 | + | * NetCDF4 might make things more complicated if you try to use all features, might not be easily mappable to WCS concept |
| − | Delivery of other data formats, other input formats | + | =Delivery of other data formats, other input formats= |
* need to map other formats to WCS and/or CF concept | * need to map other formats to WCS and/or CF concept | ||
* differentiate between format (NetCDF) and convention (CF) | * differentiate between format (NetCDF) and convention (CF) | ||
* chain with WMS server for default views/previews | * chain with WMS server for default views/previews | ||
| − | Tracability and revision tracking of Datasets | + | =Tracability and revision tracking of Datasets= |
* always try to get current data when dealing with real time data, always expect your data to be old | * always try to get current data when dealing with real time data, always expect your data to be old | ||
* would be nice to have WCS field for "last updated" date, same for NetCDF/CF (global attribute?) | * would be nice to have WCS field for "last updated" date, same for NetCDF/CF (global attribute?) | ||
| Line 29: | Line 29: | ||
** try to propose that for CF (and WCS) | ** try to propose that for CF (and WCS) | ||
| − | Delivery of Point Station data | + | =Delivery of Point Station data= |
* put config into SQL database as much as possible (views, stored procedures, etc) | * put config into SQL database as much as possible (views, stored procedures, etc) | ||
** try to maintain unit tests for this | ** try to maintain unit tests for this | ||
| + | * need to discuss in more detail how this can be served using WCS (Paul/NILU) | ||
| − | Access restrictions to WCS | + | =technical Access restrictions to WCS= |
* HTTP Basic authentication | * HTTP Basic authentication | ||
* API key | * API key | ||
| − | * does not have to be 100% secure, more about connecting with the users, knowing who they are | + | * does not have to be 100% secure, more about connecting with the users, knowing who they are and to establish an accepted way of accessing the data |
| − | * | + | * firewall whitelisting might be an option for small user groups |
| − | Relationship with other Servers | + | =Relationship with other Servers= |
| − | * write a wrapper | + | * write a wrapper to import data formats when needed |
| − | WCS 2.0 | + | =WCS 2.0= |
| − | * more modular | + | * more modular: core and extensions |
| − | * potentially easier to use/implement | + | * potentially easier to use/implement? |
| − | * CF-NetCDF extension coming | + | * proper CF-NetCDF extension coming |
| − | Processing Services | + | =Processing Services= |
| − | * community provides online processing service for their discipline, for example averaging | + | * idea: community provides online processing service for their discipline, for example averaging |
** not part of W*S, but separate service | ** not part of W*S, but separate service | ||
** protocol: web processing service http://www.opengeospatial.org/standards/wps | ** protocol: web processing service http://www.opengeospatial.org/standards/wps | ||
| − | Time filtering | + | =extended (Time) filtering= |
* day of week, hour of day, day of month,... | * day of week, hour of day, day of month,... | ||
* describe non-standard features in capabilities document? | * describe non-standard features in capabilities document? | ||
* might be difficult to get into official standard? | * might be difficult to get into official standard? | ||
| − | * does not interfere with standard if you don't use it | + | * does not/should not interfere with standard if you don't use it |
Revision as of 13:16, August 23, 2011
-- Michael Decker (MDecker) 10:03, 23 August 2011 (MDT)
CF-API
- need to read and write CF-compliant files easily
- add a python interface to ucar libcf? http://www.unidata.ucar.edu/software/libcf/
Performance/Virtual Datasets
- non-compressed data preferred
- many files vs. single file for queries
- mapping: many files -> single identifier
- some concerns that this might be too slow, will have to try and find a sensible balance
- queries might get very large (the only natural limit is the dataset/identifier)
- need to limit query size on server side (datafed browser: currently client side management)
- mapping: many files -> single identifier
Common NetCDF Python Interface, NetCDF4
- Kari cloned PyNIO interface for Windows, so no problem right now for cross platform development
- solve other problems first, keep an eye open
- NetCDF4 might make things more complicated if you try to use all features, might not be easily mappable to WCS concept
Delivery of other data formats, other input formats
- need to map other formats to WCS and/or CF concept
- differentiate between format (NetCDF) and convention (CF)
- chain with WMS server for default views/previews
Tracability and revision tracking of Datasets
- always try to get current data when dealing with real time data, always expect your data to be old
- would be nice to have WCS field for "last updated" date, same for NetCDF/CF (global attribute?)
- we can make something up on our own for a start
- try to propose that for CF (and WCS)
Delivery of Point Station data
- put config into SQL database as much as possible (views, stored procedures, etc)
- try to maintain unit tests for this
- need to discuss in more detail how this can be served using WCS (Paul/NILU)
technical Access restrictions to WCS
- HTTP Basic authentication
- API key
- does not have to be 100% secure, more about connecting with the users, knowing who they are and to establish an accepted way of accessing the data
- firewall whitelisting might be an option for small user groups
Relationship with other Servers
- write a wrapper to import data formats when needed
WCS 2.0
- more modular: core and extensions
- potentially easier to use/implement?
- proper CF-NetCDF extension coming
Processing Services
- idea: community provides online processing service for their discipline, for example averaging
- not part of W*S, but separate service
- protocol: web processing service http://www.opengeospatial.org/standards/wps
extended (Time) filtering
- day of week, hour of day, day of month,...
- describe non-standard features in capabilities document?
- might be difficult to get into official standard?
- does not/should not interfere with standard if you don't use it
