Difference between revisions of "Streaming and or netCDF File"

Latest revision as of 03:11, August 1, 2011

< Back to | Workshops | Air Quality Data Network

This page is for summarizing the discussion assessment on the topic of streaming vs file transfer. See also the extensive discussion for this page.

Background

speed up delivery of netcdf files by avoiding a local copy action before the download (if store=false)
- might not require "real streaming formats" such as ncstream
  - ncstream is a new format that differs from current netcdf 3/4, hence users would have to locally convert from ncstream to "normal" netcdf if they want a netcdf file in the end. Therefore, ncstream may be useful for specific client applications, but maybe not for general file download
- requires separation of header creation and variable data (this is probably accomplished in C API but we are not sure about Python API)

@@ Line 1: / Line 1: @@
 <noinclude>{{AQ CoP Solta2011 Backlinks}}
-This page is for summarizing the discussion on the topic of streaming vs file transfer. Se also the extensive [[http://wiki.esipfed.org/index.php/Talk:Streaming_and/or_netCDF_File | discussion]] for this page.
+This page is for summarizing the discussion assessment on the topic of streaming vs file transfer. See also the extensive [[Talk:Streaming_and_or_netCDF_File | discussion]] for this page.
-=== Data Streaming ...===
+==Background==
-* streaming,
-==== Streaming netCDF ====
-Russ Rew of Unidata points out that
-=== CF-netCDF and WCS 1.1 for data exploration and analysis ===
+* speed up delivery of netcdf files by avoiding a local copy action before the download (if store=false)
-====CF-netCDF is a strongly typed data structure for multidimensional data cubes====
+** might not require "real streaming formats" such as ncstream
-* netCDF gives the storage and data access facility;
+*** ncstream is a new format that differs from current netcdf 3/4, hence users would have to locally convert from ncstream to "normal" netcdf if they want a netcdf file in the end. Therefore, ncstream may be useful for specific client applications, but maybe not for general file download
-* CF gives it the semantic clarity and specificity.
+** requires separation of header creation and variable data (this is probably accomplished in C API but we are not sure about Python API)
-====WCS 1.1 data access protocol uses a simple parameter-space-time data query language ====
-* give me the data for specific Obs parameter, 3D Bounding box, time range.
-* The return payload is a slice or dice (sub-cube) subset of the original data along well defined dimensional axes and boundaries.
-==== CF-netCDF data model and WCS 1.1 data access protocol are structurally compatibe ====
+== Features of data streaming relevant to AQ ==
-* Observations/Models variables  >> map to CF-netCDF ???
-* WCS 1.1 '''Coverage''' >>> maps to >>> '''Dataset''', i.e. a set of homogeneous observations that share space/time dimensions. It may be a specific model, obs network etc.
+== AQ CoP Assessment ==
-* WCS 1.1 '''Field''' >>> maps to >>> '''Observation''' or model Parameter, i.e. AOT, ColumnNO2, SurfaceSo4