Group A2 - Enabling Data Access

From Earth Science Information Partners (ESIP)
Revision as of 09:53, October 8, 2021 by Anton Krom (User4) (talk | contribs) (1 revision imported)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

<Back to Data Summit Workspace

Attendees

  • Ben Domenico
  • Beth Hassett-Sipple
  • Bill Lamason
  • Brad Johns
  • Bryan Hubbell
  • Carol Meyer
  • Chet Wayland
  • David McCabe
  • Doug Solomon
  • Ed Lillis
  • Frank Lindsay
  • Gary Lear
  • Gary Walter
  • Jeff West
  • Joe Tikvart (Notes2)
  • Nancy Ritchey
  • Nick Mangus (Notes1)
  • Phil Dickerson
  • Rob Willis (Facilitator)
  • Sharon Leduc
  • Shawn McClure
  • Stefan Falke
  • Susan Lundquist
  • Tim Dye

Data System Connectivity Matrix

The connectivity matrix indicates data system connections ranging from downloading data to standards-based interoperability.

The yellow symbols <-- and --> indicate one-way connectivity, green <--> indicates bi-directional connectivity.

This matrix is incomplete and in flux as the connections among systems are identified and logged. Please add connections or make corrections.


To edit the click on the link to 'Data System Connectivity Matrix' above the matrix and then login as aqcluster PW: aqcluster
NOTE: To login, select the 'sign up now' button in the top right. Before editing, make sure that you see a 'Save' button in the toolbar.

  1. type in the arrow in the appropriate cell
  2. color the cell either yellow or green
  3. describe the connection in the appropriate on the Data System Connectivity Matrix
  • Please do not copy and paste cells. Each cell contains a hyperlink to its corresponding section of the wiki page.

Data System Profiles

 Data AccessPrimary Data Storage]Data Access Interoperability
3D-AQSNot GivenNot GivenNot Given
AIRNowAIRNow itself provides AQI data to the public. Raw, numerical data is available to stakeholders via AIRNowTech. AIRNow Gateway is a new addition that provides data streams to a variety of customers, utilizing a web service infrastructure.AIRNow is the primary store for national real-time air quality data, however the raw data exists at the State, local, or tribal agency level. It is also the store for the official air quality forecast that is issued by state/local agencies. AQS is the official store for this same data.Data is available in a variety of common formats – KML, ASCII text, and web services.
CASTNETAll data is provided on the CASTNET website after it is QA’ed. Ozone data is provided to the public through the CASTNET website and AirNowNot GivenData is available within the EPA firewall via SQL*Net and ODBC drivers. Data are available via the Web as prepackaged data sets and a data query wizard.
CMAQNot GivenNot GivenNot Given
DataFedDataFed is a mediator of data flow between providers and users. It does not primary/official data.
EMFNot a current goal of the project. Internal EPA only.National Computing Center (NCC) (HPC side). Not available outside of EPA firewallHave data interchange with EIS using formats specified (initial design work in progress).
EPA AIRQuest Data WarehouseNot GivenNoneNot Given
EPA AQSAQS provides standard and ad hoc report access to users. By policy, only state, local, tribal and federal agency employees and contractors are allowed an AQS account. External access is handled by the AQS Data Mart. The AQS Data Mart is the dissemination piece of AQS. It contains all of the data reported to AQS from 1980 to the present (even AQS does not keep that much on-line). It is updated nightly with the data added to AQS during the business day. Access requires registration but is open to anyone and there is a guest account. Access is via 5 SOAP web services or a “Direct Interface” query builder (which is just a GUI on top of the SOAP web services).Station based air quality measurements from the national ambient air quality monitoring network(s).The AQS database can be accessed by end users through the AQS application or Oracle Discoverer. The AQS Data Mart can be accessed via SOAP web services, a query interface on the EPA TTN, or via an SQL connection for EPA users and applications.
ESIPPromote the integration and flow of Earth science data from collection and analysis to end-use.No. Although many of its members provide that service.ESIP promotes the use of standards for data interoperability. The ESIP semantic web cluster is exploring new technologies for enhancing data access and interoperability.
GIOVANNIGiovanni provides access to the original and their subsets in various formats: HDF, netCDF, ASCII, kml/kmz (for Google Earth)OMI, MLS, HIRDLS, AIRS, TRMM, TOMSSOA, WMS, WCS, OPeNDAP, custom arrangements
GeoWebAll of the files indexed by the Geoweb Service provide metadata so that the user can understand the content, and are copied from their source to the user’s workstation when a visualization is requested. The visualization is performed on the user’s workstation within the World Wind Geo tool, within Google Earth, etc. Some of the files indexed by the Geoweb Service are served by OPeNDAP servers (Open-source Project for a Network Data Access Protocol) operated by NASA, NOAA, etc. Web browsers can work with OPeNDAP servers to identify a subset of the data, based on the variables in the file; and then download just that subset of data to the user’s workstation.Not GivenNot Given
HEIYes, using csv filesNot GivenNot Given
NARSTOThis is the primary purpose of the NARSTO Permanent Data ArchiveThe Langley DAAC, http://eosweb.larc.nasa.gov/PRODOCS/narsto/table_narsto.htmlData are located via the Web and are downloaded by FTP. Most users open the data files in a spreadsheet program.
NASA Atmospheric Science Data CenterSeveral custom order tools and subset tools.Store next near-online tape silo, online data pooMetadata managed in PostGres POSTGIS open source database
NEISGEIStandards based access to emissions and related data. Uses DataFed for data registration and access.NEISGEI provides standardized interfaces to emissions data stores.OGC standards (WMS, WCS, WFS)
RSIGYes. RSIG integrates incoming proprietary dataset formats into standard formats the user can save to their PC. A user can save the data or visualization--or both--to their local computer in such standard formats as portable binary, ASCII, NetCDF, binary XDR, and MPEG. The user can then export the selected datasets from RSIG into other applications--such as GIS tools--for further analysis. Also, developers can bypass the Java client totally by using OGC-WCS/WMS-compliant server scripts to programmatically capture RSIG-subsetted data for use in their applications. Includes numerous CMAQ conveniences: enables comparison (difference, absolute difference, percent difference, ratio, after regridding) of Ozone, AOD, PM25 from MODIS, FAQSD, GASP, MOZAIC, Airnow and AQS to CMAQ; displays intersection of CALIPSO and MOZAIC 4D path in CMAQ grid (that is, extract CMAQ data for each lon-lat-elevation-time of the other data).No. RSIG generally accesses data from other sources (and subsets it at the source).RSIG uses Open Geospatial Consortium (OGC) Web Mapping Service (WMS) and Web Coverage Service (WCS) standards for specifying data subsets and a supports a variety of standard output formats (ASCII, NetCDF, KMZ, etc.) requested by users to enable data to be saved and input into other applications.
Unidata IDD Data SystemThe main goal of the Unidata data systems is to make the data available to users.Not GivenUnidata has been a leader in the integration of weather data into the world of OGC and ISO standards via projects such as THREDDS (THematic Real-time Environmental Distributed Data Services) and GALEON (Gateway to Air, Land, Environment, Ocean NetCDF). Unidata also hosted an Interoperability Day at last September's OGC Technical Committee meetings.
VIEWSThe VIEWS website provides easy online access to the integrated database through a wide variety of tools and applications. Users are encouraged to register with the website, but no account or login is required to use any of the VIEWS resources. The entire contents of the integrated database are freely and openly available for ad hoc querying, display, and download. In addition, users are encouraged to submit custom requests for data and to provide additional relevant datasets for inclusion in the database.Raw data and metadata can be obtained from VIEWS in a variety of output mediums: Delimited or fixed width ASCII text files, Microsoft Excel workbooks, static HTML tables, and/or interactive HTML "data grids". In addition, the user can select the specific fields that are included in the output results and can specify a variety of output options, such as "wide" or "skinny" format, date format, missing data placholder, quoted text, etc. The VIEWS team is also working to provide the option to output data in XML and as XML SOAP responses.