Unidata IDD Data System

<Back to Data Summit Workspace <All Data Systems
Edit with Form or Submit Word Doc

General

Contact

Data System Name: Unidata Internet Data Distribution (IDD) System
Data System URL: http://www.unidata.ucar.edu/software/idd/ and http://www.unidata.ucar.edu/software/ldm/
Contact Person: Ben Domenico
Contact e-mail: ben@unidata.ucar.edu

About the Data System (Purposes, Audience)

[[About::The Unidata community of over 150 universities has built a system for disseminating real-time earth observations via the Internet. Unlike other systems, which are based on data centers where the information can be accessed, the Unidata IDD is designed so a university can request that certain data sets be delivered to computers at their site as soon as they are available from the observing system. The IDD system also allows any site with access to specialized observations to inject the dataset into the IDD for delivery to other interested sites. Many of the 250 participating sites are receiving about 2GB of real-time data per hour via the IDD -- including weather forecasts, weather station obs, data from all NEXRADS, GOES satellite imagery, lightning strike data, aircraft-borne measurements, etc. The IDD recently surpassed FTP in terms of the amount of data delivered on the Internet 2. These datasets range from huge NCEP model ouputs run on an hourly basis to data from NEXRAD radars which come in at a rate of dozens of products per second. Early experiments are underway with the DataFed group for using the underlying Unidata Local Data Manager (LDM)to deliver air quality data in real time]]

Presentation

http://www.unidata.ucar.edu/staff/mohan/Unidata Presentations/Unidata Members Meeting 2006.ppt , http://www.unidata.ucar.edu/staff/mohan/OU NWC%20Seminar.pdf

History

The Unidata IDD has been in operation on a 24x7 basis since 1995 when it replaced a real-time data delivery system that had been used a sideband on satellite TV broadcasts that were decoded by special receivers connectd to C-band receivers at user sites. The switch to IDD did away with the subscription fee which made it possible for many more sites to receive a greatly expanded menu data streams. In the last few years, the reach of the IDD has been expanded internationally and it is now being used to ship real time data in all directions among Central and South America, Europe, and a few sites in the Far East. The underlying LDM technology has been adopted by a number of different organizations to collect and disseminate their own data. Among them are the US National Weather Service which has established a collection system for the level II and level III radar data from all the NEXRAD sites. This replace the old tape-based system, greatly reducing the cost while dramatically increasing the reliability and cutting the collection time at NCDC to virtually zero. Using Unidata LDM technology , the Spanish and South Korean Weather Services have established their own internal IDD systems for internal data distribution. The THORPEX TIGGE groups are using this technology to move data among archive centers in China, Europe, and the US. The TIGGE ensemble forecasts amount to about 20 GB per hour moving on a regular basis among these centers.

Agencies

NSF, NOAA, and possibly EPA

Datasets Served

Several different types with many options within each type. Please see: http://www.unidata.ucar.edu/data/ for more details '"`UNIQ--html-00000000-QINU`"'

Parameters

Too numerous to list here. Please click on the different types of data in the section above to get details about parameters.

Spatial - Temporal Coverage

Varies greatly with different data types. Please click on the different types of data in the section above to get details about spatial-temporal coverage.

---

Health

Weather and climate data and forecasts have many health-related impacts.

Forecasting and Reanalysis

Most of the operational weather forecasts are available in real-time as are many regional and local forecasts generated at user sites. These can be coupled with plume dispersion, storm surge, hydrological, wild fire, and other models.

Model/Emissions Evaluation

See above.

Characterization, Trends, Accountability

Not sure what this means.

Other

Not Given

Data System IT

Primary/Official Store for Some data

Not Given

Data Consolidation/integration

Data from many different sources are available via the Unidata datastreams in real-time and can be analyzed in an integrated fashion via tools such as the Unidata Integrated Data Viewer

Providing Data Access to users/externals

The main goal of the Unidata data systems is to make the data available to users.

Data Processing

Many Unidata sites have elaborate processing systems, but the central Unidata dissemination facilities deliver the data in its original form.

Visualization/Analysis

Unidata supports a suite of analysis and visualization tools (GEMPAK, McIDAS, Integrated Data Viewer) that can be employed by users for analysis and visualization. Unidata also supports middleware such as the netCDF and the THREDDS Data Server that enable users to access data from remote servers via standard protocols for analysis and display using tools other than those provided by Unidata.

Decision Support (e.g. some integration into user business process)

Unidata also supports middleware such as the netCDF and the THREDDS Data Server that enable users to access data from remote servers via standard protocols for analysis and display using tools other than those provided by Unidata. Many of these are tools such as ArcGIS and IDL that are used in the GIS and Decision Support Realm

End-to-End Integration

Unidata has been involved in large integration projects such as GEOSS and the NSF ITR project called LEAD (Linked Environments for Atmospheric Discovery) that incorporate the Unidata suite of data systems and tools into end-to-end systems.

Other DS Values

Not Given

Data Access and/or Output Interoperability

Unidata has been a leader in the integration of weather data into the world of OGC and ISO standards via projects such as THREDDS (THematic Real-time Environmental Distributed Data Services) and GALEON (Gateway to Air, Land, Environment, Ocean NetCDF). Unidata also hosted an Interoperability Day at last September's OGC Technical Committee meetings.

Reusable Tools and Methods

Nearly all of Unidata's tools and software are freely available in addition to the data streams.

Security Barriers and Solutions

The IDD/LDM uses its own well known port and each site configures the software so that only an explicit list of sites can subscribe to the data.

User Feedback Approach

Unidata is completely community-driven. It has a representative Policy Committee and Users Committee. Each meets twice per year to review the status and projected direction of the Unidata community and the program center. This allows the program to be very nimble in response to changing user needs and new technological opportunities. The transition from satellite broadcast to internet data distribution in 1995 is perhaps the best example of this. Once the decision was endorsed by the governing committees, the transition was effected in a matter of months. Keep in mind that the Internet in 1995 was in the early stages of the transition from a research network to a commercial entity.

Other Architecture

Not Given