Making Science Data Easier to Use with OPeNDAP

From Earth Science Information Partners (ESIP)
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

Using Earth Science Data: Why So Difficult?

Complicated data require complicated format

  • Network Common Data Form (netCDF)
  • Hierarchical Data Format (HDF)

Complicated formats require good tools

Tools, Tools, Tools

OPeNDAP Tools.png

EOS HDF Data Readability (without OPeNDAP)

HDF Readability.png

OPeNDAP to the Rescue

Remote Access to Data--at a sub-file level

Data Access Protocol (DAP)

  • Presents data in (mostly) format-independent model
  • BUT some formats are more equal than others
    • NetCDF/CF1
    • HDF5/HDF-EOS5, HDF4/HDF-EOS2*
  • Tools can use coordinate information from more-equal formats

Examples

N.B.: Do not paste these OPeNDAP URLs into a browser!

In general, this will result in downloading the original file in its entirety. The URLs are meant to be pasted into an OPeNDAP client.

Panoply Examples

  • Tropospheric NO2 from OMI for July 7 2010: http://acdisc.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMNO2e.003//2010/OMI-Aura_L3-OMNO2e_2010m0707_v003-2010m0708t161138.he5

(Does not work in GrADS)

Ozone algorithm comparison

  • http://acdisc.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMTO3e.003/2010/OMI-Aura_L3-OMTO3e_2010m0101_v003-2010m0108t061829.he5
  • http://acdisc.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMTO3d.003/2010/OMI-Aura_L3-OMTO3d_2010m0101_v003-2010m0106t053032.he5

CO2 comparison

  • June 2010: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2010/AIRS.2010.06.01.L3.CO2Std030.v5.6.1.85.X10189104304.hdf
  • June 2003: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.06.01.L3.CO2Std030.v5.4.12.67.X09261104526.hdf

IDV Examples

Carbon Monoxide Plume from California wildfires, seen in AIRS Daily Level 3 (AIRX3STD)

  • 29 Aug: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.08.29.L3.RetStd001.v5.2.2.0.G09243131454.hdf
  • 30 Aug: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.08.30.L3.RetStd001.v5.2.2.0.G09244141238.hdf
  • 31 Aug: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.08.31.L3.RetStd001.v5.2.2.0.G09245132524.hdf
  • 01 Sep: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.09.01.L3.RetStd001.v5.2.2.0.G09246131354.hdf
  • 02 Sep: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.09.02.L3.RetStd001.v5.2.2.0.G09247131733.hdf
  • 03 Sep: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.09.03.L3.RetStd001.v5.2.2.0.G09248131356.hdf

Hint: Use ascending:CO_Total_Column_A, set range to 1E18 to 2.3E18.

Level 2 AIRS (AIRX2RET) for 01 Sep:

  • http://airspar1u.ecs.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.005//2009/244/AIRS.2009.09.01.202.L2.RetStd.v5.2.2.0.G09245202547.hdf
  • http://airspar1u.ecs.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.005//2009/244/AIRS.2009.09.01.092.L2.RetStd.v5.2.2.0.G09245130401.hdf
  • http://airspar1u.ecs.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.005//2009/244/AIRS.2009.09.01.091.L2.RetStd.v5.2.2.0.G09245130312.hdf

(The seam between granules 91 and 92 (latter two) is because it plots only half the pixel at the edges.)

Ferret/GrADS Example

CO2 and the Jetstream, based on Chahine et al., 2008, Fig.3 Links:

  • Monthly CO2 from AIRS for July 2003: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.07.01.L3.CO2Std031.v5.4.12.67.X09261105833.hdf ]
  • Geopotential height from NCEP Reanalysis]: 500 mbar = level 12, July 2003 = time 667: http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/pressure/hgt.mon.mean.nc

Ferret Commands:

set region/x=0:360
set data "http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.07.01.L3.CO2Std031.v5.4.12.67.X09261105833.hdf"
shade/lev=(.000364,.000382,.000001) MOLE_FRACTION_OF_CARBON_DIOXIDE_IN_FREE_TROPOSPHERE
set data "http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/pressure/hgt.mon.mean.nc"
contour/k=12/L=667/overlay HGT
go land

GrADS Commands

sdfopen http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.07.01.L3.CO2Std031.v5.4.12.67.X09261105833.hdf
q file
set gxout shaded
set clevs 0.000362 .000364 .000366 .000368 .000370 .000372 .000374 .000376 .000378 .000380
set csmooth on
d mole_fraction_o
run cbarm.gs
sdfopen http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/pressure/hgt.mon.mean.nc 
q file 2
set gxout contour
set cint 100
d hgt.2(t=667,z=6)

Where To Get Tools

Acknowledgments

  • Kent Yang, Joe Lee and the folks at The HDF Group for developing the enhanced HDF handlers
  • OPeNDAP.org for developing the OPeNDAP protocol and server
  • Tool developers at Goddard Institute for Space Studies (Panoply), Unidata (IDV), U. Wisconsin (McIDAS-V), PMEL (Ferret) and IGES/COLA (GrADS)
  • Fan Fang at GES DISC for deploying and testing the enhanced HDF handlers