Making Science Data Easier to Use with OPeNDAP

From Earth Science Information Partners (ESIP)
Revision as of 06:19, July 20, 2010 by Clynnes (talk | contribs) (→‎Ferret/GrADS Example)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Using Earth Science Data: Why So Difficult?

Complicated data require complicated format

  • Network Common Data Form (netCDF)
  • Hierarchical Data Format (HDF)

Complicated formats require good tools

Tools, Tools, Tools

OPeNDAP Tools.png

EOS HDF Data Readability (without OPeNDAP)

HDF Readability.png

OPeNDAP to the Rescue

Remote Access to Data--at a sub-file level

Data Access Protocol (DAP)

  • Presents data in (mostly) format-independent model
  • BUT some formats are more equal than others
    • NetCDF/CF1
    • HDF5/HDF-EOS5, HDF4/HDF-EOS2*
  • Tools can use coordinate information from more-equal formats

Examples

N.B.: Do not paste these OPeNDAP URLs into a browser!

In general, this will result in downloading the original file in its entirety. The URLs are meant to be pasted into an OPeNDAP client.

Panoply Examples

  • Tropospheric NO2 from OMI for July 7 2010: http://acdisc.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMNO2e.003//2010/OMI-Aura_L3-OMNO2e_2010m0707_v003-2010m0708t161138.he5

(Does not work in GrADS)

Ozone algorithm comparison

  • http://acdisc.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMTO3e.003/2010/OMI-Aura_L3-OMTO3e_2010m0101_v003-2010m0108t061829.he5
  • http://acdisc.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMTO3d.003/2010/OMI-Aura_L3-OMTO3d_2010m0101_v003-2010m0106t053032.he5

CO2 comparison

  • June 2010: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2010/AIRS.2010.06.01.L3.CO2Std030.v5.6.1.85.X10189104304.hdf
  • June 2003: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.06.01.L3.CO2Std030.v5.4.12.67.X09261104526.hdf

IDV Examples

Carbon Monoxide Plume from California wildfires, seen in AIRS Daily Level 3 (AIRX3STD)

  • 29 Aug: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.08.29.L3.RetStd001.v5.2.2.0.G09243131454.hdf
  • 30 Aug: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.08.30.L3.RetStd001.v5.2.2.0.G09244141238.hdf
  • 31 Aug: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.08.31.L3.RetStd001.v5.2.2.0.G09245132524.hdf
  • 01 Sep: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.09.01.L3.RetStd001.v5.2.2.0.G09246131354.hdf
  • 02 Sep: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.09.02.L3.RetStd001.v5.2.2.0.G09247131733.hdf
  • 03 Sep: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.09.03.L3.RetStd001.v5.2.2.0.G09248131356.hdf

Hint: Use ascending:CO_Total_Column_A, set range to 1E18 to 2.3E18.

Level 2 AIRS (AIRX2RET) for 01 Sep:

  • http://airspar1u.ecs.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.005//2009/244/AIRS.2009.09.01.202.L2.RetStd.v5.2.2.0.G09245202547.hdf
  • http://airspar1u.ecs.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.005//2009/244/AIRS.2009.09.01.092.L2.RetStd.v5.2.2.0.G09245130401.hdf
  • http://airspar1u.ecs.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.005//2009/244/AIRS.2009.09.01.091.L2.RetStd.v5.2.2.0.G09245130312.hdf

(The seam between granules 91 and 92 (latter two) is because it plots only half the pixel at the edges.)

Ferret/GrADS Example

CO2 and the Jetstream, based on Chahine et al., 2008, Fig.3 Links:

  • Monthly CO2 from AIRS for July 2003: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.07.01.L3.CO2Std031.v5.4.12.67.X09261105833.hdf
  • Geopotential height from NCEP Reanalysis]: 500 mbar = level 12, July 2003 = time 667: http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/pressure/hgt.mon.mean.nc

Ferret Commands:

set region/x=0:360
set data "http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.07.01.L3.CO2Std031.v5.4.12.67.X09261105833.hdf"
shade/lev=(.000364,.000382,.000001) MOLE_FRACTION_OF_CARBON_DIOXIDE_IN_FREE_TROPOSPHERE
set data "http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/pressure/hgt.mon.mean.nc"
contour/k=12/L=667/overlay HGT
go land

GrADS Commands

sdfopen http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.07.01.L3.CO2Std031.v5.4.12.67.X09261105833.hdf
q file
set gxout shaded
set clevs 0.000362 .000364 .000366 .000368 .000370 .000372 .000374 .000376 .000378 .000380
set csmooth on
d mole_fraction_o
run cbarm.gs
sdfopen http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/pressure/hgt.mon.mean.nc 
q file 2
set gxout contour
set cint 100
d hgt.2(t=667,z=6)

Where To Get Tools

Acknowledgments

  • Kent Yang, Joe Lee and the folks at The HDF Group for developing the enhanced HDF handlers
  • OPeNDAP.org for developing the OPeNDAP protocol and server
  • Tool developers at Goddard Institute for Space Studies (Panoply), Unidata (IDV), U. Wisconsin (McIDAS-V), Pacific Marine Environmental Laboratory (Ferret) and Institute of Global Environment and Society (GrADS)
  • Fan Fang at GES DISC for deploying and testing the enhanced HDF handlers