Difference between revisions of "Making Science Data Easier to Use with OPeNDAP"

From Earth Science Information Partners (ESIP)
 
(25 intermediate revisions by the same user not shown)
Line 1: Line 1:
'''N.B.: Do not paste these OPeNDAP URLs into a browser!''' In general, this will result in downloading the original file in its entirety.  The URLs are meant to be pasted into an OPeNDAP client.
+
=Using Earth Science Data: Why So Difficult? =
  
* Tropospheric NO2 from OMI for July 7 2010: <nowiki>http://acdisc.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMNO2e.003//2010/OMI-Aura_L3-OMNO2e_2010m0707_v003-2010m0708t161138.he5</nowiki>(Does not work in GrADS)
+
<div style="background:lightgreen">
 +
===Complicated data require complicated format===
 +
<big>
 +
*Network Common Data Form (netCDF)
 +
*Hierarchical Data Format (HDF)
 +
</big>
 +
 
 +
===Complicated formats require good tools===
 +
</div>
 +
 
 +
=Tools, Tools, Tools =
 +
[[Image:OPeNDAP_Tools.png]]
 +
= EOS HDF Data Readability (without OPeNDAP) =
 +
[[Image:HDF_Readability.png]]
 +
<div style="background:lightgreen">
 +
= [http://www.opendap.org OPeNDAP] to the Rescue =
 +
=== Remote Access to Data--at a sub-file level ===
 +
=== Data Access Protocol (DAP) ===
 +
<big>
 +
* Presents data in (mostly) format-independent model
 +
* BUT some formats are more equal than others
 +
** NetCDF/CF1
 +
** HDF5/HDF-EOS5, HDF4/HDF-EOS2*
 +
* Tools can use coordinate information from more-equal formats
 +
</big>
 +
</div>
 +
 
 +
= Examples =
 +
'''N.B.:  Do not paste these OPeNDAP URLs into a browser!'''
 +
 
 +
In general, this will result in downloading the original file in its entirety.  The URLs are meant to be pasted into an OPeNDAP client.
 +
 
 +
=Panoply Examples=
 +
* Tropospheric NO2 from OMI for July 7 2010: <nowiki>http://acdisc.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMNO2e.003//2010/OMI-Aura_L3-OMNO2e_2010m0707_v003-2010m0708t161138.he5</nowiki>
 +
(Does not work in GrADS)
 
=== Ozone algorithm comparison ===
 
=== Ozone algorithm comparison ===
 
* <nowiki>http://acdisc.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMTO3e.003/2010/OMI-Aura_L3-OMTO3e_2010m0101_v003-2010m0108t061829.he5</nowiki>
 
* <nowiki>http://acdisc.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMTO3e.003/2010/OMI-Aura_L3-OMTO3e_2010m0101_v003-2010m0108t061829.he5</nowiki>
Line 9: Line 43:
 
* June 2003: <nowiki>http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.06.01.L3.CO2Std030.v5.4.12.67.X09261104526.hdf</nowiki>
 
* June 2003: <nowiki>http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.06.01.L3.CO2Std030.v5.4.12.67.X09261104526.hdf</nowiki>
  
=== CO2 Chahine article ===
+
= IDV Examples =
 +
Carbon Monoxide Plume from California wildfires, seen in AIRS Daily Level 3 (AIRX3STD)
 +
* 29 Aug: <nowiki>http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.08.29.L3.RetStd001.v5.2.2.0.G09243131454.hdf</nowiki>
 +
* 30 Aug: <nowiki>http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.08.30.L3.RetStd001.v5.2.2.0.G09244141238.hdf</nowiki>
 +
* 31 Aug: <nowiki>http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.08.31.L3.RetStd001.v5.2.2.0.G09245132524.hdf</nowiki>
 +
* '''01 Sep: <nowiki>http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.09.01.L3.RetStd001.v5.2.2.0.G09246131354.hdf</nowiki>'''
 +
* 02 Sep: <nowiki>http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.09.02.L3.RetStd001.v5.2.2.0.G09247131733.hdf</nowiki>
 +
* 03 Sep: <nowiki>http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.09.03.L3.RetStd001.v5.2.2.0.G09248131356.hdf</nowiki>
 +
 
 +
Hint:  Use ascending:CO_Total_Column_A, set range to 1E18 to 2.3E18.
 +
 
 +
Level 2 AIRS (AIRX2RET) for 01 Sep:
 +
*<nowiki>http://airspar1u.ecs.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.005//2009/244/AIRS.2009.09.01.202.L2.RetStd.v5.2.2.0.G09245202547.hdf</nowiki>
 +
*<nowiki>http://airspar1u.ecs.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.005//2009/244/AIRS.2009.09.01.092.L2.RetStd.v5.2.2.0.G09245130401.hdf</nowiki>
 +
*<nowiki>http://airspar1u.ecs.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.005//2009/244/AIRS.2009.09.01.091.L2.RetStd.v5.2.2.0.G09245130312.hdf</nowiki>
 +
 
 +
(The seam between granules 91 and 92 (latter two) is because it plots only half the pixel at the edges.)
 +
 
 +
= Ferret/GrADS Example =
 +
CO2 and the Jetstream, based on [http://yly-mac.gps.caltech.edu/OCO/4raphael/Chahine_CO2_08%20copy.pdf Chahine et al., 2008, Fig.3]
 
Links:
 
Links:
* Monthly CO2 from AIRS for July 2003: <nowiki>http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.07.01.L3.CO2Std031.v5.4.12.67.X09261105833.hdf</nowiki> ]
+
* Monthly CO2 from AIRS for July 2003: <nowiki>http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.07.01.L3.CO2Std031.v5.4.12.67.X09261105833.hdf</nowiki>
 
* Geopotential height from NCEP Reanalysis]: 500 mbar = level 12, July 2003 = time 667: <nowiki>http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/pressure/hgt.mon.mean.nc</nowiki>  
 
* Geopotential height from NCEP Reanalysis]: 500 mbar = level 12, July 2003 = time 667: <nowiki>http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/pressure/hgt.mon.mean.nc</nowiki>  
 
Ferret Commands:
 
Ferret Commands:
 
  set region/x=0:360
 
  set region/x=0:360
  set data "http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.07.01.L3.CO2Std031.v5.4.12.67.X09261105833.hdf"
+
  set data "<nowiki>http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.07.01.L3.CO2Std031.v5.4.12.67.X09261105833.hdf</nowiki>"
 
  shade/lev=(.000364,.000382,.000001) MOLE_FRACTION_OF_CARBON_DIOXIDE_IN_FREE_TROPOSPHERE
 
  shade/lev=(.000364,.000382,.000001) MOLE_FRACTION_OF_CARBON_DIOXIDE_IN_FREE_TROPOSPHERE
  set data "http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/pressure/hgt.mon.mean.nc"
+
  set data "<nowiki>http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/pressure/hgt.mon.mean.nc</nowiki>"
 
  contour/k=12/L=667/overlay HGT
 
  contour/k=12/L=667/overlay HGT
 
  go land
 
  go land
 +
 +
GrADS Commands
 +
sdfopen http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.07.01.L3.CO2Std031.v5.4.12.67.X09261105833.hdf
 +
q file
 +
set gxout shaded
 +
set clevs 0.000362 .000364 .000366 .000368 .000370 .000372 .000374 .000376 .000378 .000380
 +
set csmooth on
 +
d mole_fraction_o
 +
run cbarm.gs
 +
sdfopen http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/pressure/hgt.mon.mean.nc
 +
q file 2
 +
set gxout contour
 +
set cint 100
 +
d hgt.2(t=667,z=6)
 +
 +
= Where To Get Tools =
 +
* Panoply: http://www.giss.nasa.gov/tools/panoply/
 +
* IDV (Integrated Data Viewer): http://www.unidata.ucar.edu/software/idv/ (Registration required)
 +
* McIDAS-V (IDV sibling): http://www.ssec.wisc.edu/mcidas/software/v/
 +
* Ferret: http://www.ferret.noaa.gov/Ferret/
 +
* GrADS (Gridded Analysis and Display System): http://www.iges.org/grads/
 +
 +
= Acknowledgments =
 +
* Kent Yang, Joe Lee and the folks at [http://www.hdfgroup.org The HDF Group] for developing the enhanced HDF handlers
 +
* [http://www.opendap.org OPeNDAP.org] for developing the OPeNDAP protocol and server
 +
* Tool developers at Goddard Institute for Space Studies (Panoply), Unidata (IDV), U. Wisconsin (McIDAS-V), Pacific Marine Environmental Laboratory (Ferret) and Institute of Global Environment and Society (GrADS)
 +
* Fan Fang at GES DISC for deploying and testing the enhanced HDF handlers

Latest revision as of 06:19, July 20, 2010

Using Earth Science Data: Why So Difficult?

Complicated data require complicated format

  • Network Common Data Form (netCDF)
  • Hierarchical Data Format (HDF)

Complicated formats require good tools

Tools, Tools, Tools

OPeNDAP Tools.png

EOS HDF Data Readability (without OPeNDAP)

HDF Readability.png

OPeNDAP to the Rescue

Remote Access to Data--at a sub-file level

Data Access Protocol (DAP)

  • Presents data in (mostly) format-independent model
  • BUT some formats are more equal than others
    • NetCDF/CF1
    • HDF5/HDF-EOS5, HDF4/HDF-EOS2*
  • Tools can use coordinate information from more-equal formats

Examples

N.B.: Do not paste these OPeNDAP URLs into a browser!

In general, this will result in downloading the original file in its entirety. The URLs are meant to be pasted into an OPeNDAP client.

Panoply Examples

  • Tropospheric NO2 from OMI for July 7 2010: http://acdisc.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMNO2e.003//2010/OMI-Aura_L3-OMNO2e_2010m0707_v003-2010m0708t161138.he5

(Does not work in GrADS)

Ozone algorithm comparison

  • http://acdisc.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMTO3e.003/2010/OMI-Aura_L3-OMTO3e_2010m0101_v003-2010m0108t061829.he5
  • http://acdisc.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMTO3d.003/2010/OMI-Aura_L3-OMTO3d_2010m0101_v003-2010m0106t053032.he5

CO2 comparison

  • June 2010: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2010/AIRS.2010.06.01.L3.CO2Std030.v5.6.1.85.X10189104304.hdf
  • June 2003: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.06.01.L3.CO2Std030.v5.4.12.67.X09261104526.hdf

IDV Examples

Carbon Monoxide Plume from California wildfires, seen in AIRS Daily Level 3 (AIRX3STD)

  • 29 Aug: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.08.29.L3.RetStd001.v5.2.2.0.G09243131454.hdf
  • 30 Aug: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.08.30.L3.RetStd001.v5.2.2.0.G09244141238.hdf
  • 31 Aug: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.08.31.L3.RetStd001.v5.2.2.0.G09245132524.hdf
  • 01 Sep: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.09.01.L3.RetStd001.v5.2.2.0.G09246131354.hdf
  • 02 Sep: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.09.02.L3.RetStd001.v5.2.2.0.G09247131733.hdf
  • 03 Sep: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.09.03.L3.RetStd001.v5.2.2.0.G09248131356.hdf

Hint: Use ascending:CO_Total_Column_A, set range to 1E18 to 2.3E18.

Level 2 AIRS (AIRX2RET) for 01 Sep:

  • http://airspar1u.ecs.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.005//2009/244/AIRS.2009.09.01.202.L2.RetStd.v5.2.2.0.G09245202547.hdf
  • http://airspar1u.ecs.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.005//2009/244/AIRS.2009.09.01.092.L2.RetStd.v5.2.2.0.G09245130401.hdf
  • http://airspar1u.ecs.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.005//2009/244/AIRS.2009.09.01.091.L2.RetStd.v5.2.2.0.G09245130312.hdf

(The seam between granules 91 and 92 (latter two) is because it plots only half the pixel at the edges.)

Ferret/GrADS Example

CO2 and the Jetstream, based on Chahine et al., 2008, Fig.3 Links:

  • Monthly CO2 from AIRS for July 2003: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.07.01.L3.CO2Std031.v5.4.12.67.X09261105833.hdf
  • Geopotential height from NCEP Reanalysis]: 500 mbar = level 12, July 2003 = time 667: http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/pressure/hgt.mon.mean.nc

Ferret Commands:

set region/x=0:360
set data "http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.07.01.L3.CO2Std031.v5.4.12.67.X09261105833.hdf"
shade/lev=(.000364,.000382,.000001) MOLE_FRACTION_OF_CARBON_DIOXIDE_IN_FREE_TROPOSPHERE
set data "http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/pressure/hgt.mon.mean.nc"
contour/k=12/L=667/overlay HGT
go land

GrADS Commands

sdfopen http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.07.01.L3.CO2Std031.v5.4.12.67.X09261105833.hdf
q file
set gxout shaded
set clevs 0.000362 .000364 .000366 .000368 .000370 .000372 .000374 .000376 .000378 .000380
set csmooth on
d mole_fraction_o
run cbarm.gs
sdfopen http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/pressure/hgt.mon.mean.nc 
q file 2
set gxout contour
set cint 100
d hgt.2(t=667,z=6)

Where To Get Tools

Acknowledgments

  • Kent Yang, Joe Lee and the folks at The HDF Group for developing the enhanced HDF handlers
  • OPeNDAP.org for developing the OPeNDAP protocol and server
  • Tool developers at Goddard Institute for Space Studies (Panoply), Unidata (IDV), U. Wisconsin (McIDAS-V), Pacific Marine Environmental Laboratory (Ferret) and Institute of Global Environment and Society (GrADS)
  • Fan Fang at GES DISC for deploying and testing the enhanced HDF handlers