Making Science Data Easier to Use with OPeNDAP

From Federation of Earth Science Information Partners
(Difference between revisions)
Jump to: navigation, search
(Complicated data require complicated format)
(Ferret/GrADS Example)
 
(4 intermediate revisions by one user not shown)
Line 15: Line 15:
 
= EOS HDF Data Readability (without OPeNDAP) =
 
= EOS HDF Data Readability (without OPeNDAP) =
 
[[Image:HDF_Readability.png]]
 
[[Image:HDF_Readability.png]]
 +
<div style="background:lightgreen">
 +
= [http://www.opendap.org OPeNDAP] to the Rescue =
 +
=== Remote Access to Data--at a sub-file level ===
 +
=== Data Access Protocol (DAP) ===
 +
<big>
 +
* Presents data in (mostly) format-independent model
 +
* BUT some formats are more equal than others
 +
** NetCDF/CF1
 +
** HDF5/HDF-EOS5, HDF4/HDF-EOS2*
 +
* Tools can use coordinate information from more-equal formats
 +
</big>
 +
</div>
  
 
= Examples =
 
= Examples =
Line 22: Line 34:
  
 
=Panoply Examples=
 
=Panoply Examples=
* Tropospheric NO2 from OMI for July 7 2010: <nowiki>http://acdisc.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMNO2e.003//2010/OMI-Aura_L3-OMNO2e_2010m0707_v003-2010m0708t161138.he5</nowiki>(Does not work in GrADS)
+
* Tropospheric NO2 from OMI for July 7 2010: <nowiki>http://acdisc.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMNO2e.003//2010/OMI-Aura_L3-OMNO2e_2010m0707_v003-2010m0708t161138.he5</nowiki>
 +
(Does not work in GrADS)
 
=== Ozone algorithm comparison ===
 
=== Ozone algorithm comparison ===
 
* <nowiki>http://acdisc.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMTO3e.003/2010/OMI-Aura_L3-OMTO3e_2010m0101_v003-2010m0108t061829.he5</nowiki>
 
* <nowiki>http://acdisc.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMTO3e.003/2010/OMI-Aura_L3-OMTO3e_2010m0101_v003-2010m0108t061829.he5</nowiki>
Line 29: Line 42:
 
* June 2010: <nowiki>http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2010/AIRS.2010.06.01.L3.CO2Std030.v5.6.1.85.X10189104304.hdf</nowiki>
 
* June 2010: <nowiki>http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2010/AIRS.2010.06.01.L3.CO2Std030.v5.6.1.85.X10189104304.hdf</nowiki>
 
* June 2003: <nowiki>http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.06.01.L3.CO2Std030.v5.4.12.67.X09261104526.hdf</nowiki>
 
* June 2003: <nowiki>http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.06.01.L3.CO2Std030.v5.4.12.67.X09261104526.hdf</nowiki>
 +
 
= IDV Examples =
 
= IDV Examples =
 
Carbon Monoxide Plume from California wildfires, seen in AIRS Daily Level 3 (AIRX3STD)
 
Carbon Monoxide Plume from California wildfires, seen in AIRS Daily Level 3 (AIRX3STD)
Line 50: Line 64:
 
CO2 and the Jetstream, based on [http://yly-mac.gps.caltech.edu/OCO/4raphael/Chahine_CO2_08%20copy.pdf Chahine et al., 2008, Fig.3]
 
CO2 and the Jetstream, based on [http://yly-mac.gps.caltech.edu/OCO/4raphael/Chahine_CO2_08%20copy.pdf Chahine et al., 2008, Fig.3]
 
Links:
 
Links:
* Monthly CO2 from AIRS for July 2003: <nowiki>http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.07.01.L3.CO2Std031.v5.4.12.67.X09261105833.hdf</nowiki> ]
+
* Monthly CO2 from AIRS for July 2003: <nowiki>http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.07.01.L3.CO2Std031.v5.4.12.67.X09261105833.hdf</nowiki>
 
* Geopotential height from NCEP Reanalysis]: 500 mbar = level 12, July 2003 = time 667: <nowiki>http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/pressure/hgt.mon.mean.nc</nowiki>  
 
* Geopotential height from NCEP Reanalysis]: 500 mbar = level 12, July 2003 = time 667: <nowiki>http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/pressure/hgt.mon.mean.nc</nowiki>  
 
Ferret Commands:
 
Ferret Commands:
Line 84: Line 98:
 
* Kent Yang, Joe Lee and the folks at [http://www.hdfgroup.org The HDF Group] for developing the enhanced HDF handlers
 
* Kent Yang, Joe Lee and the folks at [http://www.hdfgroup.org The HDF Group] for developing the enhanced HDF handlers
 
* [http://www.opendap.org OPeNDAP.org] for developing the OPeNDAP protocol and server
 
* [http://www.opendap.org OPeNDAP.org] for developing the OPeNDAP protocol and server
* Tool developers at Goddard Institute for Space Studies (Panoply), Unidata (IDV), U. Wisconsin (McIDAS-V), PMEL (Ferret) and IGES/COLA (GrADS)
+
* Tool developers at Goddard Institute for Space Studies (Panoply), Unidata (IDV), U. Wisconsin (McIDAS-V), Pacific Marine Environmental Laboratory (Ferret) and Institute of Global Environment and Society (GrADS)
 
* Fan Fang at GES DISC for deploying and testing the enhanced HDF handlers
 
* Fan Fang at GES DISC for deploying and testing the enhanced HDF handlers

Latest revision as of 05:19, 20 July 2010

Contents

[edit] Using Earth Science Data: Why So Difficult?

[edit] Complicated data require complicated format

  • Network Common Data Form (netCDF)
  • Hierarchical Data Format (HDF)

[edit] Complicated formats require good tools

[edit] Tools, Tools, Tools

OPeNDAP Tools.png

[edit] EOS HDF Data Readability (without OPeNDAP)

HDF Readability.png

[edit] OPeNDAP to the Rescue

[edit] Remote Access to Data--at a sub-file level

[edit] Data Access Protocol (DAP)

  • Presents data in (mostly) format-independent model
  • BUT some formats are more equal than others
    • NetCDF/CF1
    • HDF5/HDF-EOS5, HDF4/HDF-EOS2*
  • Tools can use coordinate information from more-equal formats

[edit] Examples

N.B.: Do not paste these OPeNDAP URLs into a browser!

In general, this will result in downloading the original file in its entirety. The URLs are meant to be pasted into an OPeNDAP client.

[edit] Panoply Examples

  • Tropospheric NO2 from OMI for July 7 2010: http://acdisc.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMNO2e.003//2010/OMI-Aura_L3-OMNO2e_2010m0707_v003-2010m0708t161138.he5

(Does not work in GrADS)

[edit] Ozone algorithm comparison

  • http://acdisc.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMTO3e.003/2010/OMI-Aura_L3-OMTO3e_2010m0101_v003-2010m0108t061829.he5
  • http://acdisc.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMTO3d.003/2010/OMI-Aura_L3-OMTO3d_2010m0101_v003-2010m0106t053032.he5

[edit] CO2 comparison

  • June 2010: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2010/AIRS.2010.06.01.L3.CO2Std030.v5.6.1.85.X10189104304.hdf
  • June 2003: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.06.01.L3.CO2Std030.v5.4.12.67.X09261104526.hdf

[edit] IDV Examples

Carbon Monoxide Plume from California wildfires, seen in AIRS Daily Level 3 (AIRX3STD)

  • 29 Aug: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.08.29.L3.RetStd001.v5.2.2.0.G09243131454.hdf
  • 30 Aug: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.08.30.L3.RetStd001.v5.2.2.0.G09244141238.hdf
  • 31 Aug: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.08.31.L3.RetStd001.v5.2.2.0.G09245132524.hdf
  • 01 Sep: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.09.01.L3.RetStd001.v5.2.2.0.G09246131354.hdf
  • 02 Sep: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.09.02.L3.RetStd001.v5.2.2.0.G09247131733.hdf
  • 03 Sep: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3STD.005/2009/AIRS.2009.09.03.L3.RetStd001.v5.2.2.0.G09248131356.hdf

Hint: Use ascending:CO_Total_Column_A, set range to 1E18 to 2.3E18.

Level 2 AIRS (AIRX2RET) for 01 Sep:

  • http://airspar1u.ecs.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.005//2009/244/AIRS.2009.09.01.202.L2.RetStd.v5.2.2.0.G09245202547.hdf
  • http://airspar1u.ecs.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.005//2009/244/AIRS.2009.09.01.092.L2.RetStd.v5.2.2.0.G09245130401.hdf
  • http://airspar1u.ecs.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.005//2009/244/AIRS.2009.09.01.091.L2.RetStd.v5.2.2.0.G09245130312.hdf

(The seam between granules 91 and 92 (latter two) is because it plots only half the pixel at the edges.)

[edit] Ferret/GrADS Example

CO2 and the Jetstream, based on Chahine et al., 2008, Fig.3 Links:

  • Monthly CO2 from AIRS for July 2003: http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.07.01.L3.CO2Std031.v5.4.12.67.X09261105833.hdf
  • Geopotential height from NCEP Reanalysis]: 500 mbar = level 12, July 2003 = time 667: http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/pressure/hgt.mon.mean.nc

Ferret Commands:

set region/x=0:360
set data "http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.07.01.L3.CO2Std031.v5.4.12.67.X09261105833.hdf"
shade/lev=(.000364,.000382,.000001) MOLE_FRACTION_OF_CARBON_DIOXIDE_IN_FREE_TROPOSPHERE
set data "http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/pressure/hgt.mon.mean.nc"
contour/k=12/L=667/overlay HGT
go land

GrADS Commands

sdfopen http://acdisc.gsfc.nasa.gov/opendap/Aqua_AIRS_Level3/AIRX3C2M.005/2003/AIRS.2003.07.01.L3.CO2Std031.v5.4.12.67.X09261105833.hdf
q file
set gxout shaded
set clevs 0.000362 .000364 .000366 .000368 .000370 .000372 .000374 .000376 .000378 .000380
set csmooth on
d mole_fraction_o
run cbarm.gs
sdfopen http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/pressure/hgt.mon.mean.nc 
q file 2
set gxout contour
set cint 100
d hgt.2(t=667,z=6)

[edit] Where To Get Tools

[edit] Acknowledgments

  • Kent Yang, Joe Lee and the folks at The HDF Group for developing the enhanced HDF handlers
  • OPeNDAP.org for developing the OPeNDAP protocol and server
  • Tool developers at Goddard Institute for Space Studies (Panoply), Unidata (IDV), U. Wisconsin (McIDAS-V), Pacific Marine Environmental Laboratory (Ferret) and Institute of Global Environment and Society (GrADS)
  • Fan Fang at GES DISC for deploying and testing the enhanced HDF handlers
Personal tools
Namespaces

Variants
Actions
Navigation
Toolbox