Difference between revisions of "Earth Science Data: Why So Difficult?"

From Earth Science Information Partners (ESIP)
 
(10 intermediate revisions by the same user not shown)
Line 7: Line 7:
 
Interested in learning more about the technologies mentioned in this presentation?  Here are some places to go and things to do:
 
Interested in learning more about the technologies mentioned in this presentation?  Here are some places to go and things to do:
 
=== Check out the [[Federated_Search|ESIP Federated Search cluster]] for distributed search on a shoestring ===
 
=== Check out the [[Federated_Search|ESIP Federated Search cluster]] for distributed search on a shoestring ===
=== Explore the wonders of OPeNDAP ===
+
=== Explore the wonders of [http://opendap.org/ OPeNDAP] ===
* Get ASCII subsets of the data! Go to [http://acdisc.sci.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMAERUVd.003//2010/OMI-Aura_L3-OMAERUVd_2010m0131_v003-2010m0202t014811.he5.html http://acdisc.sci.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMAERUVd.003//2010/OMI-Aura_L3-OMAERUVd_2010m0131_v003-2010m0202t014811.he5.html], select the variables of interest and click ascii.
+
* Get ASCII subsets of the data! Go to [http://acdisc.sci.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMAERUVd.003//2010/OMI-Aura_L3-OMAERUVd_2010m0131_v003-2010m0202t014811.he5.html http://acdisc.sci.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMAERUVd.003//2010/OMI-Aura_L3-OMAERUVd_2010m0131_v003-2010m0202t014811.he5.html], select the variables of interest and click Get ASCII.
 
* View Level 3 gridded data in Panoply!
 
* View Level 3 gridded data in Panoply!
 
** First, [http://www.giss.nasa.gov/tools/panoply/ download Panoply], an easy to use Java-based viewer for netCDF, HDF and GRIB data
 
** First, [http://www.giss.nasa.gov/tools/panoply/ download Panoply], an easy to use Java-based viewer for netCDF, HDF and GRIB data
** Open up Panoply, go to File-> Open Remote File, and paste in the following URL (N.B.: without .html extension):  'http://acdisc.sci.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMAERUVd.003//2010/OMI-Aura_L3-OMAERUVd_2010m0131_v003-2010m0202t014811.he5'
+
** Open up Panoply, go to File-> Open Remote File, and paste in the following URL (N.B.: without .html extension):  <nowiki>http://acdisc.sci.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMDOAO3e.003/2004/OMI-Aura_L3-OMDOAO3e_2004m1001_v003-2008m0222t095640.he5</nowiki>. (Hint:  Try viewing ColumnAmountO3 with a Map Projection of Azimuthal Equidistant Centered on 0 E, -90 N, for a good look at the ozone hole.)
 +
** Here's a URL that shows a dust storm in the eastern Mediterranean region on 24 Feb 2007: <nowiki>http://acdisc.sci.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMAERUVd.003/2007/OMI-Aura_L3-OMAERUVd_2007m0224_v003-2009m0714t140856.he5</nowiki>.  Plot FinalAerosolExtOpticalDepth500 (Aerosol Extinction Optical Depth at 500 nm).
 +
* Level 2 swath data coming soon to OPeNDAP!!  (Allows IDV to display swaths.)
 +
 
 +
=== See what you can do with netCDF/CF-1 formatted data ===
 +
* Convert AIRS L2 Swath to netCDF/CF-1 and view in IDV!
 +
** First, [http://www.unidata.ucar.edu/software/idv/ download IDV]
 +
** Then, [http://aurapar2u.ecs.nasa.gov/airspar1/daac-bin/OTF/HTTP_services.cgi?FILENAME=%2Fdata%2Fs4pa%2FAqua_AIRS_Level2%2FAIRX2RET.005%2F2007%2F238%2FAIRS.2007.08.26.128.L2.RetStd.v5.0.14.0.G07240214754.hdf&LABEL=AIRS.2007.08.26.128.L2.RetStd.v5.0.14.0.G07240214754.nc&SHORTNAME=AIRX2RET&SERVICE=NetCDF&VERSION=1.02 acquire the AIRS data in netCDF form] (this URL will execute an on-the-fly reformatting.)
 +
** Then use IDV's Data Chooser to read the data in.
 +
** Select a parameter under 2D-Grids in the Field Selector, and "Color-Shaded Plan View" under Displays.  Hint:  the CO_total_column parameter under the 2D grids shows carbon monoxide emanating from fires in Greece in Aug 2007.
 +
=== Take a walk down memory lane with [http://www.esdswg.org/spg/docindexfolder/heritage/eosdis-v0 EOSDIS Version 0] ===
 +
Yes, the user interface could have used some work, and it could have been faster, and the protocol was a bit clunky, and...  But perhaps ahead of its time for federated search frameworks...
 +
 
 +
[[User:Clynnes|Clynnes]] 18:41, 3 February 2010 (EST)

Latest revision as of 17:59, February 3, 2010

Abstract

Assembling a multi-sensor set of data for, say, a volcanic eruption means days of wandering cyberspace to find data and pry it loose from the custom search interfaces that jealously guard it. Following the Search is a slog of weeks reading format and QA documents, then cobbling together code to beat the data into a form that can be used safely in analysis tools. Why so difficult?$%#@^%!???

Above rant to be followed by a (mostly) calm and rational discussion of ways toward a solution.

Presentation (PDF)

Further Exploration

Interested in learning more about the technologies mentioned in this presentation? Here are some places to go and things to do:

Check out the ESIP Federated Search cluster for distributed search on a shoestring

Explore the wonders of OPeNDAP

  • Get ASCII subsets of the data! Go to http://acdisc.sci.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMAERUVd.003//2010/OMI-Aura_L3-OMAERUVd_2010m0131_v003-2010m0202t014811.he5.html, select the variables of interest and click Get ASCII.
  • View Level 3 gridded data in Panoply!
    • First, download Panoply, an easy to use Java-based viewer for netCDF, HDF and GRIB data
    • Open up Panoply, go to File-> Open Remote File, and paste in the following URL (N.B.: without .html extension): http://acdisc.sci.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMDOAO3e.003/2004/OMI-Aura_L3-OMDOAO3e_2004m1001_v003-2008m0222t095640.he5. (Hint: Try viewing ColumnAmountO3 with a Map Projection of Azimuthal Equidistant Centered on 0 E, -90 N, for a good look at the ozone hole.)
    • Here's a URL that shows a dust storm in the eastern Mediterranean region on 24 Feb 2007: http://acdisc.sci.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMAERUVd.003/2007/OMI-Aura_L3-OMAERUVd_2007m0224_v003-2009m0714t140856.he5. Plot FinalAerosolExtOpticalDepth500 (Aerosol Extinction Optical Depth at 500 nm).
  • Level 2 swath data coming soon to OPeNDAP!! (Allows IDV to display swaths.)

See what you can do with netCDF/CF-1 formatted data

  • Convert AIRS L2 Swath to netCDF/CF-1 and view in IDV!
    • First, download IDV
    • Then, acquire the AIRS data in netCDF form (this URL will execute an on-the-fly reformatting.)
    • Then use IDV's Data Chooser to read the data in.
    • Select a parameter under 2D-Grids in the Field Selector, and "Color-Shaded Plan View" under Displays. Hint: the CO_total_column parameter under the 2D grids shows carbon monoxide emanating from fires in Greece in Aug 2007.

Take a walk down memory lane with EOSDIS Version 0

Yes, the user interface could have used some work, and it could have been faster, and the protocol was a bit clunky, and... But perhaps ahead of its time for federated search frameworks...

Clynnes 18:41, 3 February 2010 (EST)