Difference between revisions of "Earth Science Data: Why So Difficult?"

From Earth Science Information Partners (ESIP)
(New page: ==Abstract== Assembling a multi-sensor set of data for, say, a volcanic eruption means days of wandering cyberspace to find data and pry it loose from the custom search interfaces that jea...)
 
 
(13 intermediate revisions by the same user not shown)
Line 3: Line 3:
  
 
Above rant to be followed by a (mostly) calm and rational discussion of ways toward a solution.
 
Above rant to be followed by a (mostly) calm and rational discussion of ways toward a solution.
==[[EarthScienceData_WhySoDifficult.pdf| Presentation (PDF)]]==
+
==[[Media:EarthScienceData_WhySoDifficult.pdf| Presentation (PDF)]]==
 +
==Further Exploration==
 +
Interested in learning more about the technologies mentioned in this presentation?  Here are some places to go and things to do:
 +
=== Check out the [[Federated_Search|ESIP Federated Search cluster]] for distributed search on a shoestring ===
 +
=== Explore the wonders of [http://opendap.org/ OPeNDAP] ===
 +
* Get ASCII subsets of the data! Go to [http://acdisc.sci.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMAERUVd.003//2010/OMI-Aura_L3-OMAERUVd_2010m0131_v003-2010m0202t014811.he5.html http://acdisc.sci.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMAERUVd.003//2010/OMI-Aura_L3-OMAERUVd_2010m0131_v003-2010m0202t014811.he5.html], select the variables of interest and click Get ASCII.
 +
* View Level 3 gridded data in Panoply!
 +
** First, [http://www.giss.nasa.gov/tools/panoply/ download Panoply], an easy to use Java-based viewer for netCDF, HDF and GRIB data
 +
** Open up Panoply, go to File-> Open Remote File, and paste in the following URL (N.B.: without .html extension):  <nowiki>http://acdisc.sci.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMDOAO3e.003/2004/OMI-Aura_L3-OMDOAO3e_2004m1001_v003-2008m0222t095640.he5</nowiki>. (Hint:  Try viewing ColumnAmountO3 with a Map Projection of Azimuthal Equidistant Centered on 0 E, -90 N, for a good look at the ozone hole.)
 +
** Here's a URL that shows a dust storm in the eastern Mediterranean region on 24 Feb 2007: <nowiki>http://acdisc.sci.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMAERUVd.003/2007/OMI-Aura_L3-OMAERUVd_2007m0224_v003-2009m0714t140856.he5</nowiki>.  Plot FinalAerosolExtOpticalDepth500 (Aerosol Extinction Optical Depth at 500 nm).
 +
* Level 2 swath data coming soon to OPeNDAP!!  (Allows IDV to display swaths.)
 +
 
 +
=== See what you can do with netCDF/CF-1 formatted data ===
 +
* Convert AIRS L2 Swath to netCDF/CF-1 and view in IDV!
 +
** First, [http://www.unidata.ucar.edu/software/idv/ download IDV]
 +
** Then, [http://aurapar2u.ecs.nasa.gov/airspar1/daac-bin/OTF/HTTP_services.cgi?FILENAME=%2Fdata%2Fs4pa%2FAqua_AIRS_Level2%2FAIRX2RET.005%2F2007%2F238%2FAIRS.2007.08.26.128.L2.RetStd.v5.0.14.0.G07240214754.hdf&LABEL=AIRS.2007.08.26.128.L2.RetStd.v5.0.14.0.G07240214754.nc&SHORTNAME=AIRX2RET&SERVICE=NetCDF&VERSION=1.02 acquire the AIRS data in netCDF form] (this URL will execute an on-the-fly reformatting.)
 +
** Then use IDV's Data Chooser to read the data in.
 +
** Select a parameter under 2D-Grids in the Field Selector, and "Color-Shaded Plan View" under Displays.  Hint:  the CO_total_column parameter under the 2D grids shows carbon monoxide emanating from fires in Greece in Aug 2007.
 +
=== Take a walk down memory lane with [http://www.esdswg.org/spg/docindexfolder/heritage/eosdis-v0 EOSDIS Version 0] ===
 +
Yes, the user interface could have used some work, and it could have been faster, and the protocol was a bit clunky, and...  But perhaps ahead of its time for federated search frameworks...
 +
 
 +
[[User:Clynnes|Clynnes]] 18:41, 3 February 2010 (EST)

Latest revision as of 17:59, February 3, 2010

Abstract

Assembling a multi-sensor set of data for, say, a volcanic eruption means days of wandering cyberspace to find data and pry it loose from the custom search interfaces that jealously guard it. Following the Search is a slog of weeks reading format and QA documents, then cobbling together code to beat the data into a form that can be used safely in analysis tools. Why so difficult?$%#@^%!???

Above rant to be followed by a (mostly) calm and rational discussion of ways toward a solution.

Presentation (PDF)

Further Exploration

Interested in learning more about the technologies mentioned in this presentation? Here are some places to go and things to do:

Check out the ESIP Federated Search cluster for distributed search on a shoestring

Explore the wonders of OPeNDAP

  • Get ASCII subsets of the data! Go to http://acdisc.sci.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMAERUVd.003//2010/OMI-Aura_L3-OMAERUVd_2010m0131_v003-2010m0202t014811.he5.html, select the variables of interest and click Get ASCII.
  • View Level 3 gridded data in Panoply!
    • First, download Panoply, an easy to use Java-based viewer for netCDF, HDF and GRIB data
    • Open up Panoply, go to File-> Open Remote File, and paste in the following URL (N.B.: without .html extension): http://acdisc.sci.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMDOAO3e.003/2004/OMI-Aura_L3-OMDOAO3e_2004m1001_v003-2008m0222t095640.he5. (Hint: Try viewing ColumnAmountO3 with a Map Projection of Azimuthal Equidistant Centered on 0 E, -90 N, for a good look at the ozone hole.)
    • Here's a URL that shows a dust storm in the eastern Mediterranean region on 24 Feb 2007: http://acdisc.sci.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMAERUVd.003/2007/OMI-Aura_L3-OMAERUVd_2007m0224_v003-2009m0714t140856.he5. Plot FinalAerosolExtOpticalDepth500 (Aerosol Extinction Optical Depth at 500 nm).
  • Level 2 swath data coming soon to OPeNDAP!! (Allows IDV to display swaths.)

See what you can do with netCDF/CF-1 formatted data

  • Convert AIRS L2 Swath to netCDF/CF-1 and view in IDV!
    • First, download IDV
    • Then, acquire the AIRS data in netCDF form (this URL will execute an on-the-fly reformatting.)
    • Then use IDV's Data Chooser to read the data in.
    • Select a parameter under 2D-Grids in the Field Selector, and "Color-Shaded Plan View" under Displays. Hint: the CO_total_column parameter under the 2D grids shows carbon monoxide emanating from fires in Greece in Aug 2007.

Take a walk down memory lane with EOSDIS Version 0

Yes, the user interface could have used some work, and it could have been faster, and the protocol was a bit clunky, and... But perhaps ahead of its time for federated search frameworks...

Clynnes 18:41, 3 February 2010 (EST)