Difference between revisions of "Discovery Hack-a-thon Details"

From Earth Science Information Partners (ESIP)
(Reverted edits by Nchung (talk) to last revision by Ctilmes)
 
(34 intermediate revisions by 7 users not shown)
Line 1: Line 1:
 
= Overview =
 
= Overview =
 +
 +
[https://docs.google.com/presentation/d/1uMzMKnjbYkB-ULwjMMOawxHigbUVwZjgskOCAUsPHho/edit#slide=id.p Overview Slides]
  
 
What's the Plan?  Get together to make some simple Discovery clients! All are welcome, no previous experience or coding skills necessary!
 
What's the Plan?  Get together to make some simple Discovery clients! All are welcome, no previous experience or coding skills necessary!
Line 8: Line 10:
 
'''Abstract''': The set of ESIP Discovery services encompass the overlapping conventions of Earth science federated OpenSearch, Collection Casting, Granule Casting, and Service Casting feed standards. To help lower the barrier of entry, we will provide a set of hands-on and simple approaches to using Discovery services. These include walking through some "low-hanging fruit" approaches to calling OpenSearch, Collection Casting, Granule Casting, and Service Casting.
 
'''Abstract''': The set of ESIP Discovery services encompass the overlapping conventions of Earth science federated OpenSearch, Collection Casting, Granule Casting, and Service Casting feed standards. To help lower the barrier of entry, we will provide a set of hands-on and simple approaches to using Discovery services. These include walking through some "low-hanging fruit" approaches to calling OpenSearch, Collection Casting, Granule Casting, and Service Casting.
  
== Introduction (5-mins) ==
+
== [http://commons.esipfed.org/node/424 Tuesday, July 17, 2012. 1:30pm-3:00pm]: Non-Coders ==
 +
 
 +
=== Discovery Hack-a-thon Overview (20-mins) ===
 +
 
 +
Quick overview of ESIP Discovery services and set the stage for call the ESIP Discovery services from various simple clients.
  
Give quick overview of ESIP Discovery services and set the stage for call the ESIP Discovery services from various simple clients.
+
Poll the audience for interest.
  
== Non-Coders ==
+
=== Hack-a-thon Breakout (70-mins) ===
  
=== Geoportal (20-mins) ===
+
==== Geoportal ====
  
 
We will have a Geoportal from our testbed up and running.  You'll learn what protocols the geoportal already supports for service validation, how to use the geoportal to validate and register data and services, and how to add configurations for validation to a geoportal instance. Or, just take the [http://23.23.211.222:8080/geoportal Discovery Cluster geoportal instance] out for a spin!
 
We will have a Geoportal from our testbed up and running.  You'll learn what protocols the geoportal already supports for service validation, how to use the geoportal to validate and register data and services, and how to add configurations for validation to a geoportal instance. Or, just take the [http://23.23.211.222:8080/geoportal Discovery Cluster geoportal instance] out for a spin!
Line 20: Line 26:
 
'''Mentor:'''  Christine White
 
'''Mentor:'''  Christine White
  
=== Browsers and News Readers (15-mins) ===
+
==== Browsers and News Readers ====
  
 
It's even possible to interact with both Data "casts" and OpenSearch servers using a simple browser.  Also, News Readers, especially useful for Data casts!
 
It's even possible to interact with both Data "casts" and OpenSearch servers using a simple browser.  Also, News Readers, especially useful for Data casts!
Line 26: Line 32:
 
'''Mentors:'''  Ruth Duerr and ...
 
'''Mentors:'''  Ruth Duerr and ...
  
=== Command line (15-mins) ===
+
==== Cast Publishing and Aggregation ====
 +
 
 +
Learn how to use existing web apps create 1-off data casts.  Come, create a cast or two and then see it found (that's the idea anyway).
 +
 +
'''Mentors:''' Ruth Duerr and ...
 +
 
 +
==== Command line ====
  
 
Yes, you can interact with an OpenSearch server or a Data Cast using basic command-line URL getters, like wget (available for all platforms) and curl.
 
Yes, you can interact with an OpenSearch server or a Data Cast using basic command-line URL getters, like wget (available for all platforms) and curl.
Line 32: Line 44:
 
'''Mentors:''' Hook Hua and Chris Lynnes
 
'''Mentors:''' Hook Hua and Chris Lynnes
  
== Coders ==
 
  
=== Perl Monks (15-mins) ===
+
== [http://commons.esipfed.org/node/424 Tuesday, July 17, 2012. 3:30pm-5:00pm]: Coders ==
 +
 
 +
=== Discovery Hack-a-thon Overview (15-mins) ===
 +
 
 +
Quick overview of ESIP Discovery services and set the stage for call the ESIP Discovery services from various simple clients.
 +
 
 +
Poll the audience for interest.
 +
 
 +
=== Hack-a-thon Breakout (75-mins) ===
 +
 
 +
==== Perl Monks ====
 
Learn how to hack a quick client together.  Particularly useful for doing scripted search/acquire for datasets following your own, possibly idiosyncratic needs.
 
Learn how to hack a quick client together.  Particularly useful for doing scripted search/acquire for datasets following your own, possibly idiosyncratic needs.
  
'''Mentors:''' Chris Lynnes and ...
+
Sample code:  [[Media:esip_fedsearch2.pl.txt]]
 +
 
 +
'''Mentors:''' Chris Lynnes and Brian Duggan
 +
 
 +
[[http://mojolicio.us Mojolicious]] is a nice tool for interacting with web services.  Install it like so:
 +
 
 +
  curl http://get.mojolicio.us | sh
 +
 
 +
If you are not root, also do this :
 +
 
 +
  cpanm --local-lib=~/perl5 local::lib && eval $(perl -I ~/perl5/lib/perl5/ -Mlocal::lib)
 +
 
 +
Then you will have the command line tool "mojo" which can send requests and extract using CSS selectors.  For more documentation
 +
try:
 +
 
 +
  mojo help get
 +
  perldoc Mojo::DOM
 +
  perldoc Mojo::UserAgent
 +
 
 +
Here are some sample queries:
 +
 
 +
  # dataset level
 +
  mojo get http://mirador.gsfc.nasa.gov/mirador_dataset_opensearch.xml 'OpenSearchDescription > Url' attr template
 +
  http://mirador.gsfc.nasa.gov/cgi-bin/mirador/collectionlist.pl?keyword={searchTerms}&page=1&count={count}&osLocation={geo:box}&startTime={time:start}&endTime={time:end}&format=rss
 +
  http://mirador.gsfc.nasa.gov/cgi-bin/mirador/collectionlist.pl?keyword={searchTerms}&page=1&count={count}&osLocation={geo:box}&startTime={time:start}&endTime={time:end}&format=atom
 +
 
 +
  # collection level search 
 +
  mojo get 'http://mirador.gsfc.nasa.gov/cgi-bin/mirador/collectionlist.pl?keyword=ozone;format=atom' 'entry > link[type="application/opensearchdescription+xml"]' attr href
 +
  http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_OMTO3.003.xml
 +
  http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_SSBUVO3.008.xml
 +
  http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_SBUV2N09O3.008.xml
 +
  http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_SBUV2N11O3.008.xml
 +
  http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_SBUV2N16O3.008.xml
 +
  http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_OMTO3G.003.xml
 +
  http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_OMDOAO3Z.003.xml
 +
  http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_SBUVN7O3.008.xml
 +
  http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_OMDOAO3G.003.xml
 +
  http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_OMDOAO3.003.xml
 +
 
 +
  # Granule level search
 +
  mojo get http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_OMTO3.003.xml 'Url[type="application/atom+xml"]' attr template
 +
  http://mirador.gsfc.nasa.gov/cgi-bin/mirador/granlist.pl?dataSet=OMTO3.003&page=1&maxgranules={os:count}&osLocation={geo:box}&order=a&endTime={time:end}&startTime={time:start}&format=atom
 +
 
 +
  # Get the url for a specific granule
 +
  mojo get 'http://mirador.gsfc.nasa.gov/cgi-bin/mirador/granlist.pl?format=rss&startTime=2010-01-01&endTime=2010-01-02&order=a&osLocation=&maxgranules=1&page=1&dataSet=OMTO3.003' 'item > link' text
 +
  ftp://aurapar2u.ecs.nasa.gov/data/s4pa///Aura_OMI_Level2/OMTO3.003//2009/365/OMI-Aura_L2-OMTO3_2009m1231t2226-o29062_v003-2012m0331t213901.he5
 +
  http://aurapar2u.ecs.nasa.gov/opendap/Aura_OMI_Level2/OMTO3.003//2009/365/OMI-Aura_L2-OMTO3_2009m1231t2226-o29062_v003-2012m0331t213901.he5
  
=== Python Scripters (15-mins) ===
+
  # Get the granule itself
 +
  mojo get http://aurapar2u.ecs.nasa.gov/opendap/Aura_OMI_Level2/OMTO3.003//2009/365/OMI-Aura_L2-OMTO3_2009m1231t2226-o29062_v003-2012m0331t213901.he5 > data.he5
 +
 
 +
==== Python Scripters ====
 
Learn how to use Python, an easy to use, but powerful scripting language, to interact with OpenSearch servers and find the data you need.
 
Learn how to use Python, an easy to use, but powerful scripting language, to interact with OpenSearch servers and find the data you need.
  
 
'''Mentors:''' Nga Chung and ...
 
'''Mentors:''' Nga Chung and ...
  
=== Java Programmers (15-mins) ===
+
==== Java Programmers ====
  
 
'''Mentors:''' Eric Rozell and ...
 
'''Mentors:''' Eric Rozell and ...
  
=== XSL Transforms (15-mins) ===
+
==== XSL Transforms ====
  
 
Learn how to generate simple interfaces to OpenSearch cast feeds. ESIP Discovery casts extend the Atom casts format and thus these transforms will expose additional information to the user interface.
 
Learn how to generate simple interfaces to OpenSearch cast feeds. ESIP Discovery casts extend the Atom casts format and thus these transforms will expose additional information to the user interface.
  
'''Mentors:''' Hook Hua and ...
+
The XSLT processors we'll try out are
 +
# your modern browser
 +
# [http://saxon.sourceforge.net/#F9.4HE Saxon-HE 9.4] (open source home edition)
 +
 
 +
 
 +
Getting Saxon
 +
* download: [http://sourceforge.net/projects/saxon/files/Saxon-HE/9.4/SaxonHE9-4-0-4J.zip/download SaxonHE9-4-0-4J.zip]
 +
* reference: [http://www.saxonica.com/documentation/index.html Saxon documentation]
 +
* unzip SaxonHE9-4-0-4J.zip
 +
* we only need to use the "saxon9he.jar" library
 +
 
 +
 
 +
A complete working example tutorial of an XSL to process Atom 1.0 to XHTML
 +
* [http://www.ibm.com/developerworks/xml/tutorials/x-atomxsl/section3.html Tutorial on Processing Atom 1.0 to XHTML]
 +
* [http://www.ibm.com/developerworks/xml/tutorials/x-atomxsl/section3.html#listing6 listing 6] has a complete XSLT transform providing an XHTML view of an Atom feed. save as "atom2xhtml.xsl"
 +
 
 +
 
 +
Edit the xsl to add namespaces relevant for OpenSearch. Add namespaces to the top-level <xsl:stylesheet> element:
 +
* xmlns:time="http://a9.com/-/opensearch/extensions/time/1.0/"
 +
* xmlns:geo="http://a9.com/-/opensearch/extensions/geo/1.0/"
 +
<pre>
 +
<?xml version="1.0" encoding="utf-8"?>
 +
<xsl:stylesheet version="1.0"
 +
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
 +
  xmlns:a="http://www.w3.org/2005/Atom"
 +
  xmlns:xhtml="http://www.w3.org/1999/xhtml"
 +
  xmlns="http://www.w3.org/1999/xhtml"
 +
  xmlns:time="http://a9.com/-/opensearch/extensions/time/1.0/"
 +
  xmlns:geo="http://a9.com/-/opensearch/extensions/geo/1.0/"
 +
  exclude-result-prefixes="a xhtml time">
 +
 
 +
...
 +
</pre>
 +
 
 +
 
 +
Download a sample ESIP OpenSearch granule response.
 +
 
 +
Edit the OpenSearch response to include the sytle-sheet directive.
 +
<pre>
 +
<?xml version='1.0' encoding='UTF-8'?>
 +
 
 +
<?xml-stylesheet type="text/xsl" href="opensearch2xhtml.xsl"?>
 +
 
 +
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:os="http://a9.com/-/spec/opensearch/1.1/" xmlns:time="http://a9.com/-/opensearch/extensions/time/1.0/" xmlns:geo="http://a9.com/-/opensearch/extensions/geo/1.0/" xmlns:nsidc="http://nsidc.org/ns/opensearch/1.1/">
 +
 
 +
...
 +
</pre>
 +
 
 +
Open the local OpenSearch feed xml with your browser or transform with Saxon command-line.
 +
 
 +
 
 +
Saxon command-line interface
 +
* [http://www.saxonica.com/documentation/using-xsl/commandline.xml Saxon command-line options]
 +
<pre>
 +
# -t  Display version and timing information to the standard error output. The output also traces the files that are read and writing, and extension modules that are loaded.
 +
# -tree:(linked|tiny|tinyc)  Selects the implementation of the internal tree model. -tree:tiny selects the "tiny tree" model (the default). -tree:linked selects the linked tree model. -tree:tinyc selects the "condensed tiny tree" model.  
 +
 
 +
java -cp saxon9he.jar  net.sf.saxon.Transform  -tree:linked -t -s:{source-xml-filename}  -xsl:{xsl-filename}  -o:{output-filename}
 +
</pre>
 +
 
  
== Cast Publishing and Aggregation (20-mins) ==
 
  
Learn how to use existing web apps create 1-off data casts.  Come, create a cast or two and then see it found (that's the idea anyway).
+
'''Mentors:''' Hook Hua and ...
 
'''Mentors:''' Ruth Duer
 
  
 
= Hack-a-thon Resources =
 
= Hack-a-thon Resources =
  
 
== Casts ==
 
== Casts ==
 +
 +
=== Collection Casts ===
 +
 +
*[http://nsidc.org/feeds/data/nsidc.xml Feed of all NSIDC data sets]
 +
 +
=== Granule-level Casts ===
 +
 +
*[http://nsidc.org/feeds/data/AE_Ocean.xml  Feed of AMSR-E Ocean data]
 +
*[http://nsidc.org/feeds/data/MYD29V5.xml Feed of AQUA/MODIS sea ice data]
 +
 +
=== Service Casts ===
  
 
== OpenSearch Description Documents ==
 
== OpenSearch Description Documents ==
  
 
=== Top Level (search for datasets) ===
 
=== Top Level (search for datasets) ===
 +
*[http://api.echo.nasa.gov/echo-esip/echo-esip-dataset.xml NASA EOS Clearinghouse (ECHO)]
 +
*[http://nsidc.org/api/opensearch/1.1/dataset/description NSIDC Data Set Level OpenSearch Description Document]
 +
*[http://nsidc.org/libre/services/aggregator/description NSIDC Data Set Level OpenSearch Description Document for the Aggregator underlying the OIB Portal]
 
*[http://23.23.211.222:8080/geoportal/openSearchDescription Discovery Cluster Geoportal OpenSearch Description Document]  
 
*[http://23.23.211.222:8080/geoportal/openSearchDescription Discovery Cluster Geoportal OpenSearch Description Document]  
 
*[http://podaac.jpl.nasa.gov:8890/ws/search/podaac-dataset-osd.xml PO.DAAC Dataset Level OpenSearch Description Document]
 
*[http://podaac.jpl.nasa.gov:8890/ws/search/podaac-dataset-osd.xml PO.DAAC Dataset Level OpenSearch Description Document]
Line 73: Line 213:
  
 
=== Granule-level Examples ===
 
=== Granule-level Examples ===
 +
*[http://nsidc.org./api/opensearch/1.1/dataset/NSIDC-0301/granule/description NSIDC Granule Level OpenSearch Description Document for AMSR-E/Aqua Daily EASE-Grid Brightness Temperatures]
 +
*[http://nsidc.org./api/opensearch/1.1/dataset/NSIDC-0218/granule/description NSIDC Granule Level OpenSearch Description Document for Greenland Ice Sheet Melt Characteristics Derived from Passive Microwave Data]
 +
*[http://nsidc.org./api/opensearch/1.1/dataset/IFLTRPT/granule/description NSIDC OpenSearch Description Document for IceBridge Mission Flight Reports]
 +
*[http://nsidc.org/api/opensearch/1.1/dataset/NSIDC-0305/granule/description NSIDC OpenSearch Description Document for GLAS/ICESat 1 km Laser Altimetry Digital Elevation Model of Greenland]
 +
*[http://nsidc.org/api/opensearch/1.1/dataset/NSIDC-0342/granule/description NSIDC OpenSearch Description Document for Near-Real-Time DMSP SSM/I-SSMIS Pathfinder Daily EASE-Grid Brightness Temperatures]
 
*[http://podaac.jpl.nasa.gov:8890/ws/search/podaac-granule-osd.xml PO.DAAC Granule Level OpenSearch Description Document]
 
*[http://podaac.jpl.nasa.gov:8890/ws/search/podaac-granule-osd.xml PO.DAAC Granule Level OpenSearch Description Document]
 
*[http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_AIRX3STD.005.xml OpenSearch Description Document for AIRS Level 3 Daily products]
 
*[http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_AIRX3STD.005.xml OpenSearch Description Document for AIRS Level 3 Daily products]
*[http://wsnews.jpl.nasa.gov:8100/opensearch/osdd_granule?dataset=NEWS%20Regrid%20airs.aqua%201.0hr%20Nearest%20Neighbor%200.50x0.50degrees%20v1.0 NEWS AIRS/Aqua Regrid 1-hour 0.5x0.5 degrees v1.0 OpenSearch Description Document]
+
*[http://wsnews.jpl.nasa.gov:8100/opensearch/osdd_granule?dataset=NEWS%20Regrid%20airs.aqua%201.0hr%20Nearest%20Neighbor%200.50x0.50degrees%20v1.0 JPL NEWS AIRS/Aqua Regrid 1-hour 0.5x0.5 degrees v1.0 OpenSearch Description Document]
 +
*[https://api.echo.nasa.gov:443/echo-esip/search/granuleOsdd.xml?dataCenter=LARC&shortName=MIL3DAE&versionId=4&spatialType=NORMAL&clientId= NASA ECHO Granule-level Search (MISR L3 Aerosols)] Note that ECHO asks clients to supply a client_id, like the name of your application
 +
 
 +
===Sample Code===
 +
* Java - [https://scm.escience.rpi.edu/svn/public/esip/discovery/java-tutorial/]
  
 
= Example URLs =
 
= Example URLs =
== Casting ==
 
  
 
== OpenSearch ==
 
== OpenSearch ==
 +
 +
* http://nsidc.org/api/opensearch/1.1/dataset?searchterms=Air%20Temperature&dtstart=2010-01-01T00:00:00.000Z&dtend=2011-02-01T00:00:00.000Z&bbox=-128.549015,54.665619,37.898712,72.32832
 +
* http://nsidc.org/libre/services/aggregator/feeds/custom?dtstart=2010-01-01T07:00:00.000Z&dtend=2012-07-11T05:59:59.999Z&bbox=-128.549015,54.665619,37.898712,72.32832
 +
* http://nsidc.org./api/opensearch/1.1/dataset/NSIDC-0301/granule/?dtstart=2005-01-01T00:00:00.000Z&dtend=2005-02-01T00:00:00.000Z&polarization=Horizontal&frequency=36GHz&pass=Ascending
 +
* http://nsidc.org./api/opensearch/1.1/dataset/NSIDC-0218/granule/?dtstart=2005-01-01T00:00:00.000Z&dtend=2005-02-01T00:00:00.000Z&bbox=-128.549015,54.665619,37.898712,72.32832
 +
* http://nsidc.org./api/opensearch/1.1/dataset/IFLTRPT/granule/?dtstart=2010-01-01T00:00:00.000Z&dtend=2011-02-01T00:00:00.000Z&bbox=-128.549015,54.665619,37.898712,72.32832
 +
* http://nsidc.org/api/opensearch/1.1/dataset/NSIDC-0305/granule/?dtstart=2005-01-01T00:00:00.000Z&dtend=2005-02-01T00:00:00.000Z&datum=WGS84%20Ellipsoid
 +
* http://nsidc.org/api/opensearch/1.1/dataset/NSIDC-0342/granule/?dtstart=2012-06-01T00:00:00.000Z&dtend=2012-08-01T00:00:00.000Z&bbox=-128.549015,54.665619,37.898712,72.32832&projection=EASE-Grid%20Northern&frequency=22GHz&pass=Ascending
 +
* http://wsnews.jpl.nasa.gov:8100/opensearch/granule?dataset=NEWS%20Regrid%20airs.aqua%201.0hr%20Nearest%20Neighbor%200.50x0.50degrees%20v1.0&q=temperature&bbox=-180,-90,180,90&datetimestart=2000-01-1T00:00:00Z&datetimeend=2010-01-1T00:00:00Z&startindex=0&itemsperpage=10&format=atom
  
 
= How-To Guides =
 
= How-To Guides =
*[http://sourceforge.net/apps/mediawiki/geoportal/index.php?title=Main_Page Esri Geoportal Server wiki]
+
 
*[http://23.23.211.222:8080/geoportal Discovery Custer Geoportal deployed on Amazon Instance]
+
* [http://sourceforge.net/apps/mediawiki/geoportal/index.php?title=Main_Page Esri Geoportal Server wiki]
 +
* [http://23.23.211.222:8080/geoportal Discovery Custer Geoportal deployed on Amazon Instance]
 +
* [http://www.ibm.com/developerworks/xml/tutorials/x-atomxsl/section3.html Tutorial on Processing Atom 1.0 to XHTML]
 +
 
 
----
 
----
 
Back to [[Discovery_Hack-a-thon]]
 
Back to [[Discovery_Hack-a-thon]]

Latest revision as of 17:58, July 22, 2012

Overview

Overview Slides

What's the Plan? Get together to make some simple Discovery clients! All are welcome, no previous experience or coding skills necessary!

  • Two back-to-back sessions:
  1. Tuesday, July 17, 2012. 1:30pm-3:00pm
  2. Tuesday, July 17, 2012. 3:30pm-5:00pm

Abstract: The set of ESIP Discovery services encompass the overlapping conventions of Earth science federated OpenSearch, Collection Casting, Granule Casting, and Service Casting feed standards. To help lower the barrier of entry, we will provide a set of hands-on and simple approaches to using Discovery services. These include walking through some "low-hanging fruit" approaches to calling OpenSearch, Collection Casting, Granule Casting, and Service Casting.

Tuesday, July 17, 2012. 1:30pm-3:00pm: Non-Coders

Discovery Hack-a-thon Overview (20-mins)

Quick overview of ESIP Discovery services and set the stage for call the ESIP Discovery services from various simple clients.

Poll the audience for interest.

Hack-a-thon Breakout (70-mins)

Geoportal

We will have a Geoportal from our testbed up and running. You'll learn what protocols the geoportal already supports for service validation, how to use the geoportal to validate and register data and services, and how to add configurations for validation to a geoportal instance. Or, just take the Discovery Cluster geoportal instance out for a spin!

Mentor: Christine White

Browsers and News Readers

It's even possible to interact with both Data "casts" and OpenSearch servers using a simple browser. Also, News Readers, especially useful for Data casts!

Mentors: Ruth Duerr and ...

Cast Publishing and Aggregation

Learn how to use existing web apps create 1-off data casts. Come, create a cast or two and then see it found (that's the idea anyway).

Mentors: Ruth Duerr and ...

Command line

Yes, you can interact with an OpenSearch server or a Data Cast using basic command-line URL getters, like wget (available for all platforms) and curl.

Mentors: Hook Hua and Chris Lynnes


Tuesday, July 17, 2012. 3:30pm-5:00pm: Coders

Discovery Hack-a-thon Overview (15-mins)

Quick overview of ESIP Discovery services and set the stage for call the ESIP Discovery services from various simple clients.

Poll the audience for interest.

Hack-a-thon Breakout (75-mins)

Perl Monks

Learn how to hack a quick client together. Particularly useful for doing scripted search/acquire for datasets following your own, possibly idiosyncratic needs.

Sample code: Media:esip_fedsearch2.pl.txt

Mentors: Chris Lynnes and Brian Duggan

[Mojolicious] is a nice tool for interacting with web services. Install it like so:

 curl http://get.mojolicio.us | sh

If you are not root, also do this :

  cpanm --local-lib=~/perl5 local::lib && eval $(perl -I ~/perl5/lib/perl5/ -Mlocal::lib)

Then you will have the command line tool "mojo" which can send requests and extract using CSS selectors. For more documentation try:

 mojo help get
 perldoc Mojo::DOM
 perldoc Mojo::UserAgent

Here are some sample queries:

 # dataset level
 mojo get http://mirador.gsfc.nasa.gov/mirador_dataset_opensearch.xml 'OpenSearchDescription > Url' attr template 
 http://mirador.gsfc.nasa.gov/cgi-bin/mirador/collectionlist.pl?keyword={searchTerms}&page=1&count={count}&osLocation={geo:box}&startTime={time:start}&endTime={time:end}&format=rss
 http://mirador.gsfc.nasa.gov/cgi-bin/mirador/collectionlist.pl?keyword={searchTerms}&page=1&count={count}&osLocation={geo:box}&startTime={time:start}&endTime={time:end}&format=atom
 # collection level search  
 mojo get 'http://mirador.gsfc.nasa.gov/cgi-bin/mirador/collectionlist.pl?keyword=ozone;format=atom' 'entry > link[type="application/opensearchdescription+xml"]' attr href
 http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_OMTO3.003.xml
 http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_SSBUVO3.008.xml
 http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_SBUV2N09O3.008.xml
 http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_SBUV2N11O3.008.xml
 http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_SBUV2N16O3.008.xml
 http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_OMTO3G.003.xml
 http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_OMDOAO3Z.003.xml
 http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_SBUVN7O3.008.xml
 http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_OMDOAO3G.003.xml
 http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_OMDOAO3.003.xml
 # Granule level search
 mojo get http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_OMTO3.003.xml 'Url[type="application/atom+xml"]' attr template
 http://mirador.gsfc.nasa.gov/cgi-bin/mirador/granlist.pl?dataSet=OMTO3.003&page=1&maxgranules={os:count}&osLocation={geo:box}&order=a&endTime={time:end}&startTime={time:start}&format=atom
 # Get the url for a specific granule
 mojo get 'http://mirador.gsfc.nasa.gov/cgi-bin/mirador/granlist.pl?format=rss&startTime=2010-01-01&endTime=2010-01-02&order=a&osLocation=&maxgranules=1&page=1&dataSet=OMTO3.003' 'item > link' text
 ftp://aurapar2u.ecs.nasa.gov/data/s4pa///Aura_OMI_Level2/OMTO3.003//2009/365/OMI-Aura_L2-OMTO3_2009m1231t2226-o29062_v003-2012m0331t213901.he5
 http://aurapar2u.ecs.nasa.gov/opendap/Aura_OMI_Level2/OMTO3.003//2009/365/OMI-Aura_L2-OMTO3_2009m1231t2226-o29062_v003-2012m0331t213901.he5
 # Get the granule itself
 mojo get http://aurapar2u.ecs.nasa.gov/opendap/Aura_OMI_Level2/OMTO3.003//2009/365/OMI-Aura_L2-OMTO3_2009m1231t2226-o29062_v003-2012m0331t213901.he5 > data.he5

Python Scripters

Learn how to use Python, an easy to use, but powerful scripting language, to interact with OpenSearch servers and find the data you need.

Mentors: Nga Chung and ...

Java Programmers

Mentors: Eric Rozell and ...

XSL Transforms

Learn how to generate simple interfaces to OpenSearch cast feeds. ESIP Discovery casts extend the Atom casts format and thus these transforms will expose additional information to the user interface.

The XSLT processors we'll try out are

  1. your modern browser
  2. Saxon-HE 9.4 (open source home edition)


Getting Saxon


A complete working example tutorial of an XSL to process Atom 1.0 to XHTML


Edit the xsl to add namespaces relevant for OpenSearch. Add namespaces to the top-level <xsl:stylesheet> element:

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:a="http://www.w3.org/2005/Atom"
  xmlns:xhtml="http://www.w3.org/1999/xhtml" 
  xmlns="http://www.w3.org/1999/xhtml" 
  xmlns:time="http://a9.com/-/opensearch/extensions/time/1.0/"
  xmlns:geo="http://a9.com/-/opensearch/extensions/geo/1.0/"
  exclude-result-prefixes="a xhtml time">

...


Download a sample ESIP OpenSearch granule response.

Edit the OpenSearch response to include the sytle-sheet directive.

<?xml version='1.0' encoding='UTF-8'?>

<?xml-stylesheet type="text/xsl" href="opensearch2xhtml.xsl"?>

<feed xmlns="http://www.w3.org/2005/Atom" xmlns:os="http://a9.com/-/spec/opensearch/1.1/" xmlns:time="http://a9.com/-/opensearch/extensions/time/1.0/" xmlns:geo="http://a9.com/-/opensearch/extensions/geo/1.0/" xmlns:nsidc="http://nsidc.org/ns/opensearch/1.1/">

...

Open the local OpenSearch feed xml with your browser or transform with Saxon command-line.


Saxon command-line interface

# -t  Display version and timing information to the standard error output. The output also traces the files that are read and writing, and extension modules that are loaded.
# -tree:(linked|tiny|tinyc)  Selects the implementation of the internal tree model. -tree:tiny selects the "tiny tree" model (the default). -tree:linked selects the linked tree model. -tree:tinyc selects the "condensed tiny tree" model. 

java -cp saxon9he.jar  net.sf.saxon.Transform  -tree:linked -t -s:{source-xml-filename}  -xsl:{xsl-filename}  -o:{output-filename}


Mentors: Hook Hua and ...

Hack-a-thon Resources

Casts

Collection Casts

Granule-level Casts

Service Casts

OpenSearch Description Documents

Top Level (search for datasets)

Granule-level Examples

Sample Code

Example URLs

OpenSearch

How-To Guides


Back to Discovery_Hack-a-thon