Discovery Hack-a-thon Details

Overview

Overview Slides

What's the Plan? Get together to make some simple Discovery clients! All are welcome, no previous experience or coding skills necessary!

• Two back-to-back sessions:

1. Tuesday, July 17, 2012. 1:30pm-3:00pm
2. Tuesday, July 17, 2012. 3:30pm-5:00pm

Abstract: The set of ESIP Discovery services encompass the overlapping conventions of Earth science federated OpenSearch, Collection Casting, Granule Casting, and Service Casting feed standards. To help lower the barrier of entry, we will provide a set of hands-on and simple approaches to using Discovery services. These include walking through some "low-hanging fruit" approaches to calling OpenSearch, Collection Casting, Granule Casting, and Service Casting.

Tuesday, July 17, 2012. 1:30pm-3:00pm: Non-Coders

Discovery Hack-a-thon Overview (20-mins)

Quick overview of ESIP Discovery services and set the stage for call the ESIP Discovery services from various simple clients.

Poll the audience for interest.

Hack-a-thon Breakout (70-mins)

Geoportal

We will have a Geoportal from our testbed up and running. You'll learn what protocols the geoportal already supports for service validation, how to use the geoportal to validate and register data and services, and how to add configurations for validation to a geoportal instance. Or, just take the Discovery Cluster geoportal instance out for a spin!

Mentor: Christine White

Browsers and News Readers

It's even possible to interact with both Data "casts" and OpenSearch servers using a simple browser. Also, News Readers, especially useful for Data casts!

Mentors: Ruth Duerr and ...

Cast Publishing and Aggregation

Learn how to use existing web apps create 1-off data casts. Come, create a cast or two and then see it found (that's the idea anyway).

Mentors: Ruth Duerr and ...

Command line

Yes, you can interact with an OpenSearch server or a Data Cast using basic command-line URL getters, like wget (available for all platforms) and curl.

Mentors: Hook Hua and Chris Lynnes

Tuesday, July 17, 2012. 3:30pm-5:00pm: Coders

Discovery Hack-a-thon Overview (15-mins)

Quick overview of ESIP Discovery services and set the stage for call the ESIP Discovery services from various simple clients.

Poll the audience for interest.

Hack-a-thon Breakout (75-mins)

Perl Monks

Learn how to hack a quick client together. Particularly useful for doing scripted search/acquire for datasets following your own, possibly idiosyncratic needs.

Sample code: Media:esip_fedsearch2.pl.txt

Mentors: Chris Lynnes and Brian Duggan

[Mojolicious] is a nice tool for interacting with web services. Install it like so:

 curl http://get.mojolicio.us | sh

If you are not root, also do this :

  cpanm --local-lib=~/perl5 local::lib && eval $(perl -I ~/perl5/lib/perl5/ -Mlocal::lib)

Then you will have the command line tool "mojo" which can send requests and extract using CSS selectors. For more documentation try:

 mojo help get
 perldoc Mojo::DOM
 perldoc Mojo::UserAgent

Here are some sample queries:

 # dataset level
 mojo get http://mirador.gsfc.nasa.gov/mirador_dataset_opensearch.xml 'OpenSearchDescription > Url' attr template 
 http://mirador.gsfc.nasa.gov/cgi-bin/mirador/collectionlist.pl?keyword={searchTerms}&page=1&count={count}&osLocation={geo:box}&startTime={time:start}&endTime={time:end}&format=rss
 http://mirador.gsfc.nasa.gov/cgi-bin/mirador/collectionlist.pl?keyword={searchTerms}&page=1&count={count}&osLocation={geo:box}&startTime={time:start}&endTime={time:end}&format=atom

 # collection level search  
 mojo get 'http://mirador.gsfc.nasa.gov/cgi-bin/mirador/collectionlist.pl?keyword=ozone;format=atom' 'entry > link[type="application/opensearchdescription+xml"]' attr href
 http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_OMTO3.003.xml
 http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_SSBUVO3.008.xml
 http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_SBUV2N09O3.008.xml
 http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_SBUV2N11O3.008.xml
 http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_SBUV2N16O3.008.xml
 http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_OMTO3G.003.xml
 http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_OMDOAO3Z.003.xml
 http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_SBUVN7O3.008.xml
 http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_OMDOAO3G.003.xml
 http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_OMDOAO3.003.xml

 # Granule level search
 mojo get http://mirador.gsfc.nasa.gov/OpenSearch/mirador_opensearch_OMTO3.003.xml 'Url[type="application/atom+xml"]' attr template
 http://mirador.gsfc.nasa.gov/cgi-bin/mirador/granlist.pl?dataSet=OMTO3.003&page=1&maxgranules={os:count}&osLocation={geo:box}&order=a&endTime={time:end}&startTime={time:start}&format=atom

 # Get the url for a specific granule
 mojo get 'http://mirador.gsfc.nasa.gov/cgi-bin/mirador/granlist.pl?format=rss&startTime=2010-01-01&endTime=2010-01-02&order=a&osLocation=&maxgranules=1&page=1&dataSet=OMTO3.003' 'item > link' text
 ftp://aurapar2u.ecs.nasa.gov/data/s4pa///Aura_OMI_Level2/OMTO3.003//2009/365/OMI-Aura_L2-OMTO3_2009m1231t2226-o29062_v003-2012m0331t213901.he5
 http://aurapar2u.ecs.nasa.gov/opendap/Aura_OMI_Level2/OMTO3.003//2009/365/OMI-Aura_L2-OMTO3_2009m1231t2226-o29062_v003-2012m0331t213901.he5

 # Get the granule itself
 mojo get http://aurapar2u.ecs.nasa.gov/opendap/Aura_OMI_Level2/OMTO3.003//2009/365/OMI-Aura_L2-OMTO3_2009m1231t2226-o29062_v003-2012m0331t213901.he5 > data.he5

Python Scripters

Learn how to use Python, an easy to use, but powerful scripting language, to interact with OpenSearch servers and find the data you need.

Mentors: Nga Chung and ...

Java Programmers

Mentors: Eric Rozell and ...

XSL Transforms

Learn how to generate simple interfaces to OpenSearch cast feeds. ESIP Discovery casts extend the Atom casts format and thus these transforms will expose additional information to the user interface.

The XSLT processors we'll try out are

1. your modern browser
2. Saxon-HE 9.4 (open source home edition)

Getting Saxon

• download: SaxonHE9-4-0-4J.zip
• reference: Saxon documentation
• unzip SaxonHE9-4-0-4J.zip
• we only need to use the "saxon9he.jar" library

A complete working example tutorial of an XSL to process Atom 1.0 to XHTML

• Tutorial on Processing Atom 1.0 to XHTML
• listing 6 has a complete XSLT transform providing an XHTML view of an Atom feed. save as "atom2xhtml.xsl"

Edit the xsl to add namespaces relevant for OpenSearch. Add namespaces to the top-level <xsl:stylesheet> element:

• xmlns:time="http://a9.com/-/opensearch/extensions/time/1.0/"
• xmlns:geo="http://a9.com/-/opensearch/extensions/geo/1.0/"

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:a="http://www.w3.org/2005/Atom"
  xmlns:xhtml="http://www.w3.org/1999/xhtml" 
  xmlns="http://www.w3.org/1999/xhtml" 
  xmlns:time="http://a9.com/-/opensearch/extensions/time/1.0/"
  xmlns:geo="http://a9.com/-/opensearch/extensions/geo/1.0/"
  exclude-result-prefixes="a xhtml time">

...

Download a sample ESIP OpenSearch granule response.

Edit the OpenSearch response to include the sytle-sheet directive.

<?xml version='1.0' encoding='UTF-8'?>

<?xml-stylesheet type="text/xsl" href="opensearch2xhtml.xsl"?>

<feed xmlns="http://www.w3.org/2005/Atom" xmlns:os="http://a9.com/-/spec/opensearch/1.1/" xmlns:time="http://a9.com/-/opensearch/extensions/time/1.0/" xmlns:geo="http://a9.com/-/opensearch/extensions/geo/1.0/" xmlns:nsidc="http://nsidc.org/ns/opensearch/1.1/">

...

Open the local OpenSearch feed xml with your browser or transform with Saxon command-line.

Saxon command-line interface

• Saxon command-line options

# -t  Display version and timing information to the standard error output. The output also traces the files that are read and writing, and extension modules that are loaded.
# -tree:(linked|tiny|tinyc)  Selects the implementation of the internal tree model. -tree:tiny selects the "tiny tree" model (the default). -tree:linked selects the linked tree model. -tree:tinyc selects the "condensed tiny tree" model. 

java -cp saxon9he.jar  net.sf.saxon.Transform  -tree:linked -t -s:{source-xml-filename}  -xsl:{xsl-filename}  -o:{output-filename}

Mentors: Hook Hua and ...

Hack-a-thon Resources

Casts

Collection Casts

• Feed of all NSIDC data sets

Granule-level Casts

• Feed of AMSR-E Ocean data
• Feed of AQUA/MODIS sea ice data

OpenSearch Description Documents

Top Level (search for datasets)

• NASA EOS Clearinghouse (ECHO)
• NSIDC Data Set Level OpenSearch Description Document
• NSIDC Data Set Level OpenSearch Description Document for the Aggregator underlying the OIB Portal
• Discovery Cluster Geoportal OpenSearch Description Document
• PO.DAAC Dataset Level OpenSearch Description Document
• GES DISC Mirador search engine OpenSearch Description Document
• NASA Energy Water-cycle Study A-Train Dataset OpenSearch Description Document (JPL WSNEWS)

Granule-level Examples

• NSIDC Granule Level OpenSearch Description Document for AMSR-E/Aqua Daily EASE-Grid Brightness Temperatures
• NSIDC Granule Level OpenSearch Description Document for Greenland Ice Sheet Melt Characteristics Derived from Passive Microwave Data
• NSIDC OpenSearch Description Document for IceBridge Mission Flight Reports
• NSIDC OpenSearch Description Document for GLAS/ICESat 1 km Laser Altimetry Digital Elevation Model of Greenland
• NSIDC OpenSearch Description Document for Near-Real-Time DMSP SSM/I-SSMIS Pathfinder Daily EASE-Grid Brightness Temperatures
• PO.DAAC Granule Level OpenSearch Description Document
• OpenSearch Description Document for AIRS Level 3 Daily products
• JPL NEWS AIRS/Aqua Regrid 1-hour 0.5x0.5 degrees v1.0 OpenSearch Description Document
• NASA ECHO Granule-level Search (MISR L3 Aerosols) Note that ECHO asks clients to supply a client_id, like the name of your application

Sample Code

• Java - [1]

Example URLs

OpenSearch

• http://nsidc.org/api/opensearch/1.1/dataset?searchterms=Air%20Temperature&dtstart=2010-01-01T00:00:00.000Z&dtend=2011-02-01T00:00:00.000Z&bbox=-128.549015,54.665619,37.898712,72.32832
• http://nsidc.org/libre/services/aggregator/feeds/custom?dtstart=2010-01-01T07:00:00.000Z&dtend=2012-07-11T05:59:59.999Z&bbox=-128.549015,54.665619,37.898712,72.32832
• http://nsidc.org./api/opensearch/1.1/dataset/NSIDC-0301/granule/?dtstart=2005-01-01T00:00:00.000Z&dtend=2005-02-01T00:00:00.000Z&polarization=Horizontal&frequency=36GHz&pass=Ascending
• http://nsidc.org./api/opensearch/1.1/dataset/NSIDC-0218/granule/?dtstart=2005-01-01T00:00:00.000Z&dtend=2005-02-01T00:00:00.000Z&bbox=-128.549015,54.665619,37.898712,72.32832
• http://nsidc.org./api/opensearch/1.1/dataset/IFLTRPT/granule/?dtstart=2010-01-01T00:00:00.000Z&dtend=2011-02-01T00:00:00.000Z&bbox=-128.549015,54.665619,37.898712,72.32832
• http://nsidc.org/api/opensearch/1.1/dataset/NSIDC-0305/granule/?dtstart=2005-01-01T00:00:00.000Z&dtend=2005-02-01T00:00:00.000Z&datum=WGS84%20Ellipsoid
• http://nsidc.org/api/opensearch/1.1/dataset/NSIDC-0342/granule/?dtstart=2012-06-01T00:00:00.000Z&dtend=2012-08-01T00:00:00.000Z&bbox=-128.549015,54.665619,37.898712,72.32832&projection=EASE-Grid%20Northern&frequency=22GHz&pass=Ascending
• http://wsnews.jpl.nasa.gov:8100/opensearch/granule?dataset=NEWS%20Regrid%20airs.aqua%201.0hr%20Nearest%20Neighbor%200.50x0.50degrees%20v1.0&q=temperature&bbox=-180,-90,180,90&datetimestart=2000-01-1T00:00:00Z&datetimeend=2010-01-1T00:00:00Z&startindex=0&itemsperpage=10&format=atom

How-To Guides

• Esri Geoportal Server wiki
• Discovery Custer Geoportal deployed on Amazon Instance
• Tutorial on Processing Atom 1.0 to XHTML

---

Back to Discovery_Hack-a-thon