Difference between revisions of "WCS Wrapper Installation LinuxOP"

From Earth Science Information Partners (ESIP)
Line 1: Line 1:
 
[[WCS_Access_to_netCDF_Files| Back to WCS Wrapper]]   
 
[[WCS_Access_to_netCDF_Files| Back to WCS Wrapper]]   
  
Questions and Comments should be directed to Kari Hoijarvi [mailto:hoijarvi@me.wustl.edu hoijarvi@me.wustl.edu 314-935 5772]
+
Questions and Comments should be directed to Kari Hoijarvi [mailto:hoijarvi@me.wustl.edu hoijarvi@me.wustl.edu 314-935 5772] and Michael Decker [mailto:m.decker(at)fz-juelich.de m.decker(at)fz-juelich.de]
  
==Installation on a Linux workstation or server:==
+
= Installation on a Linux workstation or server: =
 +
== notes/before you begin ==
 +
* this package has been tested with Python 2.5 and should also run on version 2.6 without problems, 3.x is not supported yet
 +
* system-wide installation of packages will require root access to the system
 +
** it is theoretically possible to install all packages to a user directory, but it might be somewhat tricky and will not be covered here
 +
* for installation of 3rd party python packages it is recommended to use [http://pypi.python.org/pypi/setuptools  easy_install]
 +
** download the installer file matching your Python version and execute it on a shell as instructed
 +
* 3rd party packages will receive updates over time. the last tested version of every package is noted below
 +
* all required packages except for PyNIO can be installed via '''easy_install''', so don't bother to download them in advance
 +
* unfortunately you will have to jump through some hoops for PyNIO. check out the [http://www.pyngl.ucar.edu/Download/ PyNIO/NGL download page] for instructions on how to obtain and install it
  
WCS Wrapper uses several open source packages. Since we did not check the licences,
+
== required 3rd party software ==
we're not including them to setup package, you have to get them yourself.  
+
You have to install the packages listed below first in order to run the OWS server. Use ''easy_install'' for all except PyNIO (see above)
 +
* [http://webpy.org/ webpy] (0.32)
 +
* [http://codespeak.net/lxml/ lxml] (2.2.2)
 +
** might require additional development/header packages to be installed on your system (''python-dev, libxml2-dev, libxslt1-dev, zlib1g-dev'' on Debian)
 +
* [http://numpy.scipy.org/ NumPy] (1.3.0)
 +
** you should have a fortran compiler installed for that, for example ''gfortran'' on Debian
 +
* [http://pypi.python.org/pypi/wsgilog/ WsgiLog] (0.1) (optional, only if you want to log webpy output to a file)
 +
* [http://www.pyngl.ucar.edu/Nio.shtml PyNIO] (1.3.0b1)
 +
** 1.3.0b1 depends on ''libgfortran1'' which is not part of debian stable/testing any more, you will have to pull it from oldstable (http://mirrors.kernel.org/debian/pool/main/g/gcc-4.1/libgfortran1_4.1.1-21_i386.deb for i386) and install it manually by copying the library file to ''/usr/lib'' and running ''ldconfig''
 +
** newer releases of PyNIO will hopefully depend on a more recent libgfortran2/3
  
==Get Python==
+
== install and start OWS server ==
Preferably '''2.5.1''' since this system has not been tested with earlier versions. You can get it from [http://python.org/download/ python.org/download/]. Build instructions for Python on Linux are included in the README file. The download is free.
+
* get the [http://htap.icg.fz-juelich.de/downloads/ows_linux.tar OWS code package for Linux]
 +
* unpack to your desired working directory, the contents of the archive will end up in ''./ows''
 +
** these instructions assume that your working directory is ''/usr/local/'', so the package contents would end up in ''/usr/local/ows''
 +
* add the path of the extracted ''ows'' directory to your ''PYTHONPATH'' environment variable
 +
** if you followed the above example, ''PYTHONPATH'' should contain ''/usr/local/ows''
 +
* cd to ''ows/web'' and execute ''python ows.py''
 +
* a server should be running on http://localhost:8080/ now
 +
** a different port can be passed to the server as an argument
 +
** you will have to create the log directory ''/var/log/ows'' before running the server for the first time
 +
**** if the server is not being run as root, the user that runs it needs write access to the log dir
 +
* you can stop the server by pressing ctrl-c in the console window
 +
* to make the server available automatically at boot time, you can use an init script
 +
** an example script that starts the OWS server in a screen session as another user is available [http://htap.icg.fz-juelich.de/downloads/ows.sh here]
  
Since the C and C++ object files are 32 bit, get a '''32 bit''' python even if you have 64 bit machine.
+
== example installations ==
 +
=== Debian ===
 +
* typescript of installation on fresh and minimal [http://debian.org Debian] stable: http://htap.icg.fz-juelich.de/downloads/debian_typescript.txt
  
== Get Web.py framework ==
+
= Creating Metadata =
 
 
WCS Wrapper uses version 0.22 of the web.py framework, which is not the newest version, so get it from [http://webpy.org/static/web.py-0.22.tar.gz here].
 
 
 
After extraction, go to main directory and type '''python setup.py install'''
 
 
 
 
 
<html>
 
<p>
 
<a href="http://webpy.org"><img src="http://webpy.unleashed.pl/logo/7.png" />
 
</p>
 
</html>
 
 
 
== Get the Python Xml package, lxml ==
 
 
 
lxml requires two packages before installation: libxslt and libxml
 
 
 
Datafed OWS is tested with [ftp://xmlsoft.org/libxslt/libxslt-1.1.22.tar.gz Libxslt 1.1.22] and [ftp://xmlsoft.org/libxslt/libxml2-2.6.30.tar.gz Libxml 2.6.30].  Other versions are available from [ftp://xmlsoft.org/libxslt/ here].  Both packages include a makefile and INSTALL file with instructions.
 
 
 
After running make install, the packages are copied to /usr/local/lib/python2.5/site-packages.  This folder needs to be on the python path, so run export PYTHONPATH="$PYTHONPATH:/usr/local/lib/python2.5/site-packages".
 
 
 
Lxml is hosted at [http://codespeak.net/lxml/ codespeak.net/lxml/]. Somehow, I have had difficulties accessing it with Firefox, IE seems to work.  WCS Wrapper uses it in XML processing.
 
 
 
== Get NetCDF ==
 
 
 
[http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-install Main installation page]
 
 
 
Get the source package [ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf.tar.gz here].
 
 
 
Quick installation instructions are available
 
[http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-install/Quick-Instructions-for-Installing-NetCDF-on-Unix.html#Quick-Instructions-for-Installing-NetCDF-on-Unix here]
 
 
 
WCS Wrapper uses netcdf.lib, which has to be on path, so run export PATH="$PATH:/path/to/netcdf"
 
For example, if netcdf was installed to /usr/local/src/netcdf-3.6.2,
 
you would run "export PATH="$PATH:/usr/local/src/netcdf-3.6.2"
 
 
 
== Get WCS Wrapper ==
 
 
 
From [http://datafed.net/ows/datafed_ows.zip datafed.net/ows/datafed_ows.zip]. Unzip
 
it into your installation folder, let's use '''/usr/local/OWS''' as an example.
 
 
 
You should have two directories '''/usr/local/OWS/datafed''' and '''/usr/local/OWS/web'''
 
 
 
Download [http://niceguy.wustl.edu/nc3/nc3.so The Linux nc3 package] to your datafed directory.
 
 
 
The folder datafed contains ''datafed'' package, which contains tools for ISO 8601 time
 
parsing and python-NetCDF bridge. ''Web'' is the WCS framework.
 
 
 
Make sure that '''/usr/local/OWS''' is in your '''PYTHONPATH''' variable. Add it with export PYTHONPATH="$PYTHONPATH:/usr/local/OWS"
 
 
 
run "export" and check the PYTHONPATH.  You should see at least "/usr/local/OWS/:/usr/local/lib/python2.5/site-packages"
 
 
 
Change Directory to '''/usr/local/OWS/web''' folder and type '''python ows.py'''. You have now your server running, listening port 8080.
 
 
 
You can pass the port as parameter. '''python ows.py 80''' listens the standard port 80.
 
 
 
Congratulations, you have your server running. Now we need to install some data to serve.
 
 
 
== Creating Service from a single NetCDF file ==
 
 
 
'''/OWS/web/static/testprovider/''' contains files '''CubeA_ncml.xml''', '''CubeA.py''' and '''CubeA.nc'''
 
 
 
*'''CubeA_ncml.xml''' is a [http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Tutorial.html NetCDF Markup Language] description of the test coverage.
 
*'''CubeA.py''' is a script that creates the empty '''.nc''' file and fills it with test data. It servers as a good example code how to create your own CF 1.0 compatible files. Run it.
 
*'''CubeA.nc''' is the output of two previous, and serves as our test coverage.
 
 
 
Sections of the WCS url:
 
*'''http://capita.wustl.edu:8080''' is the server url. It returns a static index.html page of providers using this server.
 
*'''/testprovider''' is the provider name. There must be a folder '''/usr/OWS/web/static/testprovider/'''. *'''&identifier=CubeA''' is the name of the NetCDF file. So in this example, '''/OWS/web/static/testprovider/''' contains file '''CubeA.nc''' and it becomes a coverage. There can be any number of NetCDF files. Coverages are all listed in the GetCoverage request.
 
*RangeSubset='''fieldSubset:*''' selects variables to be returned from the NetCFD, '*' gets all, &RangeSubset='''fieldSubset:Spam;fieldSubset:Eggs''' gets variables '''Spam''' and '''Eggs'''
 
 
 
'''Try this:''' [http://localhost:8080/testprovider?service=WCS&version=1.1.0&Request=GetCoverage&identifier=CubeA&BoundingBox=-180,-90,180,90,urn:ogc:def:crs:OGC:2:84&TimeSequence=2007-01-01&RangeSubset=fieldSubset:*&format=image/netcdf&store=true test query]
 
 
 
The wcs GetCoverage returns an Xml envelope
 
<html>
 
    <pre><span class="pi">&lt;?xml version="1.0" encoding="UTF-8"?&gt;</span>
 
&lt;<span class="start-tag">Coverages</span><span class="attribute-name"> xmlns</span>=<span
 
    class="attribute-value">"http://www.opengis.net/wcs/1.1/ows" </span><span class="attribute-name">
 
        xmlns:ows</span>=<span class="attribute-value">"http://www.opengis.net/ows" </span><span
 
            class="attribute-name">xmlns:xlink</span>=<span class="attribute-value">"http://www.w3.org/1999/xlink"</span>&gt;
 
    &lt;<span class="start-tag">Coverage</span>&gt;
 
        &lt;<span class="start-tag">ows:Title</span>&gt;Select * From CubeA&lt;/<span class="end-tag">ows:Title</span>&gt;
 
        &lt;<span class="start-tag">ows:Abstract</span>&gt;Where boundingbox=-180,-90,180,90,urn:ogc:def:crs:OGC:2:84 and timesequence=2007-01-01&lt;/<span class="end-tag">ows:Abstract</span>&gt;
 
        &lt;<span class="start-tag">Identifier</span>&gt;<strong>CubeA</strong>&lt;/<span class="end-tag">Identifier</span>&gt;
 
        &lt;<span class="start-tag">Reference</span><span class="attribute-name"> xlink:href</span>=<span
 
            class="attribute-value">"<strong>http://localhost:8080/static/cache/CubeA_1.nc</strong>" </span><span
 
                class="attribute-name">xlink:role</span>=<span class="attribute-value">"urn:ogc:def:role:WCS:1.1:coverage"</span><span
 
                    class="attribute-name">/</span>&gt;
 
    &lt;/<span class="end-tag">Coverage</span>&gt;
 
&lt;/<span class="end-tag">Coverages</span>&gt;
 
</pre>
 
</html>
 
 
 
Of all the standard supported options, '''&format=image/netcdf&store=true'''
 
is the only supported one. The request writes out a netcdf file '''/OWS/web/static/cache/CubeA_1.nc''' and so it can be retrieved later with url '''http://localhost:8080/static/cache/CubeA_1.nc'''
 
and it will be deleted in ten minutes. The number after coverage is a generated sequence number.
 
 
 
'''In short:'''
 
 
 
*Every '''folder''' under '''static''' becomes a '''provider'''. Providers are unrelated. There can be any number of providers.
 
*Every '''NetCDF file''' in the '''provider folder''' becomes a '''coverage'''. The WCS Capabilities document is automatically compiled from all the NetCDF files.
 
*Every '''variable''' inside '''NetCDF''' becomes a '''field'''
 
 
 
== Creating Metadata ==
 
  
 
The metadata queries of WCS, '''GetCoverage''' and '''DescribeCoverage''' need to be precompiled.
 
The metadata queries of WCS, '''GetCoverage''' and '''DescribeCoverage''' need to be precompiled.
Line 125: Line 54:
 
:: python owsadmin.py wcs_desc -v 1.1.0 testprovider
 
:: python owsadmin.py wcs_desc -v 1.1.0 testprovider
  
They will create files '''/usr/local/OWS/web/static/testprovider/wcs_1.1.0_capabilities.xml'''
+
They will create files '''/usr/local/ows/web/static/testprovider/wcs_1.1.0_capabilities.xml'''
and '''/OWS/web/static/testprovider/wcs_1.1.0_desc_CubeA.xml'''. The capabilities file contains all the coverages that this provider has, and each coverage gets its own desc file.
+
and '''/ows/web/static/testprovider/wcs_1.1.0_desc_CubeA.xml'''. The capabilities file contains all the coverages that this provider has, and each coverage gets its own desc file.
  
 
Try now test queries [http://localhost:8080/testprovider?service=WCS&version=1.1.0&Request=GetCapabilities GetCapabilities] and [http://localhost:8080/testprovider?service=WCS&version=1.1.0&Request=DescribeCoverage&identifier=CubeA DescribeCoverage CubeA]
 
Try now test queries [http://localhost:8080/testprovider?service=WCS&version=1.1.0&Request=GetCapabilities GetCapabilities] and [http://localhost:8080/testprovider?service=WCS&version=1.1.0&Request=DescribeCoverage&identifier=CubeA DescribeCoverage CubeA]
Line 132: Line 61:
 
Press ctrl+C to terminate the server. Although this server is intended for development and debugging only, you can use it to server data from your own workstation just fine. For high load sites, look [http://webpy.org/install webpy.org/install] documentation how to set up a high load site.
 
Press ctrl+C to terminate the server. Although this server is intended for development and debugging only, you can use it to server data from your own workstation just fine. For high load sites, look [http://webpy.org/install webpy.org/install] documentation how to set up a high load site.
  
Notice: WCS Wrapper core is written in C++, enabling large amount of data delivery. Python is used only for query parsing, making minimal impact in speed.
+
Notice: WCS core functions are written in C/C++, enabling large amount of data delivery. Python is used only for query parsing, making minimal impact in speed.
 
 
== Creating a Service from periodical NetCDF files. ==
 
 
 
We are serving some HTAP ozone model data with this method. Our test servers are a Windows XP workstation and a Linux 64 bit server.
 
 
 
The windows version runs on the default port 80, server was started '''python ows.py 80'''. Linux uses port 81, which has to be in the query url.
 
 
 
'''Live Query Examples:'''
 
 
 
Linux Server:
 
:[http://niceguy.wustl.edu:81/HTAP?service=WCS&version=1.1.0&Request=GetCapabilities GetCapabilities]
 
:[http://niceguy.wustl.edu:81/HTAP?service=WCS&version=1.1.0&Request=DescribeCoverage&identifier=HTAP_GEMAQ-v1p0 DescribeCoverage]
 
:[http://niceguy.wustl.edu:81/HTAP?service=WCS&version=1.1.0&Request=GetCoverage&identifier=GEMAQ-v1p0&BoundingBox=260,32,280,48,urn:ogc:def:crs:OGC:2:84&TimeSequence=2001-01-01T12:00:00Z&RangeSubset=fieldSubset:vmr_no2&format=image/netcdf&store=true GetCoverage]
 
 
 
Windows Workstation:
 
:[http://128.252.167.125/HTAP?service=WCS&version=1.1.0&Request=GetCapabilities GetCapabilities]
 
:[http://128.252.167.125/HTAP?service=WCS&version=1.1.0&Request=DescribeCoverage&identifier=HTAP_GEMAQ-v1p0 DescribeCoverage]
 
:[http://128.252.167.125/HTAP?service=WCS&version=1.1.0&Request=GetCoverage&identifier=GEMAQ-v1p0&BoundingBox=260,32,280,48,urn:ogc:def:crs:OGC:2:84&TimeSequence=2001-01-01T12:00:00Z&RangeSubset=fieldSubset:vmr_no2&format=image/netcdf&store=true GetCoverage]
 
 
 
How to set this up on your site:
 
 
 
Download our HTAP test package from [http://datafed.net/ows/datafed_ows_HTAP.zip" datafed.net/ows/datafed_ows_HTAP.zip]. It only has two days of data to make download small.
 
 
 
Unzip it to the '''/usr/local/OWS''' folder, the zip file contains directories
 
'''/usr/local/OWS/GEMAQ-v1p0''', '''/usr/local/OWS/GEOSChem-v45''' and '''/usr/local/OWS/web/static/HTAP/'''
 
 
 
The dataset name is '''HTAP'''. The coverages are '''GEMAQ-v1p0''' and '''GEOSChem-v45'''.
 
 
 
Then you need to edit '''/usr/local/OWS/web/static/HTAP/HTAP_config.py''' and change the variable '''netcdf_root_folder''' to point the directory where chose to install (so far, /usr/local/OWS).
 
 
 
Try this [http://localhost:8080/HTAP?service=WCS&version=1.1.0&Request=GetCoverage&identifier=GEMAQ-v1p0&BoundingBox=260,32,280,48,urn:ogc:def:crs:OGC:2:84&TimeSequence=2001-01-01T12:00:00Z&RangeSubset=fieldSubset:vmr_no2&format=image/netcdf&store=true test query].
 
 
 
Notice that it starts with http://localhost:8080/
 
  
 
The server returns a short XML document, which contains a link into the query result,
 
The server returns a short XML document, which contains a link into the query result,
 
you can download it by copying the url to your browser.
 
you can download it by copying the url to your browser.
 
Test from another computer using your own IP address or domain name http://123.456.789.123:8080/
 
  
 
== index.html Pages ==
 
== index.html Pages ==
Line 176: Line 70:
 
If no query is present, the server gives a default page '''index.html'''. You should provide pages for your server and for all the providers.
 
If no query is present, the server gives a default page '''index.html'''. You should provide pages for your server and for all the providers.
  
The server is at '''/usr/local/OWS/web/static/index.html''', which will be displayed from url [http://localhost:8080/ http://localhost:8080/]
+
The server is at '''/usr/local/ows/web/static/index.html''', which will be displayed from url [http://localhost:8080/ http://localhost:8080/]
  
Every provider should have a index.html like  '''/usr/local/OWS/web/static/testprovider/index.html''' which will be displayed from [http://localhost:8080/testprovider http://localhost:8080/testprovider]
+
Every provider can have an own index.html like  '''/usr/local/ows/web/static/testprovider/index.html''' which will be displayed from [http://localhost:8080/testprovider http://localhost:8080/testprovider]
  
 
If you modify the default index.html, make sure not to overwrite it next time you upgrade the WCS Wrapper package. Keep the main copy elsewhere and copy it back.
 
If you modify the default index.html, make sure not to overwrite it next time you upgrade the WCS Wrapper package. Keep the main copy elsewhere and copy it back.

Revision as of 06:39, September 9, 2009

Back to WCS Wrapper

Questions and Comments should be directed to Kari Hoijarvi hoijarvi@me.wustl.edu 314-935 5772 and Michael Decker m.decker(at)fz-juelich.de

Installation on a Linux workstation or server:

notes/before you begin

  • this package has been tested with Python 2.5 and should also run on version 2.6 without problems, 3.x is not supported yet
  • system-wide installation of packages will require root access to the system
    • it is theoretically possible to install all packages to a user directory, but it might be somewhat tricky and will not be covered here
  • for installation of 3rd party python packages it is recommended to use easy_install
    • download the installer file matching your Python version and execute it on a shell as instructed
  • 3rd party packages will receive updates over time. the last tested version of every package is noted below
  • all required packages except for PyNIO can be installed via easy_install, so don't bother to download them in advance
  • unfortunately you will have to jump through some hoops for PyNIO. check out the PyNIO/NGL download page for instructions on how to obtain and install it

required 3rd party software

You have to install the packages listed below first in order to run the OWS server. Use easy_install for all except PyNIO (see above)

  • webpy (0.32)
  • lxml (2.2.2)
    • might require additional development/header packages to be installed on your system (python-dev, libxml2-dev, libxslt1-dev, zlib1g-dev on Debian)
  • NumPy (1.3.0)
    • you should have a fortran compiler installed for that, for example gfortran on Debian
  • WsgiLog (0.1) (optional, only if you want to log webpy output to a file)
  • PyNIO (1.3.0b1)

install and start OWS server

  • get the OWS code package for Linux
  • unpack to your desired working directory, the contents of the archive will end up in ./ows
    • these instructions assume that your working directory is /usr/local/, so the package contents would end up in /usr/local/ows
  • add the path of the extracted ows directory to your PYTHONPATH environment variable
    • if you followed the above example, PYTHONPATH should contain /usr/local/ows
  • cd to ows/web and execute python ows.py
  • a server should be running on http://localhost:8080/ now
    • a different port can be passed to the server as an argument
    • you will have to create the log directory /var/log/ows before running the server for the first time
        • if the server is not being run as root, the user that runs it needs write access to the log dir
  • you can stop the server by pressing ctrl-c in the console window
  • to make the server available automatically at boot time, you can use an init script
    • an example script that starts the OWS server in a screen session as another user is available here

example installations

Debian

Creating Metadata

The metadata queries of WCS, GetCoverage and DescribeCoverage need to be precompiled.

Run the following commands:

cd /usr/local/OWS/web
python owsadmin.py wcs_cap -v 1.1.0 -d localhost testprovider
python owsadmin.py wcs_desc -v 1.1.0 testprovider

They will create files /usr/local/ows/web/static/testprovider/wcs_1.1.0_capabilities.xml and /ows/web/static/testprovider/wcs_1.1.0_desc_CubeA.xml. The capabilities file contains all the coverages that this provider has, and each coverage gets its own desc file.

Try now test queries GetCapabilities and DescribeCoverage CubeA

Press ctrl+C to terminate the server. Although this server is intended for development and debugging only, you can use it to server data from your own workstation just fine. For high load sites, look webpy.org/install documentation how to set up a high load site.

Notice: WCS core functions are written in C/C++, enabling large amount of data delivery. Python is used only for query parsing, making minimal impact in speed.

The server returns a short XML document, which contains a link into the query result, you can download it by copying the url to your browser.

index.html Pages

If no query is present, the server gives a default page index.html. You should provide pages for your server and for all the providers.

The server is at /usr/local/ows/web/static/index.html, which will be displayed from url http://localhost:8080/

Every provider can have an own index.html like /usr/local/ows/web/static/testprovider/index.html which will be displayed from http://localhost:8080/testprovider

If you modify the default index.html, make sure not to overwrite it next time you upgrade the WCS Wrapper package. Keep the main copy elsewhere and copy it back.

Important! web.py development server requires, that text files have Unix-style end of line marker, plain LF and not the Windows convention CR-LF. The owsadmin.py script can be used to fix this:

python /usr/local/OWS/web/owsadmin.py unix_nl "/usr/local/OWS/web/static/testprovider/index.html"

Use your html editing tool support if possible.