DataFed Service Chain Example

From Federation of Earth Science Information Partners
Revision as of 21:55, 10 June 2007 by Uy3Fo0 (Talk | contribs)

Jump to: navigation, search
cheap diazepam cheap xenical nokia ringtones qwest ringtones zanaflex online diazepam online verizon ringtones free motorola ringtones order lortab prozac online ericsson ringtones but ortho cheap cyclobenzaprine cheap celexa free ringtones phentermine free cool ringtones cheap prozac free qwest ringtones cialis online sprint ringtones nexium online real ringtones sagem ringtones tracfone ringtones free nokia ringtones free punk ringtones mp3 ringtones sony ringtones ativan online but hgh free midi ringtones alltel ringtones cheap flexeril sony ericsson ringtones motorola ringtones vicodin online free tracfone ringtones buy soma levitra online free motorola ringtones cheap didrex cheap sildenafil pharmacy online online zoloft online hydrocodone online free sony ringtones cingular ringtones viagra buy paxil ambien online buy lortab order hydrocodone vigrx online cheap ativan hgh online sildenafil online buy viagra ortho online nextel ringtones lisinopril free funny ringtones free sony ericsson ringtones buy vicodin hydrocodone order ativan mono ringtones clonazepam levitra free sprint ringtones cialis online sony ringtones cheap levitra zanaflex online phentermine celexa online ultracet online wwe ringtones free sharp ringtones buy norco soma online cheap ultracet free cingular ringtones cheap adipex cheap xenical cheap albuterol cheap tenuate cheap ortho free music ringtones sagem ringtones cheap lisinopril meridia online cheap propecia cheap vigrx cheap diethylpropion cheap albuterol mp3 ringtones meridia online free ringtones cheap ultram soma online cheap lortab cheap sildenafil free nextel ringtones cheap lorazepam free funny ringtones diethylpropion online free ringtones funny ringtones cingular ringtones cheap soma nokia ringtones clonazepam online zyban online clomid online buy prozac online valium order lisinopril meridia online buy tramadol sharp ringtones free midi ringtones flexeril online real ringtones celexa online zoloft online free nextel ringtones ericsson ringtones free music ringtones mtv ringtones order ultram buy valium viagra online lorazepam online cheap cialis cheap phentermine buy nexium samsung ringtones cheap norco meridia online sprint ringtones buy rivotril order carisoprodol carisoprodol online cyclobenzaprine online free qwest ringtones free qwest ringtones cheap fioricet verizon ringtones punk ringtones cheap hoodia pharmacy online online tracfone ringtones tramadol cheap rivotril tramadol online didrex online cheap fioricet music ringtones cheap albuterol free nokia ringtones buy viagra buy alprazolam buy xenical pharmacy online online free funny ringtones order zanaflex free ericsson ringtones cyclobenzaprine online adipex online cheap xanax free alltel ringtones free qwest ringtones ultram online sonyericsson ringtones free samsung ringtones cheap lortab free mono ringtones free samsung ringtones zyban online cheap celexa free funny ringtones wwe ringtones cheap rivotril but clomid buy fioricet buy hgh cheap lipitor cheap hoodia clonazepam online cool ringtones tenuate online cheap vicodin real ringtones free sonyericsson ringtones buy hoodia buy tramadol nextel ringtones cheap alprazolam ambien online
WCS python demos by Hoijarvi
For questions regarding this page e-mail Hoijarvi

Contents

Background

Service Chaining Example: Outline

This example uses python to

  • get point data
  • render it as an image
  • store the image on local computer

See WCS page SOAPifying_WCS

Getting Demo Script

Download http://datafed.net/demo/soapchain.txt and save it with extension .py

type from command line python soapchain.py and watch it run.

There are no parameters, the script is hard coded for demo purposes.

SOAP problems

There are two problems with SOAP and large amounts of data.

  • First, data has to be passed as xml, which results to enormous amounts of data for large tables, causing performance problems.
  • Second, SOAP defines 4 MB size limit for messages, so large data transfers have to partitioned, making things more complicated.
  • Third problem is WCS binary data, although binary encoding with MIME attachments is possoble, I do not know any system that would use it.

Datafed services have solved the issue by using common two phase pattern. The SOAP services return an envelope with small amount of metadata describing the result, and an uri pointer to the cached result.

Datafed SOAP services use this pattern also while passing data into services. For example in this chase, the table url is passed from WCS to RenderMapPoint directly. Since the services are located in the same machine, they can access the table directly, without turning it into xml ever.

Service Chain Example: Python Script

Function execute_chain

This is the main program for the service chain.

  • Query data with WCS,
  • Render it,
  • save locally

Since the output envelope does not contain data, but a reference to data, get_table_url function returns the reference url to the data table, which is then passed into the rendering service. It also returns a reference to the image, which is still in the server.

So the services don't pass data to each others, they pass a little metadata and a url to the data. If the services are in the same server, this bypasses reading and writing huge amounts of xml data.

def execute_chain():
    print "querying table"
    table_url = get_table_url()

    # table_url == http://webapps.datafed.net/storage.aspx?ID=GetCoverage_91
    # rendering service can get this directly from the cache with the ID GetCoverage_91

    print "rendering table "   table_url
    image_url = get_image_url(table_url)

    # image_url == http://webapps.datafed.net/storage.aspx?ID=RenderMapPoint_92
    # image processing services could again get the big image directly from
    # the server cache with ID RenderMapPoint_92
    #
    # Now we get the result from the server and save it on the local drive.

    print "fetching image "   image_url
    image_stream = urllib2.urlopen(image_url)
    try:
        print "writing image to file"
        dump_stream(image_stream, "soapchain.png")
    finally:
        image_stream.close()

Query Template for WCS

get_coverage_query = """
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <soap:Body>
        <wcs:GetCoverage version="1.0.0" service="WCS"
                xmlns:gml="http://www.opengis.net/gml" xmlns:wcs="http://www.opengis.net/wcs">
            <!-- datafed dataset_abbr.param_abbr defines the coverage name -->
            <wcs:sourceCoverage>[[dataset_abbr]].[[param_abbr]]</wcs:sourceCoverage> 
            <wcs:domainSubset>
                <wcs:spatialSubset>
                    <!--
                        This element queries the USA.
                        The dataset has no elevation, so only lat and lon
                        limits are needed. 
                    -->
                    <gml:Envelope srsName="WGS84(DD)">
                        <gml:pos>[[lon_min]] [[lat_min]]</gml:pos>
                        <gml:pos>[[lon_max]] [[lat_max]]</gml:pos>
                    </gml:Envelope>
                    <gml:Grid dimension="2">
                        <gml:limits>
                            <!--
                                grid size. This is a point dataset, so these numbers have no meaning.
                            -->
                            <gml:GridEnvelope>
                                <gml:low>0 0</gml:low>
                                <gml:high>99 99</gml:high>
                            </gml:GridEnvelope>
                        </gml:limits>
                        <gml:axisName>lat</gml:axisName>
                        <gml:axisName>lon</gml:axisName>
                    </gml:Grid>
                </wcs:spatialSubset>
                <wcs:temporalSubset>
                    <!--
                        query data for one time only.
                    -->
                    <gml:timePosition>[[datetime]]</gml:timePosition>
                </wcs:temporalSubset>
            </wcs:domainSubset>
            <wcs:output>
                <!--
                .NET dataset is a good format for point datasets
                -->
                <wcs:format>dataset-schema</wcs:format>
            </wcs:output>
        </wcs:GetCoverage>
    </soap:Body>
</soap:Envelope>
"""

Query Template for Render Call

render_point_query = """
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <soap:Body>
        <rmp:Render xmlns:rmp="http://datafed.net/xs/RenderMapPoint">
            <Table xmlns="http://datafed.net/xs/Table">
                <TableRef>[[tableref]]</TableRef>
            </Table>
            <rmp:Settings xmlns:mi="http://datafed.net/xs/MapImageLatLon"
                 xmlns:ip="http://datafed.net/xs/ImagePrimitives">
                <rmp:image_desc>
                    <mi:zoom>
                        <mi:image_width>400</mi:image_width>
                        <mi:image_height>200</mi:image_height>
                        <mi:lat_min>[[lat_min]]</mi:lat_min>
                        <mi:lat_max>[[lat_max]]</mi:lat_max>
                        <mi:lon_min>[[lon_min]]</mi:lon_min>
                        <mi:lon_max>[[lon_max]]</mi:lon_max>
                    </mi:zoom>
                    <mi:bgcolor>0xE1FFF0</mi:bgcolor>
                    <mi:image_format>image/png</mi:image_format>
                </rmp:image_desc>
                <rmp:data_column>[[param_abbr]]</rmp:data_column>
                <rmp:scale_min>0</rmp:scale_min>
                <rmp:scale_max>[[scale_max]]</rmp:scale_max>
                <rmp:sqrt>false</rmp:sqrt>
                <rmp:symbol>
                    <ip:width>10</ip:width>
                    <ip:height>10</ip:height>
                    <ip:offset_x>0</ip:offset_x>
                    <ip:offset_y>0</ip:offset_y>
                    <ip:shape>circle</ip:shape>
                    <ip:num_of_sides>4</ip:num_of_sides>
                    <ip:baseline>false</ip:baseline>
                </rmp:symbol>
                <rmp:pen>
                    <ip:width>0.5</ip:width>
                    <ip:style>solid</ip:style>
                    <ip:color>red</ip:color>
                </rmp:pen>
                <rmp:brush>
                    <ip:style>solid</ip:style>
                    <ip:color>yellow</ip:color>
                </rmp:brush>
                <rmp:script>
                    used.symbol.width=symbol.width*norm_param_value;
                    used.symbol.height=symbol.height*norm_param_value;
                </rmp:script>
            </rmp:Settings>
        </rmp:Render>
    </soap:Body>
</soap:Envelope>
"""

Other Utility Functions

  • query_datafed: A generic soap call to datafed server.
  • replace_parameters: If you want to change most common parameters.
  • look_for_ns_name: xml utility
  • dump_stream: write image stream to file
  • get_table_url, get_image_url: soap calls
Personal tools
Namespaces

Variants
Actions
Navigation
Toolbox