WCS Development Issues
Miscellaneous Things that have come up in design, development and deployment of datafed WCS system. This includes email-conversations and personal observations.
Using Datafed client on NILU test DB with slightly irregular time dimension
http://webapps.datafed.net/datafed.aspx?page=test/NILU
The map view is an aggregate over 30 months, so it's kind of slow.
Since there is some problem with the time periodicity in EBAS, I made the map view to aggregate over one month. This demonstrated seasonal browsing.
http://webapps.datafed.net/datafed.aspx?page=test/NILU_month
Running WCS on NILU EBAS schema using sybase
Hello,
It looks actually pretty good. I can browse it better than I expected with the first try. But please check your DB connection. Currently your server reports an error:
Layer: 0, Origin: 0 Unable to connect: Adaptive Server is unavailable or does not exist
I made a test page for tweaking and testing the settings:
http://webapps.datafed.net/datafed.aspx?page=test/NILU
As soon as you have the DB online, I'll improve it. Currently there is time range aggregate on map view so that you can see something.
Other than that, the thing looks good, here are my first comments:
On 10/27/2010 8:27 AM, Paul Eckhardt wrote: > Those tests revealed some more questions: > > 1) for one coverage you assume a constant measurement interval, not only > across all locations, but also over the whole time series.
This is can be dealt in many ways. In general, the data fields in a coverage should share dimensions. So if you have data that actually does share dimensions, put them in one coverage. If you have hourly data and aggregated daily averages, make two coverages. If most of the locations are the same, data can be in one coverage. If location tables are different, make separate coverages. That's the easy part.
It seems, that EMEP data does not fit into this simplistic form, and therefore we have to think how to configure the WCS.
> This is generally not the case for emep data: > 1a) every station has their own measurement cycle (some changing filters > at midnight, others at 6am, this will again differ for the precipitation > samplers. No comment about the importance or consequences about this. > 1b) not all measurements have the same resolution (everything from > minute resolution to mothly averages is in principle possible)
This probably results in separating data by time resolution to separate coverages.
> 1b) some stations change their schedule over time. > 1c) there is no guarantee, that a timeseries is complete (i.e. no gaps)
This has not been a problem. For example CIRA/VIEWS has data twice a week, Still, we consider it a daily dataset.
In general, when the field is normalized into triple (loc_code, datetime, value), missing data is just a missing row.
> 1d) time series are not necessarily unique (there might me multiple > timeseries for one parameter at one location)
Does this mean, that there could be hourly and daily SO4? That's no problem, again those two are separate coverages.
If the data is truly random, like 1-20 measures at random times during the day, and separate in each location, then we cannot have periodic time dimensions. The time dimension must then be just a range, not with periodicity or enumerated times.
I haven't tested such coverages, yours might then be the first one. There's no technical reason why it should be a huge problem.
> To address 1b, I added a filter criteria to only use timeseries with a > resolution of 1 day.
Seems to work for this test!
> For the rest, I'm a bit lost. Maybe we need to aggregate (homogenize the > intervals) for all measurements in an intermediate database, in order to > fit in the OGC model? > Do we need to aggregate a unique value for each station, parameter, and day?
The wonderful thing about SQL is, that it gives you this option but does not enforce it. We compiled GSOD, Global Summary Of the Day, and did a ton of on-the-fly calculations using SQL views, producing what we needed directly from the original data. Later, some views were materialized for performance reasons. If you feel so, go ahead and create tables for WCS views and populating them from data.
> 2) the parameter naming conventions are not easily convertible with the > emep naming. > As an example there is SO4f (which I interpreted as SO4 particles in the > pm2.5 fraction - is this correct?), but emep distinguishes between > sea-salt-corrected SO4 and total SO4. Same for SO4t (which I understand > is the whole aerosol fraction, without a size cutoff?). > Please see http://tarantula.nilu.no/projects/ccc/nasaames/complist.html > for a (hopefully up-to-date?) list of components/matrixes used in emep.
We are using CF standard names:
http://webapps.datafed.net/table.aspx?database=catalog&table=standard_names
compiled from http://cf-pcmdi.llnl.gov/documents/cf-standard-names/standard-name-table/15/cf-standard-name-table.xml
If you have two same measures of SO4 done with two different type instruments and you want to publish both, go ahead and call the field SO4_instrA and SO4_instrB but choose the standard_name attribute from the CF table if possible.
I'm not an expert in CF naming, but sea-salt-corrected SO4 and total SO4 sound like two different CF standard names. Anyway, your fields can be called sea_salt_corrected_SO4 and total_SO4, that's fine. They may have same or different standard name, that's no problem.
> > Minor issues: > If I use the datafed browser, I can see some hiccups. I'm not sure if > this is due to my wrong configuration of the ows software... > > 1) the map layer is empty. Are there no features for the european > region, or is there a problem with the geo-references from my side? I had left North America map to the default view. I changed it into world map. The browser should pick a proper map based of catalog defaults. > 2) the defuault time displayed in the browser is 2010-07-29T06:00:00, > which i can not relate to any response from the wcs server. The > describeCoverage response contains: > <TemporalDomain> > <TimePeriod> > <BeginPosition>1972-01-01T06:00:00Z</BeginPosition> > <EndPosition>2007-01-07T06:00:00Z</EndPosition> > <TimeResolution>P1D</TimeResolution> > </TimePeriod> > </TemporalDomain>
There is a problem in "make_time_dimension" function in NILU_config.py. I filled the test database with daily data.
The time_min must have the same resolution as the data. So if you have daily data, it must not have hours. If it's hourly, it cannot have minutes.
So change
time_min = iso_time.parse(row[i_time_min]).dt time_max = iso_time.parse(row[i_time_max]).dt
to
time_min = iso_time.parse(row[i_time_min]).dt.date() time_max = iso_time.parse(row[i_time_max]).dt.date()
and the periodic dimension should work. I have to add a check for this in the iso_time module
The SQL query is then compiled by datetime='2010-07-29T00:00:00' and requires precision, one second off and it won't return anything. If the data is not on precise time intervals, you can in the browser set time window access for map view, to get all the data from datetime plus minus short period.
> 3) changing the time field in the browser updates the map, but somehow > the timeseries display is not changed (still showing 2010) > I can manually "zoom" out& in the time axes of the time series plot though. Timeseries display should update the cursor, blue vertical line. It does not update the time zoom. > 4) once I get the timeseries displayed, the y-scale seems to be fixed to > [0.0, 1.0]. What might be the reason for this? The values reported by > the wcs are correct, and agree with what is plotted (at least the values > below 1.0)
You can set a different scale. For each field, I have to set a default zoom, currently manually, but I'm planning to monitor registered WCS's and then I can have a script generating a better default. Click on the "Service Program" button to change settings.
> > However, here's the best display I could get: >> This is a display for 15th Jan 2006, 06:00. The cross at the location of > Birkenes, Norway and the corresponding timeseries plotted correctly.
That's better than I expected. As soon as you have the DB online, I'll take another look.
> Kind regards > > Paul > >
Two more things, make sure you don't have critical passwords in your python code. The sources are http accessible. Create a read-only account and for paranoia hide the connection module outside OWS/web/static.
Also in http://knulp.nilu.no:8080/static/NILU/wcs_capabilities.conf change the "Distributor:DataFed" to "Distributor:NILU"
You should add "IS NOT NULL" in the filters for location and data views, as well as filter out data with -999 as missing marker. I never use nulls or -999 except when I have to filter them out at the lowest level data access.
Greetings, Kari
Existing HTAP Services
Look into the point provider how to set up database for point WCS
The CIRA provider shows how to make a custom processor for a database where you cannot create views.
Kari
On 10/27/2010 2:57 AM, Michael Decker wrote: > Hi, > > those should all be valid and unique providers. HTAP_monthly* should be fully CF-compliant, in the others that might not be entirely so, but that might improve in the future. > > Another thing: > I guess we will also have a look at point station data soon. Do you have some documentation about what you have done there so far or some helpful query links for me to look at? I did not look at the source code for that at all so far but probably will within the next few weeks... > > Michael > > On 10/26/10 17:37, Kari Hoijarvi wrote: >> Hi, >> >> I have registered these services into our catalog: >> >> http://webapps.datafed.net/HTAP.uFIND >> http://webapps.datafed.net/catalog.aspx?table=WCS_services&distributor=juelich >> >> >> Are all these valid, or are some of them duplicates? >> >> HTAP_FC_pressure >> HTAP_monthly >> MACC_bnds >> HTAP_FC >> HTAP_FE >> HTAP_FE_hourly >> HTAP_monthly_pressure >> HTAP_FE_pressure >> > >
Getting Timeseries from WCS
Expand the bbox a little bit:
There is a known problem with rounding. In the version of OWS I should fix this.
By the way, you can call the wcs 1.1 services via our 1.0 proxy, but don't send these links anywhere. It's only set up so that our browser will work. The id 73556 is completely random and may change any time. So use the real service instead:
store=false works fine with firefox.
Kari
On 10/26/2010 3:53 PM, Falke, Stefan R (IS) wrote: > > Kari, > > > > I’m trying to get a GetCoverage example for a time series request on the degree day dataset. I’ve created the following using the TimeSeries WCS Query form in the DataFed Browser but it doesn’t return anything. Do you see anything wrong with the query? I made the request for a single lat-lon point, which creates an unusal bbound in the WCS request but I don’t know if that’s the issue: > > > > http://webapps.datafed.net/cov_73556.ogc?SERVICE=WCS&REQUEST=GetCoverage&VERSION=1.0.0&CRS=EPSG:4326&COVERAGE=DD&TIME=2001-01-01T00:00:00/2001-01-31T00:00:00&BBOX=-118.6109,37.2301,-118.6109,37.2301,0,0&WIDTH=1&HEIGHT=1&DEPTH=-1&FORMAT=CSV > > > > I generated this from this page: > > http://webapps.datafed.net/datafed.aspx?page=NGC/DegreeDays > > > > Thanks, > > Stefan
GeoEnterpiseLab got new service online
You could create yourself an organization here:
http://webapps.datafed.net/table.aspx?database=catalog&table=organizations&mode=edit&edit=7019&edit_mode=dup&message=Creating+new+from+existing+item.
Choose a nice looking abbreviation like GEOEL, enter the new information and save. Then you can use that abbreviation as distributor and originator.
Kari
On 10/26/2010 12:55 PM, Roberts, Gregory (IS) wrote: > Kari, > > I updated the wcs_capabilities.conf and also added index.html to each wcs directory. One thing though, I had to remove the Distributor:NorthropGrumman and Originator:NorthropGrumman from the keywords, since we are not allowed to show where the data is from. Let me know if these are correct. >
> They're there. > > http://webapps.datafed.net/catalog.aspx?table=WCS_services&sort=domain+asc > > Greg, could you please add the keywords to wcs_capabilities.conf so that the catalog search facility can be used. > > > Kari >
http://webapps.datafed.net/catalog.aspx?table=WCS_coverages&originator=GEL&pagesize=50
Looks good.
Kari
On 10/26/2010 2:08 PM, Roberts, Gregory (IS) wrote: > I created GeoEnterprise Lab (GEL and updated all the wcs_capabilities.conf with Distributor:GEL, Originator:GEL > >
Next Heading
Hello
Next Heading
Hello
