Networking impediments and opportunities

From Earth Science Information Partners (ESIP)

< Back to AQ CoP.png | Workshops | Air Quality Data Network


Session Intro

Martin: This third day will be a wrap-up day. The first day we have dealt with technical topics, the second day was about social, community and programmatic networking. This third day we would like to address the workshop motto: "from virtual to real". Now that the AQ Data network is in gear, what are the impediments toward further progress and what are the opportunities .. within the constrains of funding, manpower, knowledge. In the second morning session, we would like to listen to the participants: what at is their specific interest in participating in the network? The participants here are practitioners and they already have certain ways of doing things. What are the added values that the Network could contribute? What are the things that they could do with the network that they can not do now? It would be nice to take home ideas about showcasing these added values and opportunities. Finally for the afternoon, we have scheduled workshop outputs, outcomes and plans. In fact much of the day should be about outputs outcomes and plans.

Since we are now entering the discussion mode, and to capture the essence of those discussions, I am now suggesting to move from the PowerPoint to the wiki as the channel of communication. This will relieve us from the pressures of generating a workshop report after the fact. Furthermore, though the wiki everyone can participate and make contributions, now and in the future. So, a short wiki tutorial by here.

General Discussion

R.Husar: Voluntary human networks such as the AQ CoP have very special characteristics and impediments that have been well characterized by Bogardi:

Nature of Networking Projects: Complex funding; mixture of paid and voluntary; multiple obligations; international; differing project maturity; governance, cultures
Which glue keeps it together? Trust and personal affinity. Common objectives and scientific values. Mutual respect. Mutual benefit (win-win). Complementarity. Donor dictate
Lethal ingredients. Turf mentality. Budget discrepancies. Too much competition. Lack of data and information exchange. Donor jealousy
Particularly intriguing is the list of "Lethal ingredients" that can cause these networks to be dysfunctional.

B. Domenico: The Unidata governance consist of a user community that brainstorms all the opportunities and leads that from the users point of view. The compliment of that is that we have a policy committee that considers the resource constraints, focus areas and opportunities that should be given off. SO the group in this room, augmented with others could be the policy committee to make decisions on priorities and directions.

R.Husar: The Unidata model of governance is indeed very attractive, practical and effective approach. The entire data community operations that includes both software developments, outreach/education and governance is maintained by a strong financial and moral support by NSF, spanning the past 25 years. This AQ group does not yet have such sponsoring support, neither does it have a deep and broad buy-in by the potential users.

B.Domenico: Unidata did not start out with major NSF funding. There were meetings of the users that voiced the need for real time access to meteorological data particularly by the academic communities. At the same time there was a small group looking into the networking and communication technology. The initial proposals to NSF were modest. Your AQ community would also pursue a similar gradual act. NSF is pretty good at supporting proposal where the community get together to define their needs.

{Comment by Erin on NSF RFP on Cyber.. }

M.Schultz: A governance model just as outlined by Ben has been applied in the ??? projects. We did form a steering group composed of representatives of the partnering institutions. Decisions were made by this group on how to move forward. There is a programmers committee that needs to coordinate software development activities.

All these governance is organized without funding for the development of the network.

{RBH: Were the users represented in the governance? was the outcome a network of autonomous systems or a joint development of a model?}

T. Dye: Governance is imperative for us from a program point of view so that we know whats coming up over the next 10 years. Telecons and technical meetings would be useful to give life to this activity and make the operational folks more involved. This should be a major recommendation of this workshop. This should be of interest for funding agencies because ultimately this would save them money.

R. Husar: GEO is not the funding station. But recently, they have been exploring ways to use GEO as broker mediator between proposers with ideas and potential funding agencies. As one of its organs, the GEO AQ CoP could be a broker between proposers of specific network projects and potential funding agencies of such networking activities.

DLR Status, Impediments, Opportunities

M. Schultz: We now invite participants to share with us their current status of networking, their current impediments and opportunities. Lets start with DLR and Oleg. As we understand you have expressed interest in joining the AQ data network. Your group at DLR is not using the community WCS server but nevertheless you are offering data to the network using a WCS user interface.

O.Goussev: I can comment on technical aspects. Based on suggestions from NASA, we have implemented a WCS 1.0 server which was simpler to implement. If the AQ network can make use of WCS 1.0 then you can use our service now.

R.Husar: The AQ data network can indeed make use of DLR WCS service by adding a small converter/adapter at DataFed. Infact the GOME 2 satellite data are registered in the AQ community catalog. When a user accesses the GOME data it is obtained by calling the DLR WCS 1.0 server through the adapter piece. The limitations include the lack of the data access along the time dimension.

M.Schultz: From Rudy's comment I understand that if your server and the client are WCS 1.0 then they are interoperable. If the server is 1.0 and the client is 1.x they can still be interoperable through the adapter piece at the client or the mediator.

O.Goussev: In my opinion clients should be able to support interoperability with WCS 1.0, 1.1 and 2.0 as it emerge.


RHusar: Interoperability may involve keeping the server protocols flexible which also means that the clients will have to be flexible to understand the different dialects. Conversely by making the servers harmonious, i.e by co sharing, the clients need only understand that particular dialect. The third approach used in the DataFed is for the clients to access all of the data services through a mediator which performs the adaptations and homogenization of the diverse servers. By examining the realities, we as a community and individuals should evaluate which of these network architectural approaches are suitable under different conditions. My feeling is that for now we should retain all three possibilities.

NILU Status, Impediments, Opportunities

PEckhardt: We have implemented the AQ Community WCS server at NILU as a prototype. ... server separate from the operational/production server. A small fraction of the production database is mirrored at the WCS prototype. The system has been running since the beginning of 2011. The served EMEP AQ monitoring data data has been Explored through the WCS client. With regard to the impediment, I have three main points:

  • As many of you know, I had a problem with the WCS protocol. For our purposes, the data acces protocol must support 'point observation' data. The current WCS protocol is geared toward delivering spatial coverages, i.e the concentration of pollutants at specific discrete locations. However, beyond the concentration spatial or temporal coverage we also need to deliver considerable amount of metadata related to the monitoring station, the measurement method, and the sensor information. Over the past two days I have learned that new developments on the standards now allow the delivery of much richer pay load. We still need to evaluate if the new extended CF-netCDF binary file format will allow the incorporation of the most important metadata. This would mean that we have to pay more attention to the standardization of the pay load in addition with standardized data query language.
  • Funding for adoption of the protocol is another impediment. At this time there is no firm commitment. However, the strong interest at NILU in this standard based networking process which then may lead to more specific commitments by Kjetil and Asmund
  • A special NILU-specific issue is that whatever we do will need to be usable and applicable immediately

B.Domenico: There are two perspectives here. Client and the Server. From the point of view of servers we can think of the payload that is being returned as a result of data request. There are many protocols that can produce a specific payload WCS, WFS, SOS. From the point of view of user that is creating a client, it needs to address servers like WCS 1.0, WCS 1.1,SOS. This makes the plans more complicated because there are lot of options.

M. Schultz: So one thing we learned at this meeting is that flexibility can be either at the client side or at the server side. We as a community would then need to evaluate what is the best way to apportion the networking efforts between the clients and the servers.

B. Domenico: As a community may also decide that you may not want to have a full flexibility to cover all protocols and all payloads. You may find that WCS is adequate to cover most of your needs.


T. Dye: We will have server up and running in an year. Funding is going to be an issue But we see this as an opportunity to work together with government partners, university partners so that we don't have to learn everything ourselves. We will need to reach back and talk to EPA for establishing priorities. There are currently five to eight data distribution efforts underway just like this one that we have evaluated. Another key for us is to make sure that this effort has a life so that when we are back in 10 years, it has grown just as Unidata has grown. If this effort only last for 2 years, it is tough for us. It will help lower the cost to get some of the things we want into the software. We will also need meeting and talking of the programmers we should not forget the human element.

We will also need help on the data use end. We would like DataFed to connect toward WCS service rather than accessing the daily observation files. That will give us feedback on the quality of the service.

E.Robinson: Tim, for your programmers to collaborate you will need to give them permission to work within this group. I totally agree with you for the human element and meeting face to face. But its funny that Michel Decker saw Kari Hoijarvi first time on camera this week.

R.Husar: If programmers are co-developing the community software then the communication is mostly regarding implementation details not about philosophical or major design issues. For that, the ad-hoc just-in-time electronic communication is more effective than meetings. Even more important is the role of wikis and community list servers that can capture the interactions and makes open participation by the community possible.

E. Fialkowski: We have been putting in lot of efforts into interoperabiltiy and modularity. Our big problem is that we don't hear much back from our users. We don't have enough users. We are providers linking datasets, standardize their format and show what can be done with it. Most everything I do plugs into DataFed.


L.Bigagli: INSPIRE... mandates that for the EU countries the geospatial information should be expressed using the standards specified by the INSPIRE...So, if this AQ network team efforts adheres to the INSPIRE guidelines that will continue to the stability and graceful evolution.

B. Domenico: net-CDF data packaging and its API also follows a graceful evolution. All the versions of net-CDF are backward compatible. So net-CDF 4 can read net-CDF 1 files. This was not the case for WCS evolution. WCS 1.1 is not directly compatible with WCS 1.0. The time scale of CF convention evolution is this: the discrete sampling feature of CF was proposed 4 years ago before it was finally adapted this year. So there is plenty of time and opportunity for input.

R.Husar: Since the AQ network is meant to be compliant with a set of standards and conventions such as net-CDF, CF, WCS, ISO 19115. The stability of network operation will depend on the stability of these standards.

B.Domenico: One of the advantages of focusing on the payload like CF-netCDF is that when you are building your analysis or visualization software you can use data access API of net-CDF and you don't have to worry about whats on disk or service payload. You can have HDF, grid or other files and you can read them through the net-CDF API. This means that the analysis package ,i.e the client, can be separated from the server that accesses the data.

S.Falke: We are implementing the WCS through the community server and we are learning from each other through interaction like this meeting. From the client side it would be desirable to have more automated way. One possibility would be to publish these developments through the catalog so they could be discovered and used more effectively.

M.Schultz: This is a very good suggestion, in fact after the break we could spend some time discussing the various metadata tags that were proposed through AQ community catalog.

    • STI - commitment to adopt community server; need help on technical level
    • EEA - wants to add WCS service but needs to determine how to do this (e.g. decide which standard(s) to support)
    • NG - community definition of metadata, in particular for client development; how can INSPIRE etc. help us?
    • JRC - no impediments, all ready to go
  • Mismatched funding / mismatched development cycles
    • NILU - must answer immediate needs; this drives development efforts
  • Open source community efforts versus off-the-shelf software
    • Difficult to keep open source software going over the years
    • Require resources staff/$
    • DLR - limited resources - limit development efforts, not unwilling to install and run community server
    • NILU - limited funding, but maybe not impossible
    • STI - limited funding, need to prioritize
    • Is it reliable?
    • Don't trust open source
    • STI and EEA - must be certain that system survives ~10 years, not only on technical level but as a community
    • Unidata - backward compatibility is part of netcdf philosophy, not so in WCS (example multi-part mime wrapper); CF has slow development cycle on purpose, largely backward compatible although newer versions resolve ambiguities and therefore may break old code
  • Unclear expectations/requirements of the community
  • Need for clear, simple documentation/instructions
    • Group needs to add feedback/experience to documentation
    • Responsibility to maintain structure
    • STI - communication tools, repository+version control are critical elements
  • Changing/evolving standards and approach - Interoperability burnout
    • Too many new things - how to know where to focus?
    • Governance needs - limited resources, growing needs
    • STI - structured community software development would probably work
    • Support vs. documentation vs. development
    • Organisation of community software development: formal vs. informal; need for some sort of steering group with a mandate to take decisions
    • Unidata - started as small grant to look at needs
  • Challenges of real-time systems
    • Modifications needed to be made to the software for data coming in frequently
    • performance
    • STI - must persuade users to go through services rather than data downloads
  • Insufficient user feedback/ user involvement
  • Agreement on protocols (i.e. WCS x.y, W*S) vs. payload (i.e. adopt new netcdf-cf standard as community standard)