Networking impediments and opportunities

< Back to | Workshops | Air Quality Data Network

Kjeld/Ludewig

Session Intro

Martin: This third day will be a wrap-up day. The first day we have dealt with technical topics, the second day was about social, community and programmatic networking. This third day we would like to address the workshop motto: "from virtual to real". Now that the AQ Data network is in gear, what are the impediments toward further progress and what are the opportunities .. within the constrains of funding, manpower, knowledge. In the second morning session, we would like to listen to the participants: what at is their specific interest in participating in the network? The participants here are practitioners and they already have certain ways of doing things. What are the added values that the Network could contribute? What are the things that they could do with the network that they can not do now? It would be nice to take home ideas about showcasing these added values and opportunities. Finally for the afternoon, we have scheduled workshop outputs, outcomes and plans. In fact much of the day should be about outputs outcomes and plans.

Since we are now entering the discussion mode, and to capture the essence of those discussions, I am now suggesting to move from the PowerPoint to the wiki as the channel of communication. This will relieve us from the pressures of generating a workshop report after the fact. Furthermore, though the wiki everyone can participate and make contributions, now and in the future. So, a short wiki tutorial by Erin...link here.

General Discussion

R.Husar: Voluntary human networks such as the AQ CoP have very special characteristics and impediments that have been well characterized by Bogardi:

Nature of Networking Projects: Complex funding; mixture of paid and voluntary; multiple obligations; international; differing project maturity; governance, cultures
Which glue keeps it together? Trust and personal affinity. Common objectives and scientific values. Mutual respect. Mutual benefit (win-win). Complementarity. Donor dictate
Lethal ingredients. Turf mentality. Budget discrepancies. Too much competition. Lack of data and information exchange. Donor jealousy
Particularly intriguing is the list of "Lethal ingredients" that can cause these networks to be dysfunctional.

B. Domenico: The Unidata governance consist of a user community that brainstorms all the opportunities and leads that from the users point of view. The compliment of that is that we have a policy committee that considers the resource constraints, focus areas and opportunities that should be given off. SO the group in this room, augmented with others could be the policy committee to make decisions on priorities and directions.

R.Husar: The Unidata model of governance is indeed very attractive, practical and effective approach. The entire data community operations that includes both software developments, outreach/education and governance is maintained by a strong financial and moral support by NSF, spanning the past 25 years. This AQ group does not yet have such sponsoring support, neither does it have a deep and broad buy-in by the potential users.

B.Domenico: Unidata did not start out with major NSF funding. There were meetings of the users that voiced the need for real time access to meteorological data particularly by the academic communities. At the same time there was a small group looking into the networking and communication technology. The initial proposals to NSF were modest. Your AQ community would also pursue a similar gradual act. NSF is pretty good at supporting proposal where the community get together to define their needs.

{Comment by Erin on NSF RFP on Cyber.. http://www.nsf.gov/pubs/2011/nsf11581/nsf11581.htm?WT.mc_id=USNSF_25&WT.mc_ev=click }

M.Schultz: A governance model just as outlined by Ben has been applied in the ??? projects. We did form a steering group composed of representatives of the partnering institutions. Decisions were made by this group on how to move forward. There is a programmers committee that needs to coordinate software development activities.

All these governance is organized without funding for the development of the network.

{RBH: Were the users represented in the governance? was the outcome a network of autonomous systems or a joint development of a model?}

T. Dye: Governance is imperative for us from a program point of view so that we know whats coming up over the next 10 years. Telecons and technical meetings would be useful to give life to this activity and make the operational folks more involved. This should be a major recommendation of this workshop. This should be of interest for funding agencies because ultimately this would save them money.

R. Husar: GEO is not the funding station. But recently, they have been exploring ways to use GEO as broker mediator between proposers with ideas and potential funding agencies. As one of its organs, the GEO AQ CoP could be a broker between proposers of specific network projects and potential funding agencies of such networking activities.

DLR Status, Impediments, Opportunities

M. Schultz: We now invite participants to share with us their current status of networking, their current impediments and opportunities. Lets start with DLR and Oleg. As we understand you have expressed interest in joining the AQ data network. Your group at DLR is not using the community WCS server but nevertheless you are offering data to the network using a WCS user interface.

O.Goussev: I can comment on technical aspects. Based on suggestions from NASA, we have implemented a WCS 1.0 server which was simpler to implement. If the AQ network can make use of WCS 1.0 then you can use our service now.

R.Husar: The AQ data network can indeed make use of DLR WCS service by adding a small converter/adapter at DataFed. Infact the GOME 2 satellite data are registered in the AQ community catalog. When a user accesses the GOME data it is obtained by calling the DLR WCS 1.0 server through the adapter piece. The limitations include the lack of the data access along the time dimension.

M.Schultz: From Rudy's comment I understand that if your server and the client are WCS 1.0 then they are interoperable. If the server is 1.0 and the client is 1.x they can still be interoperable through the adapter piece at the client or the mediator.

O.Goussev: In my opinion clients should be able to support interoperability with WCS 1.0, 1.1 and 2.0 as it emerge.

BDomenico:

RHusar: Interoperability may involve keeping the server protocols flexible which also means that the clients will have to be flexible to understand the different dialects. Conversely by making the servers harmonious, i.e by co sharing, the clients need only understand that particular dialect. The third approach used in the DataFed is for the clients to access all of the data services through a mediator which performs the adaptations and homogenization of the diverse servers. By examining the realities, we as a community and individuals should evaluate which of these network architectural approaches are suitable under different conditions. My feeling is that for now we should retain all three possibilities.

NILU Status, Impediments, Opportunities

PEckhardt: We have implemented the AQ Community WCS server at NILU as a prototype. ... server separate from the operational/production server. A small fraction of the production database is mirrored at the WCS prototype. The system has been running since the beginning of 2011. The served EMEP AQ monitoring data data has been Explored through the WCS client. With regard to the impediment, I have three main points:

• As many of you know, I had a problem with the WCS protocol. For our purposes, the data acces protocol must support 'point observation' data. The current WCS protocol is geared toward delivering spatial coverages, i.e the concentration of pollutants at specific discrete locations. However, beyond the concentration spatial or temporal coverage we also need to deliver considerable amount of metadata related to the monitoring station, the measurement method, and the sensor information. Over the past two days I have learned that new developments on the standards now allow the delivery of much richer pay load. We still need to evaluate if the new extended CF-netCDF binary file format will allow the incorporation of the most important metadata.

New possiblities into WCS convention . Put into the payload. We also need the

• Funding - conversation , funding .. no answer The fact that initially Kjetil and Assmund
• Special to NILU - immediately

STI

• • STI - commitment to adopt community server; need help on technical level

• • EEA - wants to add WCS service but needs to determine how to do this (e.g. decide which standard(s) to support)
  • CNR comment - INSPIRE can add stability to standards
  • NG - community definition of metadata, in particular for client development; how can INSPIRE etc. help us?
  • JRC - no impediments, all ready to go
• Mismatched funding / mismatched development cycles
  • NILU - must answer immediate needs; this drives development efforts
• Open source community efforts versus off-the-shelf software
  • Difficult to keep open source software going over the years
  • Require resources staff/$
  • DLR - limited resources - limit development efforts, not unwilling to install and run community server
  • NILU - limited funding, but maybe not impossible
  • STI - limited funding, need to prioritize
  • Is it reliable?
  • Don't trust open source
  • STI and EEA - must be certain that system survives ~10 years, not only on technical level but as a community
  • Unidata - backward compatibility is part of netcdf philosophy, not so in WCS (example multi-part mime wrapper); CF has slow development cycle on purpose, largely backward compatible although newer versions resolve ambiguities and therefore may break old code
• Unclear expectations/requirements of the community
• Need for clear, simple documentation/instructions
  • Group needs to add feedback/experience to documentation
  • Responsibility to maintain structure
  • STI - communication tools, repository+version control are critical elements
• Changing/evolving standards and approach - Interoperability burnout
  • Too many new things - how to know where to focus?
  • Governance needs - limited resources, growing needs
  • STI - structured community software development would probably work
  • Support vs. documentation vs. development
  • Organisation of community software development: formal vs. informal; need for some sort of steering group with a mandate to take decisions
  • Unidata - started as small grant to look at needs
• Challenges of real-time systems
  • Modifications needed to be made to the software for data coming in frequently
  • performance
  • STI - must persuade users to go through services rather than data downloads

• Insufficient user feedback/ user involvement

• Agreement on protocols (i.e. WCS x.y, W*S) vs. payload (i.e. adopt new netcdf-cf standard as community standard)