https://wiki.esipfed.org/w/api.php?action=feedcontributions&user=128.252.167.79&feedformat=atomEarth Science Information Partners (ESIP) - User contributions [en]2024-03-29T10:23:06ZUser contributionsMediaWiki 1.35.14https://wiki.esipfed.org/w/index.php?title=HTAP_Report,_Sub-Chap._6_-_Data/Info_System&diff=7995HTAP Report, Sub-Chap. 6 - Data/Info System2007-04-14T20:28:17Z<p>128.252.167.79: /* Introduction */</p>
<hr />
<div><noinclude>[[Integrated_Global_Dataset| < Back to Integrated Global Dataset on ESIP wiki]] <br></noinclude> <br />
<br />
This wiki page is inteded to be the collaborative workspace for Task Force members interested in the HTAP Information System development, implementation and use. The contents can be freely edited by anyone who is logged in. Comments and feedback can also be sent by email to rhusar@me.wustl.edu. The content of this page is is on the '''''Information System''''' that supposts Chapter 6. The subsections are action-oriented. A draft [[HTAP Report Chap. 6 - Jan 07 Outline]] is also on this wiki.<br />
<br />
==Introduction==<br />
The purpose of '''[http://www.htap.org/activities/2007_interim_report/TF%20HTAP%202007%20Interim%20Report%20Prospectus%20Oct%202006.pdf Chapter 6]''' is to discuss the need to integrate information from observations, models, and emissions inventories to better understand intercontinental transport. This section focuses on the information system in support of the integration of observations, emissions and models for HTAP. <br />
<br />
Recent developments in air quality monitoring, modeling and information technologies offer outstanding opportunities to fulfill the information needs for the HTAP integration effort. The data from surface-based air pollution monitoring networks now routinely provide spatio-temporal and chemical patterns of ozone and PM. Satellite sensors with global coverage and kilometer-scale spatial resolution now provide real-time snapshots which depict the pattern of industrial haze, smoke, dust, as well as some gaseous species in stunning detail. Detailed phisico-chemical models are now capable of simulating the spatial-temporal pollutant pattern on regional and global scales. The ‘terabytes’ of data from observations and models can now be stored, processed and delivered in near-real time. The instantaneous ‘horizontal’ diffusion of information via the Internet now permits, in principle, the delivery of the right information to the right people at the right place and time. Standardized computer-computer communication languages and Service-Oriented Architectures (SOA) now facilitate the flexible access, quality assurance and processing of raw data into high-grade ‘actionable’ knowledge suitable for HTAP policy decisions. Last but not least, the World Wide Web has opened the way to generous sharing of data and tools leading to collaborative analysis and virtual workgroups. Nevertheless, air quality data analyses and data model integration face significant hurdles. The section below present an architectural framework and implementation strategy for the proposed HTAP information system.<br />
<br />
==HTAP Information System==<br />
The term architecture here refers to both function and form of the HTAP Information System (HTAP IS). The key function of the HTAP IS is to provide information technology support to the researches performing their task while seeking to better understand transcontinental air pollution transport. <br />
<br />
It is a tools set that is to empower the participating and collaborating analysts. The HTAP IS will not compete with existing tools of its members. Rather it will embrace and leverage those resources through an open, collaborative federation philosophy. Its main contribution is to connect the TF participating analysts and their information resources <br />
<br />
The information system HTAP IS consists of the following: <br />
* Provide tools and methods for data access, processing and integration. This will be performed by the IT of the HTAP Federated Data System. <br />
* Facilitate open communication between the collaborating analysts. The IT technologies include wikis and other groupware, social software, blog, Skype, etc.. <br />
* Provide a shared workspace and where the above activities take place <br />
<br />
===Homogenize the data into an interoperable dataset===<br />
* facilitate access to the distributed data<br />
* ensure the seamless flow of data through interoperable interfaces<br />
* provide a set of basic tools for data exploration and processing<br />
<br />
[[Image:HTAP_IntDataSys.png|600px|left]] The primary goal of the data system is to allow the flow and integration of observational, emission and model data. The model evaluation requires that both the observations and if possible the emissions are fixed. For this reason it is desirable to prepare an integrated observational database to which the various model implementations can be compared to. On one hand, the integrated data set should be as inclusive as possible. On the other hand, such a goal needs to be tempered by many limitations that preclude a broad, inclusive data integration<br />
<br />
The proposed HTAP data system will be a hybrid combination of both distributed and fixed components as illustrated schematically in Figure ??. Both the data providers as well as the HTAP analysts-users will be distributed. However, they will be connected through an integrated HTAP database which should be a stable, virtually fixed database. The section below describes the main component of this distributed data system. <br />
<br />
The primary function of the HTAP IDS is to facilitate the creation of an integrated data set suitable for model evaluation and pollutant characterization.<br />
<br />
The multiple steps that are required for this functionality are shown on the left. The sequence of operations can be viewed as a value chain that transform the raw input data into a highly organized integrated data set. These value adding steps have to be performed for each candidate data set. <br />
<br />
The operations that prepare the integrated data set can be viewed as services that sequentially operate on the data stream. Each of the services with its clearly defined functionality and firmly specified service interface. In principle, the standards-based interface allows a connection of service chains using formal work flow software. The main benefit of Service Oriented Architecture (SOA) is that allows the building of agile application programs that can respond to changes in the data input conditions as well as the output requirements. <br />
<br />
The service oriented software architecture is an implementation of the System of Systems approach, which is the design philosophy of GEOSS. Each service can be operated by autonomous providers and the "system" that implements the service is behind the service interface combining the independent services constitutes System of Systems. In other words, following the SOA approach, not only the data providers are distributed but also the processing services as well. This flexible approach to distributed computing allows the distribution of labor (chain of services) in many different configurations. <br />
<br />
For instance, wrapping the existing data with standards based interfaces for data access can be performed for an entire cluster of data. This is the approach taken in the federated data system DataFed. Given a standard interface to a variety of data sets, the Quality Assurance service can be performed by another service provider that is seamlessly connected to the data access mediator. Similarly, the service that prepares a data set for data integration can be provided by another service in the data flow network. This flexibility offered through the chaining and orchestration of distributed, loosely coupled web services is the architectural support for the building of agile data systems for the support of future demanding applications.<br />
<br />
===Interoperability Standards and Data Wrapping===<br />
<br />
The methods and tools for model inter comparisons was a subject of a productive workshop at GRC Ispra in March 2006. European and American members of the HTAP TF presented their respective approaches as part of the model and data intercoperisons as part of their respective projects ENSEMBLES, Eurodelta, ACCENT, AEROCOM, GEMS, and DataFed. Several recommendations were made to improve future use of intercomparison data. The most important recommendation was to agree on a common data format, NetCDF CF conventions, which would allow a variety of tools to work with the data.<br />
<br />
Interoperable data access will be accomplished through the use of a suite of international standards. The naming of individual chemical parameters will follow the CF convention used by the Climate and Forecast (CF) communities. The existing names for atmospheric chemicals in the CF convention were inadequate to accommodate all the parameters used in the HTAP modeling. in order to remedy this shortcoming the list of standard names was extended by the HTAP community under leadership of C. Textor. She also became a member of the CF convention board that is the custodian of the standard names. The standard names for HTAP models were developed using a collaborative wiki workspace. <br />
<br />
The data transfer from providers to users also benefit from standard data formats. For modeling data the use of netCDF-CF as a standard format is recommended. The use of a standard physical data format and the CF naming conventions allows, in principle, the seamless connection between data provider and consumer services. It should be noted, however, that at this time the CF naming convention has only been developed for the model parameters and not for the various observational parameters. Also, the nedCDF CF is primarily used as a model data exchange format, while the transfer of surface monitoring data and satellite data is less standardized.<br />
<br />
The third aspect of data interoperability is a standard query language through which user services request specific data from the provider services. It is proposed that for the HTAP data information system adapts the Web Coverage Service (WCS) as the standard data query language. The WCS data access protocol is defined by the international Open Geospatial Consortium (OGC), which is also the key organization responsible for interoperability standards in GEOSS. Since the WCS protocol was originally developed for the GIS community, it was necessary to adapt it to the needs of "fluid Earth sciences". Members of the HTAP group have been actively participating in the development and testing of the WCS interoperability standards. <br />
<br />
Standards-based data access can be accomplished by ‘wrapping’ the heterogeneous data into standardized web services, callable through well-defined Internet protocols. The result of this ‘wrapping’ process is an array of homogeneous, virtual datasets that can be accessed through a standard query language and the returned data are packaged in standard format, directly usable by the consuming services.<br />
<br />
===Quality Assurance===<br />
This is just a place holder. Here we need to explain the quality assurance steps that are needed to prepare the HTAP Integrated Dataset.<br />
For true quality assurance and for data homogenization the data flow channels for individual data sets need to be evaluated in the context of other data.<br />
<br />
===Homogenization and Integration===<br />
The HTAP Integrated Dataset (HID) will be used to compare models to observations. It will be created from the proper combination of a variety of surface, upper air and satellite observations. However, before the inclusion into HID, each dataset will need to be scrutinized to make it suitable for model comparison. The scrutiny may include filtering, aggregation and possibly fusion operations.<br />
<br />
A good example is the 400-station AIRNOW network reporting hourly ozone and PM2.5 concentrations over the US. The AIRNOW network includes both urban sites that are strongly influenced by local sources. Urban stations need to be removed from HID dataset.<br />
<br />
The Service Oriented Architecture (SOA) allows connecting the web service components (e.g. services for data access, transformation, fusion, rendering, etc.) using work flow software.<br />
<br />
Developing the specific criteria and procedures for the HTAP integrated dataset will require the attention of a HTAP subgroup.<br />
<br />
==== Data Selection Criteria====<br />
Monitoring data for atmospheric constituents are now available from a variety of sources, not all of which are suitable for the integrated HTAP data set. Given the limited scope and resources of HTAP, it will be necessary to select a subset of the available observational data that would be most appropriate for the preparation of the 2009 assessment. A set of criteria for the data selection is given below.<br />
* The suitability criteria may include the measured parameters, their spatial extent and coverage density, as well as the time range and sampling frequency with major emphasis on data quality (defined as???).<br />
* The compilation of the global integrated data sets should initially focus on PM and ozone, including the gaseous precursors.<br />
* Subsequent data collection should also include observations on atmospheric mercury and persistent organic pollutants (POPS). <br />
* In order to cover hemispheric transport of air pollutants the data set system should accept and utilize data from outside the geographical scope of EMEP.<br />
* The data gathering should begin the compilation using the of existing databases in Europe (RETRO, TRADEOFF, QUANTIFY, CRATE, AEROCOM) and virtual federated systems in the US (DataFed, Giovanni and others). <br />
* TF data integration should also contribute to and benefit from other ongoing data integration efforts to integrate the global data resources, most notably, with ACCENT project in Europe, and similar efforts in the US.<br />
* Special emphasis should be placed on the collection of suitable vertical profiles from aircraft measurements as well as from surface and space-borne lidars.<br />
<br />
The evaluation of suitable observational data sets for model validation and fusion will require close interaction between the modeling and observational communities. An wiki workspace for open collaborative evaluation of different datasets is desirable.<br />
<br />
==Integrated Data Set==<br />
<br />
The providers of observational, emission, and modeling data are distributed. The individual data sets will be funneled into the integrated HTAP data set. The data transfer will be accomplished using standard data transfer protocols described above. Transferring data from the providers to users will include two operations for each data set: quality assurance and semantic transformation/integration.<br />
<br />
In the next, upcoming phase of HTAP the existing modeling and observing systems will be integrated to yield a deeper and more robust understanding of hemispheric transport. Both the modeling of the chemical constituents as well as the Earth observations used to document the hemispheric transport are currently being performed by individual projects and programs in the US and Europe. These constitute autonomous systems with well defined purpose and functionality. The key role of the Task Force is to assess the contributions of the individual systems and to integrate those into a System of Systems. It is believed that the GEO System of Systems architecture is an attractive framework for HTAP integration.<br />
<br />
====HTAP Information Network====<br />
[[Image:HTAP Network.png|600px|left]]<br />
<br />
* Federated Data System DataFed<br />
* NASA Data System Giovanni<br />
* Emission Data System NEISGEI<br />
* Juelich Data System<br />
* AeorCom<br />
* EMEP<br />
<br />
<br />
Data Networks. Connected hubs; Show Core HTAP network<br />
<br />
<br />
<br />
<br />
<br />
<br />
====HTAP Data Sets - need list of others====<br />
[http://datafedwiki.wustl.edu/index.php/2007-03-26:_Datasets_for_HTAP_through_DataFed See 20 selected datasets] in the federated data systems, DataFed<br />
* TOMS_AI_G - Satellite <br />
* SURF_MET - Surface <br />
* SEAW_US - Satellite <br />
* SCIAMACHYm - Satellite <br />
* RETRO ANTHRO - Emission <br />
* OnEarth JPL - Satellite <br />
* OMId - Satellite <br />
* NAAPS GLOBAL - Model <br />
* MOPITT Day - Satellite <br />
* MODISd G - Satellite <br />
* MODIS Global Fire - Satellite <br />
* MISRm G - Satellite <br />
* GOMEm G - Satellite <br />
* GOCART G OL - Model <br />
* EDGAR - Emission <br />
* CALIPSO - Satellite <br />
* AIRNOW - Surface <br />
* VIEWS OL - Surface <br />
* AERONETd - Surface <br />
* AEROCOM LOA - Model<br />
<br />
==Reconciliation and Integration of Observations, Emissions and Models==<br />
===Compare observations to models and emissions===<br />
<br />
===Characterize pollutant pattern and SSR=== <br />
===Develop real-time data assimilation===<br />
===Perform reanalysis with assimilated data===<br />
<br />
==Organizational Aspects==<br />
===HTAP Relationship to GEO and GEOSS===<br />
There is an outstanding opportunity to develop a mutually beneficial and supportive relationship between the activities of the HTAP Task Force and that of GEO. The Group of Earth Observations with its broad range of national and organizational members has developed a general architectural framework for turning Earth observations into societal benefits. The three main components of this architecture are models, and observations, which then feed into decision support systems used in a variety of societal decision making. This general GEO framework is well suited as an architectural guide to the HTAP program. However, this framework lacks specific guidelines and implementation details that are needed for practical Earth observing and decision support systems. <br />
<br />
[[Image:400px HTAP GEOSS Arch.png|400px|left]] The HTAP program provides an opportunity to extend the GEO framework features that arise in specific HTAP application context. In case of HTAP, the major activities are shown in the architectural diagram of Figure ?? The HTAP modeling is conducted through global scale chemical transport models that simulate or forecast the atmospheric chemical composition as a consequence of natural and human induced emissions. The observations arise from satellite, ground-based and airborne observations of chemical constituents and their hemispheric transport. The decision support system consists of the scientists as members of the HTAP task force, and the task force co-chairs as the intermediaries between LRTP and the TF. (Terry, Andre this description of the HTAP DSS needs your help). <br />
<br />
Representing the Decision Support System (DSS) for HTAP is an important aspect because it can guide the design and and implementation of the information system for the support of the decision processes. A higher resolution DSS design document is also beneficial as a communications channel for the interacting system of systems components. A domain-specific DSS may also allow the generalization and applicability to the design of DSS structures for similar application domains. <br />
<br />
The implementation of the GEO framework utilizes the concept of Global Observing System of Systems (GEOSS). Traditionally, Earth observations were performed by well defined systems such as specific satellites and monitoring networks which were designed and operated using systems engineering principles. However, GEO recognized that the understanding of Earth system requires the utilization and integration of the individual, autonomous systems. The key difference between the Systems and System of System approaches are highlighted in Table 1.<br />
<br />
The HTAP program can also be considered an early demonstration of the GEO concepts through its end to end approach. Both the atmospheric modeling, as well as the observations currently exist through the operation of existing modeling and observation systems. The Task Force has agreed organizing and evaluating those models, assembling and integrating the observational data sets and then reconciling the models with the observations will be the focus of the Phase II effort. The Task Force will also prepare and deliver a report to LRTP to aid its deliberations and its decision making processes. This sequence of activities constitutes an end to end approach that turns observations and models into actionable knowledge for societal decision making. One could say that this is an octagonal approach to more deliberate step by step development of GEOSS where in Phase I interoperability, in Phase II ???<br />
<br />
===HTAP Relationship to IGACO=== <br />
[[Image:IGACO Framework.png|300px|left]] The HTAP program can also have a mutually beneficial interaction with the Integrated Global Atmospheric Chemistry Observations (IGACO) program. The IGACO is part of the Integrated Global Observing Strategy (IGOS). IGACO proposes a specific framework for combining observational data and models. It also specifies the higher level information products that can be used for creating social benefit. <br />
<br />
In the IGACO framework, the data products from satellite, surface, and aircraft observations are collected for inclusion into an integrated data archive. A key component in the IGACO data flow is the mandatory quality assurance and QA QC that precedes the archiving. This is particularly important for HTAP where multi-sensory data from many different providers are to be combined into an integrated database. In the IGACO framework, a key output from the data system is a high integrated spatial-temporal data set which can be used in a multitude of applications that produce socal benefit. The goal of creating such an integrated data set is shared by the IGACO and the HTAP programs (?? Len, this section could benefit from your input)<br />
<br />
===HTAP Relationship with Other Programs===<br />
---<br />
---<br />
<br />
<br />
<br />
==Links==<br />
[[IGACO Framework]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=HTAP_Report,_Sub-Chap._6_-_Data/Info_System&diff=7978HTAP Report, Sub-Chap. 6 - Data/Info System2007-04-13T15:58:08Z<p>128.252.167.79: /* Homogenize the data into an interoperable dataset */</p>
<hr />
<div><noinclude>[[Integrated_Global_Dataset| < Back to Integrated Global Dataset on ESIP wiki]] <br></noinclude> <br />
<br />
This wiki page is inteded to be the collaborative workspace for Task Force members interested in the HTAP Information System development, implementation and use. The contents can be freely edited by anyone who is logged in. Comments and feedback can also be sent by email to rhusar@me.wustl.edu. The content of this page is is on the '''''Information System''''' that supposts Chapter 6. The subsections are action-oriented. A draft [[HTAP Report Chap. 6 - Jan 07 Outline]] is also on this wiki.<br />
<br />
==Introduction==<br />
The purpose of '''[http://www.htap.org/activities/2007_interim_report/TF%20HTAP%202007%20Interim%20Report%20Prospectus%20Oct%202006.pdf Chapter 6]''' is to discuss the need to integrate information from observations, models, and emissions inventories to better understand intercontinental transport. This section focuses on the information system that is to support the processes for the integration of observations, emissions and models for HTAP. <br />
<br />
Recent developments in air quality monitoring, modeling and information technologies offer outstanding opportunities to fulfill the information needs for the HTAP integration effort. The data from surface-based air pollution monitoring networks now routinely provide spatio-temporal and chemical patterns of ozone and PM. Satellite sensors with global coverage and kilometer-scale spatial resolution now provide real-time snapshots which depict the pattern of industrial haze, smoke and dust in stunning detail. The ‘terabytes’ of data from these surface and remote sensors can now be stored, processed and delivered in near-real time. The instantaneous ‘horizontal’ diffusion of information via the Internet now permits, in principle, the delivery of the right information to the right people at the right place and time. Standardized computer-computer communication languages and Service-Oriented Architectures (SOA) now facilitate the flexible access, quality assurance and processing of raw data into high-grade ‘actionable’ knowledge suitable for HTAP policy decissions. Last but not least, the World Wide Web has opened the way to generous sharing of data and tools leading to collaborative analysis and virtual workgroups. Nevertheless, air quality data analyses and data model integration face significant hurdles. The section below present an architectral framework and implementation strategyfor the proposed HTAP information system.<br />
<br />
==HTAP Information System==<br />
The term achitecture here refers to both function and form of the HTAP Informations System (HTAP IS). The key function of the HTAP IS is to provide information technology support to the researches performing their task while sseking to better understand transcontinental air pollution transport. <br />
<br />
It is a tools set that is to empower the participating and collaborating analysts. The HTAP IS will not compete with existing tools of its members. Rather it will embrace and levarage those resources through an open, collaborative federation philosophy. Its main contribution is to connect the TF participating analysts and their information resources <br />
<br />
The information system HTAP IS consits of the following: <br />
* Porovide tools and methods for data access, processing and integration. This will be performed by the IT of the HTAP Federated Data System. <br />
* Facilitate open communication between the collaboing analysts. The IT technologies include wikis and other groupware, social software, blog, Skype, etc.. <br />
* Provide a shared workspace and where the above activitie take place <br />
<br />
===Homogenize the data into an interoperable dataset===<br />
* facilitate accsess to the distributed data<br />
* ensure the seamless flow of data through interoperable interfaces<br />
* provide a set of basic tools for data exploration and processing<br />
<br />
[[Image:HTAP_IntDataSys.png|600px|left]] The primary goal of the data system is to allow the flow and integration of observational, emission and model data. The model evaluation requires that both the observations and if possible the emissions are fixed. For this reason it is desirable to prepare an integrated observational database to which the various model implementations can be compared to. On one hand, the integrated data set should be as inclusive as possible. On the other hand, such a goal needs to be tempered by many limitations that preclude a broad, inclusive data integration<br />
<br />
The proposed HTAP data system will be a hybrid combination of both distributed and fixed components as illustrated schematically in Figure ??. Both the data providers as well as the HTAP analysts-users will be distributed. However, they will be connected through an integrated HTAP database which should be a stable, virtually fixed database. The section below describes the main component of this distributed data system. <br />
<br />
The primary function of the HTAP IDS is to facilitate the creation of an integrated data set suitable for model evaluation and pollutant characterization.<br />
<br />
The multiple steps that are required for this functionality are shown on the left. The sequence of operations can be viewed as a value chain that transform the raw input data into a highly organized integrated data set. These value adding steps have to be performed for each candidate data set. <br />
<br />
The operations that prepare the integrated data set can be viewed as services that sequentially operate on the data stream. Each of the services with its clearly defined functionality and firmly specified service interface. In principle, the standards-based interface allows a connection of service chains using formal work flow software. The main benefit of Service Oriented Architecture (SOA) is that allows the building of agile application programs that can respond to changes in the data input conditions as well as the output requirements. <br />
<br />
The service oriented software architecture is an implementation of the System of Systems approach, which is the desgn phylosophy of GEOSS. Each service can be operated by autonomous providers and the "system" that implements the service is behind the service interfece combining the independent services constitutes System of Systems. In other words, following the SOA approach, not only the data providers are distributed but also the processing services as well. This flexible approach to distributed computing allows the distribution of labor (chain of services) in many different configurations. <br />
<br />
For instance, wrapping the existing data with standards based interfaces for data access can be performed for an entire cluster of data. This is the approach taken in the federated data system DataFed. Given a standard interface to a variety of data sets, the Quality Assurance service can be performed by another srvice provider that is seamlesly connected to the data access mediator. Similarly, the service that prepares a data set for data integration can be provided by another service in the data flow network. This flexibility offered through the chaining and orchestration of distributed, loosely coupled web services is the architectural support for the building of agile data systems for the support of future demanding applications.<br />
<br />
===Interoperability Standards and Data Wrapping===<br />
<br />
The methods and tools for model intercomparisons was a subject of a productive workshop at GRC Ispra in March 2006. European and American members of the HTAP TF presented their respective approaches as part of the model and data intercoperisons as part of their respecive projects ENSEMBLES, Eurodelta, ACCENT, AEROCOM, GEMS, and DataFed. Several recommendations were made to improve future use of intercomparison data. The most important recommendation was to agree on a common data format, NetCDF CF conventions, which would allow a variety of tools to work with the data.<br />
<br />
Interoperable data access will be accomplished through the use of a suite of international standards. The naming of individual chemical parameters will follow the CF convention used by the Climate and Forecast (CF) communities. The existing names for atmospheric chemicals in the CF convention were inadequate to accomodate all the parameters used in the HTAP modeling. in order to remedy this shortcomming the list of standard names was extended by the HTAP community under leadership of C. Textor. She also became a memeber of the CF convention board that is the custodian of the standard names. The standard names for HTAP models were developed using a collaborative wiki workspace. <br />
<br />
The data transfer from providers to users also benefit from standard data formats. For modeling data the use of netCDF-CF as a standard format is recomended. The use of a standard physical data format and the CF naming conventions allows, in principle, the seamless connection between data provider and consumer services. It should be noted, however, that at this time the CF naming convention has only been developed for the model parameters and not for the various observational parameters. Also, the nedCDF CF is primarily used as a model data exchange format, while the transfer of surface monitoring data and satellite data is less standardized.<br />
<br />
The third aspect of data interoperability is a standard query language through which user services request specific data from the provider services. It is proposed that for the HTAP data information system adapts the Web Coverage Service (WCS) as the standard data query language. The WCS data access protocol is defined by the international Open Geospatial Consortium (OGC), which is also the key organization responsible for interoperability standards in GEOSS. Since the WCS protocol was originally developed for the GIS community, it was necessary to adapt it to the needs of "fluid Earth sciences". Members of the HTAP group have been actively participating in the development and testing of the WCS interoperability standards. <br />
<br />
Standards-based data access can be accomplished by ‘wrapping’ the heterogeneous data into standardized web services, callable through well-defined Internet protocols. The result of this ‘wrapping’ process is an array of homogeneous, virtual datasets that can be accessed through a standard querry language and the returned data are packaged in standard format, directly usable by the consuming services.<br />
<br />
===Quality Assurance===<br />
This is just a place holder. Here we need to explain the quality assurance steps that are needed to prepare the HTAP Integrated Dataset.<br />
For true quality assurance and for data homogenization the data flow channels for individual data sets need to be evaluated in the context of other data.<br />
<br />
===Homogenization and Integration===<br />
The HTAP Integrated Dataset (HID) will be used to compare models to observations. It will be created from the proper combination of a variety of surface, upper air and satellite observations. However, before the inclusion into HID, each dataset will need to be scrutinezed to make it suitable for model comparison. The scrutiny may include filtering, aggregation and possibly fusion operations.<br />
<br />
A good example is the 400-station AIRNOW network reporting hourly ozone and PM2.5 concentrations over the US. The AIRNOW network includes both urban sites that are strongly influenced by local sources. Urban stations need to be removed from HID dataset.<br />
<br />
The Service Oriented Architecture (SOA) allows connecting the web service components (e.g. services for data access, transformation, fusion, rendering, etc.) using work flow software.<br />
<br />
Developing the specific criteria and procedures for the HTAP integrated dataset will require the attention of a HTAP subgroup.<br />
<br />
==== Data Selection Criteria====<br />
Monitoring data for atmospheric constituents are now available from a variety of sources, not all of which are suitable for the integrated HTAP data set. Given the limited scope and resources of HTAP, it will be necessary to select a subset of the available observational data that would be most appropriate for the preparation of the 2009 assessment. A set of criteria for the data selection is given below.<br />
* The suitability criteria may include the measured parameters, their spatial extent and coverage density, as well as the time range and sampling frequency with major emphasis on data quality (defined as???).<br />
* The compilation of the global integrated data sets should initially focus on PM and ozone, including the gaseous precursors.<br />
* Subsequent data collaction should also include observations on atmospheric mercury and persistent organic pollutants (POPS). <br />
* In order to cover hemispheric transport of air pollutants the data set system should accept and utilize data from outside the geographical scope of EMEP.<br />
* The data gathering should begin the compilation using the of existing databases in Europe (RETRO, TRADEOFF, QUANTIFY, CRATE, AEROCOM) and virtual federated systems in the US (DataFed, Giovanni and others). <br />
* TF data integration should also contribute to and benefit from other ongoing data integration efforts to integrate the global data resources, most notably, with ACCENT project in Europe, and similar efforts in the US.<br />
* Special emphasis should be placed on the collection of suitable vertical profiles from aircraft measurements as well as from surface and space-borne lidars.<br />
<br />
The evaluation of suitabile observational data sets for model validation and fusion will require close interaction between the modeling and observational communities. An wiki workspace for open collaborative evaluation of different datasets is desirebale.<br />
<br />
==Integrated Data Set==<br />
<br />
The providers of observational, emission, and modeling data are distributed. The individual data sets will be funeled into the integrated HTAP data set. This data set transfer will be accomplished using standard data transfer protocols. Transfering the data from the provider to the user will include two additional observations for each data set, quality assurance and semantic transformation.<br />
<br />
On the left are the inputs from observations, emissions, and model data.<br />
<br />
<br />
<br />
<br />
<br />
<br />
In order to accomplish the objectives set for the HTAP TF it is evident that it will require the System of Systems approach as envisioned by GEO. Both the modeling of the chemical constituents as well as the Earth observations used to document the hemispheric transport are being performed by individual autonomous projects and programs (Systems). A key function of the Task Force is to assess the contributions of the individual systems and then integrate those into a coherent assessment. Much of this interim report is in fact such an assessment of the current knowledge based on previous or past models and observations.<br />
<br />
In the next, upcoming phase of HTAP is to actually integrate the existing modeling and observing systems such that their combination and fusion can yield a deeper and more robust understanding of HTAP. since this next phase will involve integration of multiple relevant data sets, comparison and evaluation of multiple models, as well as the reconciliation of the observations and models to create the new and deeper understanding . In order to execute this extensive integration, it will be necessary to conduct this integration using a generally accepted framework as well as accepted scientific methods. It is believed that the GEO architecture and the System of Systems is an attractive approach to HTAP integration.<br />
<br />
The HTAP program can also be considered an early demonstration of the GEO concepts through its end to end approach. Both the atmospheric modeling, as well as the observations currently exist through the operation of existing modeling and observation systems. The Task Force has agreed organizing and evaluating those models, assembling and integrating the observational data sets and then reconciling the models with the observations will be the focus of the Phase II effort. The Task Force will also prepare and deliver a report to LRTP to aid its deliberations and its decision making processes. This sequence of activities constitutes an end to end approach that turns observations and models into actionable knowledge for societal decision making. One could say that this is an octagonal approach to more deliberate step by step development of GEOSS where in Phase I interoperability, in Phase II ???<br />
<br />
====HTAP Information Network====<br />
[[Image:HTAP Network.png|600px|left]]<br />
<br />
* Federated Data System DataFed<br />
* NASA Data System Giovanni<br />
* Emission Data System NEISGEI<br />
* Juelich Data System<br />
* AeorCom<br />
* EMEP<br />
<br />
<br />
Data Networks. Connected hubs; Show Core HTAP network<br />
<br />
<br />
<br />
<br />
<br />
<br />
====HTAP Data Sets - need list of others====<br />
[http://datafedwiki.wustl.edu/index.php/2007-03-26:_Datasets_for_HTAP_through_DataFed See 20 selected datasets] in the federated data systems, DataFed<br />
* TOMS_AI_G - Satellite <br />
* SURF_MET - Surface <br />
* SEAW_US - Satellite <br />
* SCIAMACHYm - Satellite <br />
* RETRO ANTHRO - Emission <br />
* OnEarth JPL - Satellite <br />
* OMId - Satellite <br />
* NAAPS GLOBAL - Model <br />
* MOPITT Day - Satellite <br />
* MODISd G - Satellite <br />
* MODIS Global Fire - Satellite <br />
* MISRm G - Satellite <br />
* GOMEm G - Satellite <br />
* GOCART G OL - Model <br />
* EDGAR - Emission <br />
* CALIPSO - Satellite <br />
* AIRNOW - Surface <br />
* VIEWS OL - Surface <br />
* AERONETd - Surface <br />
* AEROCOM LOA - Model<br />
<br />
==Reconciliation and Integration of Observations, Emissions and Models==<br />
===Compare observations to models and emissions===<br />
===Characterize pollutant pattern and SSR=== <br />
===Develop real-time data assimilation===<br />
===Perform reanalysis with assimilated data=== <br />
<br />
==Organizational Aspects==<br />
===HTAP Relationship to GEO and GEOSS===<br />
There is an outstanding opportunity to develop a mutually beneficial and supportive relationship between the activities of the HTAP Task Force and that of GEO. The Group of Earth Observations with its broad range of national and organizational members has developed a general architectural framework for turning Earth observations into societal benefits. The three main components of this architecture are models, and observations, which then feed into decision support systems used in a variety of societal decision making. This general GEO framework is well suited as an architectural guide to the HTAP program. However, this framework lacks specific guidelines and implementation details that are needed for practical Earth observing and decision support systems. <br />
<br />
[[Image:400px HTAP GEOSS Arch.png|400px|left]] The HTAP program provides an opportunity to extend the GEO framework features that arise in specific HTAP application context. In case of HTAP, the major activities are shown in the architectural diagram of Figure ?? The HTAP modeling is conducted through global scale chemical transport models that simulate or forecast the atmospheric chemical composition as a consequence of natural and human induced emissions. The observations arise from satellite, ground-based and airborne observations of chemical constituents and their hemispheric transport. The decision support system consists of the scientists as members of the HTAP task force, and the task force co-chairs as the intermediaries between LRTP and the TF. (Terry, Andre this description of the HTAP DSS needs your help). <br />
<br />
Representing the Decision Support System (DSS) for HTAP is an important aspect because it can guide the design and and implementation of the information system for the support of the decission processes. A higher resolution DSS design document is also beneficial as a communications channel for the interacting system of systems components. A domain-specific DSS may also allow the generalization and applicability to the design of DSS structures for similar application domains. <br />
<br />
The implementation of the GEO framework utilizes the concept of Global Observing System of Systems (GEOSS). Traditionally, Earth observations were performed by well defined systems such as specific satellites and monitoring networks which were designed and operated using systems engineering principles. However, GEO recognized that the understanding of Earth system requires the utilization and integration of the individual, autonomous systems. The key difference between the Systems and System of System approaches are highlighted in Table 1.<br />
<br />
===HTAP Relationship to IGACO=== <br />
[[Image:IGACO Framework.png|300px|left]] The HTAP program can also have a mutually beneficial interaction with the Integrated Global Atmospheric Chemistry Observations (IGACO) program. The IGACO is part of the Integrated Global Observing Strategy (IGOS). IGACO proposes a specific framework for combining observational data and models. It also specifies the higher level information products that can be used for creating social benefit. <br />
<br />
In the IGACO framework, the data products from satellite, surface, and aircraft observations are collected for inclusion into an integrated data archive. A key component in the IGACO data flow is the mandatory quality assurance and QA QC that precedes the archiving. This is particularly important for HTAP where multi-sensory data from many different providers are to be combined into an integrated database. In the IGACO framework, a key output from the data system is a high integrated spatial-temporal data set which can be used in a multitude of applications that produce socal benefit. The goal of creating such an integrated data set is shared by the IGACO and the HTAP programs (?? Len, this section could benefit from your input)<br />
<br />
===HTAP Relationship with Other Programs===<br />
---<br />
---<br />
<br />
<br />
<br />
==Links==<br />
[[IGACO Framework]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=HTAP_Report,_Sub-Chap._6_-_Data/Info_System&diff=7977HTAP Report, Sub-Chap. 6 - Data/Info System2007-04-13T15:51:46Z<p>128.252.167.79: /* Quality Assurance */</p>
<hr />
<div><noinclude>[[Integrated_Global_Dataset| < Back to Integrated Global Dataset on ESIP wiki]] <br></noinclude> <br />
<br />
This wiki page is inteded to be the collaborative workspace for Task Force members interested in the HTAP Information System development, implementation and use. The contents can be freely edited by anyone who is logged in. Comments and feedback can also be sent by email to rhusar@me.wustl.edu. The content of this page is is on the '''''Information System''''' that supposts Chapter 6. The subsections are action-oriented. A draft [[HTAP Report Chap. 6 - Jan 07 Outline]] is also on this wiki.<br />
<br />
==Introduction==<br />
The purpose of '''[http://www.htap.org/activities/2007_interim_report/TF%20HTAP%202007%20Interim%20Report%20Prospectus%20Oct%202006.pdf Chapter 6]''' is to discuss the need to integrate information from observations, models, and emissions inventories to better understand intercontinental transport. This section focuses on the information system that is to support the processes for the integration of observations, emissions and models for HTAP. <br />
<br />
Recent developments in air quality monitoring, modeling and information technologies offer outstanding opportunities to fulfill the information needs for the HTAP integration effort. The data from surface-based air pollution monitoring networks now routinely provide spatio-temporal and chemical patterns of ozone and PM. Satellite sensors with global coverage and kilometer-scale spatial resolution now provide real-time snapshots which depict the pattern of industrial haze, smoke and dust in stunning detail. The ‘terabytes’ of data from these surface and remote sensors can now be stored, processed and delivered in near-real time. The instantaneous ‘horizontal’ diffusion of information via the Internet now permits, in principle, the delivery of the right information to the right people at the right place and time. Standardized computer-computer communication languages and Service-Oriented Architectures (SOA) now facilitate the flexible access, quality assurance and processing of raw data into high-grade ‘actionable’ knowledge suitable for HTAP policy decissions. Last but not least, the World Wide Web has opened the way to generous sharing of data and tools leading to collaborative analysis and virtual workgroups. Nevertheless, air quality data analyses and data model integration face significant hurdles. The section below present an architectral framework and implementation strategyfor the proposed HTAP information system.<br />
<br />
==HTAP Information System==<br />
The term achitecture here refers to both function and form of the HTAP Informations System (HTAP IS). The key function of the HTAP IS is to provide information technology support to the researches performing their task while sseking to better understand transcontinental air pollution transport. <br />
<br />
It is a tools set that is to empower the participating and collaborating analysts. The HTAP IS will not compete with existing tools of its members. Rather it will embrace and levarage those resources through an open, collaborative federation philosophy. Its main contribution is to connect the TF participating analysts and their information resources <br />
<br />
The information system HTAP IS consits of the following: <br />
* Porovide tools and methods for data access, processing and integration. This will be performed by the IT of the HTAP Federated Data System. <br />
* Facilitate open communication between the collaboing analysts. The IT technologies include wikis and other groupware, social software, blog, Skype, etc.. <br />
* Provide a shared workspace and where the above activitie take place <br />
<br />
===Homogenize the data into an interoperable dataset===<br />
* facilitate accsess to the distributed data<br />
* ensure the seamless flow of data through interoperable interfaces<br />
* provide a set of basic tools for data exploration and processing<br />
<br />
[[Image:HTAP_IntDataSys.png|600px|left]] The primary goal of the data system is to allow the flow and integration of observational, emission and model data. The model evaluation requires that both the observations and if possible the emissions are fixed. For this reason it is desirable to prepare an integrated observational database to which the various model implementations can be compared to. On one hand, the integrated data set should be as inclusive as possible. On the other hand, such a goal needs to be tempered by many limitations that preclude a broad, inclusive data integration<br />
<br />
The proposed HTAP data system will be a hybrid combination of both distributed and fixed components as illustrated schematically in Figure ??. Both the data providers as well as the HTAP analysts-users will be distributed. However, they will be connected through an integrated HTAP database which should be a stable, virtually fixed database. The section below describes the main component of this distributed data system. <br />
<br />
The primary function of the HTAP IDS is to facilitate the creation of an integrated data set suitable for model evaluation and pollutant characterization.<br />
<br />
The multiple steps that are required for this functionality are shown on the left. The sequence of operations can be viewed as a value chain that transform the raw input data into a highly organized integrated data set. These value adding steps have to be performed for each candidate data set. <br />
<br />
The operations that prepare the integrated data set can be viewed as services that operate on the data stream using the principles of modern Service Oriented Architecure (SOA). Each of the services with its clearly defined functionality and firmly specified service interface. In principle, the standards-based interface allows a connection of service chains using formal work flow software. The main benefit of service oriented architecture is that allows the building of agile application programs that can respond to changes in the data input conditions as well as the output requirements. The service oriented software architecture is an implementation of the System of Systems approach. Furthermore, since each of the services can be operated by autonomous providers and the "system" that implements the service is behind the service interfece. In other words, following the SOA approach, not only the data providers are distributed but also the processing services as well. This flexible approach to distributed computing allows the distribution of labor (chain of services) in many different configurations. For instance, wrapping the existing data with standards based interfaces for data access can be performed for an entire cluster of data. This is the approach taken in the federated data system DataFed. Given a standard interface to a variety of data sets, the Quality Assurance service can be performed by another srvice provider that is seamlesly connected to the data access mediator. Similarly, the service that prepares a data set for data integration can be provided by another service in the data flow network. The flexibility offered through the chaining and orchestration of distributed, loosely coupled web services is the architectural support for the building of agile data systems.<br />
<br />
===Interoperability Standards and Data Wrapping===<br />
<br />
The methods and tools for model intercomparisons was a subject of a productive workshop at GRC Ispra in March 2006. European and American members of the HTAP TF presented their respective approaches as part of the model and data intercoperisons as part of their respecive projects ENSEMBLES, Eurodelta, ACCENT, AEROCOM, GEMS, and DataFed. Several recommendations were made to improve future use of intercomparison data. The most important recommendation was to agree on a common data format, NetCDF CF conventions, which would allow a variety of tools to work with the data.<br />
<br />
Interoperable data access will be accomplished through the use of a suite of international standards. The naming of individual chemical parameters will follow the CF convention used by the Climate and Forecast (CF) communities. The existing names for atmospheric chemicals in the CF convention were inadequate to accomodate all the parameters used in the HTAP modeling. in order to remedy this shortcomming the list of standard names was extended by the HTAP community under leadership of C. Textor. She also became a memeber of the CF convention board that is the custodian of the standard names. The standard names for HTAP models were developed using a collaborative wiki workspace. <br />
<br />
The data transfer from providers to users also benefit from standard data formats. For modeling data the use of netCDF-CF as a standard format is recomended. The use of a standard physical data format and the CF naming conventions allows, in principle, the seamless connection between data provider and consumer services. It should be noted, however, that at this time the CF naming convention has only been developed for the model parameters and not for the various observational parameters. Also, the nedCDF CF is primarily used as a model data exchange format, while the transfer of surface monitoring data and satellite data is less standardized.<br />
<br />
The third aspect of data interoperability is a standard query language through which user services request specific data from the provider services. It is proposed that for the HTAP data information system adapts the Web Coverage Service (WCS) as the standard data query language. The WCS data access protocol is defined by the international Open Geospatial Consortium (OGC), which is also the key organization responsible for interoperability standards in GEOSS. Since the WCS protocol was originally developed for the GIS community, it was necessary to adapt it to the needs of "fluid Earth sciences". Members of the HTAP group have been actively participating in the development and testing of the WCS interoperability standards. <br />
<br />
Standards-based data access can be accomplished by ‘wrapping’ the heterogeneous data into standardized web services, callable through well-defined Internet protocols. The result of this ‘wrapping’ process is an array of homogeneous, virtual datasets that can be accessed through a standard querry language and the returned data are packaged in standard format, directly usable by the consuming services.<br />
<br />
===Quality Assurance===<br />
This is just a place holder. Here we need to explain the quality assurance steps that are needed to prepare the HTAP Integrated Dataset.<br />
For true quality assurance and for data homogenization the data flow channels for individual data sets need to be evaluated in the context of other data.<br />
<br />
===Homogenization and Integration===<br />
The HTAP Integrated Dataset (HID) will be used to compare models to observations. It will be created from the proper combination of a variety of surface, upper air and satellite observations. However, before the inclusion into HID, each dataset will need to be scrutinezed to make it suitable for model comparison. The scrutiny may include filtering, aggregation and possibly fusion operations.<br />
<br />
A good example is the 400-station AIRNOW network reporting hourly ozone and PM2.5 concentrations over the US. The AIRNOW network includes both urban sites that are strongly influenced by local sources. Urban stations need to be removed from HID dataset.<br />
<br />
The Service Oriented Architecture (SOA) allows connecting the web service components (e.g. services for data access, transformation, fusion, rendering, etc.) using work flow software.<br />
<br />
Developing the specific criteria and procedures for the HTAP integrated dataset will require the attention of a HTAP subgroup.<br />
<br />
==== Data Selection Criteria====<br />
Monitoring data for atmospheric constituents are now available from a variety of sources, not all of which are suitable for the integrated HTAP data set. Given the limited scope and resources of HTAP, it will be necessary to select a subset of the available observational data that would be most appropriate for the preparation of the 2009 assessment. A set of criteria for the data selection is given below.<br />
* The suitability criteria may include the measured parameters, their spatial extent and coverage density, as well as the time range and sampling frequency with major emphasis on data quality (defined as???).<br />
* The compilation of the global integrated data sets should initially focus on PM and ozone, including the gaseous precursors.<br />
* Subsequent data collaction should also include observations on atmospheric mercury and persistent organic pollutants (POPS). <br />
* In order to cover hemispheric transport of air pollutants the data set system should accept and utilize data from outside the geographical scope of EMEP.<br />
* The data gathering should begin the compilation using the of existing databases in Europe (RETRO, TRADEOFF, QUANTIFY, CRATE, AEROCOM) and virtual federated systems in the US (DataFed, Giovanni and others). <br />
* TF data integration should also contribute to and benefit from other ongoing data integration efforts to integrate the global data resources, most notably, with ACCENT project in Europe, and similar efforts in the US.<br />
* Special emphasis should be placed on the collection of suitable vertical profiles from aircraft measurements as well as from surface and space-borne lidars.<br />
<br />
The evaluation of suitabile observational data sets for model validation and fusion will require close interaction between the modeling and observational communities. An wiki workspace for open collaborative evaluation of different datasets is desirebale.<br />
<br />
==Integrated Data Set==<br />
<br />
The providers of observational, emission, and modeling data are distributed. The individual data sets will be funeled into the integrated HTAP data set. This data set transfer will be accomplished using standard data transfer protocols. Transfering the data from the provider to the user will include two additional observations for each data set, quality assurance and semantic transformation.<br />
<br />
On the left are the inputs from observations, emissions, and model data.<br />
<br />
<br />
<br />
<br />
<br />
<br />
In order to accomplish the objectives set for the HTAP TF it is evident that it will require the System of Systems approach as envisioned by GEO. Both the modeling of the chemical constituents as well as the Earth observations used to document the hemispheric transport are being performed by individual autonomous projects and programs (Systems). A key function of the Task Force is to assess the contributions of the individual systems and then integrate those into a coherent assessment. Much of this interim report is in fact such an assessment of the current knowledge based on previous or past models and observations.<br />
<br />
In the next, upcoming phase of HTAP is to actually integrate the existing modeling and observing systems such that their combination and fusion can yield a deeper and more robust understanding of HTAP. since this next phase will involve integration of multiple relevant data sets, comparison and evaluation of multiple models, as well as the reconciliation of the observations and models to create the new and deeper understanding . In order to execute this extensive integration, it will be necessary to conduct this integration using a generally accepted framework as well as accepted scientific methods. It is believed that the GEO architecture and the System of Systems is an attractive approach to HTAP integration.<br />
<br />
The HTAP program can also be considered an early demonstration of the GEO concepts through its end to end approach. Both the atmospheric modeling, as well as the observations currently exist through the operation of existing modeling and observation systems. The Task Force has agreed organizing and evaluating those models, assembling and integrating the observational data sets and then reconciling the models with the observations will be the focus of the Phase II effort. The Task Force will also prepare and deliver a report to LRTP to aid its deliberations and its decision making processes. This sequence of activities constitutes an end to end approach that turns observations and models into actionable knowledge for societal decision making. One could say that this is an octagonal approach to more deliberate step by step development of GEOSS where in Phase I interoperability, in Phase II ???<br />
<br />
====HTAP Information Network====<br />
[[Image:HTAP Network.png|600px|left]]<br />
<br />
* Federated Data System DataFed<br />
* NASA Data System Giovanni<br />
* Emission Data System NEISGEI<br />
* Juelich Data System<br />
* AeorCom<br />
* EMEP<br />
<br />
<br />
Data Networks. Connected hubs; Show Core HTAP network<br />
<br />
<br />
<br />
<br />
<br />
<br />
====HTAP Data Sets - need list of others====<br />
[http://datafedwiki.wustl.edu/index.php/2007-03-26:_Datasets_for_HTAP_through_DataFed See 20 selected datasets] in the federated data systems, DataFed<br />
* TOMS_AI_G - Satellite <br />
* SURF_MET - Surface <br />
* SEAW_US - Satellite <br />
* SCIAMACHYm - Satellite <br />
* RETRO ANTHRO - Emission <br />
* OnEarth JPL - Satellite <br />
* OMId - Satellite <br />
* NAAPS GLOBAL - Model <br />
* MOPITT Day - Satellite <br />
* MODISd G - Satellite <br />
* MODIS Global Fire - Satellite <br />
* MISRm G - Satellite <br />
* GOMEm G - Satellite <br />
* GOCART G OL - Model <br />
* EDGAR - Emission <br />
* CALIPSO - Satellite <br />
* AIRNOW - Surface <br />
* VIEWS OL - Surface <br />
* AERONETd - Surface <br />
* AEROCOM LOA - Model<br />
<br />
==Reconciliation and Integration of Observations, Emissions and Models==<br />
===Compare observations to models and emissions===<br />
===Characterize pollutant pattern and SSR=== <br />
===Develop real-time data assimilation===<br />
===Perform reanalysis with assimilated data=== <br />
<br />
==Organizational Aspects==<br />
===HTAP Relationship to GEO and GEOSS===<br />
There is an outstanding opportunity to develop a mutually beneficial and supportive relationship between the activities of the HTAP Task Force and that of GEO. The Group of Earth Observations with its broad range of national and organizational members has developed a general architectural framework for turning Earth observations into societal benefits. The three main components of this architecture are models, and observations, which then feed into decision support systems used in a variety of societal decision making. This general GEO framework is well suited as an architectural guide to the HTAP program. However, this framework lacks specific guidelines and implementation details that are needed for practical Earth observing and decision support systems. <br />
<br />
[[Image:400px HTAP GEOSS Arch.png|400px|left]] The HTAP program provides an opportunity to extend the GEO framework features that arise in specific HTAP application context. In case of HTAP, the major activities are shown in the architectural diagram of Figure ?? The HTAP modeling is conducted through global scale chemical transport models that simulate or forecast the atmospheric chemical composition as a consequence of natural and human induced emissions. The observations arise from satellite, ground-based and airborne observations of chemical constituents and their hemispheric transport. The decision support system consists of the scientists as members of the HTAP task force, and the task force co-chairs as the intermediaries between LRTP and the TF. (Terry, Andre this description of the HTAP DSS needs your help). <br />
<br />
Representing the Decision Support System (DSS) for HTAP is an important aspect because it can guide the design and and implementation of the information system for the support of the decission processes. A higher resolution DSS design document is also beneficial as a communications channel for the interacting system of systems components. A domain-specific DSS may also allow the generalization and applicability to the design of DSS structures for similar application domains. <br />
<br />
The implementation of the GEO framework utilizes the concept of Global Observing System of Systems (GEOSS). Traditionally, Earth observations were performed by well defined systems such as specific satellites and monitoring networks which were designed and operated using systems engineering principles. However, GEO recognized that the understanding of Earth system requires the utilization and integration of the individual, autonomous systems. The key difference between the Systems and System of System approaches are highlighted in Table 1.<br />
<br />
===HTAP Relationship to IGACO=== <br />
[[Image:IGACO Framework.png|300px|left]] The HTAP program can also have a mutually beneficial interaction with the Integrated Global Atmospheric Chemistry Observations (IGACO) program. The IGACO is part of the Integrated Global Observing Strategy (IGOS). IGACO proposes a specific framework for combining observational data and models. It also specifies the higher level information products that can be used for creating social benefit. <br />
<br />
In the IGACO framework, the data products from satellite, surface, and aircraft observations are collected for inclusion into an integrated data archive. A key component in the IGACO data flow is the mandatory quality assurance and QA QC that precedes the archiving. This is particularly important for HTAP where multi-sensory data from many different providers are to be combined into an integrated database. In the IGACO framework, a key output from the data system is a high integrated spatial-temporal data set which can be used in a multitude of applications that produce socal benefit. The goal of creating such an integrated data set is shared by the IGACO and the HTAP programs (?? Len, this section could benefit from your input)<br />
<br />
===HTAP Relationship with Other Programs===<br />
---<br />
---<br />
<br />
<br />
<br />
==Links==<br />
[[IGACO Framework]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=HTAP_Report,_Sub-Chap._6_-_Data/Info_System&diff=7976HTAP Report, Sub-Chap. 6 - Data/Info System2007-04-13T15:51:31Z<p>128.252.167.79: /* Homogenize the data into an interoperable dataset */</p>
<hr />
<div><noinclude>[[Integrated_Global_Dataset| < Back to Integrated Global Dataset on ESIP wiki]] <br></noinclude> <br />
<br />
This wiki page is inteded to be the collaborative workspace for Task Force members interested in the HTAP Information System development, implementation and use. The contents can be freely edited by anyone who is logged in. Comments and feedback can also be sent by email to rhusar@me.wustl.edu. The content of this page is is on the '''''Information System''''' that supposts Chapter 6. The subsections are action-oriented. A draft [[HTAP Report Chap. 6 - Jan 07 Outline]] is also on this wiki.<br />
<br />
==Introduction==<br />
The purpose of '''[http://www.htap.org/activities/2007_interim_report/TF%20HTAP%202007%20Interim%20Report%20Prospectus%20Oct%202006.pdf Chapter 6]''' is to discuss the need to integrate information from observations, models, and emissions inventories to better understand intercontinental transport. This section focuses on the information system that is to support the processes for the integration of observations, emissions and models for HTAP. <br />
<br />
Recent developments in air quality monitoring, modeling and information technologies offer outstanding opportunities to fulfill the information needs for the HTAP integration effort. The data from surface-based air pollution monitoring networks now routinely provide spatio-temporal and chemical patterns of ozone and PM. Satellite sensors with global coverage and kilometer-scale spatial resolution now provide real-time snapshots which depict the pattern of industrial haze, smoke and dust in stunning detail. The ‘terabytes’ of data from these surface and remote sensors can now be stored, processed and delivered in near-real time. The instantaneous ‘horizontal’ diffusion of information via the Internet now permits, in principle, the delivery of the right information to the right people at the right place and time. Standardized computer-computer communication languages and Service-Oriented Architectures (SOA) now facilitate the flexible access, quality assurance and processing of raw data into high-grade ‘actionable’ knowledge suitable for HTAP policy decissions. Last but not least, the World Wide Web has opened the way to generous sharing of data and tools leading to collaborative analysis and virtual workgroups. Nevertheless, air quality data analyses and data model integration face significant hurdles. The section below present an architectral framework and implementation strategyfor the proposed HTAP information system.<br />
<br />
==HTAP Information System==<br />
The term achitecture here refers to both function and form of the HTAP Informations System (HTAP IS). The key function of the HTAP IS is to provide information technology support to the researches performing their task while sseking to better understand transcontinental air pollution transport. <br />
<br />
It is a tools set that is to empower the participating and collaborating analysts. The HTAP IS will not compete with existing tools of its members. Rather it will embrace and levarage those resources through an open, collaborative federation philosophy. Its main contribution is to connect the TF participating analysts and their information resources <br />
<br />
The information system HTAP IS consits of the following: <br />
* Porovide tools and methods for data access, processing and integration. This will be performed by the IT of the HTAP Federated Data System. <br />
* Facilitate open communication between the collaboing analysts. The IT technologies include wikis and other groupware, social software, blog, Skype, etc.. <br />
* Provide a shared workspace and where the above activitie take place <br />
<br />
===Homogenize the data into an interoperable dataset===<br />
* facilitate accsess to the distributed data<br />
* ensure the seamless flow of data through interoperable interfaces<br />
* provide a set of basic tools for data exploration and processing<br />
<br />
[[Image:HTAP_IntDataSys.png|600px|left]] The primary goal of the data system is to allow the flow and integration of observational, emission and model data. The model evaluation requires that both the observations and if possible the emissions are fixed. For this reason it is desirable to prepare an integrated observational database to which the various model implementations can be compared to. On one hand, the integrated data set should be as inclusive as possible. On the other hand, such a goal needs to be tempered by many limitations that preclude a broad, inclusive data integration<br />
<br />
The proposed HTAP data system will be a hybrid combination of both distributed and fixed components as illustrated schematically in Figure ??. Both the data providers as well as the HTAP analysts-users will be distributed. However, they will be connected through an integrated HTAP database which should be a stable, virtually fixed database. The section below describes the main component of this distributed data system. <br />
<br />
The primary function of the HTAP IDS is to facilitate the creation of an integrated data set suitable for model evaluation and pollutant characterization.<br />
<br />
The multiple steps that are required for this functionality are shown on the left. The sequence of operations can be viewed as a value chain that transform the raw input data into a highly organized integrated data set. These value adding steps have to be performed for each candidate data set. <br />
<br />
The operations that prepare the integrated data set can be viewed as services that operate on the data stream using the principles of modern Service Oriented Architecure (SOA). Each of the services with its clearly defined functionality and firmly specified service interface. In principle, the standards-based interface allows a connection of service chains using formal work flow software. The main benefit of service oriented architecture is that allows the building of agile application programs that can respond to changes in the data input conditions as well as the output requirements. The service oriented software architecture is an implementation of the System of Systems approach. Furthermore, since each of the services can be operated by autonomous providers and the "system" that implements the service is behind the service interfece. In other words, following the SOA approach, not only the data providers are distributed but also the processing services as well. This flexible approach to distributed computing allows the distribution of labor (chain of services) in many different configurations. For instance, wrapping the existing data with standards based interfaces for data access can be performed for an entire cluster of data. This is the approach taken in the federated data system DataFed. Given a standard interface to a variety of data sets, the Quality Assurance service can be performed by another srvice provider that is seamlesly connected to the data access mediator. Similarly, the service that prepares a data set for data integration can be provided by another service in the data flow network. The flexibility offered through the chaining and orchestration of distributed, loosely coupled web services is the architectural support for the building of agile data systems.<br />
<br />
===Interoperability Standards and Data Wrapping===<br />
<br />
The methods and tools for model intercomparisons was a subject of a productive workshop at GRC Ispra in March 2006. European and American members of the HTAP TF presented their respective approaches as part of the model and data intercoperisons as part of their respecive projects ENSEMBLES, Eurodelta, ACCENT, AEROCOM, GEMS, and DataFed. Several recommendations were made to improve future use of intercomparison data. The most important recommendation was to agree on a common data format, NetCDF CF conventions, which would allow a variety of tools to work with the data.<br />
<br />
Interoperable data access will be accomplished through the use of a suite of international standards. The naming of individual chemical parameters will follow the CF convention used by the Climate and Forecast (CF) communities. The existing names for atmospheric chemicals in the CF convention were inadequate to accomodate all the parameters used in the HTAP modeling. in order to remedy this shortcomming the list of standard names was extended by the HTAP community under leadership of C. Textor. She also became a memeber of the CF convention board that is the custodian of the standard names. The standard names for HTAP models were developed using a collaborative wiki workspace. <br />
<br />
The data transfer from providers to users also benefit from standard data formats. For modeling data the use of netCDF-CF as a standard format is recomended. The use of a standard physical data format and the CF naming conventions allows, in principle, the seamless connection between data provider and consumer services. It should be noted, however, that at this time the CF naming convention has only been developed for the model parameters and not for the various observational parameters. Also, the nedCDF CF is primarily used as a model data exchange format, while the transfer of surface monitoring data and satellite data is less standardized.<br />
<br />
The third aspect of data interoperability is a standard query language through which user services request specific data from the provider services. It is proposed that for the HTAP data information system adapts the Web Coverage Service (WCS) as the standard data query language. The WCS data access protocol is defined by the international Open Geospatial Consortium (OGC), which is also the key organization responsible for interoperability standards in GEOSS. Since the WCS protocol was originally developed for the GIS community, it was necessary to adapt it to the needs of "fluid Earth sciences". Members of the HTAP group have been actively participating in the development and testing of the WCS interoperability standards. <br />
<br />
Standards-based data access can be accomplished by ‘wrapping’ the heterogeneous data into standardized web services, callable through well-defined Internet protocols. The result of this ‘wrapping’ process is an array of homogeneous, virtual datasets that can be accessed through a standard querry language and the returned data are packaged in standard format, directly usable by the consuming services.<br />
<br />
===Quality Assurance===<br />
This is just a place holder. Here we need to explain the quality assurance steps that are needed to prepare the HTAP Integrated Dataset.<br />
<br />
===Homogenization and Integration===<br />
The HTAP Integrated Dataset (HID) will be used to compare models to observations. It will be created from the proper combination of a variety of surface, upper air and satellite observations. However, before the inclusion into HID, each dataset will need to be scrutinezed to make it suitable for model comparison. The scrutiny may include filtering, aggregation and possibly fusion operations.<br />
<br />
A good example is the 400-station AIRNOW network reporting hourly ozone and PM2.5 concentrations over the US. The AIRNOW network includes both urban sites that are strongly influenced by local sources. Urban stations need to be removed from HID dataset.<br />
<br />
The Service Oriented Architecture (SOA) allows connecting the web service components (e.g. services for data access, transformation, fusion, rendering, etc.) using work flow software.<br />
<br />
Developing the specific criteria and procedures for the HTAP integrated dataset will require the attention of a HTAP subgroup.<br />
<br />
==== Data Selection Criteria====<br />
Monitoring data for atmospheric constituents are now available from a variety of sources, not all of which are suitable for the integrated HTAP data set. Given the limited scope and resources of HTAP, it will be necessary to select a subset of the available observational data that would be most appropriate for the preparation of the 2009 assessment. A set of criteria for the data selection is given below.<br />
* The suitability criteria may include the measured parameters, their spatial extent and coverage density, as well as the time range and sampling frequency with major emphasis on data quality (defined as???).<br />
* The compilation of the global integrated data sets should initially focus on PM and ozone, including the gaseous precursors.<br />
* Subsequent data collaction should also include observations on atmospheric mercury and persistent organic pollutants (POPS). <br />
* In order to cover hemispheric transport of air pollutants the data set system should accept and utilize data from outside the geographical scope of EMEP.<br />
* The data gathering should begin the compilation using the of existing databases in Europe (RETRO, TRADEOFF, QUANTIFY, CRATE, AEROCOM) and virtual federated systems in the US (DataFed, Giovanni and others). <br />
* TF data integration should also contribute to and benefit from other ongoing data integration efforts to integrate the global data resources, most notably, with ACCENT project in Europe, and similar efforts in the US.<br />
* Special emphasis should be placed on the collection of suitable vertical profiles from aircraft measurements as well as from surface and space-borne lidars.<br />
<br />
The evaluation of suitabile observational data sets for model validation and fusion will require close interaction between the modeling and observational communities. An wiki workspace for open collaborative evaluation of different datasets is desirebale.<br />
<br />
==Integrated Data Set==<br />
<br />
The providers of observational, emission, and modeling data are distributed. The individual data sets will be funeled into the integrated HTAP data set. This data set transfer will be accomplished using standard data transfer protocols. Transfering the data from the provider to the user will include two additional observations for each data set, quality assurance and semantic transformation.<br />
<br />
On the left are the inputs from observations, emissions, and model data.<br />
<br />
<br />
<br />
<br />
<br />
<br />
In order to accomplish the objectives set for the HTAP TF it is evident that it will require the System of Systems approach as envisioned by GEO. Both the modeling of the chemical constituents as well as the Earth observations used to document the hemispheric transport are being performed by individual autonomous projects and programs (Systems). A key function of the Task Force is to assess the contributions of the individual systems and then integrate those into a coherent assessment. Much of this interim report is in fact such an assessment of the current knowledge based on previous or past models and observations.<br />
<br />
In the next, upcoming phase of HTAP is to actually integrate the existing modeling and observing systems such that their combination and fusion can yield a deeper and more robust understanding of HTAP. since this next phase will involve integration of multiple relevant data sets, comparison and evaluation of multiple models, as well as the reconciliation of the observations and models to create the new and deeper understanding . In order to execute this extensive integration, it will be necessary to conduct this integration using a generally accepted framework as well as accepted scientific methods. It is believed that the GEO architecture and the System of Systems is an attractive approach to HTAP integration.<br />
<br />
The HTAP program can also be considered an early demonstration of the GEO concepts through its end to end approach. Both the atmospheric modeling, as well as the observations currently exist through the operation of existing modeling and observation systems. The Task Force has agreed organizing and evaluating those models, assembling and integrating the observational data sets and then reconciling the models with the observations will be the focus of the Phase II effort. The Task Force will also prepare and deliver a report to LRTP to aid its deliberations and its decision making processes. This sequence of activities constitutes an end to end approach that turns observations and models into actionable knowledge for societal decision making. One could say that this is an octagonal approach to more deliberate step by step development of GEOSS where in Phase I interoperability, in Phase II ???<br />
<br />
====HTAP Information Network====<br />
[[Image:HTAP Network.png|600px|left]]<br />
<br />
* Federated Data System DataFed<br />
* NASA Data System Giovanni<br />
* Emission Data System NEISGEI<br />
* Juelich Data System<br />
* AeorCom<br />
* EMEP<br />
<br />
<br />
Data Networks. Connected hubs; Show Core HTAP network<br />
<br />
<br />
<br />
<br />
<br />
<br />
====HTAP Data Sets - need list of others====<br />
[http://datafedwiki.wustl.edu/index.php/2007-03-26:_Datasets_for_HTAP_through_DataFed See 20 selected datasets] in the federated data systems, DataFed<br />
* TOMS_AI_G - Satellite <br />
* SURF_MET - Surface <br />
* SEAW_US - Satellite <br />
* SCIAMACHYm - Satellite <br />
* RETRO ANTHRO - Emission <br />
* OnEarth JPL - Satellite <br />
* OMId - Satellite <br />
* NAAPS GLOBAL - Model <br />
* MOPITT Day - Satellite <br />
* MODISd G - Satellite <br />
* MODIS Global Fire - Satellite <br />
* MISRm G - Satellite <br />
* GOMEm G - Satellite <br />
* GOCART G OL - Model <br />
* EDGAR - Emission <br />
* CALIPSO - Satellite <br />
* AIRNOW - Surface <br />
* VIEWS OL - Surface <br />
* AERONETd - Surface <br />
* AEROCOM LOA - Model<br />
<br />
==Reconciliation and Integration of Observations, Emissions and Models==<br />
===Compare observations to models and emissions===<br />
===Characterize pollutant pattern and SSR=== <br />
===Develop real-time data assimilation===<br />
===Perform reanalysis with assimilated data=== <br />
<br />
==Organizational Aspects==<br />
===HTAP Relationship to GEO and GEOSS===<br />
There is an outstanding opportunity to develop a mutually beneficial and supportive relationship between the activities of the HTAP Task Force and that of GEO. The Group of Earth Observations with its broad range of national and organizational members has developed a general architectural framework for turning Earth observations into societal benefits. The three main components of this architecture are models, and observations, which then feed into decision support systems used in a variety of societal decision making. This general GEO framework is well suited as an architectural guide to the HTAP program. However, this framework lacks specific guidelines and implementation details that are needed for practical Earth observing and decision support systems. <br />
<br />
[[Image:400px HTAP GEOSS Arch.png|400px|left]] The HTAP program provides an opportunity to extend the GEO framework features that arise in specific HTAP application context. In case of HTAP, the major activities are shown in the architectural diagram of Figure ?? The HTAP modeling is conducted through global scale chemical transport models that simulate or forecast the atmospheric chemical composition as a consequence of natural and human induced emissions. The observations arise from satellite, ground-based and airborne observations of chemical constituents and their hemispheric transport. The decision support system consists of the scientists as members of the HTAP task force, and the task force co-chairs as the intermediaries between LRTP and the TF. (Terry, Andre this description of the HTAP DSS needs your help). <br />
<br />
Representing the Decision Support System (DSS) for HTAP is an important aspect because it can guide the design and and implementation of the information system for the support of the decission processes. A higher resolution DSS design document is also beneficial as a communications channel for the interacting system of systems components. A domain-specific DSS may also allow the generalization and applicability to the design of DSS structures for similar application domains. <br />
<br />
The implementation of the GEO framework utilizes the concept of Global Observing System of Systems (GEOSS). Traditionally, Earth observations were performed by well defined systems such as specific satellites and monitoring networks which were designed and operated using systems engineering principles. However, GEO recognized that the understanding of Earth system requires the utilization and integration of the individual, autonomous systems. The key difference between the Systems and System of System approaches are highlighted in Table 1.<br />
<br />
===HTAP Relationship to IGACO=== <br />
[[Image:IGACO Framework.png|300px|left]] The HTAP program can also have a mutually beneficial interaction with the Integrated Global Atmospheric Chemistry Observations (IGACO) program. The IGACO is part of the Integrated Global Observing Strategy (IGOS). IGACO proposes a specific framework for combining observational data and models. It also specifies the higher level information products that can be used for creating social benefit. <br />
<br />
In the IGACO framework, the data products from satellite, surface, and aircraft observations are collected for inclusion into an integrated data archive. A key component in the IGACO data flow is the mandatory quality assurance and QA QC that precedes the archiving. This is particularly important for HTAP where multi-sensory data from many different providers are to be combined into an integrated database. In the IGACO framework, a key output from the data system is a high integrated spatial-temporal data set which can be used in a multitude of applications that produce socal benefit. The goal of creating such an integrated data set is shared by the IGACO and the HTAP programs (?? Len, this section could benefit from your input)<br />
<br />
===HTAP Relationship with Other Programs===<br />
---<br />
---<br />
<br />
<br />
<br />
==Links==<br />
[[IGACO Framework]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=HTAP_Report,_Sub-Chap._6_-_Data/Info_System&diff=7975HTAP Report, Sub-Chap. 6 - Data/Info System2007-04-13T15:47:56Z<p>128.252.167.79: /* HTAP Information System */</p>
<hr />
<div><noinclude>[[Integrated_Global_Dataset| < Back to Integrated Global Dataset on ESIP wiki]] <br></noinclude> <br />
<br />
This wiki page is inteded to be the collaborative workspace for Task Force members interested in the HTAP Information System development, implementation and use. The contents can be freely edited by anyone who is logged in. Comments and feedback can also be sent by email to rhusar@me.wustl.edu. The content of this page is is on the '''''Information System''''' that supposts Chapter 6. The subsections are action-oriented. A draft [[HTAP Report Chap. 6 - Jan 07 Outline]] is also on this wiki.<br />
<br />
==Introduction==<br />
The purpose of '''[http://www.htap.org/activities/2007_interim_report/TF%20HTAP%202007%20Interim%20Report%20Prospectus%20Oct%202006.pdf Chapter 6]''' is to discuss the need to integrate information from observations, models, and emissions inventories to better understand intercontinental transport. This section focuses on the information system that is to support the processes for the integration of observations, emissions and models for HTAP. <br />
<br />
Recent developments in air quality monitoring, modeling and information technologies offer outstanding opportunities to fulfill the information needs for the HTAP integration effort. The data from surface-based air pollution monitoring networks now routinely provide spatio-temporal and chemical patterns of ozone and PM. Satellite sensors with global coverage and kilometer-scale spatial resolution now provide real-time snapshots which depict the pattern of industrial haze, smoke and dust in stunning detail. The ‘terabytes’ of data from these surface and remote sensors can now be stored, processed and delivered in near-real time. The instantaneous ‘horizontal’ diffusion of information via the Internet now permits, in principle, the delivery of the right information to the right people at the right place and time. Standardized computer-computer communication languages and Service-Oriented Architectures (SOA) now facilitate the flexible access, quality assurance and processing of raw data into high-grade ‘actionable’ knowledge suitable for HTAP policy decissions. Last but not least, the World Wide Web has opened the way to generous sharing of data and tools leading to collaborative analysis and virtual workgroups. Nevertheless, air quality data analyses and data model integration face significant hurdles. The section below present an architectral framework and implementation strategyfor the proposed HTAP information system.<br />
<br />
==HTAP Information System==<br />
The term achitecture here refers to both function and form of the HTAP Informations System (HTAP IS). The key function of the HTAP IS is to provide information technology support to the researches performing their task while sseking to better understand transcontinental air pollution transport. <br />
<br />
It is a tools set that is to empower the participating and collaborating analysts. The HTAP IS will not compete with existing tools of its members. Rather it will embrace and levarage those resources through an open, collaborative federation philosophy. Its main contribution is to connect the TF participating analysts and their information resources <br />
<br />
The information system HTAP IS consits of the following: <br />
* Porovide tools and methods for data access, processing and integration. This will be performed by the IT of the HTAP Federated Data System. <br />
* Facilitate open communication between the collaboing analysts. The IT technologies include wikis and other groupware, social software, blog, Skype, etc.. <br />
* Provide a shared workspace and where the above activitie take place <br />
<br />
===Homogenize the data into an interoperable dataset===<br />
* facilitate accsess to the distributed data<br />
* ensure the seamless flow of data through interoperable interfaces<br />
* provide a set of basic tools for data exploration and processing<br />
<br />
[[Image:HTAP_IntDataSys.png|600px|left]] The primary goal of the data system is to allow the integration of observational emission and model data. The model evaluation requires that both the observations and if possible the emissions are fixed. For this reason it is desirable to prepare an integrated observational database to which the various model implementations can be compared to. On one hand, the integrated data set should be as inclusive as possible. On the other hand, such a goal needs to be tempered by many limitations that preclude a broad, inclusive data integration<br />
<br />
The proposed HTAP data system will be a hybrid combination of both distributed and fixed components as illustrated schematically in Figure ??. Both the data providers as well as the HTAP analysts-users will be distributed. However, they will be connected through an integrated HTAP database which should be a stable, virtually fixed database. The section below describes the main component of this distributed data system. <br />
<br />
The HTAP integrated data system has to perform two major functions. The primary function is to facilitate the creation of an integrated data set suitable for model evaluation and pollutant characterization. The multiple steps that are required for this functionality are shown on the left. The sequence of operations can be viewed as a value chain that transform the raw input data into a highly organized integrated data set. Since these value adding steps have to be performed for each candidate data set the operations can be performed in sequence following the rules of data flow programming. For true quality assurance and for data homogenization the data flow channels for individual data sets need to be evaluated in the context of other data. <br />
<br />
The operations that prepare the integrated data set can be viewed as services that operate on the data stream using the principles of modern Service Oriented Architecure (SOA). Each of the services with its clearly defined functionality and firmly specified service interface. In principle, the standards-based interface allows a connection of service chains using formal work flow software. The main benefit of service oriented architecture is that allows the building of agile application programs that can respond to changes in the data input conditions as well as the output requirements. The service oriented software architecture is an implementation of the System of Systems approach. Furthermore, since each of the services can be operated by autonomous providers and the "system" that implements the service is behind the service interfece. In other words, following the SOA approach, not only the data providers are distributed but also the processing services as well. This flexible approach to distributed computing allows the distribution of labor (chain of services) in many different configurations. For instance, wrapping the existing data with standards based interfaces for data access can be performed for an entire cluster of data. This is the approach taken in the federated data system DataFed. Given a standard interface to a variety of data sets, the Quality Assurance service can be performed by another srvice provider that is seamlesly connected to the data access mediator. Similarly, the service that prepares a data set for data integration can be provided by another service in the data flow network. The flexibility offered through the chaining and orchestration of distributed, loosely coupled web services is the architectural support for the building of agile data systems.<br />
<br />
===Interoperability Standards and Data Wrapping===<br />
<br />
The methods and tools for model intercomparisons was a subject of a productive workshop at GRC Ispra in March 2006. European and American members of the HTAP TF presented their respective approaches as part of the model and data intercoperisons as part of their respecive projects ENSEMBLES, Eurodelta, ACCENT, AEROCOM, GEMS, and DataFed. Several recommendations were made to improve future use of intercomparison data. The most important recommendation was to agree on a common data format, NetCDF CF conventions, which would allow a variety of tools to work with the data.<br />
<br />
Interoperable data access will be accomplished through the use of a suite of international standards. The naming of individual chemical parameters will follow the CF convention used by the Climate and Forecast (CF) communities. The existing names for atmospheric chemicals in the CF convention were inadequate to accomodate all the parameters used in the HTAP modeling. in order to remedy this shortcomming the list of standard names was extended by the HTAP community under leadership of C. Textor. She also became a memeber of the CF convention board that is the custodian of the standard names. The standard names for HTAP models were developed using a collaborative wiki workspace. <br />
<br />
The data transfer from providers to users also benefit from standard data formats. For modeling data the use of netCDF-CF as a standard format is recomended. The use of a standard physical data format and the CF naming conventions allows, in principle, the seamless connection between data provider and consumer services. It should be noted, however, that at this time the CF naming convention has only been developed for the model parameters and not for the various observational parameters. Also, the nedCDF CF is primarily used as a model data exchange format, while the transfer of surface monitoring data and satellite data is less standardized.<br />
<br />
The third aspect of data interoperability is a standard query language through which user services request specific data from the provider services. It is proposed that for the HTAP data information system adapts the Web Coverage Service (WCS) as the standard data query language. The WCS data access protocol is defined by the international Open Geospatial Consortium (OGC), which is also the key organization responsible for interoperability standards in GEOSS. Since the WCS protocol was originally developed for the GIS community, it was necessary to adapt it to the needs of "fluid Earth sciences". Members of the HTAP group have been actively participating in the development and testing of the WCS interoperability standards. <br />
<br />
Standards-based data access can be accomplished by ‘wrapping’ the heterogeneous data into standardized web services, callable through well-defined Internet protocols. The result of this ‘wrapping’ process is an array of homogeneous, virtual datasets that can be accessed through a standard querry language and the returned data are packaged in standard format, directly usable by the consuming services.<br />
<br />
===Quality Assurance===<br />
This is just a place holder. Here we need to explain the quality assurance steps that are needed to prepare the HTAP Integrated Dataset.<br />
<br />
===Homogenization and Integration===<br />
The HTAP Integrated Dataset (HID) will be used to compare models to observations. It will be created from the proper combination of a variety of surface, upper air and satellite observations. However, before the inclusion into HID, each dataset will need to be scrutinezed to make it suitable for model comparison. The scrutiny may include filtering, aggregation and possibly fusion operations.<br />
<br />
A good example is the 400-station AIRNOW network reporting hourly ozone and PM2.5 concentrations over the US. The AIRNOW network includes both urban sites that are strongly influenced by local sources. Urban stations need to be removed from HID dataset.<br />
<br />
The Service Oriented Architecture (SOA) allows connecting the web service components (e.g. services for data access, transformation, fusion, rendering, etc.) using work flow software.<br />
<br />
Developing the specific criteria and procedures for the HTAP integrated dataset will require the attention of a HTAP subgroup.<br />
<br />
==== Data Selection Criteria====<br />
Monitoring data for atmospheric constituents are now available from a variety of sources, not all of which are suitable for the integrated HTAP data set. Given the limited scope and resources of HTAP, it will be necessary to select a subset of the available observational data that would be most appropriate for the preparation of the 2009 assessment. A set of criteria for the data selection is given below.<br />
* The suitability criteria may include the measured parameters, their spatial extent and coverage density, as well as the time range and sampling frequency with major emphasis on data quality (defined as???).<br />
* The compilation of the global integrated data sets should initially focus on PM and ozone, including the gaseous precursors.<br />
* Subsequent data collaction should also include observations on atmospheric mercury and persistent organic pollutants (POPS). <br />
* In order to cover hemispheric transport of air pollutants the data set system should accept and utilize data from outside the geographical scope of EMEP.<br />
* The data gathering should begin the compilation using the of existing databases in Europe (RETRO, TRADEOFF, QUANTIFY, CRATE, AEROCOM) and virtual federated systems in the US (DataFed, Giovanni and others). <br />
* TF data integration should also contribute to and benefit from other ongoing data integration efforts to integrate the global data resources, most notably, with ACCENT project in Europe, and similar efforts in the US.<br />
* Special emphasis should be placed on the collection of suitable vertical profiles from aircraft measurements as well as from surface and space-borne lidars.<br />
<br />
The evaluation of suitabile observational data sets for model validation and fusion will require close interaction between the modeling and observational communities. An wiki workspace for open collaborative evaluation of different datasets is desirebale.<br />
<br />
==Integrated Data Set==<br />
<br />
The providers of observational, emission, and modeling data are distributed. The individual data sets will be funeled into the integrated HTAP data set. This data set transfer will be accomplished using standard data transfer protocols. Transfering the data from the provider to the user will include two additional observations for each data set, quality assurance and semantic transformation.<br />
<br />
On the left are the inputs from observations, emissions, and model data.<br />
<br />
<br />
<br />
<br />
<br />
<br />
In order to accomplish the objectives set for the HTAP TF it is evident that it will require the System of Systems approach as envisioned by GEO. Both the modeling of the chemical constituents as well as the Earth observations used to document the hemispheric transport are being performed by individual autonomous projects and programs (Systems). A key function of the Task Force is to assess the contributions of the individual systems and then integrate those into a coherent assessment. Much of this interim report is in fact such an assessment of the current knowledge based on previous or past models and observations.<br />
<br />
In the next, upcoming phase of HTAP is to actually integrate the existing modeling and observing systems such that their combination and fusion can yield a deeper and more robust understanding of HTAP. since this next phase will involve integration of multiple relevant data sets, comparison and evaluation of multiple models, as well as the reconciliation of the observations and models to create the new and deeper understanding . In order to execute this extensive integration, it will be necessary to conduct this integration using a generally accepted framework as well as accepted scientific methods. It is believed that the GEO architecture and the System of Systems is an attractive approach to HTAP integration.<br />
<br />
The HTAP program can also be considered an early demonstration of the GEO concepts through its end to end approach. Both the atmospheric modeling, as well as the observations currently exist through the operation of existing modeling and observation systems. The Task Force has agreed organizing and evaluating those models, assembling and integrating the observational data sets and then reconciling the models with the observations will be the focus of the Phase II effort. The Task Force will also prepare and deliver a report to LRTP to aid its deliberations and its decision making processes. This sequence of activities constitutes an end to end approach that turns observations and models into actionable knowledge for societal decision making. One could say that this is an octagonal approach to more deliberate step by step development of GEOSS where in Phase I interoperability, in Phase II ???<br />
<br />
====HTAP Information Network====<br />
[[Image:HTAP Network.png|600px|left]]<br />
<br />
* Federated Data System DataFed<br />
* NASA Data System Giovanni<br />
* Emission Data System NEISGEI<br />
* Juelich Data System<br />
* AeorCom<br />
* EMEP<br />
<br />
<br />
Data Networks. Connected hubs; Show Core HTAP network<br />
<br />
<br />
<br />
<br />
<br />
<br />
====HTAP Data Sets - need list of others====<br />
[http://datafedwiki.wustl.edu/index.php/2007-03-26:_Datasets_for_HTAP_through_DataFed See 20 selected datasets] in the federated data systems, DataFed<br />
* TOMS_AI_G - Satellite <br />
* SURF_MET - Surface <br />
* SEAW_US - Satellite <br />
* SCIAMACHYm - Satellite <br />
* RETRO ANTHRO - Emission <br />
* OnEarth JPL - Satellite <br />
* OMId - Satellite <br />
* NAAPS GLOBAL - Model <br />
* MOPITT Day - Satellite <br />
* MODISd G - Satellite <br />
* MODIS Global Fire - Satellite <br />
* MISRm G - Satellite <br />
* GOMEm G - Satellite <br />
* GOCART G OL - Model <br />
* EDGAR - Emission <br />
* CALIPSO - Satellite <br />
* AIRNOW - Surface <br />
* VIEWS OL - Surface <br />
* AERONETd - Surface <br />
* AEROCOM LOA - Model<br />
<br />
==Reconciliation and Integration of Observations, Emissions and Models==<br />
===Compare observations to models and emissions===<br />
===Characterize pollutant pattern and SSR=== <br />
===Develop real-time data assimilation===<br />
===Perform reanalysis with assimilated data=== <br />
<br />
==Organizational Aspects==<br />
===HTAP Relationship to GEO and GEOSS===<br />
There is an outstanding opportunity to develop a mutually beneficial and supportive relationship between the activities of the HTAP Task Force and that of GEO. The Group of Earth Observations with its broad range of national and organizational members has developed a general architectural framework for turning Earth observations into societal benefits. The three main components of this architecture are models, and observations, which then feed into decision support systems used in a variety of societal decision making. This general GEO framework is well suited as an architectural guide to the HTAP program. However, this framework lacks specific guidelines and implementation details that are needed for practical Earth observing and decision support systems. <br />
<br />
[[Image:400px HTAP GEOSS Arch.png|400px|left]] The HTAP program provides an opportunity to extend the GEO framework features that arise in specific HTAP application context. In case of HTAP, the major activities are shown in the architectural diagram of Figure ?? The HTAP modeling is conducted through global scale chemical transport models that simulate or forecast the atmospheric chemical composition as a consequence of natural and human induced emissions. The observations arise from satellite, ground-based and airborne observations of chemical constituents and their hemispheric transport. The decision support system consists of the scientists as members of the HTAP task force, and the task force co-chairs as the intermediaries between LRTP and the TF. (Terry, Andre this description of the HTAP DSS needs your help). <br />
<br />
Representing the Decision Support System (DSS) for HTAP is an important aspect because it can guide the design and and implementation of the information system for the support of the decission processes. A higher resolution DSS design document is also beneficial as a communications channel for the interacting system of systems components. A domain-specific DSS may also allow the generalization and applicability to the design of DSS structures for similar application domains. <br />
<br />
The implementation of the GEO framework utilizes the concept of Global Observing System of Systems (GEOSS). Traditionally, Earth observations were performed by well defined systems such as specific satellites and monitoring networks which were designed and operated using systems engineering principles. However, GEO recognized that the understanding of Earth system requires the utilization and integration of the individual, autonomous systems. The key difference between the Systems and System of System approaches are highlighted in Table 1.<br />
<br />
===HTAP Relationship to IGACO=== <br />
[[Image:IGACO Framework.png|300px|left]] The HTAP program can also have a mutually beneficial interaction with the Integrated Global Atmospheric Chemistry Observations (IGACO) program. The IGACO is part of the Integrated Global Observing Strategy (IGOS). IGACO proposes a specific framework for combining observational data and models. It also specifies the higher level information products that can be used for creating social benefit. <br />
<br />
In the IGACO framework, the data products from satellite, surface, and aircraft observations are collected for inclusion into an integrated data archive. A key component in the IGACO data flow is the mandatory quality assurance and QA QC that precedes the archiving. This is particularly important for HTAP where multi-sensory data from many different providers are to be combined into an integrated database. In the IGACO framework, a key output from the data system is a high integrated spatial-temporal data set which can be used in a multitude of applications that produce socal benefit. The goal of creating such an integrated data set is shared by the IGACO and the HTAP programs (?? Len, this section could benefit from your input)<br />
<br />
===HTAP Relationship with Other Programs===<br />
---<br />
---<br />
<br />
<br />
<br />
==Links==<br />
[[IGACO Framework]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=HTAP_Report,_Sub-Chap._6_-_Data/Info_System&diff=7974HTAP Report, Sub-Chap. 6 - Data/Info System2007-04-13T15:40:44Z<p>128.252.167.79: /* Introduction */</p>
<hr />
<div><noinclude>[[Integrated_Global_Dataset| < Back to Integrated Global Dataset on ESIP wiki]] <br></noinclude> <br />
<br />
This wiki page is inteded to be the collaborative workspace for Task Force members interested in the HTAP Information System development, implementation and use. The contents can be freely edited by anyone who is logged in. Comments and feedback can also be sent by email to rhusar@me.wustl.edu. The content of this page is is on the '''''Information System''''' that supposts Chapter 6. The subsections are action-oriented. A draft [[HTAP Report Chap. 6 - Jan 07 Outline]] is also on this wiki.<br />
<br />
==Introduction==<br />
The purpose of '''[http://www.htap.org/activities/2007_interim_report/TF%20HTAP%202007%20Interim%20Report%20Prospectus%20Oct%202006.pdf Chapter 6]''' is to discuss the need to integrate information from observations, models, and emissions inventories to better understand intercontinental transport. This section focuses on the information system that is to support the processes for the integration of observations, emissions and models for HTAP. <br />
<br />
Recent developments in air quality monitoring, modeling and information technologies offer outstanding opportunities to fulfill the information needs for the HTAP integration effort. The data from surface-based air pollution monitoring networks now routinely provide spatio-temporal and chemical patterns of ozone and PM. Satellite sensors with global coverage and kilometer-scale spatial resolution now provide real-time snapshots which depict the pattern of industrial haze, smoke and dust in stunning detail. The ‘terabytes’ of data from these surface and remote sensors can now be stored, processed and delivered in near-real time. The instantaneous ‘horizontal’ diffusion of information via the Internet now permits, in principle, the delivery of the right information to the right people at the right place and time. Standardized computer-computer communication languages and Service-Oriented Architectures (SOA) now facilitate the flexible access, quality assurance and processing of raw data into high-grade ‘actionable’ knowledge suitable for HTAP policy decissions. Last but not least, the World Wide Web has opened the way to generous sharing of data and tools leading to collaborative analysis and virtual workgroups. Nevertheless, air quality data analyses and data model integration face significant hurdles. The section below present an architectral framework and implementation strategyfor the proposed HTAP information system.<br />
<br />
==HTAP Information System==<br />
The term achitecture here refers to both function and form of the HTAP informations system. The key function of the system is to provide information technology support to the researches performing their task while sseking to better understand transcontinental air pollution transport. It is a tools set that is to empower the participating and collaborating analysts. The key roles of the federated data system are envisioned as: <br />
* Porovide tools and methods for data access, processing and integration. This will be performed by the IT of the HTAP Federated Data System. <br />
* Facilitate open communication between the collaboing analysts. The IT technologies include wikis and other groupware, social software, blog, Skype, etc.. <br />
* Provide a shared workspace and where the above activitie take place <br />
<br />
The HTAP IS will not compete with existing tools of its members. Rather it will embrace and levarage those resources through an open, collaborative federation philosophy. A connective tissue that is to connect the diverse distributed resources. <br />
<br />
<br />
<br />
<br />
<br />
===Homogenize the data into an interoperable dataset===<br />
* facilitate accsess to the distributed data<br />
* ensure the seamless flow of data through interoperable interfaces<br />
* provide a set of basic tools for data exploration and processing<br />
<br />
[[Image:HTAP_IntDataSys.png|600px|left]] The primary goal of the data system is to allow the integration of observational emission and model data. The model evaluation requires that both the observations and if possible the emissions are fixed. For this reason it is desirable to prepare an integrated observational database to which the various model implementations can be compared to. On one hand, the integrated data set should be as inclusive as possible. On the other hand, such a goal needs to be tempered by many limitations that preclude a broad, inclusive data integration<br />
<br />
The proposed HTAP data system will be a hybrid combination of both distributed and fixed components as illustrated schematically in Figure ??. Both the data providers as well as the HTAP analysts-users will be distributed. However, they will be connected through an integrated HTAP database which should be a stable, virtually fixed database. The section below describes the main component of this distributed data system. <br />
<br />
The HTAP integrated data system has to perform two major functions. The primary function is to facilitate the creation of an integrated data set suitable for model evaluation and pollutant characterization. The multiple steps that are required for this functionality are shown on the left. The sequence of operations can be viewed as a value chain that transform the raw input data into a highly organized integrated data set. Since these value adding steps have to be performed for each candidate data set the operations can be performed in sequence following the rules of data flow programming. For true quality assurance and for data homogenization the data flow channels for individual data sets need to be evaluated in the context of other data. <br />
<br />
The operations that prepare the integrated data set can be viewed as services that operate on the data stream using the principles of modern Service Oriented Architecure (SOA). Each of the services with its clearly defined functionality and firmly specified service interface. In principle, the standards-based interface allows a connection of service chains using formal work flow software. The main benefit of service oriented architecture is that allows the building of agile application programs that can respond to changes in the data input conditions as well as the output requirements. The service oriented software architecture is an implementation of the System of Systems approach. Furthermore, since each of the services can be operated by autonomous providers and the "system" that implements the service is behind the service interfece. In other words, following the SOA approach, not only the data providers are distributed but also the processing services as well. This flexible approach to distributed computing allows the distribution of labor (chain of services) in many different configurations. For instance, wrapping the existing data with standards based interfaces for data access can be performed for an entire cluster of data. This is the approach taken in the federated data system DataFed. Given a standard interface to a variety of data sets, the Quality Assurance service can be performed by another srvice provider that is seamlesly connected to the data access mediator. Similarly, the service that prepares a data set for data integration can be provided by another service in the data flow network. The flexibility offered through the chaining and orchestration of distributed, loosely coupled web services is the architectural support for the building of agile data systems.<br />
<br />
===Interoperability Standards and Data Wrapping===<br />
<br />
The methods and tools for model intercomparisons was a subject of a productive workshop at GRC Ispra in March 2006. European and American members of the HTAP TF presented their respective approaches as part of the model and data intercoperisons as part of their respecive projects ENSEMBLES, Eurodelta, ACCENT, AEROCOM, GEMS, and DataFed. Several recommendations were made to improve future use of intercomparison data. The most important recommendation was to agree on a common data format, NetCDF CF conventions, which would allow a variety of tools to work with the data.<br />
<br />
Interoperable data access will be accomplished through the use of a suite of international standards. The naming of individual chemical parameters will follow the CF convention used by the Climate and Forecast (CF) communities. The existing names for atmospheric chemicals in the CF convention were inadequate to accomodate all the parameters used in the HTAP modeling. in order to remedy this shortcomming the list of standard names was extended by the HTAP community under leadership of C. Textor. She also became a memeber of the CF convention board that is the custodian of the standard names. The standard names for HTAP models were developed using a collaborative wiki workspace. <br />
<br />
The data transfer from providers to users also benefit from standard data formats. For modeling data the use of netCDF-CF as a standard format is recomended. The use of a standard physical data format and the CF naming conventions allows, in principle, the seamless connection between data provider and consumer services. It should be noted, however, that at this time the CF naming convention has only been developed for the model parameters and not for the various observational parameters. Also, the nedCDF CF is primarily used as a model data exchange format, while the transfer of surface monitoring data and satellite data is less standardized.<br />
<br />
The third aspect of data interoperability is a standard query language through which user services request specific data from the provider services. It is proposed that for the HTAP data information system adapts the Web Coverage Service (WCS) as the standard data query language. The WCS data access protocol is defined by the international Open Geospatial Consortium (OGC), which is also the key organization responsible for interoperability standards in GEOSS. Since the WCS protocol was originally developed for the GIS community, it was necessary to adapt it to the needs of "fluid Earth sciences". Members of the HTAP group have been actively participating in the development and testing of the WCS interoperability standards. <br />
<br />
Standards-based data access can be accomplished by ‘wrapping’ the heterogeneous data into standardized web services, callable through well-defined Internet protocols. The result of this ‘wrapping’ process is an array of homogeneous, virtual datasets that can be accessed through a standard querry language and the returned data are packaged in standard format, directly usable by the consuming services.<br />
<br />
===Quality Assurance===<br />
This is just a place holder. Here we need to explain the quality assurance steps that are needed to prepare the HTAP Integrated Dataset.<br />
<br />
===Homogenization and Integration===<br />
The HTAP Integrated Dataset (HID) will be used to compare models to observations. It will be created from the proper combination of a variety of surface, upper air and satellite observations. However, before the inclusion into HID, each dataset will need to be scrutinezed to make it suitable for model comparison. The scrutiny may include filtering, aggregation and possibly fusion operations.<br />
<br />
A good example is the 400-station AIRNOW network reporting hourly ozone and PM2.5 concentrations over the US. The AIRNOW network includes both urban sites that are strongly influenced by local sources. Urban stations need to be removed from HID dataset.<br />
<br />
The Service Oriented Architecture (SOA) allows connecting the web service components (e.g. services for data access, transformation, fusion, rendering, etc.) using work flow software.<br />
<br />
Developing the specific criteria and procedures for the HTAP integrated dataset will require the attention of a HTAP subgroup.<br />
<br />
==== Data Selection Criteria====<br />
Monitoring data for atmospheric constituents are now available from a variety of sources, not all of which are suitable for the integrated HTAP data set. Given the limited scope and resources of HTAP, it will be necessary to select a subset of the available observational data that would be most appropriate for the preparation of the 2009 assessment. A set of criteria for the data selection is given below.<br />
* The suitability criteria may include the measured parameters, their spatial extent and coverage density, as well as the time range and sampling frequency with major emphasis on data quality (defined as???).<br />
* The compilation of the global integrated data sets should initially focus on PM and ozone, including the gaseous precursors.<br />
* Subsequent data collaction should also include observations on atmospheric mercury and persistent organic pollutants (POPS). <br />
* In order to cover hemispheric transport of air pollutants the data set system should accept and utilize data from outside the geographical scope of EMEP.<br />
* The data gathering should begin the compilation using the of existing databases in Europe (RETRO, TRADEOFF, QUANTIFY, CRATE, AEROCOM) and virtual federated systems in the US (DataFed, Giovanni and others). <br />
* TF data integration should also contribute to and benefit from other ongoing data integration efforts to integrate the global data resources, most notably, with ACCENT project in Europe, and similar efforts in the US.<br />
* Special emphasis should be placed on the collection of suitable vertical profiles from aircraft measurements as well as from surface and space-borne lidars.<br />
<br />
The evaluation of suitabile observational data sets for model validation and fusion will require close interaction between the modeling and observational communities. An wiki workspace for open collaborative evaluation of different datasets is desirebale.<br />
<br />
==Integrated Data Set==<br />
<br />
The providers of observational, emission, and modeling data are distributed. The individual data sets will be funeled into the integrated HTAP data set. This data set transfer will be accomplished using standard data transfer protocols. Transfering the data from the provider to the user will include two additional observations for each data set, quality assurance and semantic transformation.<br />
<br />
On the left are the inputs from observations, emissions, and model data.<br />
<br />
<br />
<br />
<br />
<br />
<br />
In order to accomplish the objectives set for the HTAP TF it is evident that it will require the System of Systems approach as envisioned by GEO. Both the modeling of the chemical constituents as well as the Earth observations used to document the hemispheric transport are being performed by individual autonomous projects and programs (Systems). A key function of the Task Force is to assess the contributions of the individual systems and then integrate those into a coherent assessment. Much of this interim report is in fact such an assessment of the current knowledge based on previous or past models and observations.<br />
<br />
In the next, upcoming phase of HTAP is to actually integrate the existing modeling and observing systems such that their combination and fusion can yield a deeper and more robust understanding of HTAP. since this next phase will involve integration of multiple relevant data sets, comparison and evaluation of multiple models, as well as the reconciliation of the observations and models to create the new and deeper understanding . In order to execute this extensive integration, it will be necessary to conduct this integration using a generally accepted framework as well as accepted scientific methods. It is believed that the GEO architecture and the System of Systems is an attractive approach to HTAP integration.<br />
<br />
The HTAP program can also be considered an early demonstration of the GEO concepts through its end to end approach. Both the atmospheric modeling, as well as the observations currently exist through the operation of existing modeling and observation systems. The Task Force has agreed organizing and evaluating those models, assembling and integrating the observational data sets and then reconciling the models with the observations will be the focus of the Phase II effort. The Task Force will also prepare and deliver a report to LRTP to aid its deliberations and its decision making processes. This sequence of activities constitutes an end to end approach that turns observations and models into actionable knowledge for societal decision making. One could say that this is an octagonal approach to more deliberate step by step development of GEOSS where in Phase I interoperability, in Phase II ???<br />
<br />
====HTAP Information Network====<br />
[[Image:HTAP Network.png|600px|left]]<br />
<br />
* Federated Data System DataFed<br />
* NASA Data System Giovanni<br />
* Emission Data System NEISGEI<br />
* Juelich Data System<br />
* AeorCom<br />
* EMEP<br />
<br />
<br />
Data Networks. Connected hubs; Show Core HTAP network<br />
<br />
<br />
<br />
<br />
<br />
<br />
====HTAP Data Sets - need list of others====<br />
[http://datafedwiki.wustl.edu/index.php/2007-03-26:_Datasets_for_HTAP_through_DataFed See 20 selected datasets] in the federated data systems, DataFed<br />
* TOMS_AI_G - Satellite <br />
* SURF_MET - Surface <br />
* SEAW_US - Satellite <br />
* SCIAMACHYm - Satellite <br />
* RETRO ANTHRO - Emission <br />
* OnEarth JPL - Satellite <br />
* OMId - Satellite <br />
* NAAPS GLOBAL - Model <br />
* MOPITT Day - Satellite <br />
* MODISd G - Satellite <br />
* MODIS Global Fire - Satellite <br />
* MISRm G - Satellite <br />
* GOMEm G - Satellite <br />
* GOCART G OL - Model <br />
* EDGAR - Emission <br />
* CALIPSO - Satellite <br />
* AIRNOW - Surface <br />
* VIEWS OL - Surface <br />
* AERONETd - Surface <br />
* AEROCOM LOA - Model<br />
<br />
==Reconciliation and Integration of Observations, Emissions and Models==<br />
===Compare observations to models and emissions===<br />
===Characterize pollutant pattern and SSR=== <br />
===Develop real-time data assimilation===<br />
===Perform reanalysis with assimilated data=== <br />
<br />
==Organizational Aspects==<br />
===HTAP Relationship to GEO and GEOSS===<br />
There is an outstanding opportunity to develop a mutually beneficial and supportive relationship between the activities of the HTAP Task Force and that of GEO. The Group of Earth Observations with its broad range of national and organizational members has developed a general architectural framework for turning Earth observations into societal benefits. The three main components of this architecture are models, and observations, which then feed into decision support systems used in a variety of societal decision making. This general GEO framework is well suited as an architectural guide to the HTAP program. However, this framework lacks specific guidelines and implementation details that are needed for practical Earth observing and decision support systems. <br />
<br />
[[Image:400px HTAP GEOSS Arch.png|400px|left]] The HTAP program provides an opportunity to extend the GEO framework features that arise in specific HTAP application context. In case of HTAP, the major activities are shown in the architectural diagram of Figure ?? The HTAP modeling is conducted through global scale chemical transport models that simulate or forecast the atmospheric chemical composition as a consequence of natural and human induced emissions. The observations arise from satellite, ground-based and airborne observations of chemical constituents and their hemispheric transport. The decision support system consists of the scientists as members of the HTAP task force, and the task force co-chairs as the intermediaries between LRTP and the TF. (Terry, Andre this description of the HTAP DSS needs your help). <br />
<br />
Representing the Decision Support System (DSS) for HTAP is an important aspect because it can guide the design and and implementation of the information system for the support of the decission processes. A higher resolution DSS design document is also beneficial as a communications channel for the interacting system of systems components. A domain-specific DSS may also allow the generalization and applicability to the design of DSS structures for similar application domains. <br />
<br />
The implementation of the GEO framework utilizes the concept of Global Observing System of Systems (GEOSS). Traditionally, Earth observations were performed by well defined systems such as specific satellites and monitoring networks which were designed and operated using systems engineering principles. However, GEO recognized that the understanding of Earth system requires the utilization and integration of the individual, autonomous systems. The key difference between the Systems and System of System approaches are highlighted in Table 1.<br />
<br />
===HTAP Relationship to IGACO=== <br />
[[Image:IGACO Framework.png|300px|left]] The HTAP program can also have a mutually beneficial interaction with the Integrated Global Atmospheric Chemistry Observations (IGACO) program. The IGACO is part of the Integrated Global Observing Strategy (IGOS). IGACO proposes a specific framework for combining observational data and models. It also specifies the higher level information products that can be used for creating social benefit. <br />
<br />
In the IGACO framework, the data products from satellite, surface, and aircraft observations are collected for inclusion into an integrated data archive. A key component in the IGACO data flow is the mandatory quality assurance and QA QC that precedes the archiving. This is particularly important for HTAP where multi-sensory data from many different providers are to be combined into an integrated database. In the IGACO framework, a key output from the data system is a high integrated spatial-temporal data set which can be used in a multitude of applications that produce socal benefit. The goal of creating such an integrated data set is shared by the IGACO and the HTAP programs (?? Len, this section could benefit from your input)<br />
<br />
===HTAP Relationship with Other Programs===<br />
---<br />
---<br />
<br />
<br />
<br />
==Links==<br />
[[IGACO Framework]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=HTAP_Report,_Sub-Chap._6_-_Data/Info_System&diff=7953HTAP Report, Sub-Chap. 6 - Data/Info System2007-04-12T05:16:46Z<p>128.252.167.79: /* HTAP Relationship to IGACO */</p>
<hr />
<div><noinclude>[[Integrated_Global_Dataset| < Back to Integrated Global Dataset on ESIP wiki]] <br></noinclude> <br />
<br />
This wiki page is inteded to be the collaborative workspace for Task Force members interested in the HTAP Information System development, implementation and use. The contents can be freely edited by anyone who is logged in. Comments and feedback can also be sent by email to rhusar@me.wustl.edu. The content of this page is is on the '''''Information System''''' that supposts Chapter 6. The subsections are action-oriented. <br />
<br />
From [http://www.htap.org/activities/2007_interim_report/TF%20HTAP%202007%20Interim%20Report%20Prospectus%20Oct%202006.pdf Oct 2006 Prospectus]: This chapter (Chapter 6: '''Integration of Observations, Emissions and Modeling'''.) will discuss the need to integrate information from observations, models, and emissions inventories to better understand intercontinental transport. A draft [[HTAP Report Chap. 6 - Jan 07 Outline]] is also on this wiki.<br />
<br />
==Introduction==<br />
The goal of the HTAP data system is to aid the assembly of an appropriate set of observations and models, and to facilitate the integration and fusion of observations with models. <br />
<br />
Recent developments in air quality monitoring, modeling and information technologies offer outstanding<br />
opportunities to fulfill the information needs for the HTAP integration effort. The data from surface-based air pollution monitoring networks now routinely provide spatio-temporal and chemical patterns of ozone and PM. Satellite<br />
sensors with global coverage and kilometer-scale spatial resolution now provide real-time snapshots which depict the pattern of industrial haze, smoke and dust in stunning detail. The ‘terabytes’ of data from these surface and remote sensors can now be stored, processed and delivered in near-real time. The instantaneous ‘horizontal’ diffusion of information via the Internet now permits, in principle, the delivery of the right information to the right people at the right place and time. Standardized computer-computer communication languages and Service-Oriented Architectures (SOA) now facilitate the flexible access, quality assurance and processing of raw data into high-grade ‘actionable’ knowledge suitable for HTAP policy decissions. Last but not least, the World Wide Web has opened the way to generous sharing of data and tools leading to collaborative analysis and virtual workgroups. Nevertheless, air quality data analyses and data model integration face significant hurdles. The “data deluge” problem is especially acute for satellite observations and model outputs. <br />
<br />
<br />
<br />
====HTAP Data System Architecture====<br />
The proposed HTAP data system will integrate the emerging observational and modeling resources by not a centrally planned and powerful, dynamic technologies and through a collaborative federation philosophy. The key roles of the federated data system are to (1) facilitate accsess to the distributed data; (2) ensure the seamless flow of data through interoperable interfaces; (3) provide a set of basic tools for data exploration and processing. <br />
<br />
The Service Oriented Architecture (SOA) allows connecting the web service components (e.g. services for data access, transformation, fusion, rendering, etc.) using work flow software.<br />
<br />
===Homogenize the data into an interoperable dataset===<br />
The primary goal of the data system is to allow the integration of observational emission and model data. The model evaluation requires that both the observations and if possible the emissions are fixed. For this reason it is desirable to prepare an integrated observational database to which the various model implementations can be compared to. On one hand, the integrated data set should be as inclusive as possible. On the other hand, such a goal needs to be tempered by many limitations that preclude a broad, inclusive data integration<br />
<br />
The proposed HTAP data system will be a hybrid combination of both distributed and fixed components as illustrated schematically in Figure ??. Both the data providers as well as the HTAP analysts-users will be distributed. However, they will be connected through an integrated HTAP database which should be a stable, virtually fixed database. The section below describes the main component of this distributed data system. <br />
<br />
The HTAP integrated data system has to perform two major functions. The primary function is to facilitate the creation of an integrated data set suitable for model evaluation and pollutant characterization. The multiple steps that are required for this functionality are shown on the left. The sequence of operations can be viewed as a value chain that transform the raw input data into a highly organized integrated data set. Since these value adding steps have to be performed for each candidate data set the operations can be performed in sequence following the rules of data flow programming. For true quality assurance and for data homogenization the data flow channels for individual data sets need to be evaluated in the context of other data. <br />
<br />
The operations that prepare the integrated data set can be viewed as services that operate on the data stream using the principles of modern Service Oriented Architecure (SOA). Each of the services with its clearly defined functionality and firmly specified service interface. In principle, the standards-based interface allows a connection of service chains using formal work flow software. The main benefit of service oriented architecture is that allows the building of agile application programs that can respond to changes in the data input conditions as well as the output requirements. The service oriented software architecture is an implementation of the System of Systems approach. Furthermore, since each of the services can be operated by autonomous providers and the "system" that implements the service is behind the service interfece. In other words, following the SOA approach, not only the data providers are distributed but also the processing services as well. This flexible approach to distributed computing allows the distribution of labor (chain of services) in many different configurations. For instance, wrapping the existing data with standards based interfaces for data access can be performed for an entire cluster of data. This is the approach taken in the federated data system DataFed. Given a standard interface to a variety of data sets, the Quality Assurance service can be performed by another srvice provider that is seamlesly connected to the data access mediator. Similarly, the service that prepares a data set for data integration can be provided by another service in the data flow network. The flexibility offered through the chaining and orchestration of distributed, loosely coupled web services is the architectural support for the building of agile data systems.<br />
<br />
===Interoperability Standards and Data Wrapping===<br />
Interoperable data access will be accomplished through the use of a suite of international standards. The naming of individual chemical parameters will follow the CF convention used by the Climate and Forecast (CF) communities. The existing names for atmospheric chemicals in the CF convention were inadequate to accomodate all the parameters used in the HTAP modeling. in order to remedy this shortcomming the list of standard names was extended by the HTAP community under leadership of C. Textor. She also became a memeber of the CF convention board that is the custodian of the standard names. The standard names for HTAP models were developed using a collaborative wiki workspace. <br />
<br />
The data transfer from providers to users also benefit from standard data formats. For modeling data the use of netCDF-CF as a standard format is recomended. The use of a standard physical data format and the CF naming conventions allows, in principle, the seamless connection between data provider and consumer services. It should be noted, however, that at this time the CF naming convention has only been developed for the model parameters and not for the various observational parameters. Also, the nedCDF CF is primarily used as a model data exchange format, while the transfer of surface monitoring data and satellite data is less standardized.<br />
<br />
The third aspect of data interoperability is a standard query language through which user services request specific data from the provider services. It is proposed that for the HTAP data information system adapts the Web Coverage Service (WCS) as the standard data query language. The WCS data access protocol is defined by the international Open Geospatial Consortium (OGC), which is also the key organization responsible for interoperability standards in GEOSS. Since the WCS protocol was originally developed for the GIS community, it was necessary to adapt it to the needs of "fluid Earth sciences". Members of the HTAP group have been actively participating in the development and testing of the WCS interoperability standards. <br />
<br />
Standards-based data access can be accomplished by ‘wrapping’ the heterogeneous data into standardized web services, callable through well-defined Internet protocols. The result of this ‘wrapping’ process is an array of homogeneous, virtual datasets that can be accessed through a standard querry language and the returned data are packaged in standard format, directly usable by the consuming services.<br />
<br />
===Quality Assurance===<br />
This is just a place holder. Here we need to explain the quality assurance steps that are needed to prepare the HTAP Integrated Dataset.<br />
<br />
===Homogenization and Integration===<br />
The HTAP Integrated Dataset (HID) will be used to compare models to observations. It will be created from the proper combination of a variety of surface, upper air and satellite observations. However, before the inclusion into HID, each dataset will need to be scrutinezed to make it suitable for model comparison. The scrutiny may include filtering, aggregation and possibly fusion operations.<br />
<br />
A good example is the 400-station AIRNOW network reporting hourly ozone and PM2.5 concentrations over the US. The AIRNOW network includes both urban sites that are strongly influenced by local sources. Urban stations need to be removed from HID dataset.<br />
<br />
Developing the specific criteria and procedures for the HTAP integrated dataset will require the attention of a HTAP subgroup.<br />
==== Data Selection Criteria====<br />
Monitoring data for atmospheric constituents are now available from a variety of sources, not all of which are suitable for the integrated HTAP data set. Given the limited scope and resources of HTAP, it will be necessary to select a subset of the available observational data that would be most appropriate for the preparation of the 2009 assessment. A set of criteria for the data selection is given below.<br />
* The suitability criteria may include the measured parameters, their spatial extent and coverage density, as well as the time range and sampling frequency with major emphasis on data quality (defined as???).<br />
* The compilation of the global integrated data sets should initially focus on PM and ozone, including the gaseous precursors.<br />
* Subsequent data collaction should also include observations on atmospheric mercury and persistent organic pollutants (POPS). <br />
* In order to cover hemispheric transport of air pollutants the data set system should accept and utilize data from outside the geographical scope of EMEP.<br />
* The data gathering should begin the compilation using the of existing databases in Europe (RETRO, TRADEOFF, QUANTIFY, CRATE, AEROCOM) and virtual federated systems in the US (DataFed, Giovanni and others). <br />
* TF data integration should also contribute to and benefit from other ongoing data integration efforts to integrate the global data resources, most notably, with ACCENT project in Europe, and similar efforts in the US.<br />
* Special emphasis should be placed on the collection of suitable vertical profiles from aircraft measurements as well as from surface and space-borne lidars.<br />
<br />
The evaluation of suitabile observational data sets for model validation and fusion will require close interaction between the modeling and observational communities. An wiki workspace for open collaborative evaluation of different datasets is desirebale.<br />
<br />
==Integrated Data Set==<br />
<br />
The providers of observational, emission, and modeling data are distributed. The individual data sets will be funeled into the integrated HTAP data set. This data set transfer will be accomplished using standard data transfer protocols. Transfering the data from the provider to the user will include two additional observations for each data set, quality assurance and semantic transformation.<br />
<br />
On the left are the inputs from observations, emissions, and model data.<br />
<br />
<br />
<br />
<br />
<br />
<br />
[[Image:HTAP_IntDataSys.png|600px|left]] In order to accomplish the objectives set for the HTAP TF it is evident that it will require the System of Systems approach as envisioned by GEO. Both the modeling of the chemical constituents as well as the Earth observations used to document the hemispheric transport are being performed by individual autonomous projects and programs (Systems). A key function of the Task Force is to assess the contributions of the individual systems and then integrate those into a coherent assessment. Much of this interim report is in fact such an assessment of the current knowledge based on previous or past models and observations.<br />
<br />
In the next, upcoming phase of HTAP is to actually integrate the existing modeling and observing systems such that their combination and fusion can yield a deeper and more robust understanding of HTAP. since this next phase will involve integration of multiple relevant data sets, comparison and evaluation of multiple models, as well as the reconciliation of the observations and models to create the new and deeper understanding . In order to execute this extensive integration, it will be necessary to conduct this integration using a generally accepted framework as well as accepted scientific methods. It is believed that the GEO architecture and the System of Systems is an attractive approach to HTAP integration.<br />
<br />
The HTAP program can also be considered an early demonstration of the GEO concepts through its end to end approach. Both the atmospheric modeling, as well as the observations currently exist through the operation of existing modeling and observation systems. The Task Force has agreed organizing and evaluating those models, assembling and integrating the observational data sets and then reconciling the models with the observations will be the focus of the Phase II effort. The Task Force will also prepare and deliver a report to LRTP to aid its deliberations and its decision making processes. This sequence of activities constitutes an end to end approach that turns observations and models into actionable knowledge for societal decision making. One could say that this is an octagonal approach to more deliberate step by step development of GEOSS where in Phase I interoperability, in Phase II ???<br />
<br />
Provider nodes <br />
* Federated Data System DataFed<br />
* NASA Data System Giovanni<br />
* Emission Data System NEISGEI<br />
* Juelich Data System<br />
* AeorCom<br />
* EMEP<br />
<br />
<br />
Data Networks. Connected hubs; Show Core HTAP network<br />
<br />
[http://datafedwiki.wustl.edu/index.php/2007-03-26:_Datasets_for_HTAP_through_DataFed See 20 selected datasets] in the federated data systems, DataFed<br />
<br />
<br />
<br />
==Reconciliation and Integration of Observations, Emissions and Models==<br />
===Compare observations to models and emissions===<br />
===Characterize pollutant pattern and SSR=== <br />
===Develop real-time data assimilation===<br />
===Perform reanalysis with assimilated data=== <br />
<br />
==Organizational Aspects==<br />
===HTAP Relationship to GEO and GEOSS===<br />
There is an outstanding opportunity to develop a mutually beneficial and supportive relationship between the activities of the HTAP Task Force and that of GEO. The Group of Earth Observations with its broad range of national and organizational members has developed a general architectural framework for turning Earth observations into societal benefits. The three main components of this architecture are models, and observations, which then feed into decision support systems used in a variety of societal decision making. This general GEO framework is well suited as an architectural guide to the HTAP program. However, this framework lacks specific guidelines and implementation details that are needed for practical Earth observing and decision support systems. <br />
<br />
[[Image:400px HTAP GEOSS Arch.png|400px|left]] The HTAP program provides an opportunity to extend the GEO framework features that arise in specific HTAP application context. In case of HTAP, the major activities are shown in the architectural diagram of Figure ?? The HTAP modeling is conducted through global scale chemical transport models that simulate or forecast the atmospheric chemical composition as a consequence of natural and human induced emissions. The observations arise from satellite, ground-based and airborne observations of chemical constituents and their hemispheric transport. The decision support system consists of the scientists as members of the HTAP task force, and the task force co-chairs as the intermediaries between LRTP and the TF. (Terry, Andre this description of the HTAP DSS needs your help). <br />
<br />
Representing the Decision Support System (DSS) for HTAP is an important aspect because it can guide the design and and implementation of the information system for the support of the decission processes. A higher resolution DSS design document is also beneficial as a communications channel for the interacting system of systems components. A domain-specific DSS may also allow the generalization and applicability to the design of DSS structures for similar application domains. <br />
<br />
The implementation of the GEO framework utilizes the concept of Global Observing System of Systems (GEOSS). Traditionally, Earth observations were performed by well defined systems such as specific satellites and monitoring networks which were designed and operated using systems engineering principles. However, GEO recognized that the understanding of Earth system requires the utilization and integration of the individual, autonomous systems. The key difference between the Systems and System of System approaches are highlighted in Table 1.<br />
<br />
===HTAP Relationship to IGACO=== <br />
[[Image:IGACO Framework.png|300px|left]] The HTAP program can also have a mutually beneficial interaction with the Integrated Global Atmospheric Chemistry Observations (IGACO) program. The IGACO is part of the Integrated Global Observing Strategy (IGOS). IGACO proposes a specific framework for combining observational data and models. It also specifies the higher level information products that can be used for creating social benefit. <br />
<br />
In the IGACO framework, the data products from satellite, surface, and aircraft observations are collected for inclusion into an integrated data archive. A key component in the IGACO data flow is the mandatory quality assurance and QA QC that precedes the archiving. This is particularly important for HTAP where multi-sensory data from many different providers are to be combined into an integrated database. In the IGACO framework, a key output from the data system is a high integrated spatial-temporal data set which can be used in a multitude of applications that produce socal benefit. The goal of creating such an integrated data set is shared by the IGACO and the HTAP programs (?? Len, this section could benefit from your input)<br />
<br />
===HTAP Relationship with Other Programs===<br />
---<br />
---<br />
<br />
<br />
<br />
==Links==<br />
[[IGACO Framework]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=HTAP_Report,_Sub-Chap._6_-_Data/Info_System&diff=7952HTAP Report, Sub-Chap. 6 - Data/Info System2007-04-12T05:13:28Z<p>128.252.167.79: /* HTAP Relationship to GEO and GEOSS */</p>
<hr />
<div><noinclude>[[Integrated_Global_Dataset| < Back to Integrated Global Dataset on ESIP wiki]] <br></noinclude> <br />
<br />
This wiki page is inteded to be the collaborative workspace for Task Force members interested in the HTAP Information System development, implementation and use. The contents can be freely edited by anyone who is logged in. Comments and feedback can also be sent by email to rhusar@me.wustl.edu. The content of this page is is on the '''''Information System''''' that supposts Chapter 6. The subsections are action-oriented. <br />
<br />
From [http://www.htap.org/activities/2007_interim_report/TF%20HTAP%202007%20Interim%20Report%20Prospectus%20Oct%202006.pdf Oct 2006 Prospectus]: This chapter (Chapter 6: '''Integration of Observations, Emissions and Modeling'''.) will discuss the need to integrate information from observations, models, and emissions inventories to better understand intercontinental transport. A draft [[HTAP Report Chap. 6 - Jan 07 Outline]] is also on this wiki.<br />
<br />
==Introduction==<br />
The goal of the HTAP data system is to aid the assembly of an appropriate set of observations and models, and to facilitate the integration and fusion of observations with models. <br />
<br />
Recent developments in air quality monitoring, modeling and information technologies offer outstanding<br />
opportunities to fulfill the information needs for the HTAP integration effort. The data from surface-based air pollution monitoring networks now routinely provide spatio-temporal and chemical patterns of ozone and PM. Satellite<br />
sensors with global coverage and kilometer-scale spatial resolution now provide real-time snapshots which depict the pattern of industrial haze, smoke and dust in stunning detail. The ‘terabytes’ of data from these surface and remote sensors can now be stored, processed and delivered in near-real time. The instantaneous ‘horizontal’ diffusion of information via the Internet now permits, in principle, the delivery of the right information to the right people at the right place and time. Standardized computer-computer communication languages and Service-Oriented Architectures (SOA) now facilitate the flexible access, quality assurance and processing of raw data into high-grade ‘actionable’ knowledge suitable for HTAP policy decissions. Last but not least, the World Wide Web has opened the way to generous sharing of data and tools leading to collaborative analysis and virtual workgroups. Nevertheless, air quality data analyses and data model integration face significant hurdles. The “data deluge” problem is especially acute for satellite observations and model outputs. <br />
<br />
<br />
<br />
====HTAP Data System Architecture====<br />
The proposed HTAP data system will integrate the emerging observational and modeling resources by not a centrally planned and powerful, dynamic technologies and through a collaborative federation philosophy. The key roles of the federated data system are to (1) facilitate accsess to the distributed data; (2) ensure the seamless flow of data through interoperable interfaces; (3) provide a set of basic tools for data exploration and processing. <br />
<br />
The Service Oriented Architecture (SOA) allows connecting the web service components (e.g. services for data access, transformation, fusion, rendering, etc.) using work flow software.<br />
<br />
===Homogenize the data into an interoperable dataset===<br />
The primary goal of the data system is to allow the integration of observational emission and model data. The model evaluation requires that both the observations and if possible the emissions are fixed. For this reason it is desirable to prepare an integrated observational database to which the various model implementations can be compared to. On one hand, the integrated data set should be as inclusive as possible. On the other hand, such a goal needs to be tempered by many limitations that preclude a broad, inclusive data integration<br />
<br />
The proposed HTAP data system will be a hybrid combination of both distributed and fixed components as illustrated schematically in Figure ??. Both the data providers as well as the HTAP analysts-users will be distributed. However, they will be connected through an integrated HTAP database which should be a stable, virtually fixed database. The section below describes the main component of this distributed data system. <br />
<br />
The HTAP integrated data system has to perform two major functions. The primary function is to facilitate the creation of an integrated data set suitable for model evaluation and pollutant characterization. The multiple steps that are required for this functionality are shown on the left. The sequence of operations can be viewed as a value chain that transform the raw input data into a highly organized integrated data set. Since these value adding steps have to be performed for each candidate data set the operations can be performed in sequence following the rules of data flow programming. For true quality assurance and for data homogenization the data flow channels for individual data sets need to be evaluated in the context of other data. <br />
<br />
The operations that prepare the integrated data set can be viewed as services that operate on the data stream using the principles of modern Service Oriented Architecure (SOA). Each of the services with its clearly defined functionality and firmly specified service interface. In principle, the standards-based interface allows a connection of service chains using formal work flow software. The main benefit of service oriented architecture is that allows the building of agile application programs that can respond to changes in the data input conditions as well as the output requirements. The service oriented software architecture is an implementation of the System of Systems approach. Furthermore, since each of the services can be operated by autonomous providers and the "system" that implements the service is behind the service interfece. In other words, following the SOA approach, not only the data providers are distributed but also the processing services as well. This flexible approach to distributed computing allows the distribution of labor (chain of services) in many different configurations. For instance, wrapping the existing data with standards based interfaces for data access can be performed for an entire cluster of data. This is the approach taken in the federated data system DataFed. Given a standard interface to a variety of data sets, the Quality Assurance service can be performed by another srvice provider that is seamlesly connected to the data access mediator. Similarly, the service that prepares a data set for data integration can be provided by another service in the data flow network. The flexibility offered through the chaining and orchestration of distributed, loosely coupled web services is the architectural support for the building of agile data systems.<br />
<br />
===Interoperability Standards and Data Wrapping===<br />
Interoperable data access will be accomplished through the use of a suite of international standards. The naming of individual chemical parameters will follow the CF convention used by the Climate and Forecast (CF) communities. The existing names for atmospheric chemicals in the CF convention were inadequate to accomodate all the parameters used in the HTAP modeling. in order to remedy this shortcomming the list of standard names was extended by the HTAP community under leadership of C. Textor. She also became a memeber of the CF convention board that is the custodian of the standard names. The standard names for HTAP models were developed using a collaborative wiki workspace. <br />
<br />
The data transfer from providers to users also benefit from standard data formats. For modeling data the use of netCDF-CF as a standard format is recomended. The use of a standard physical data format and the CF naming conventions allows, in principle, the seamless connection between data provider and consumer services. It should be noted, however, that at this time the CF naming convention has only been developed for the model parameters and not for the various observational parameters. Also, the nedCDF CF is primarily used as a model data exchange format, while the transfer of surface monitoring data and satellite data is less standardized.<br />
<br />
The third aspect of data interoperability is a standard query language through which user services request specific data from the provider services. It is proposed that for the HTAP data information system adapts the Web Coverage Service (WCS) as the standard data query language. The WCS data access protocol is defined by the international Open Geospatial Consortium (OGC), which is also the key organization responsible for interoperability standards in GEOSS. Since the WCS protocol was originally developed for the GIS community, it was necessary to adapt it to the needs of "fluid Earth sciences". Members of the HTAP group have been actively participating in the development and testing of the WCS interoperability standards. <br />
<br />
Standards-based data access can be accomplished by ‘wrapping’ the heterogeneous data into standardized web services, callable through well-defined Internet protocols. The result of this ‘wrapping’ process is an array of homogeneous, virtual datasets that can be accessed through a standard querry language and the returned data are packaged in standard format, directly usable by the consuming services.<br />
<br />
===Quality Assurance===<br />
This is just a place holder. Here we need to explain the quality assurance steps that are needed to prepare the HTAP Integrated Dataset.<br />
<br />
===Homogenization and Integration===<br />
The HTAP Integrated Dataset (HID) will be used to compare models to observations. It will be created from the proper combination of a variety of surface, upper air and satellite observations. However, before the inclusion into HID, each dataset will need to be scrutinezed to make it suitable for model comparison. The scrutiny may include filtering, aggregation and possibly fusion operations.<br />
<br />
A good example is the 400-station AIRNOW network reporting hourly ozone and PM2.5 concentrations over the US. The AIRNOW network includes both urban sites that are strongly influenced by local sources. Urban stations need to be removed from HID dataset.<br />
<br />
Developing the specific criteria and procedures for the HTAP integrated dataset will require the attention of a HTAP subgroup.<br />
==== Data Selection Criteria====<br />
Monitoring data for atmospheric constituents are now available from a variety of sources, not all of which are suitable for the integrated HTAP data set. Given the limited scope and resources of HTAP, it will be necessary to select a subset of the available observational data that would be most appropriate for the preparation of the 2009 assessment. A set of criteria for the data selection is given below.<br />
* The suitability criteria may include the measured parameters, their spatial extent and coverage density, as well as the time range and sampling frequency with major emphasis on data quality (defined as???).<br />
* The compilation of the global integrated data sets should initially focus on PM and ozone, including the gaseous precursors.<br />
* Subsequent data collaction should also include observations on atmospheric mercury and persistent organic pollutants (POPS). <br />
* In order to cover hemispheric transport of air pollutants the data set system should accept and utilize data from outside the geographical scope of EMEP.<br />
* The data gathering should begin the compilation using the of existing databases in Europe (RETRO, TRADEOFF, QUANTIFY, CRATE, AEROCOM) and virtual federated systems in the US (DataFed, Giovanni and others). <br />
* TF data integration should also contribute to and benefit from other ongoing data integration efforts to integrate the global data resources, most notably, with ACCENT project in Europe, and similar efforts in the US.<br />
* Special emphasis should be placed on the collection of suitable vertical profiles from aircraft measurements as well as from surface and space-borne lidars.<br />
<br />
The evaluation of suitabile observational data sets for model validation and fusion will require close interaction between the modeling and observational communities. An wiki workspace for open collaborative evaluation of different datasets is desirebale.<br />
<br />
==Integrated Data Set==<br />
<br />
The providers of observational, emission, and modeling data are distributed. The individual data sets will be funeled into the integrated HTAP data set. This data set transfer will be accomplished using standard data transfer protocols. Transfering the data from the provider to the user will include two additional observations for each data set, quality assurance and semantic transformation.<br />
<br />
On the left are the inputs from observations, emissions, and model data.<br />
<br />
<br />
<br />
<br />
<br />
<br />
[[Image:HTAP_IntDataSys.png|600px|left]] In order to accomplish the objectives set for the HTAP TF it is evident that it will require the System of Systems approach as envisioned by GEO. Both the modeling of the chemical constituents as well as the Earth observations used to document the hemispheric transport are being performed by individual autonomous projects and programs (Systems). A key function of the Task Force is to assess the contributions of the individual systems and then integrate those into a coherent assessment. Much of this interim report is in fact such an assessment of the current knowledge based on previous or past models and observations.<br />
<br />
In the next, upcoming phase of HTAP is to actually integrate the existing modeling and observing systems such that their combination and fusion can yield a deeper and more robust understanding of HTAP. since this next phase will involve integration of multiple relevant data sets, comparison and evaluation of multiple models, as well as the reconciliation of the observations and models to create the new and deeper understanding . In order to execute this extensive integration, it will be necessary to conduct this integration using a generally accepted framework as well as accepted scientific methods. It is believed that the GEO architecture and the System of Systems is an attractive approach to HTAP integration.<br />
<br />
The HTAP program can also be considered an early demonstration of the GEO concepts through its end to end approach. Both the atmospheric modeling, as well as the observations currently exist through the operation of existing modeling and observation systems. The Task Force has agreed organizing and evaluating those models, assembling and integrating the observational data sets and then reconciling the models with the observations will be the focus of the Phase II effort. The Task Force will also prepare and deliver a report to LRTP to aid its deliberations and its decision making processes. This sequence of activities constitutes an end to end approach that turns observations and models into actionable knowledge for societal decision making. One could say that this is an octagonal approach to more deliberate step by step development of GEOSS where in Phase I interoperability, in Phase II ???<br />
<br />
Provider nodes <br />
* Federated Data System DataFed<br />
* NASA Data System Giovanni<br />
* Emission Data System NEISGEI<br />
* Juelich Data System<br />
* AeorCom<br />
* EMEP<br />
<br />
<br />
Data Networks. Connected hubs; Show Core HTAP network<br />
<br />
[http://datafedwiki.wustl.edu/index.php/2007-03-26:_Datasets_for_HTAP_through_DataFed See 20 selected datasets] in the federated data systems, DataFed<br />
<br />
<br />
<br />
==Reconciliation and Integration of Observations, Emissions and Models==<br />
===Compare observations to models and emissions===<br />
===Characterize pollutant pattern and SSR=== <br />
===Develop real-time data assimilation===<br />
===Perform reanalysis with assimilated data=== <br />
<br />
==Organizational Aspects==<br />
===HTAP Relationship to GEO and GEOSS===<br />
There is an outstanding opportunity to develop a mutually beneficial and supportive relationship between the activities of the HTAP Task Force and that of GEO. The Group of Earth Observations with its broad range of national and organizational members has developed a general architectural framework for turning Earth observations into societal benefits. The three main components of this architecture are models, and observations, which then feed into decision support systems used in a variety of societal decision making. This general GEO framework is well suited as an architectural guide to the HTAP program. However, this framework lacks specific guidelines and implementation details that are needed for practical Earth observing and decision support systems. <br />
<br />
[[Image:400px HTAP GEOSS Arch.png|400px|left]] The HTAP program provides an opportunity to extend the GEO framework features that arise in specific HTAP application context. In case of HTAP, the major activities are shown in the architectural diagram of Figure ?? The HTAP modeling is conducted through global scale chemical transport models that simulate or forecast the atmospheric chemical composition as a consequence of natural and human induced emissions. The observations arise from satellite, ground-based and airborne observations of chemical constituents and their hemispheric transport. The decision support system consists of the scientists as members of the HTAP task force, and the task force co-chairs as the intermediaries between LRTP and the TF. (Terry, Andre this description of the HTAP DSS needs your help). <br />
<br />
Representing the Decision Support System (DSS) for HTAP is an important aspect because it can guide the design and and implementation of the information system for the support of the decission processes. A higher resolution DSS design document is also beneficial as a communications channel for the interacting system of systems components. A domain-specific DSS may also allow the generalization and applicability to the design of DSS structures for similar application domains. <br />
<br />
The implementation of the GEO framework utilizes the concept of Global Observing System of Systems (GEOSS). Traditionally, Earth observations were performed by well defined systems such as specific satellites and monitoring networks which were designed and operated using systems engineering principles. However, GEO recognized that the understanding of Earth system requires the utilization and integration of the individual, autonomous systems. The key difference between the Systems and System of System approaches are highlighted in Table 1.<br />
<br />
===HTAP Relationship to IGACO=== <br />
[[Image:IGACO Framework.png|300px|left]] The HTAP program can also have a mutually beneficial interaction with the IGACO program. Integrated Global Atmospheric Chemistry Observations (IGACO) Theme is part of the Integrated Global Observing Strategy (IGOS). The IGACO program is formulated by an array of experienced atmospheric scientists. IGACO proposes a specific framework for the system components that processes the observational data and models. It also specifies the higher level information products that can be used for creating social benefit. <br />
<br />
In the IGACO framework, the data products from satellite, surface, and aircraft observations are collected for inclusion into an integrated data archive. A key component in the IGACO data flow is the mandatory quality assurance and QA QC that precedes the archiving. This is particularly important for HTAP where multi-sensory data from many different providers are to be combined into an integrated database. In the IGACO framework, a key output from the data system is a high integrated spatial-temporal data set which can be used in a multitude of applications that produce socal benefit. The goal of creating such an integrated data set is shared by the IGACO and the HTAP programs (?? Len, this section could benefit from your input)<br />
<br />
===HTAP Relationship with Other Programs===<br />
---<br />
---<br />
<br />
<br />
<br />
==Links==<br />
[[IGACO Framework]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=HTAP_Report,_Sub-Chap._6_-_Data/Info_System&diff=7951HTAP Report, Sub-Chap. 6 - Data/Info System2007-04-12T05:09:33Z<p>128.252.167.79: /* Perform QA at all steps */</p>
<hr />
<div><noinclude>[[Integrated_Global_Dataset| < Back to Integrated Global Dataset on ESIP wiki]] <br></noinclude> <br />
<br />
This wiki page is inteded to be the collaborative workspace for Task Force members interested in the HTAP Information System development, implementation and use. The contents can be freely edited by anyone who is logged in. Comments and feedback can also be sent by email to rhusar@me.wustl.edu. The content of this page is is on the '''''Information System''''' that supposts Chapter 6. The subsections are action-oriented. <br />
<br />
From [http://www.htap.org/activities/2007_interim_report/TF%20HTAP%202007%20Interim%20Report%20Prospectus%20Oct%202006.pdf Oct 2006 Prospectus]: This chapter (Chapter 6: '''Integration of Observations, Emissions and Modeling'''.) will discuss the need to integrate information from observations, models, and emissions inventories to better understand intercontinental transport. A draft [[HTAP Report Chap. 6 - Jan 07 Outline]] is also on this wiki.<br />
<br />
==Introduction==<br />
The goal of the HTAP data system is to aid the assembly of an appropriate set of observations and models, and to facilitate the integration and fusion of observations with models. <br />
<br />
Recent developments in air quality monitoring, modeling and information technologies offer outstanding<br />
opportunities to fulfill the information needs for the HTAP integration effort. The data from surface-based air pollution monitoring networks now routinely provide spatio-temporal and chemical patterns of ozone and PM. Satellite<br />
sensors with global coverage and kilometer-scale spatial resolution now provide real-time snapshots which depict the pattern of industrial haze, smoke and dust in stunning detail. The ‘terabytes’ of data from these surface and remote sensors can now be stored, processed and delivered in near-real time. The instantaneous ‘horizontal’ diffusion of information via the Internet now permits, in principle, the delivery of the right information to the right people at the right place and time. Standardized computer-computer communication languages and Service-Oriented Architectures (SOA) now facilitate the flexible access, quality assurance and processing of raw data into high-grade ‘actionable’ knowledge suitable for HTAP policy decissions. Last but not least, the World Wide Web has opened the way to generous sharing of data and tools leading to collaborative analysis and virtual workgroups. Nevertheless, air quality data analyses and data model integration face significant hurdles. The “data deluge” problem is especially acute for satellite observations and model outputs. <br />
<br />
<br />
<br />
====HTAP Data System Architecture====<br />
The proposed HTAP data system will integrate the emerging observational and modeling resources by not a centrally planned and powerful, dynamic technologies and through a collaborative federation philosophy. The key roles of the federated data system are to (1) facilitate accsess to the distributed data; (2) ensure the seamless flow of data through interoperable interfaces; (3) provide a set of basic tools for data exploration and processing. <br />
<br />
The Service Oriented Architecture (SOA) allows connecting the web service components (e.g. services for data access, transformation, fusion, rendering, etc.) using work flow software.<br />
<br />
===Homogenize the data into an interoperable dataset===<br />
The primary goal of the data system is to allow the integration of observational emission and model data. The model evaluation requires that both the observations and if possible the emissions are fixed. For this reason it is desirable to prepare an integrated observational database to which the various model implementations can be compared to. On one hand, the integrated data set should be as inclusive as possible. On the other hand, such a goal needs to be tempered by many limitations that preclude a broad, inclusive data integration<br />
<br />
The proposed HTAP data system will be a hybrid combination of both distributed and fixed components as illustrated schematically in Figure ??. Both the data providers as well as the HTAP analysts-users will be distributed. However, they will be connected through an integrated HTAP database which should be a stable, virtually fixed database. The section below describes the main component of this distributed data system. <br />
<br />
The HTAP integrated data system has to perform two major functions. The primary function is to facilitate the creation of an integrated data set suitable for model evaluation and pollutant characterization. The multiple steps that are required for this functionality are shown on the left. The sequence of operations can be viewed as a value chain that transform the raw input data into a highly organized integrated data set. Since these value adding steps have to be performed for each candidate data set the operations can be performed in sequence following the rules of data flow programming. For true quality assurance and for data homogenization the data flow channels for individual data sets need to be evaluated in the context of other data. <br />
<br />
The operations that prepare the integrated data set can be viewed as services that operate on the data stream using the principles of modern Service Oriented Architecure (SOA). Each of the services with its clearly defined functionality and firmly specified service interface. In principle, the standards-based interface allows a connection of service chains using formal work flow software. The main benefit of service oriented architecture is that allows the building of agile application programs that can respond to changes in the data input conditions as well as the output requirements. The service oriented software architecture is an implementation of the System of Systems approach. Furthermore, since each of the services can be operated by autonomous providers and the "system" that implements the service is behind the service interfece. In other words, following the SOA approach, not only the data providers are distributed but also the processing services as well. This flexible approach to distributed computing allows the distribution of labor (chain of services) in many different configurations. For instance, wrapping the existing data with standards based interfaces for data access can be performed for an entire cluster of data. This is the approach taken in the federated data system DataFed. Given a standard interface to a variety of data sets, the Quality Assurance service can be performed by another srvice provider that is seamlesly connected to the data access mediator. Similarly, the service that prepares a data set for data integration can be provided by another service in the data flow network. The flexibility offered through the chaining and orchestration of distributed, loosely coupled web services is the architectural support for the building of agile data systems.<br />
<br />
===Interoperability Standards and Data Wrapping===<br />
Interoperable data access will be accomplished through the use of a suite of international standards. The naming of individual chemical parameters will follow the CF convention used by the Climate and Forecast (CF) communities. The existing names for atmospheric chemicals in the CF convention were inadequate to accomodate all the parameters used in the HTAP modeling. in order to remedy this shortcomming the list of standard names was extended by the HTAP community under leadership of C. Textor. She also became a memeber of the CF convention board that is the custodian of the standard names. The standard names for HTAP models were developed using a collaborative wiki workspace. <br />
<br />
The data transfer from providers to users also benefit from standard data formats. For modeling data the use of netCDF-CF as a standard format is recomended. The use of a standard physical data format and the CF naming conventions allows, in principle, the seamless connection between data provider and consumer services. It should be noted, however, that at this time the CF naming convention has only been developed for the model parameters and not for the various observational parameters. Also, the nedCDF CF is primarily used as a model data exchange format, while the transfer of surface monitoring data and satellite data is less standardized.<br />
<br />
The third aspect of data interoperability is a standard query language through which user services request specific data from the provider services. It is proposed that for the HTAP data information system adapts the Web Coverage Service (WCS) as the standard data query language. The WCS data access protocol is defined by the international Open Geospatial Consortium (OGC), which is also the key organization responsible for interoperability standards in GEOSS. Since the WCS protocol was originally developed for the GIS community, it was necessary to adapt it to the needs of "fluid Earth sciences". Members of the HTAP group have been actively participating in the development and testing of the WCS interoperability standards. <br />
<br />
Standards-based data access can be accomplished by ‘wrapping’ the heterogeneous data into standardized web services, callable through well-defined Internet protocols. The result of this ‘wrapping’ process is an array of homogeneous, virtual datasets that can be accessed through a standard querry language and the returned data are packaged in standard format, directly usable by the consuming services.<br />
<br />
===Quality Assurance===<br />
This is just a place holder. Here we need to explain the quality assurance steps that are needed to prepare the HTAP Integrated Dataset.<br />
<br />
===Homogenization and Integration===<br />
The HTAP Integrated Dataset (HID) will be used to compare models to observations. It will be created from the proper combination of a variety of surface, upper air and satellite observations. However, before the inclusion into HID, each dataset will need to be scrutinezed to make it suitable for model comparison. The scrutiny may include filtering, aggregation and possibly fusion operations.<br />
<br />
A good example is the 400-station AIRNOW network reporting hourly ozone and PM2.5 concentrations over the US. The AIRNOW network includes both urban sites that are strongly influenced by local sources. Urban stations need to be removed from HID dataset.<br />
<br />
Developing the specific criteria and procedures for the HTAP integrated dataset will require the attention of a HTAP subgroup.<br />
==== Data Selection Criteria====<br />
Monitoring data for atmospheric constituents are now available from a variety of sources, not all of which are suitable for the integrated HTAP data set. Given the limited scope and resources of HTAP, it will be necessary to select a subset of the available observational data that would be most appropriate for the preparation of the 2009 assessment. A set of criteria for the data selection is given below.<br />
* The suitability criteria may include the measured parameters, their spatial extent and coverage density, as well as the time range and sampling frequency with major emphasis on data quality (defined as???).<br />
* The compilation of the global integrated data sets should initially focus on PM and ozone, including the gaseous precursors.<br />
* Subsequent data collaction should also include observations on atmospheric mercury and persistent organic pollutants (POPS). <br />
* In order to cover hemispheric transport of air pollutants the data set system should accept and utilize data from outside the geographical scope of EMEP.<br />
* The data gathering should begin the compilation using the of existing databases in Europe (RETRO, TRADEOFF, QUANTIFY, CRATE, AEROCOM) and virtual federated systems in the US (DataFed, Giovanni and others). <br />
* TF data integration should also contribute to and benefit from other ongoing data integration efforts to integrate the global data resources, most notably, with ACCENT project in Europe, and similar efforts in the US.<br />
* Special emphasis should be placed on the collection of suitable vertical profiles from aircraft measurements as well as from surface and space-borne lidars.<br />
<br />
The evaluation of suitabile observational data sets for model validation and fusion will require close interaction between the modeling and observational communities. An wiki workspace for open collaborative evaluation of different datasets is desirebale.<br />
<br />
==Integrated Data Set==<br />
<br />
The providers of observational, emission, and modeling data are distributed. The individual data sets will be funeled into the integrated HTAP data set. This data set transfer will be accomplished using standard data transfer protocols. Transfering the data from the provider to the user will include two additional observations for each data set, quality assurance and semantic transformation.<br />
<br />
On the left are the inputs from observations, emissions, and model data.<br />
<br />
<br />
<br />
<br />
<br />
<br />
[[Image:HTAP_IntDataSys.png|600px|left]] In order to accomplish the objectives set for the HTAP TF it is evident that it will require the System of Systems approach as envisioned by GEO. Both the modeling of the chemical constituents as well as the Earth observations used to document the hemispheric transport are being performed by individual autonomous projects and programs (Systems). A key function of the Task Force is to assess the contributions of the individual systems and then integrate those into a coherent assessment. Much of this interim report is in fact such an assessment of the current knowledge based on previous or past models and observations.<br />
<br />
In the next, upcoming phase of HTAP is to actually integrate the existing modeling and observing systems such that their combination and fusion can yield a deeper and more robust understanding of HTAP. since this next phase will involve integration of multiple relevant data sets, comparison and evaluation of multiple models, as well as the reconciliation of the observations and models to create the new and deeper understanding . In order to execute this extensive integration, it will be necessary to conduct this integration using a generally accepted framework as well as accepted scientific methods. It is believed that the GEO architecture and the System of Systems is an attractive approach to HTAP integration.<br />
<br />
The HTAP program can also be considered an early demonstration of the GEO concepts through its end to end approach. Both the atmospheric modeling, as well as the observations currently exist through the operation of existing modeling and observation systems. The Task Force has agreed organizing and evaluating those models, assembling and integrating the observational data sets and then reconciling the models with the observations will be the focus of the Phase II effort. The Task Force will also prepare and deliver a report to LRTP to aid its deliberations and its decision making processes. This sequence of activities constitutes an end to end approach that turns observations and models into actionable knowledge for societal decision making. One could say that this is an octagonal approach to more deliberate step by step development of GEOSS where in Phase I interoperability, in Phase II ???<br />
<br />
Provider nodes <br />
* Federated Data System DataFed<br />
* NASA Data System Giovanni<br />
* Emission Data System NEISGEI<br />
* Juelich Data System<br />
* AeorCom<br />
* EMEP<br />
<br />
<br />
Data Networks. Connected hubs; Show Core HTAP network<br />
<br />
[http://datafedwiki.wustl.edu/index.php/2007-03-26:_Datasets_for_HTAP_through_DataFed See 20 selected datasets] in the federated data systems, DataFed<br />
<br />
<br />
<br />
==Reconciliation and Integration of Observations, Emissions and Models==<br />
===Compare observations to models and emissions===<br />
===Characterize pollutant pattern and SSR=== <br />
===Develop real-time data assimilation===<br />
===Perform reanalysis with assimilated data=== <br />
<br />
==Organizational Aspects==<br />
===HTAP Relationship to GEO and GEOSS===<br />
There is an outstanding opportunity to develop a mutually beneficial and supportive relationship between the activities of the HTAP Task Force and that of GEO. The Group of Earth Observations with its broad range of national and organizational members has developed a general architectural framework for turning Earth observations into societal benefits. The three main components of this architecture are models, and observations, which then feed into decision support systems used in a variety of societal decision making. This general framework is well suited as an architectural guide to the HTAP program. However, this framework lacks specific guidelines and implementation details that are needed for practical Earth observing and decision support systems. <br />
<br />
[[Image:400px HTAP GEOSS Arch.png|400px|left]] The HTAP program provides an opportunity to extend the GEO framework with more detailed features that arise in specific HTAP application context. In case of HTAP, the major activities are shown in the architectural diagram of Figure ?? The HTAP modeling is conducted through global scale chemical transport models that simulate or forecast the atmospheric chemical composition as a consequence of natural and human induced emissions. The observations arise from satellite, ground-based and airborne observations of chemical constituents and their hemispheric transport. The decision support system consists of the scientists as members of the HTAP task force, and the task force co-chairs as the intermediaries between LRTP and the TF. (Terry, Andre this description of the HTAP DSS needs your help). <br />
<br />
Representing the Decision Support System (DSS) for HTAP is an important aspect because it can guide the design and and implementation of the information system for the support of the DSS processes. A higher resolution DSS design document is also beneficial as a communications medium for the interacting system of systems. A domain specific DSS may also allow the generalization and applicability to the design of DSS structures for similar application domains. <br />
<br />
The implementation of the GEO framework utilizes the concept of Global Observing System of Systems (GEOSS). Traditionally, Earth observations were performed by well defined systems such as specific satellites and monitoring networks which were designed and operated using systems engineering principles. However, GEO recognized that the understanding of Earth system requires the utilization and integration of the individual, autonomous systems. The key difference between the Systems and System of System approaches are highlighted in Table 1.<br />
<br />
===HTAP Relationship to IGACO=== <br />
[[Image:IGACO Framework.png|300px|left]] The HTAP program can also have a mutually beneficial interaction with the IGACO program. Integrated Global Atmospheric Chemistry Observations (IGACO) Theme is part of the Integrated Global Observing Strategy (IGOS). The IGACO program is formulated by an array of experienced atmospheric scientists. IGACO proposes a specific framework for the system components that processes the observational data and models. It also specifies the higher level information products that can be used for creating social benefit. <br />
<br />
In the IGACO framework, the data products from satellite, surface, and aircraft observations are collected for inclusion into an integrated data archive. A key component in the IGACO data flow is the mandatory quality assurance and QA QC that precedes the archiving. This is particularly important for HTAP where multi-sensory data from many different providers are to be combined into an integrated database. In the IGACO framework, a key output from the data system is a high integrated spatial-temporal data set which can be used in a multitude of applications that produce socal benefit. The goal of creating such an integrated data set is shared by the IGACO and the HTAP programs (?? Len, this section could benefit from your input)<br />
<br />
===HTAP Relationship with Other Programs===<br />
---<br />
---<br />
<br />
<br />
<br />
==Links==<br />
[[IGACO Framework]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=HTAP_Report,_Sub-Chap._6_-_Data/Info_System&diff=7950HTAP Report, Sub-Chap. 6 - Data/Info System2007-04-12T05:04:00Z<p>128.252.167.79: /* Integrated Data Set */</p>
<hr />
<div><noinclude>[[Integrated_Global_Dataset| < Back to Integrated Global Dataset on ESIP wiki]] <br></noinclude> <br />
<br />
This wiki page is inteded to be the collaborative workspace for Task Force members interested in the HTAP Information System development, implementation and use. The contents can be freely edited by anyone who is logged in. Comments and feedback can also be sent by email to rhusar@me.wustl.edu. The content of this page is is on the '''''Information System''''' that supposts Chapter 6. The subsections are action-oriented. <br />
<br />
From [http://www.htap.org/activities/2007_interim_report/TF%20HTAP%202007%20Interim%20Report%20Prospectus%20Oct%202006.pdf Oct 2006 Prospectus]: This chapter (Chapter 6: '''Integration of Observations, Emissions and Modeling'''.) will discuss the need to integrate information from observations, models, and emissions inventories to better understand intercontinental transport. A draft [[HTAP Report Chap. 6 - Jan 07 Outline]] is also on this wiki.<br />
<br />
==Introduction==<br />
The goal of the HTAP data system is to aid the assembly of an appropriate set of observations and models, and to facilitate the integration and fusion of observations with models. <br />
<br />
Recent developments in air quality monitoring, modeling and information technologies offer outstanding<br />
opportunities to fulfill the information needs for the HTAP integration effort. The data from surface-based air pollution monitoring networks now routinely provide spatio-temporal and chemical patterns of ozone and PM. Satellite<br />
sensors with global coverage and kilometer-scale spatial resolution now provide real-time snapshots which depict the pattern of industrial haze, smoke and dust in stunning detail. The ‘terabytes’ of data from these surface and remote sensors can now be stored, processed and delivered in near-real time. The instantaneous ‘horizontal’ diffusion of information via the Internet now permits, in principle, the delivery of the right information to the right people at the right place and time. Standardized computer-computer communication languages and Service-Oriented Architectures (SOA) now facilitate the flexible access, quality assurance and processing of raw data into high-grade ‘actionable’ knowledge suitable for HTAP policy decissions. Last but not least, the World Wide Web has opened the way to generous sharing of data and tools leading to collaborative analysis and virtual workgroups. Nevertheless, air quality data analyses and data model integration face significant hurdles. The “data deluge” problem is especially acute for satellite observations and model outputs. <br />
<br />
<br />
<br />
====HTAP Data System Architecture====<br />
The proposed HTAP data system will integrate the emerging observational and modeling resources by not a centrally planned and powerful, dynamic technologies and through a collaborative federation philosophy. The key roles of the federated data system are to (1) facilitate accsess to the distributed data; (2) ensure the seamless flow of data through interoperable interfaces; (3) provide a set of basic tools for data exploration and processing. <br />
<br />
The Service Oriented Architecture (SOA) allows connecting the web service components (e.g. services for data access, transformation, fusion, rendering, etc.) using work flow software.<br />
<br />
===Homogenize the data into an interoperable dataset===<br />
The primary goal of the data system is to allow the integration of observational emission and model data. The model evaluation requires that both the observations and if possible the emissions are fixed. For this reason it is desirable to prepare an integrated observational database to which the various model implementations can be compared to. On one hand, the integrated data set should be as inclusive as possible. On the other hand, such a goal needs to be tempered by many limitations that preclude a broad, inclusive data integration<br />
<br />
The proposed HTAP data system will be a hybrid combination of both distributed and fixed components as illustrated schematically in Figure ??. Both the data providers as well as the HTAP analysts-users will be distributed. However, they will be connected through an integrated HTAP database which should be a stable, virtually fixed database. The section below describes the main component of this distributed data system. <br />
<br />
The HTAP integrated data system has to perform two major functions. The primary function is to facilitate the creation of an integrated data set suitable for model evaluation and pollutant characterization. The multiple steps that are required for this functionality are shown on the left. The sequence of operations can be viewed as a value chain that transform the raw input data into a highly organized integrated data set. Since these value adding steps have to be performed for each candidate data set the operations can be performed in sequence following the rules of data flow programming. For true quality assurance and for data homogenization the data flow channels for individual data sets need to be evaluated in the context of other data. <br />
<br />
The operations that prepare the integrated data set can be viewed as services that operate on the data stream using the principles of modern Service Oriented Architecure (SOA). Each of the services with its clearly defined functionality and firmly specified service interface. In principle, the standards-based interface allows a connection of service chains using formal work flow software. The main benefit of service oriented architecture is that allows the building of agile application programs that can respond to changes in the data input conditions as well as the output requirements. The service oriented software architecture is an implementation of the System of Systems approach. Furthermore, since each of the services can be operated by autonomous providers and the "system" that implements the service is behind the service interfece. In other words, following the SOA approach, not only the data providers are distributed but also the processing services as well. This flexible approach to distributed computing allows the distribution of labor (chain of services) in many different configurations. For instance, wrapping the existing data with standards based interfaces for data access can be performed for an entire cluster of data. This is the approach taken in the federated data system DataFed. Given a standard interface to a variety of data sets, the Quality Assurance service can be performed by another srvice provider that is seamlesly connected to the data access mediator. Similarly, the service that prepares a data set for data integration can be provided by another service in the data flow network. The flexibility offered through the chaining and orchestration of distributed, loosely coupled web services is the architectural support for the building of agile data systems.<br />
<br />
===Interoperability Standards and Data Wrapping===<br />
Interoperable data access will be accomplished through the use of a suite of international standards. The naming of individual chemical parameters will follow the CF convention used by the Climate and Forecast (CF) communities. The existing names for atmospheric chemicals in the CF convention were inadequate to accomodate all the parameters used in the HTAP modeling. in order to remedy this shortcomming the list of standard names was extended by the HTAP community under leadership of C. Textor. She also became a memeber of the CF convention board that is the custodian of the standard names. The standard names for HTAP models were developed using a collaborative wiki workspace. <br />
<br />
The data transfer from providers to users also benefit from standard data formats. For modeling data the use of netCDF-CF as a standard format is recomended. The use of a standard physical data format and the CF naming conventions allows, in principle, the seamless connection between data provider and consumer services. It should be noted, however, that at this time the CF naming convention has only been developed for the model parameters and not for the various observational parameters. Also, the nedCDF CF is primarily used as a model data exchange format, while the transfer of surface monitoring data and satellite data is less standardized.<br />
<br />
The third aspect of data interoperability is a standard query language through which user services request specific data from the provider services. It is proposed that for the HTAP data information system adapts the Web Coverage Service (WCS) as the standard data query language. The WCS data access protocol is defined by the international Open Geospatial Consortium (OGC), which is also the key organization responsible for interoperability standards in GEOSS. Since the WCS protocol was originally developed for the GIS community, it was necessary to adapt it to the needs of "fluid Earth sciences". Members of the HTAP group have been actively participating in the development and testing of the WCS interoperability standards. <br />
<br />
Standards-based data access can be accomplished by ‘wrapping’ the heterogeneous data into standardized web services, callable through well-defined Internet protocols. The result of this ‘wrapping’ process is an array of homogeneous, virtual datasets that can be accessed through a standard querry language and the returned data are packaged in standard format, directly usable by the consuming services.<br />
<br />
===Quality Assurance===<br />
This is just a place holder. Here we need to explain the quality assurance steps that are needed to prepare the HTAP Integrated Dataset.<br />
<br />
===Homogenization and Integration===<br />
The HTAP Integrated Dataset (HID) will be used to compare models to observations. It will be created from the proper combination of a variety of surface, upper air and satellite observations. However, before the inclusion into HID, each dataset will need to be scrutinezed to make it suitable for model comparison. The scrutiny may include filtering, aggregation and possibly fusion operations.<br />
<br />
A good example is the 400-station AIRNOW network reporting hourly ozone and PM2.5 concentrations over the US. The AIRNOW network includes both urban sites that are strongly influenced by local sources. Urban stations need to be removed from HID dataset.<br />
<br />
Developing the specific criteria and procedures for the HTAP integrated dataset will require the attention of a HTAP subgroup.<br />
==== Data Selection Criteria====<br />
Monitoring data for atmospheric constituents are now available from a variety of sources, not all of which are suitable for the integrated HTAP data set. Given the limited scope and resources of HTAP, it will be necessary to select a subset of the available observational data that would be most appropriate for the preparation of the 2009 assessment. A set of criteria for the data selection is given below.<br />
* The suitability criteria may include the measured parameters, their spatial extent and coverage density, as well as the time range and sampling frequency with major emphasis on data quality (defined as???).<br />
* The compilation of the global integrated data sets should initially focus on PM and ozone, including the gaseous precursors.<br />
* Subsequent data collaction should also include observations on atmospheric mercury and persistent organic pollutants (POPS). <br />
* In order to cover hemispheric transport of air pollutants the data set system should accept and utilize data from outside the geographical scope of EMEP.<br />
* The data gathering should begin the compilation using the of existing databases in Europe (RETRO, TRADEOFF, QUANTIFY, CRATE, AEROCOM) and virtual federated systems in the US (DataFed, Giovanni and others). <br />
* TF data integration should also contribute to and benefit from other ongoing data integration efforts to integrate the global data resources, most notably, with ACCENT project in Europe, and similar efforts in the US.<br />
* Special emphasis should be placed on the collection of suitable vertical profiles from aircraft measurements as well as from surface and space-borne lidars.<br />
<br />
The evaluation of suitabile observational data sets for model validation and fusion will require close interaction between the modeling and observational communities. An wiki workspace for open collaborative evaluation of different datasets is desirebale.<br />
<br />
==Integrated Data Set==<br />
<br />
The providers of observational, emission, and modeling data are distributed. The individual data sets will be funeled into the integrated HTAP data set. This data set transfer will be accomplished using standard data transfer protocols. Transfering the data from the provider to the user will include two additional observations for each data set, quality assurance and semantic transformation.<br />
<br />
On the left are the inputs from observations, emissions, and model data.<br />
<br />
<br />
<br />
<br />
<br />
<br />
[[Image:HTAP_IntDataSys.png|600px|left]] In order to accomplish the objectives set for the HTAP TF it is evident that it will require the System of Systems approach as envisioned by GEO. Both the modeling of the chemical constituents as well as the Earth observations used to document the hemispheric transport are being performed by individual autonomous projects and programs (Systems). A key function of the Task Force is to assess the contributions of the individual systems and then integrate those into a coherent assessment. Much of this interim report is in fact such an assessment of the current knowledge based on previous or past models and observations.<br />
<br />
In the next, upcoming phase of HTAP is to actually integrate the existing modeling and observing systems such that their combination and fusion can yield a deeper and more robust understanding of HTAP. since this next phase will involve integration of multiple relevant data sets, comparison and evaluation of multiple models, as well as the reconciliation of the observations and models to create the new and deeper understanding . In order to execute this extensive integration, it will be necessary to conduct this integration using a generally accepted framework as well as accepted scientific methods. It is believed that the GEO architecture and the System of Systems is an attractive approach to HTAP integration.<br />
<br />
The HTAP program can also be considered an early demonstration of the GEO concepts through its end to end approach. Both the atmospheric modeling, as well as the observations currently exist through the operation of existing modeling and observation systems. The Task Force has agreed organizing and evaluating those models, assembling and integrating the observational data sets and then reconciling the models with the observations will be the focus of the Phase II effort. The Task Force will also prepare and deliver a report to LRTP to aid its deliberations and its decision making processes. This sequence of activities constitutes an end to end approach that turns observations and models into actionable knowledge for societal decision making. One could say that this is an octagonal approach to more deliberate step by step development of GEOSS where in Phase I interoperability, in Phase II ???<br />
<br />
Provider nodes <br />
* Federated Data System DataFed<br />
* NASA Data System Giovanni<br />
* Emission Data System NEISGEI<br />
* Juelich Data System<br />
* AeorCom<br />
* EMEP<br />
<br />
<br />
Data Networks. Connected hubs; Show Core HTAP network<br />
<br />
[http://datafedwiki.wustl.edu/index.php/2007-03-26:_Datasets_for_HTAP_through_DataFed See 20 selected datasets] in the federated data systems, DataFed<br />
<br />
===Perform QA at all steps===<br />
<br />
==Reconciliation and Integration of Observations, Emissions and Models==<br />
===Compare observations to models and emissions===<br />
===Characterize pollutant pattern and SSR=== <br />
===Develop real-time data assimilation===<br />
===Perform reanalysis with assimilated data=== <br />
<br />
==Organizational Aspects==<br />
===HTAP Relationship to GEO and GEOSS===<br />
There is an outstanding opportunity to develop a mutually beneficial and supportive relationship between the activities of the HTAP Task Force and that of GEO. The Group of Earth Observations with its broad range of national and organizational members has developed a general architectural framework for turning Earth observations into societal benefits. The three main components of this architecture are models, and observations, which then feed into decision support systems used in a variety of societal decision making. This general framework is well suited as an architectural guide to the HTAP program. However, this framework lacks specific guidelines and implementation details that are needed for practical Earth observing and decision support systems. <br />
<br />
[[Image:400px HTAP GEOSS Arch.png|400px|left]] The HTAP program provides an opportunity to extend the GEO framework with more detailed features that arise in specific HTAP application context. In case of HTAP, the major activities are shown in the architectural diagram of Figure ?? The HTAP modeling is conducted through global scale chemical transport models that simulate or forecast the atmospheric chemical composition as a consequence of natural and human induced emissions. The observations arise from satellite, ground-based and airborne observations of chemical constituents and their hemispheric transport. The decision support system consists of the scientists as members of the HTAP task force, and the task force co-chairs as the intermediaries between LRTP and the TF. (Terry, Andre this description of the HTAP DSS needs your help). <br />
<br />
Representing the Decision Support System (DSS) for HTAP is an important aspect because it can guide the design and and implementation of the information system for the support of the DSS processes. A higher resolution DSS design document is also beneficial as a communications medium for the interacting system of systems. A domain specific DSS may also allow the generalization and applicability to the design of DSS structures for similar application domains. <br />
<br />
The implementation of the GEO framework utilizes the concept of Global Observing System of Systems (GEOSS). Traditionally, Earth observations were performed by well defined systems such as specific satellites and monitoring networks which were designed and operated using systems engineering principles. However, GEO recognized that the understanding of Earth system requires the utilization and integration of the individual, autonomous systems. The key difference between the Systems and System of System approaches are highlighted in Table 1.<br />
<br />
===HTAP Relationship to IGACO=== <br />
[[Image:IGACO Framework.png|300px|left]] The HTAP program can also have a mutually beneficial interaction with the IGACO program. Integrated Global Atmospheric Chemistry Observations (IGACO) Theme is part of the Integrated Global Observing Strategy (IGOS). The IGACO program is formulated by an array of experienced atmospheric scientists. IGACO proposes a specific framework for the system components that processes the observational data and models. It also specifies the higher level information products that can be used for creating social benefit. <br />
<br />
In the IGACO framework, the data products from satellite, surface, and aircraft observations are collected for inclusion into an integrated data archive. A key component in the IGACO data flow is the mandatory quality assurance and QA QC that precedes the archiving. This is particularly important for HTAP where multi-sensory data from many different providers are to be combined into an integrated database. In the IGACO framework, a key output from the data system is a high integrated spatial-temporal data set which can be used in a multitude of applications that produce socal benefit. The goal of creating such an integrated data set is shared by the IGACO and the HTAP programs (?? Len, this section could benefit from your input)<br />
<br />
===HTAP Relationship with Other Programs===<br />
---<br />
---<br />
<br />
<br />
<br />
==Links==<br />
[[IGACO Framework]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=HTAP_Report,_Sub-Chap._6_-_Data/Info_System&diff=7949HTAP Report, Sub-Chap. 6 - Data/Info System2007-04-12T05:03:13Z<p>128.252.167.79: /* Integrate multi-sensory data into a coherent data systems */</p>
<hr />
<div><noinclude>[[Integrated_Global_Dataset| < Back to Integrated Global Dataset on ESIP wiki]] <br></noinclude> <br />
<br />
This wiki page is inteded to be the collaborative workspace for Task Force members interested in the HTAP Information System development, implementation and use. The contents can be freely edited by anyone who is logged in. Comments and feedback can also be sent by email to rhusar@me.wustl.edu. The content of this page is is on the '''''Information System''''' that supposts Chapter 6. The subsections are action-oriented. <br />
<br />
From [http://www.htap.org/activities/2007_interim_report/TF%20HTAP%202007%20Interim%20Report%20Prospectus%20Oct%202006.pdf Oct 2006 Prospectus]: This chapter (Chapter 6: '''Integration of Observations, Emissions and Modeling'''.) will discuss the need to integrate information from observations, models, and emissions inventories to better understand intercontinental transport. A draft [[HTAP Report Chap. 6 - Jan 07 Outline]] is also on this wiki.<br />
<br />
==Introduction==<br />
The goal of the HTAP data system is to aid the assembly of an appropriate set of observations and models, and to facilitate the integration and fusion of observations with models. <br />
<br />
Recent developments in air quality monitoring, modeling and information technologies offer outstanding<br />
opportunities to fulfill the information needs for the HTAP integration effort. The data from surface-based air pollution monitoring networks now routinely provide spatio-temporal and chemical patterns of ozone and PM. Satellite<br />
sensors with global coverage and kilometer-scale spatial resolution now provide real-time snapshots which depict the pattern of industrial haze, smoke and dust in stunning detail. The ‘terabytes’ of data from these surface and remote sensors can now be stored, processed and delivered in near-real time. The instantaneous ‘horizontal’ diffusion of information via the Internet now permits, in principle, the delivery of the right information to the right people at the right place and time. Standardized computer-computer communication languages and Service-Oriented Architectures (SOA) now facilitate the flexible access, quality assurance and processing of raw data into high-grade ‘actionable’ knowledge suitable for HTAP policy decissions. Last but not least, the World Wide Web has opened the way to generous sharing of data and tools leading to collaborative analysis and virtual workgroups. Nevertheless, air quality data analyses and data model integration face significant hurdles. The “data deluge” problem is especially acute for satellite observations and model outputs. <br />
<br />
<br />
<br />
====HTAP Data System Architecture====<br />
The proposed HTAP data system will integrate the emerging observational and modeling resources by not a centrally planned and powerful, dynamic technologies and through a collaborative federation philosophy. The key roles of the federated data system are to (1) facilitate accsess to the distributed data; (2) ensure the seamless flow of data through interoperable interfaces; (3) provide a set of basic tools for data exploration and processing. <br />
<br />
The Service Oriented Architecture (SOA) allows connecting the web service components (e.g. services for data access, transformation, fusion, rendering, etc.) using work flow software.<br />
<br />
===Homogenize the data into an interoperable dataset===<br />
The primary goal of the data system is to allow the integration of observational emission and model data. The model evaluation requires that both the observations and if possible the emissions are fixed. For this reason it is desirable to prepare an integrated observational database to which the various model implementations can be compared to. On one hand, the integrated data set should be as inclusive as possible. On the other hand, such a goal needs to be tempered by many limitations that preclude a broad, inclusive data integration<br />
<br />
The proposed HTAP data system will be a hybrid combination of both distributed and fixed components as illustrated schematically in Figure ??. Both the data providers as well as the HTAP analysts-users will be distributed. However, they will be connected through an integrated HTAP database which should be a stable, virtually fixed database. The section below describes the main component of this distributed data system. <br />
<br />
The HTAP integrated data system has to perform two major functions. The primary function is to facilitate the creation of an integrated data set suitable for model evaluation and pollutant characterization. The multiple steps that are required for this functionality are shown on the left. The sequence of operations can be viewed as a value chain that transform the raw input data into a highly organized integrated data set. Since these value adding steps have to be performed for each candidate data set the operations can be performed in sequence following the rules of data flow programming. For true quality assurance and for data homogenization the data flow channels for individual data sets need to be evaluated in the context of other data. <br />
<br />
The operations that prepare the integrated data set can be viewed as services that operate on the data stream using the principles of modern Service Oriented Architecure (SOA). Each of the services with its clearly defined functionality and firmly specified service interface. In principle, the standards-based interface allows a connection of service chains using formal work flow software. The main benefit of service oriented architecture is that allows the building of agile application programs that can respond to changes in the data input conditions as well as the output requirements. The service oriented software architecture is an implementation of the System of Systems approach. Furthermore, since each of the services can be operated by autonomous providers and the "system" that implements the service is behind the service interfece. In other words, following the SOA approach, not only the data providers are distributed but also the processing services as well. This flexible approach to distributed computing allows the distribution of labor (chain of services) in many different configurations. For instance, wrapping the existing data with standards based interfaces for data access can be performed for an entire cluster of data. This is the approach taken in the federated data system DataFed. Given a standard interface to a variety of data sets, the Quality Assurance service can be performed by another srvice provider that is seamlesly connected to the data access mediator. Similarly, the service that prepares a data set for data integration can be provided by another service in the data flow network. The flexibility offered through the chaining and orchestration of distributed, loosely coupled web services is the architectural support for the building of agile data systems.<br />
<br />
===Interoperability Standards and Data Wrapping===<br />
Interoperable data access will be accomplished through the use of a suite of international standards. The naming of individual chemical parameters will follow the CF convention used by the Climate and Forecast (CF) communities. The existing names for atmospheric chemicals in the CF convention were inadequate to accomodate all the parameters used in the HTAP modeling. in order to remedy this shortcomming the list of standard names was extended by the HTAP community under leadership of C. Textor. She also became a memeber of the CF convention board that is the custodian of the standard names. The standard names for HTAP models were developed using a collaborative wiki workspace. <br />
<br />
The data transfer from providers to users also benefit from standard data formats. For modeling data the use of netCDF-CF as a standard format is recomended. The use of a standard physical data format and the CF naming conventions allows, in principle, the seamless connection between data provider and consumer services. It should be noted, however, that at this time the CF naming convention has only been developed for the model parameters and not for the various observational parameters. Also, the nedCDF CF is primarily used as a model data exchange format, while the transfer of surface monitoring data and satellite data is less standardized.<br />
<br />
The third aspect of data interoperability is a standard query language through which user services request specific data from the provider services. It is proposed that for the HTAP data information system adapts the Web Coverage Service (WCS) as the standard data query language. The WCS data access protocol is defined by the international Open Geospatial Consortium (OGC), which is also the key organization responsible for interoperability standards in GEOSS. Since the WCS protocol was originally developed for the GIS community, it was necessary to adapt it to the needs of "fluid Earth sciences". Members of the HTAP group have been actively participating in the development and testing of the WCS interoperability standards. <br />
<br />
Standards-based data access can be accomplished by ‘wrapping’ the heterogeneous data into standardized web services, callable through well-defined Internet protocols. The result of this ‘wrapping’ process is an array of homogeneous, virtual datasets that can be accessed through a standard querry language and the returned data are packaged in standard format, directly usable by the consuming services.<br />
<br />
===Quality Assurance===<br />
This is just a place holder. Here we need to explain the quality assurance steps that are needed to prepare the HTAP Integrated Dataset.<br />
<br />
===Homogenization and Integration===<br />
The HTAP Integrated Dataset (HID) will be used to compare models to observations. It will be created from the proper combination of a variety of surface, upper air and satellite observations. However, before the inclusion into HID, each dataset will need to be scrutinezed to make it suitable for model comparison. The scrutiny may include filtering, aggregation and possibly fusion operations.<br />
<br />
A good example is the 400-station AIRNOW network reporting hourly ozone and PM2.5 concentrations over the US. The AIRNOW network includes both urban sites that are strongly influenced by local sources. Urban stations need to be removed from HID dataset.<br />
<br />
Developing the specific criteria and procedures for the HTAP integrated dataset will require the attention of a HTAP subgroup.<br />
==== Data Selection Criteria====<br />
Monitoring data for atmospheric constituents are now available from a variety of sources, not all of which are suitable for the integrated HTAP data set. Given the limited scope and resources of HTAP, it will be necessary to select a subset of the available observational data that would be most appropriate for the preparation of the 2009 assessment. A set of criteria for the data selection is given below.<br />
* The suitability criteria may include the measured parameters, their spatial extent and coverage density, as well as the time range and sampling frequency with major emphasis on data quality (defined as???).<br />
* The compilation of the global integrated data sets should initially focus on PM and ozone, including the gaseous precursors.<br />
* Subsequent data collaction should also include observations on atmospheric mercury and persistent organic pollutants (POPS). <br />
* In order to cover hemispheric transport of air pollutants the data set system should accept and utilize data from outside the geographical scope of EMEP.<br />
* The data gathering should begin the compilation using the of existing databases in Europe (RETRO, TRADEOFF, QUANTIFY, CRATE, AEROCOM) and virtual federated systems in the US (DataFed, Giovanni and others). <br />
* TF data integration should also contribute to and benefit from other ongoing data integration efforts to integrate the global data resources, most notably, with ACCENT project in Europe, and similar efforts in the US.<br />
* Special emphasis should be placed on the collection of suitable vertical profiles from aircraft measurements as well as from surface and space-borne lidars.<br />
<br />
The evaluation of suitabile observational data sets for model validation and fusion will require close interaction between the modeling and observational communities. An wiki workspace for open collaborative evaluation of different datasets is desirebale.<br />
<br />
==Integrated Data Set==<br />
<br />
The providers of observational, emission, and modeling data are distributed. The individual data sets will be funeled into the integrated HTAP data set. This data set transfer will be accomplished using standard data transfer protocols. Transfering the data from the provider to the user will include two additional observations for each data set, quality assurance and semantic transformation.<br />
<br />
On the left the inputs from observations, emissions, and model data<br />
<br />
<br />
<br />
<br />
<br />
<br />
[[Image:HTAP_IntDataSys.png|600px|left]] In order to accomplish the objectives set for the HTAP TF it is evident that it will require the System of Systems approach as envisioned by GEO. Both the modeling of the chemical constituents as well as the Earth observations used to document the hemispheric transport are being performed by individual autonomous projects and programs (Systems). A key function of the Task Force is to assess the contributions of the individual systems and then integrate those into a coherent assessment. Much of this interim report is in fact such an assessment of the current knowledge based on previous or past models and observations.<br />
<br />
In the next, upcoming phase of HTAP is to actually integrate the existing modeling and observing systems such that their combination and fusion can yield a deeper and more robust understanding of HTAP. since this next phase will involve integration of multiple relevant data sets, comparison and evaluation of multiple models, as well as the reconciliation of the observations and models to create the new and deeper understanding . In order to execute this extensive integration, it will be necessary to conduct this integration using a generally accepted framework as well as accepted scientific methods. It is believed that the GEO architecture and the System of Systems is an attractive approach to HTAP integration.<br />
<br />
The HTAP program can also be considered an early demonstration of the GEO concepts through its end to end approach. Both the atmospheric modeling, as well as the observations currently exist through the operation of existing modeling and observation systems. The Task Force has agreed organizing and evaluating those models, assembling and integrating the observational data sets and then reconciling the models with the observations will be the focus of the Phase II effort. The Task Force will also prepare and deliver a report to LRTP to aid its deliberations and its decision making processes. This sequence of activities constitutes an end to end approach that turns observations and models into actionable knowledge for societal decision making. One could say that this is an octagonal approach to more deliberate step by step development of GEOSS where in Phase I interoperability, in Phase II ???<br />
<br />
Provider nodes <br />
* Federated Data System DataFed<br />
* NASA Data System Giovanni<br />
* Emission Data System NEISGEI<br />
* Juelich Data System<br />
* AeorCom<br />
* EMEP<br />
<br />
<br />
Data Networks. Connected hubs; Show Core HTAP network<br />
<br />
[http://datafedwiki.wustl.edu/index.php/2007-03-26:_Datasets_for_HTAP_through_DataFed See 20 selected datasets] in the federated data systems, DataFed<br />
<br />
===Perform QA at all steps===<br />
<br />
==Reconciliation and Integration of Observations, Emissions and Models==<br />
===Compare observations to models and emissions===<br />
===Characterize pollutant pattern and SSR=== <br />
===Develop real-time data assimilation===<br />
===Perform reanalysis with assimilated data=== <br />
<br />
==Organizational Aspects==<br />
===HTAP Relationship to GEO and GEOSS===<br />
There is an outstanding opportunity to develop a mutually beneficial and supportive relationship between the activities of the HTAP Task Force and that of GEO. The Group of Earth Observations with its broad range of national and organizational members has developed a general architectural framework for turning Earth observations into societal benefits. The three main components of this architecture are models, and observations, which then feed into decision support systems used in a variety of societal decision making. This general framework is well suited as an architectural guide to the HTAP program. However, this framework lacks specific guidelines and implementation details that are needed for practical Earth observing and decision support systems. <br />
<br />
[[Image:400px HTAP GEOSS Arch.png|400px|left]] The HTAP program provides an opportunity to extend the GEO framework with more detailed features that arise in specific HTAP application context. In case of HTAP, the major activities are shown in the architectural diagram of Figure ?? The HTAP modeling is conducted through global scale chemical transport models that simulate or forecast the atmospheric chemical composition as a consequence of natural and human induced emissions. The observations arise from satellite, ground-based and airborne observations of chemical constituents and their hemispheric transport. The decision support system consists of the scientists as members of the HTAP task force, and the task force co-chairs as the intermediaries between LRTP and the TF. (Terry, Andre this description of the HTAP DSS needs your help). <br />
<br />
Representing the Decision Support System (DSS) for HTAP is an important aspect because it can guide the design and and implementation of the information system for the support of the DSS processes. A higher resolution DSS design document is also beneficial as a communications medium for the interacting system of systems. A domain specific DSS may also allow the generalization and applicability to the design of DSS structures for similar application domains. <br />
<br />
The implementation of the GEO framework utilizes the concept of Global Observing System of Systems (GEOSS). Traditionally, Earth observations were performed by well defined systems such as specific satellites and monitoring networks which were designed and operated using systems engineering principles. However, GEO recognized that the understanding of Earth system requires the utilization and integration of the individual, autonomous systems. The key difference between the Systems and System of System approaches are highlighted in Table 1.<br />
<br />
===HTAP Relationship to IGACO=== <br />
[[Image:IGACO Framework.png|300px|left]] The HTAP program can also have a mutually beneficial interaction with the IGACO program. Integrated Global Atmospheric Chemistry Observations (IGACO) Theme is part of the Integrated Global Observing Strategy (IGOS). The IGACO program is formulated by an array of experienced atmospheric scientists. IGACO proposes a specific framework for the system components that processes the observational data and models. It also specifies the higher level information products that can be used for creating social benefit. <br />
<br />
In the IGACO framework, the data products from satellite, surface, and aircraft observations are collected for inclusion into an integrated data archive. A key component in the IGACO data flow is the mandatory quality assurance and QA QC that precedes the archiving. This is particularly important for HTAP where multi-sensory data from many different providers are to be combined into an integrated database. In the IGACO framework, a key output from the data system is a high integrated spatial-temporal data set which can be used in a multitude of applications that produce socal benefit. The goal of creating such an integrated data set is shared by the IGACO and the HTAP programs (?? Len, this section could benefit from your input)<br />
<br />
===HTAP Relationship with Other Programs===<br />
---<br />
---<br />
<br />
<br />
<br />
==Links==<br />
[[IGACO Framework]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=HTAP_Report,_Sub-Chap._6_-_Data/Info_System&diff=7948HTAP Report, Sub-Chap. 6 - Data/Info System2007-04-12T05:02:35Z<p>128.252.167.79: /* Identify providers and datasets relevant to HTAP */</p>
<hr />
<div><noinclude>[[Integrated_Global_Dataset| < Back to Integrated Global Dataset on ESIP wiki]] <br></noinclude> <br />
<br />
This wiki page is inteded to be the collaborative workspace for Task Force members interested in the HTAP Information System development, implementation and use. The contents can be freely edited by anyone who is logged in. Comments and feedback can also be sent by email to rhusar@me.wustl.edu. The content of this page is is on the '''''Information System''''' that supposts Chapter 6. The subsections are action-oriented. <br />
<br />
From [http://www.htap.org/activities/2007_interim_report/TF%20HTAP%202007%20Interim%20Report%20Prospectus%20Oct%202006.pdf Oct 2006 Prospectus]: This chapter (Chapter 6: '''Integration of Observations, Emissions and Modeling'''.) will discuss the need to integrate information from observations, models, and emissions inventories to better understand intercontinental transport. A draft [[HTAP Report Chap. 6 - Jan 07 Outline]] is also on this wiki.<br />
<br />
==Introduction==<br />
The goal of the HTAP data system is to aid the assembly of an appropriate set of observations and models, and to facilitate the integration and fusion of observations with models. <br />
<br />
Recent developments in air quality monitoring, modeling and information technologies offer outstanding<br />
opportunities to fulfill the information needs for the HTAP integration effort. The data from surface-based air pollution monitoring networks now routinely provide spatio-temporal and chemical patterns of ozone and PM. Satellite<br />
sensors with global coverage and kilometer-scale spatial resolution now provide real-time snapshots which depict the pattern of industrial haze, smoke and dust in stunning detail. The ‘terabytes’ of data from these surface and remote sensors can now be stored, processed and delivered in near-real time. The instantaneous ‘horizontal’ diffusion of information via the Internet now permits, in principle, the delivery of the right information to the right people at the right place and time. Standardized computer-computer communication languages and Service-Oriented Architectures (SOA) now facilitate the flexible access, quality assurance and processing of raw data into high-grade ‘actionable’ knowledge suitable for HTAP policy decissions. Last but not least, the World Wide Web has opened the way to generous sharing of data and tools leading to collaborative analysis and virtual workgroups. Nevertheless, air quality data analyses and data model integration face significant hurdles. The “data deluge” problem is especially acute for satellite observations and model outputs. <br />
<br />
<br />
<br />
====HTAP Data System Architecture====<br />
The proposed HTAP data system will integrate the emerging observational and modeling resources by not a centrally planned and powerful, dynamic technologies and through a collaborative federation philosophy. The key roles of the federated data system are to (1) facilitate accsess to the distributed data; (2) ensure the seamless flow of data through interoperable interfaces; (3) provide a set of basic tools for data exploration and processing. <br />
<br />
The Service Oriented Architecture (SOA) allows connecting the web service components (e.g. services for data access, transformation, fusion, rendering, etc.) using work flow software.<br />
<br />
===Homogenize the data into an interoperable dataset===<br />
The primary goal of the data system is to allow the integration of observational emission and model data. The model evaluation requires that both the observations and if possible the emissions are fixed. For this reason it is desirable to prepare an integrated observational database to which the various model implementations can be compared to. On one hand, the integrated data set should be as inclusive as possible. On the other hand, such a goal needs to be tempered by many limitations that preclude a broad, inclusive data integration<br />
<br />
The proposed HTAP data system will be a hybrid combination of both distributed and fixed components as illustrated schematically in Figure ??. Both the data providers as well as the HTAP analysts-users will be distributed. However, they will be connected through an integrated HTAP database which should be a stable, virtually fixed database. The section below describes the main component of this distributed data system. <br />
<br />
The HTAP integrated data system has to perform two major functions. The primary function is to facilitate the creation of an integrated data set suitable for model evaluation and pollutant characterization. The multiple steps that are required for this functionality are shown on the left. The sequence of operations can be viewed as a value chain that transform the raw input data into a highly organized integrated data set. Since these value adding steps have to be performed for each candidate data set the operations can be performed in sequence following the rules of data flow programming. For true quality assurance and for data homogenization the data flow channels for individual data sets need to be evaluated in the context of other data. <br />
<br />
The operations that prepare the integrated data set can be viewed as services that operate on the data stream using the principles of modern Service Oriented Architecure (SOA). Each of the services with its clearly defined functionality and firmly specified service interface. In principle, the standards-based interface allows a connection of service chains using formal work flow software. The main benefit of service oriented architecture is that allows the building of agile application programs that can respond to changes in the data input conditions as well as the output requirements. The service oriented software architecture is an implementation of the System of Systems approach. Furthermore, since each of the services can be operated by autonomous providers and the "system" that implements the service is behind the service interfece. In other words, following the SOA approach, not only the data providers are distributed but also the processing services as well. This flexible approach to distributed computing allows the distribution of labor (chain of services) in many different configurations. For instance, wrapping the existing data with standards based interfaces for data access can be performed for an entire cluster of data. This is the approach taken in the federated data system DataFed. Given a standard interface to a variety of data sets, the Quality Assurance service can be performed by another srvice provider that is seamlesly connected to the data access mediator. Similarly, the service that prepares a data set for data integration can be provided by another service in the data flow network. The flexibility offered through the chaining and orchestration of distributed, loosely coupled web services is the architectural support for the building of agile data systems.<br />
<br />
===Interoperability Standards and Data Wrapping===<br />
Interoperable data access will be accomplished through the use of a suite of international standards. The naming of individual chemical parameters will follow the CF convention used by the Climate and Forecast (CF) communities. The existing names for atmospheric chemicals in the CF convention were inadequate to accomodate all the parameters used in the HTAP modeling. in order to remedy this shortcomming the list of standard names was extended by the HTAP community under leadership of C. Textor. She also became a memeber of the CF convention board that is the custodian of the standard names. The standard names for HTAP models were developed using a collaborative wiki workspace. <br />
<br />
The data transfer from providers to users also benefit from standard data formats. For modeling data the use of netCDF-CF as a standard format is recomended. The use of a standard physical data format and the CF naming conventions allows, in principle, the seamless connection between data provider and consumer services. It should be noted, however, that at this time the CF naming convention has only been developed for the model parameters and not for the various observational parameters. Also, the nedCDF CF is primarily used as a model data exchange format, while the transfer of surface monitoring data and satellite data is less standardized.<br />
<br />
The third aspect of data interoperability is a standard query language through which user services request specific data from the provider services. It is proposed that for the HTAP data information system adapts the Web Coverage Service (WCS) as the standard data query language. The WCS data access protocol is defined by the international Open Geospatial Consortium (OGC), which is also the key organization responsible for interoperability standards in GEOSS. Since the WCS protocol was originally developed for the GIS community, it was necessary to adapt it to the needs of "fluid Earth sciences". Members of the HTAP group have been actively participating in the development and testing of the WCS interoperability standards. <br />
<br />
Standards-based data access can be accomplished by ‘wrapping’ the heterogeneous data into standardized web services, callable through well-defined Internet protocols. The result of this ‘wrapping’ process is an array of homogeneous, virtual datasets that can be accessed through a standard querry language and the returned data are packaged in standard format, directly usable by the consuming services.<br />
<br />
===Quality Assurance===<br />
This is just a place holder. Here we need to explain the quality assurance steps that are needed to prepare the HTAP Integrated Dataset.<br />
<br />
===Homogenization and Integration===<br />
The HTAP Integrated Dataset (HID) will be used to compare models to observations. It will be created from the proper combination of a variety of surface, upper air and satellite observations. However, before the inclusion into HID, each dataset will need to be scrutinezed to make it suitable for model comparison. The scrutiny may include filtering, aggregation and possibly fusion operations.<br />
<br />
A good example is the 400-station AIRNOW network reporting hourly ozone and PM2.5 concentrations over the US. The AIRNOW network includes both urban sites that are strongly influenced by local sources. Urban stations need to be removed from HID dataset.<br />
<br />
Developing the specific criteria and procedures for the HTAP integrated dataset will require the attention of a HTAP subgroup.<br />
==== Data Selection Criteria====<br />
Monitoring data for atmospheric constituents are now available from a variety of sources, not all of which are suitable for the integrated HTAP data set. Given the limited scope and resources of HTAP, it will be necessary to select a subset of the available observational data that would be most appropriate for the preparation of the 2009 assessment. A set of criteria for the data selection is given below.<br />
* The suitability criteria may include the measured parameters, their spatial extent and coverage density, as well as the time range and sampling frequency with major emphasis on data quality (defined as???).<br />
* The compilation of the global integrated data sets should initially focus on PM and ozone, including the gaseous precursors.<br />
* Subsequent data collaction should also include observations on atmospheric mercury and persistent organic pollutants (POPS). <br />
* In order to cover hemispheric transport of air pollutants the data set system should accept and utilize data from outside the geographical scope of EMEP.<br />
* The data gathering should begin the compilation using the of existing databases in Europe (RETRO, TRADEOFF, QUANTIFY, CRATE, AEROCOM) and virtual federated systems in the US (DataFed, Giovanni and others). <br />
* TF data integration should also contribute to and benefit from other ongoing data integration efforts to integrate the global data resources, most notably, with ACCENT project in Europe, and similar efforts in the US.<br />
* Special emphasis should be placed on the collection of suitable vertical profiles from aircraft measurements as well as from surface and space-borne lidars.<br />
<br />
The evaluation of suitabile observational data sets for model validation and fusion will require close interaction between the modeling and observational communities. An wiki workspace for open collaborative evaluation of different datasets is desirebale.<br />
<br />
==Integrated Data Set==<br />
<br />
The providers of observational, emission, and modeling data are distributed. The individual data sets will be funeled into the integrated HTAP data set. This data set transfer will be accomplished using standard data transfer protocols. Transfering the data from the provider to the user will include two additional observations for each data set, quality assurance and semantic transformation.<br />
<br />
On the left the inputs from observations, emissions, and model data<br />
<br />
<br />
<br />
<br />
<br />
===Integrate multi-sensory data into a coherent data systems===<br />
<br />
[[Image:HTAP_IntDataSys.png|600px|left]] In order to accomplish the objectives set for the HTAP TF it is evident that it will require the System of Systems approach as envisioned by GEO. Both the modeling of the chemical constituents as well as the Earth observations used to document the hemispheric transport are being performed by individual autonomous projects and programs (Systems). A key function of the Task Force is to assess the contributions of the individual systems and then integrate those into a coherent assessment. Much of this interim report is in fact such an assessment of the current knowledge based on previous or past models and observations.<br />
<br />
In the next, upcoming phase of HTAP is to actually integrate the existing modeling and observing systems such that their combination and fusion can yield a deeper and more robust understanding of HTAP. since this next phase will involve integration of multiple relevant data sets, comparison and evaluation of multiple models, as well as the reconciliation of the observations and models to create the new and deeper understanding . In order to execute this extensive integration, it will be necessary to conduct this integration using a generally accepted framework as well as accepted scientific methods. It is believed that the GEO architecture and the System of Systems is an attractive approach to HTAP integration.<br />
<br />
The HTAP program can also be considered an early demonstration of the GEO concepts through its end to end approach. Both the atmospheric modeling, as well as the observations currently exist through the operation of existing modeling and observation systems. The Task Force has agreed organizing and evaluating those models, assembling and integrating the observational data sets and then reconciling the models with the observations will be the focus of the Phase II effort. The Task Force will also prepare and deliver a report to LRTP to aid its deliberations and its decision making processes. This sequence of activities constitutes an end to end approach that turns observations and models into actionable knowledge for societal decision making. One could say that this is an octagonal approach to more deliberate step by step development of GEOSS where in Phase I interoperability, in Phase II ???<br />
<br />
Provider nodes <br />
* Federated Data System DataFed<br />
* NASA Data System Giovanni<br />
* Emission Data System NEISGEI<br />
* Juelich Data System<br />
* AeorCom<br />
* EMEP<br />
<br />
<br />
Data Networks. Connected hubs; Show Core HTAP network<br />
<br />
[http://datafedwiki.wustl.edu/index.php/2007-03-26:_Datasets_for_HTAP_through_DataFed See 20 selected datasets] in the federated data systems, DataFed<br />
<br />
===Perform QA at all steps===<br />
<br />
==Reconciliation and Integration of Observations, Emissions and Models==<br />
===Compare observations to models and emissions===<br />
===Characterize pollutant pattern and SSR=== <br />
===Develop real-time data assimilation===<br />
===Perform reanalysis with assimilated data=== <br />
<br />
==Organizational Aspects==<br />
===HTAP Relationship to GEO and GEOSS===<br />
There is an outstanding opportunity to develop a mutually beneficial and supportive relationship between the activities of the HTAP Task Force and that of GEO. The Group of Earth Observations with its broad range of national and organizational members has developed a general architectural framework for turning Earth observations into societal benefits. The three main components of this architecture are models, and observations, which then feed into decision support systems used in a variety of societal decision making. This general framework is well suited as an architectural guide to the HTAP program. However, this framework lacks specific guidelines and implementation details that are needed for practical Earth observing and decision support systems. <br />
<br />
[[Image:400px HTAP GEOSS Arch.png|400px|left]] The HTAP program provides an opportunity to extend the GEO framework with more detailed features that arise in specific HTAP application context. In case of HTAP, the major activities are shown in the architectural diagram of Figure ?? The HTAP modeling is conducted through global scale chemical transport models that simulate or forecast the atmospheric chemical composition as a consequence of natural and human induced emissions. The observations arise from satellite, ground-based and airborne observations of chemical constituents and their hemispheric transport. The decision support system consists of the scientists as members of the HTAP task force, and the task force co-chairs as the intermediaries between LRTP and the TF. (Terry, Andre this description of the HTAP DSS needs your help). <br />
<br />
Representing the Decision Support System (DSS) for HTAP is an important aspect because it can guide the design and and implementation of the information system for the support of the DSS processes. A higher resolution DSS design document is also beneficial as a communications medium for the interacting system of systems. A domain specific DSS may also allow the generalization and applicability to the design of DSS structures for similar application domains. <br />
<br />
The implementation of the GEO framework utilizes the concept of Global Observing System of Systems (GEOSS). Traditionally, Earth observations were performed by well defined systems such as specific satellites and monitoring networks which were designed and operated using systems engineering principles. However, GEO recognized that the understanding of Earth system requires the utilization and integration of the individual, autonomous systems. The key difference between the Systems and System of System approaches are highlighted in Table 1.<br />
<br />
===HTAP Relationship to IGACO=== <br />
[[Image:IGACO Framework.png|300px|left]] The HTAP program can also have a mutually beneficial interaction with the IGACO program. Integrated Global Atmospheric Chemistry Observations (IGACO) Theme is part of the Integrated Global Observing Strategy (IGOS). The IGACO program is formulated by an array of experienced atmospheric scientists. IGACO proposes a specific framework for the system components that processes the observational data and models. It also specifies the higher level information products that can be used for creating social benefit. <br />
<br />
In the IGACO framework, the data products from satellite, surface, and aircraft observations are collected for inclusion into an integrated data archive. A key component in the IGACO data flow is the mandatory quality assurance and QA QC that precedes the archiving. This is particularly important for HTAP where multi-sensory data from many different providers are to be combined into an integrated database. In the IGACO framework, a key output from the data system is a high integrated spatial-temporal data set which can be used in a multitude of applications that produce socal benefit. The goal of creating such an integrated data set is shared by the IGACO and the HTAP programs (?? Len, this section could benefit from your input)<br />
<br />
===HTAP Relationship with Other Programs===<br />
---<br />
---<br />
<br />
<br />
<br />
==Links==<br />
[[IGACO Framework]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=HTAP_Report,_Sub-Chap._6_-_Data/Info_System&diff=7947HTAP Report, Sub-Chap. 6 - Data/Info System2007-04-12T05:01:33Z<p>128.252.167.79: /* Homogenization and Integration */</p>
<hr />
<div><noinclude>[[Integrated_Global_Dataset| < Back to Integrated Global Dataset on ESIP wiki]] <br></noinclude> <br />
<br />
This wiki page is inteded to be the collaborative workspace for Task Force members interested in the HTAP Information System development, implementation and use. The contents can be freely edited by anyone who is logged in. Comments and feedback can also be sent by email to rhusar@me.wustl.edu. The content of this page is is on the '''''Information System''''' that supposts Chapter 6. The subsections are action-oriented. <br />
<br />
From [http://www.htap.org/activities/2007_interim_report/TF%20HTAP%202007%20Interim%20Report%20Prospectus%20Oct%202006.pdf Oct 2006 Prospectus]: This chapter (Chapter 6: '''Integration of Observations, Emissions and Modeling'''.) will discuss the need to integrate information from observations, models, and emissions inventories to better understand intercontinental transport. A draft [[HTAP Report Chap. 6 - Jan 07 Outline]] is also on this wiki.<br />
<br />
==Introduction==<br />
The goal of the HTAP data system is to aid the assembly of an appropriate set of observations and models, and to facilitate the integration and fusion of observations with models. <br />
<br />
Recent developments in air quality monitoring, modeling and information technologies offer outstanding<br />
opportunities to fulfill the information needs for the HTAP integration effort. The data from surface-based air pollution monitoring networks now routinely provide spatio-temporal and chemical patterns of ozone and PM. Satellite<br />
sensors with global coverage and kilometer-scale spatial resolution now provide real-time snapshots which depict the pattern of industrial haze, smoke and dust in stunning detail. The ‘terabytes’ of data from these surface and remote sensors can now be stored, processed and delivered in near-real time. The instantaneous ‘horizontal’ diffusion of information via the Internet now permits, in principle, the delivery of the right information to the right people at the right place and time. Standardized computer-computer communication languages and Service-Oriented Architectures (SOA) now facilitate the flexible access, quality assurance and processing of raw data into high-grade ‘actionable’ knowledge suitable for HTAP policy decissions. Last but not least, the World Wide Web has opened the way to generous sharing of data and tools leading to collaborative analysis and virtual workgroups. Nevertheless, air quality data analyses and data model integration face significant hurdles. The “data deluge” problem is especially acute for satellite observations and model outputs. <br />
<br />
<br />
<br />
====HTAP Data System Architecture====<br />
The proposed HTAP data system will integrate the emerging observational and modeling resources by not a centrally planned and powerful, dynamic technologies and through a collaborative federation philosophy. The key roles of the federated data system are to (1) facilitate accsess to the distributed data; (2) ensure the seamless flow of data through interoperable interfaces; (3) provide a set of basic tools for data exploration and processing. <br />
<br />
The Service Oriented Architecture (SOA) allows connecting the web service components (e.g. services for data access, transformation, fusion, rendering, etc.) using work flow software.<br />
<br />
===Homogenize the data into an interoperable dataset===<br />
The primary goal of the data system is to allow the integration of observational emission and model data. The model evaluation requires that both the observations and if possible the emissions are fixed. For this reason it is desirable to prepare an integrated observational database to which the various model implementations can be compared to. On one hand, the integrated data set should be as inclusive as possible. On the other hand, such a goal needs to be tempered by many limitations that preclude a broad, inclusive data integration<br />
<br />
The proposed HTAP data system will be a hybrid combination of both distributed and fixed components as illustrated schematically in Figure ??. Both the data providers as well as the HTAP analysts-users will be distributed. However, they will be connected through an integrated HTAP database which should be a stable, virtually fixed database. The section below describes the main component of this distributed data system. <br />
<br />
The HTAP integrated data system has to perform two major functions. The primary function is to facilitate the creation of an integrated data set suitable for model evaluation and pollutant characterization. The multiple steps that are required for this functionality are shown on the left. The sequence of operations can be viewed as a value chain that transform the raw input data into a highly organized integrated data set. Since these value adding steps have to be performed for each candidate data set the operations can be performed in sequence following the rules of data flow programming. For true quality assurance and for data homogenization the data flow channels for individual data sets need to be evaluated in the context of other data. <br />
<br />
The operations that prepare the integrated data set can be viewed as services that operate on the data stream using the principles of modern Service Oriented Architecure (SOA). Each of the services with its clearly defined functionality and firmly specified service interface. In principle, the standards-based interface allows a connection of service chains using formal work flow software. The main benefit of service oriented architecture is that allows the building of agile application programs that can respond to changes in the data input conditions as well as the output requirements. The service oriented software architecture is an implementation of the System of Systems approach. Furthermore, since each of the services can be operated by autonomous providers and the "system" that implements the service is behind the service interfece. In other words, following the SOA approach, not only the data providers are distributed but also the processing services as well. This flexible approach to distributed computing allows the distribution of labor (chain of services) in many different configurations. For instance, wrapping the existing data with standards based interfaces for data access can be performed for an entire cluster of data. This is the approach taken in the federated data system DataFed. Given a standard interface to a variety of data sets, the Quality Assurance service can be performed by another srvice provider that is seamlesly connected to the data access mediator. Similarly, the service that prepares a data set for data integration can be provided by another service in the data flow network. The flexibility offered through the chaining and orchestration of distributed, loosely coupled web services is the architectural support for the building of agile data systems.<br />
<br />
===Interoperability Standards and Data Wrapping===<br />
Interoperable data access will be accomplished through the use of a suite of international standards. The naming of individual chemical parameters will follow the CF convention used by the Climate and Forecast (CF) communities. The existing names for atmospheric chemicals in the CF convention were inadequate to accomodate all the parameters used in the HTAP modeling. in order to remedy this shortcomming the list of standard names was extended by the HTAP community under leadership of C. Textor. She also became a memeber of the CF convention board that is the custodian of the standard names. The standard names for HTAP models were developed using a collaborative wiki workspace. <br />
<br />
The data transfer from providers to users also benefit from standard data formats. For modeling data the use of netCDF-CF as a standard format is recomended. The use of a standard physical data format and the CF naming conventions allows, in principle, the seamless connection between data provider and consumer services. It should be noted, however, that at this time the CF naming convention has only been developed for the model parameters and not for the various observational parameters. Also, the nedCDF CF is primarily used as a model data exchange format, while the transfer of surface monitoring data and satellite data is less standardized.<br />
<br />
The third aspect of data interoperability is a standard query language through which user services request specific data from the provider services. It is proposed that for the HTAP data information system adapts the Web Coverage Service (WCS) as the standard data query language. The WCS data access protocol is defined by the international Open Geospatial Consortium (OGC), which is also the key organization responsible for interoperability standards in GEOSS. Since the WCS protocol was originally developed for the GIS community, it was necessary to adapt it to the needs of "fluid Earth sciences". Members of the HTAP group have been actively participating in the development and testing of the WCS interoperability standards. <br />
<br />
Standards-based data access can be accomplished by ‘wrapping’ the heterogeneous data into standardized web services, callable through well-defined Internet protocols. The result of this ‘wrapping’ process is an array of homogeneous, virtual datasets that can be accessed through a standard querry language and the returned data are packaged in standard format, directly usable by the consuming services.<br />
<br />
===Quality Assurance===<br />
This is just a place holder. Here we need to explain the quality assurance steps that are needed to prepare the HTAP Integrated Dataset.<br />
<br />
===Homogenization and Integration===<br />
The HTAP Integrated Dataset (HID) will be used to compare models to observations. It will be created from the proper combination of a variety of surface, upper air and satellite observations. However, before the inclusion into HID, each dataset will need to be scrutinezed to make it suitable for model comparison. The scrutiny may include filtering, aggregation and possibly fusion operations.<br />
<br />
A good example is the 400-station AIRNOW network reporting hourly ozone and PM2.5 concentrations over the US. The AIRNOW network includes both urban sites that are strongly influenced by local sources. Urban stations need to be removed from HID dataset.<br />
<br />
Developing the specific criteria and procedures for the HTAP integrated dataset will require the attention of a HTAP subgroup.<br />
==== Data Selection Criteria====<br />
Monitoring data for atmospheric constituents are now available from a variety of sources, not all of which are suitable for the integrated HTAP data set. Given the limited scope and resources of HTAP, it will be necessary to select a subset of the available observational data that would be most appropriate for the preparation of the 2009 assessment. A set of criteria for the data selection is given below.<br />
* The suitability criteria may include the measured parameters, their spatial extent and coverage density, as well as the time range and sampling frequency with major emphasis on data quality (defined as???).<br />
* The compilation of the global integrated data sets should initially focus on PM and ozone, including the gaseous precursors.<br />
* Subsequent data collaction should also include observations on atmospheric mercury and persistent organic pollutants (POPS). <br />
* In order to cover hemispheric transport of air pollutants the data set system should accept and utilize data from outside the geographical scope of EMEP.<br />
* The data gathering should begin the compilation using the of existing databases in Europe (RETRO, TRADEOFF, QUANTIFY, CRATE, AEROCOM) and virtual federated systems in the US (DataFed, Giovanni and others). <br />
* TF data integration should also contribute to and benefit from other ongoing data integration efforts to integrate the global data resources, most notably, with ACCENT project in Europe, and similar efforts in the US.<br />
* Special emphasis should be placed on the collection of suitable vertical profiles from aircraft measurements as well as from surface and space-borne lidars.<br />
<br />
The evaluation of suitabile observational data sets for model validation and fusion will require close interaction between the modeling and observational communities. An wiki workspace for open collaborative evaluation of different datasets is desirebale.<br />
<br />
==Integrated Data Set==<br />
<br />
The providers of observational, emission, and modeling data are distributed. The individual data sets will be funeled into the integrated HTAP data set. This data set transfer will be accomplished using standard data transfer protocols. Transfering the data from the provider to the user will include two additional observations for each data set, quality assurance and semantic transformation.<br />
<br />
On the left the inputs from observations, emissions, and model data<br />
<br />
<br />
<br />
===Identify providers and datasets relevant to HTAP===<br />
===Integrate multi-sensory data into a coherent data systems===<br />
<br />
[[Image:HTAP_IntDataSys.png|600px|left]] In order to accomplish the objectives set for the HTAP TF it is evident that it will require the System of Systems approach as envisioned by GEO. Both the modeling of the chemical constituents as well as the Earth observations used to document the hemispheric transport are being performed by individual autonomous projects and programs (Systems). A key function of the Task Force is to assess the contributions of the individual systems and then integrate those into a coherent assessment. Much of this interim report is in fact such an assessment of the current knowledge based on previous or past models and observations.<br />
<br />
In the next, upcoming phase of HTAP is to actually integrate the existing modeling and observing systems such that their combination and fusion can yield a deeper and more robust understanding of HTAP. since this next phase will involve integration of multiple relevant data sets, comparison and evaluation of multiple models, as well as the reconciliation of the observations and models to create the new and deeper understanding . In order to execute this extensive integration, it will be necessary to conduct this integration using a generally accepted framework as well as accepted scientific methods. It is believed that the GEO architecture and the System of Systems is an attractive approach to HTAP integration.<br />
<br />
The HTAP program can also be considered an early demonstration of the GEO concepts through its end to end approach. Both the atmospheric modeling, as well as the observations currently exist through the operation of existing modeling and observation systems. The Task Force has agreed organizing and evaluating those models, assembling and integrating the observational data sets and then reconciling the models with the observations will be the focus of the Phase II effort. The Task Force will also prepare and deliver a report to LRTP to aid its deliberations and its decision making processes. This sequence of activities constitutes an end to end approach that turns observations and models into actionable knowledge for societal decision making. One could say that this is an octagonal approach to more deliberate step by step development of GEOSS where in Phase I interoperability, in Phase II ???<br />
<br />
Provider nodes <br />
* Federated Data System DataFed<br />
* NASA Data System Giovanni<br />
* Emission Data System NEISGEI<br />
* Juelich Data System<br />
* AeorCom<br />
* EMEP<br />
<br />
<br />
Data Networks. Connected hubs; Show Core HTAP network<br />
<br />
[http://datafedwiki.wustl.edu/index.php/2007-03-26:_Datasets_for_HTAP_through_DataFed See 20 selected datasets] in the federated data systems, DataFed<br />
<br />
===Perform QA at all steps===<br />
<br />
==Reconciliation and Integration of Observations, Emissions and Models==<br />
===Compare observations to models and emissions===<br />
===Characterize pollutant pattern and SSR=== <br />
===Develop real-time data assimilation===<br />
===Perform reanalysis with assimilated data=== <br />
<br />
==Organizational Aspects==<br />
===HTAP Relationship to GEO and GEOSS===<br />
There is an outstanding opportunity to develop a mutually beneficial and supportive relationship between the activities of the HTAP Task Force and that of GEO. The Group of Earth Observations with its broad range of national and organizational members has developed a general architectural framework for turning Earth observations into societal benefits. The three main components of this architecture are models, and observations, which then feed into decision support systems used in a variety of societal decision making. This general framework is well suited as an architectural guide to the HTAP program. However, this framework lacks specific guidelines and implementation details that are needed for practical Earth observing and decision support systems. <br />
<br />
[[Image:400px HTAP GEOSS Arch.png|400px|left]] The HTAP program provides an opportunity to extend the GEO framework with more detailed features that arise in specific HTAP application context. In case of HTAP, the major activities are shown in the architectural diagram of Figure ?? The HTAP modeling is conducted through global scale chemical transport models that simulate or forecast the atmospheric chemical composition as a consequence of natural and human induced emissions. The observations arise from satellite, ground-based and airborne observations of chemical constituents and their hemispheric transport. The decision support system consists of the scientists as members of the HTAP task force, and the task force co-chairs as the intermediaries between LRTP and the TF. (Terry, Andre this description of the HTAP DSS needs your help). <br />
<br />
Representing the Decision Support System (DSS) for HTAP is an important aspect because it can guide the design and and implementation of the information system for the support of the DSS processes. A higher resolution DSS design document is also beneficial as a communications medium for the interacting system of systems. A domain specific DSS may also allow the generalization and applicability to the design of DSS structures for similar application domains. <br />
<br />
The implementation of the GEO framework utilizes the concept of Global Observing System of Systems (GEOSS). Traditionally, Earth observations were performed by well defined systems such as specific satellites and monitoring networks which were designed and operated using systems engineering principles. However, GEO recognized that the understanding of Earth system requires the utilization and integration of the individual, autonomous systems. The key difference between the Systems and System of System approaches are highlighted in Table 1.<br />
<br />
===HTAP Relationship to IGACO=== <br />
[[Image:IGACO Framework.png|300px|left]] The HTAP program can also have a mutually beneficial interaction with the IGACO program. Integrated Global Atmospheric Chemistry Observations (IGACO) Theme is part of the Integrated Global Observing Strategy (IGOS). The IGACO program is formulated by an array of experienced atmospheric scientists. IGACO proposes a specific framework for the system components that processes the observational data and models. It also specifies the higher level information products that can be used for creating social benefit. <br />
<br />
In the IGACO framework, the data products from satellite, surface, and aircraft observations are collected for inclusion into an integrated data archive. A key component in the IGACO data flow is the mandatory quality assurance and QA QC that precedes the archiving. This is particularly important for HTAP where multi-sensory data from many different providers are to be combined into an integrated database. In the IGACO framework, a key output from the data system is a high integrated spatial-temporal data set which can be used in a multitude of applications that produce socal benefit. The goal of creating such an integrated data set is shared by the IGACO and the HTAP programs (?? Len, this section could benefit from your input)<br />
<br />
===HTAP Relationship with Other Programs===<br />
---<br />
---<br />
<br />
<br />
<br />
==Links==<br />
[[IGACO Framework]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=HTAP_Report,_Sub-Chap._6_-_Data/Info_System&diff=7946HTAP Report, Sub-Chap. 6 - Data/Info System2007-04-12T05:00:50Z<p>128.252.167.79: /* Data Selection Criteria */</p>
<hr />
<div><noinclude>[[Integrated_Global_Dataset| < Back to Integrated Global Dataset on ESIP wiki]] <br></noinclude> <br />
<br />
This wiki page is inteded to be the collaborative workspace for Task Force members interested in the HTAP Information System development, implementation and use. The contents can be freely edited by anyone who is logged in. Comments and feedback can also be sent by email to rhusar@me.wustl.edu. The content of this page is is on the '''''Information System''''' that supposts Chapter 6. The subsections are action-oriented. <br />
<br />
From [http://www.htap.org/activities/2007_interim_report/TF%20HTAP%202007%20Interim%20Report%20Prospectus%20Oct%202006.pdf Oct 2006 Prospectus]: This chapter (Chapter 6: '''Integration of Observations, Emissions and Modeling'''.) will discuss the need to integrate information from observations, models, and emissions inventories to better understand intercontinental transport. A draft [[HTAP Report Chap. 6 - Jan 07 Outline]] is also on this wiki.<br />
<br />
==Introduction==<br />
The goal of the HTAP data system is to aid the assembly of an appropriate set of observations and models, and to facilitate the integration and fusion of observations with models. <br />
<br />
Recent developments in air quality monitoring, modeling and information technologies offer outstanding<br />
opportunities to fulfill the information needs for the HTAP integration effort. The data from surface-based air pollution monitoring networks now routinely provide spatio-temporal and chemical patterns of ozone and PM. Satellite<br />
sensors with global coverage and kilometer-scale spatial resolution now provide real-time snapshots which depict the pattern of industrial haze, smoke and dust in stunning detail. The ‘terabytes’ of data from these surface and remote sensors can now be stored, processed and delivered in near-real time. The instantaneous ‘horizontal’ diffusion of information via the Internet now permits, in principle, the delivery of the right information to the right people at the right place and time. Standardized computer-computer communication languages and Service-Oriented Architectures (SOA) now facilitate the flexible access, quality assurance and processing of raw data into high-grade ‘actionable’ knowledge suitable for HTAP policy decissions. Last but not least, the World Wide Web has opened the way to generous sharing of data and tools leading to collaborative analysis and virtual workgroups. Nevertheless, air quality data analyses and data model integration face significant hurdles. The “data deluge” problem is especially acute for satellite observations and model outputs. <br />
<br />
<br />
<br />
====HTAP Data System Architecture====<br />
The proposed HTAP data system will integrate the emerging observational and modeling resources by not a centrally planned and powerful, dynamic technologies and through a collaborative federation philosophy. The key roles of the federated data system are to (1) facilitate accsess to the distributed data; (2) ensure the seamless flow of data through interoperable interfaces; (3) provide a set of basic tools for data exploration and processing. <br />
<br />
The Service Oriented Architecture (SOA) allows connecting the web service components (e.g. services for data access, transformation, fusion, rendering, etc.) using work flow software.<br />
<br />
===Homogenize the data into an interoperable dataset===<br />
The primary goal of the data system is to allow the integration of observational emission and model data. The model evaluation requires that both the observations and if possible the emissions are fixed. For this reason it is desirable to prepare an integrated observational database to which the various model implementations can be compared to. On one hand, the integrated data set should be as inclusive as possible. On the other hand, such a goal needs to be tempered by many limitations that preclude a broad, inclusive data integration<br />
<br />
The proposed HTAP data system will be a hybrid combination of both distributed and fixed components as illustrated schematically in Figure ??. Both the data providers as well as the HTAP analysts-users will be distributed. However, they will be connected through an integrated HTAP database which should be a stable, virtually fixed database. The section below describes the main component of this distributed data system. <br />
<br />
The HTAP integrated data system has to perform two major functions. The primary function is to facilitate the creation of an integrated data set suitable for model evaluation and pollutant characterization. The multiple steps that are required for this functionality are shown on the left. The sequence of operations can be viewed as a value chain that transform the raw input data into a highly organized integrated data set. Since these value adding steps have to be performed for each candidate data set the operations can be performed in sequence following the rules of data flow programming. For true quality assurance and for data homogenization the data flow channels for individual data sets need to be evaluated in the context of other data. <br />
<br />
The operations that prepare the integrated data set can be viewed as services that operate on the data stream using the principles of modern Service Oriented Architecure (SOA). Each of the services with its clearly defined functionality and firmly specified service interface. In principle, the standards-based interface allows a connection of service chains using formal work flow software. The main benefit of service oriented architecture is that allows the building of agile application programs that can respond to changes in the data input conditions as well as the output requirements. The service oriented software architecture is an implementation of the System of Systems approach. Furthermore, since each of the services can be operated by autonomous providers and the "system" that implements the service is behind the service interfece. In other words, following the SOA approach, not only the data providers are distributed but also the processing services as well. This flexible approach to distributed computing allows the distribution of labor (chain of services) in many different configurations. For instance, wrapping the existing data with standards based interfaces for data access can be performed for an entire cluster of data. This is the approach taken in the federated data system DataFed. Given a standard interface to a variety of data sets, the Quality Assurance service can be performed by another srvice provider that is seamlesly connected to the data access mediator. Similarly, the service that prepares a data set for data integration can be provided by another service in the data flow network. The flexibility offered through the chaining and orchestration of distributed, loosely coupled web services is the architectural support for the building of agile data systems.<br />
<br />
===Interoperability Standards and Data Wrapping===<br />
Interoperable data access will be accomplished through the use of a suite of international standards. The naming of individual chemical parameters will follow the CF convention used by the Climate and Forecast (CF) communities. The existing names for atmospheric chemicals in the CF convention were inadequate to accomodate all the parameters used in the HTAP modeling. in order to remedy this shortcomming the list of standard names was extended by the HTAP community under leadership of C. Textor. She also became a memeber of the CF convention board that is the custodian of the standard names. The standard names for HTAP models were developed using a collaborative wiki workspace. <br />
<br />
The data transfer from providers to users also benefit from standard data formats. For modeling data the use of netCDF-CF as a standard format is recomended. The use of a standard physical data format and the CF naming conventions allows, in principle, the seamless connection between data provider and consumer services. It should be noted, however, that at this time the CF naming convention has only been developed for the model parameters and not for the various observational parameters. Also, the nedCDF CF is primarily used as a model data exchange format, while the transfer of surface monitoring data and satellite data is less standardized.<br />
<br />
The third aspect of data interoperability is a standard query language through which user services request specific data from the provider services. It is proposed that for the HTAP data information system adapts the Web Coverage Service (WCS) as the standard data query language. The WCS data access protocol is defined by the international Open Geospatial Consortium (OGC), which is also the key organization responsible for interoperability standards in GEOSS. Since the WCS protocol was originally developed for the GIS community, it was necessary to adapt it to the needs of "fluid Earth sciences". Members of the HTAP group have been actively participating in the development and testing of the WCS interoperability standards. <br />
<br />
Standards-based data access can be accomplished by ‘wrapping’ the heterogeneous data into standardized web services, callable through well-defined Internet protocols. The result of this ‘wrapping’ process is an array of homogeneous, virtual datasets that can be accessed through a standard querry language and the returned data are packaged in standard format, directly usable by the consuming services.<br />
<br />
===Quality Assurance===<br />
This is just a place holder. Here we need to explain the quality assurance steps that are needed to prepare the HTAP Integrated Dataset.<br />
<br />
===Homogenization and Integration===<br />
The HTAP Integrated Dataset (HID) will be used to compare models to observations. It will be created from the proper combination of a variety of surface, upper air and satellite observations. However, before the inclusion into HID, each dataset will need to be scrutinezed to make it suitable for model comparison. The scrutiny may include filtering, aggregation and possibly fusion operations.<br />
<br />
A good example is the 400-station AIRNOW network reporting hourly ozone and PM2.5 concentrations over the US. The AIRNOW network includes both urban sites that are strongly influenced by local sources. Urban stations need to be removed from HID dataset.<br />
<br />
Developing the specific criteria and procedures for the HTAP integrated dataset will require the attention of a HTAP subgroup.<br />
<br />
==Integrated Data Set==<br />
<br />
The providers of observational, emission, and modeling data are distributed. The individual data sets will be funeled into the integrated HTAP data set. This data set transfer will be accomplished using standard data transfer protocols. Transfering the data from the provider to the user will include two additional observations for each data set, quality assurance and semantic transformation.<br />
<br />
On the left the inputs from observations, emissions, and model data<br />
<br />
<br />
<br />
===Identify providers and datasets relevant to HTAP===<br />
===Integrate multi-sensory data into a coherent data systems===<br />
<br />
[[Image:HTAP_IntDataSys.png|600px|left]] In order to accomplish the objectives set for the HTAP TF it is evident that it will require the System of Systems approach as envisioned by GEO. Both the modeling of the chemical constituents as well as the Earth observations used to document the hemispheric transport are being performed by individual autonomous projects and programs (Systems). A key function of the Task Force is to assess the contributions of the individual systems and then integrate those into a coherent assessment. Much of this interim report is in fact such an assessment of the current knowledge based on previous or past models and observations.<br />
<br />
In the next, upcoming phase of HTAP is to actually integrate the existing modeling and observing systems such that their combination and fusion can yield a deeper and more robust understanding of HTAP. since this next phase will involve integration of multiple relevant data sets, comparison and evaluation of multiple models, as well as the reconciliation of the observations and models to create the new and deeper understanding . In order to execute this extensive integration, it will be necessary to conduct this integration using a generally accepted framework as well as accepted scientific methods. It is believed that the GEO architecture and the System of Systems is an attractive approach to HTAP integration.<br />
<br />
The HTAP program can also be considered an early demonstration of the GEO concepts through its end to end approach. Both the atmospheric modeling, as well as the observations currently exist through the operation of existing modeling and observation systems. The Task Force has agreed organizing and evaluating those models, assembling and integrating the observational data sets and then reconciling the models with the observations will be the focus of the Phase II effort. The Task Force will also prepare and deliver a report to LRTP to aid its deliberations and its decision making processes. This sequence of activities constitutes an end to end approach that turns observations and models into actionable knowledge for societal decision making. One could say that this is an octagonal approach to more deliberate step by step development of GEOSS where in Phase I interoperability, in Phase II ???<br />
<br />
Provider nodes <br />
* Federated Data System DataFed<br />
* NASA Data System Giovanni<br />
* Emission Data System NEISGEI<br />
* Juelich Data System<br />
* AeorCom<br />
* EMEP<br />
<br />
<br />
Data Networks. Connected hubs; Show Core HTAP network<br />
<br />
[http://datafedwiki.wustl.edu/index.php/2007-03-26:_Datasets_for_HTAP_through_DataFed See 20 selected datasets] in the federated data systems, DataFed<br />
<br />
===Perform QA at all steps===<br />
<br />
==Reconciliation and Integration of Observations, Emissions and Models==<br />
===Compare observations to models and emissions===<br />
===Characterize pollutant pattern and SSR=== <br />
===Develop real-time data assimilation===<br />
===Perform reanalysis with assimilated data=== <br />
<br />
==Organizational Aspects==<br />
===HTAP Relationship to GEO and GEOSS===<br />
There is an outstanding opportunity to develop a mutually beneficial and supportive relationship between the activities of the HTAP Task Force and that of GEO. The Group of Earth Observations with its broad range of national and organizational members has developed a general architectural framework for turning Earth observations into societal benefits. The three main components of this architecture are models, and observations, which then feed into decision support systems used in a variety of societal decision making. This general framework is well suited as an architectural guide to the HTAP program. However, this framework lacks specific guidelines and implementation details that are needed for practical Earth observing and decision support systems. <br />
<br />
[[Image:400px HTAP GEOSS Arch.png|400px|left]] The HTAP program provides an opportunity to extend the GEO framework with more detailed features that arise in specific HTAP application context. In case of HTAP, the major activities are shown in the architectural diagram of Figure ?? The HTAP modeling is conducted through global scale chemical transport models that simulate or forecast the atmospheric chemical composition as a consequence of natural and human induced emissions. The observations arise from satellite, ground-based and airborne observations of chemical constituents and their hemispheric transport. The decision support system consists of the scientists as members of the HTAP task force, and the task force co-chairs as the intermediaries between LRTP and the TF. (Terry, Andre this description of the HTAP DSS needs your help). <br />
<br />
Representing the Decision Support System (DSS) for HTAP is an important aspect because it can guide the design and and implementation of the information system for the support of the DSS processes. A higher resolution DSS design document is also beneficial as a communications medium for the interacting system of systems. A domain specific DSS may also allow the generalization and applicability to the design of DSS structures for similar application domains. <br />
<br />
The implementation of the GEO framework utilizes the concept of Global Observing System of Systems (GEOSS). Traditionally, Earth observations were performed by well defined systems such as specific satellites and monitoring networks which were designed and operated using systems engineering principles. However, GEO recognized that the understanding of Earth system requires the utilization and integration of the individual, autonomous systems. The key difference between the Systems and System of System approaches are highlighted in Table 1.<br />
<br />
===HTAP Relationship to IGACO=== <br />
[[Image:IGACO Framework.png|300px|left]] The HTAP program can also have a mutually beneficial interaction with the IGACO program. Integrated Global Atmospheric Chemistry Observations (IGACO) Theme is part of the Integrated Global Observing Strategy (IGOS). The IGACO program is formulated by an array of experienced atmospheric scientists. IGACO proposes a specific framework for the system components that processes the observational data and models. It also specifies the higher level information products that can be used for creating social benefit. <br />
<br />
In the IGACO framework, the data products from satellite, surface, and aircraft observations are collected for inclusion into an integrated data archive. A key component in the IGACO data flow is the mandatory quality assurance and QA QC that precedes the archiving. This is particularly important for HTAP where multi-sensory data from many different providers are to be combined into an integrated database. In the IGACO framework, a key output from the data system is a high integrated spatial-temporal data set which can be used in a multitude of applications that produce socal benefit. The goal of creating such an integrated data set is shared by the IGACO and the HTAP programs (?? Len, this section could benefit from your input)<br />
<br />
===HTAP Relationship with Other Programs===<br />
---<br />
---<br />
<br />
<br />
<br />
==Links==<br />
[[IGACO Framework]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=HTAP_Report,_Sub-Chap._6_-_Data/Info_System&diff=7945HTAP Report, Sub-Chap. 6 - Data/Info System2007-04-12T04:58:42Z<p>128.252.167.79: /* Introduction */</p>
<hr />
<div><noinclude>[[Integrated_Global_Dataset| < Back to Integrated Global Dataset on ESIP wiki]] <br></noinclude> <br />
<br />
This wiki page is inteded to be the collaborative workspace for Task Force members interested in the HTAP Information System development, implementation and use. The contents can be freely edited by anyone who is logged in. Comments and feedback can also be sent by email to rhusar@me.wustl.edu. The content of this page is is on the '''''Information System''''' that supposts Chapter 6. The subsections are action-oriented. <br />
<br />
From [http://www.htap.org/activities/2007_interim_report/TF%20HTAP%202007%20Interim%20Report%20Prospectus%20Oct%202006.pdf Oct 2006 Prospectus]: This chapter (Chapter 6: '''Integration of Observations, Emissions and Modeling'''.) will discuss the need to integrate information from observations, models, and emissions inventories to better understand intercontinental transport. A draft [[HTAP Report Chap. 6 - Jan 07 Outline]] is also on this wiki.<br />
<br />
==Introduction==<br />
The goal of the HTAP data system is to aid the assembly of an appropriate set of observations and models, and to facilitate the integration and fusion of observations with models. <br />
<br />
Recent developments in air quality monitoring, modeling and information technologies offer outstanding<br />
opportunities to fulfill the information needs for the HTAP integration effort. The data from surface-based air pollution monitoring networks now routinely provide spatio-temporal and chemical patterns of ozone and PM. Satellite<br />
sensors with global coverage and kilometer-scale spatial resolution now provide real-time snapshots which depict the pattern of industrial haze, smoke and dust in stunning detail. The ‘terabytes’ of data from these surface and remote sensors can now be stored, processed and delivered in near-real time. The instantaneous ‘horizontal’ diffusion of information via the Internet now permits, in principle, the delivery of the right information to the right people at the right place and time. Standardized computer-computer communication languages and Service-Oriented Architectures (SOA) now facilitate the flexible access, quality assurance and processing of raw data into high-grade ‘actionable’ knowledge suitable for HTAP policy decissions. Last but not least, the World Wide Web has opened the way to generous sharing of data and tools leading to collaborative analysis and virtual workgroups. Nevertheless, air quality data analyses and data model integration face significant hurdles. The “data deluge” problem is especially acute for satellite observations and model outputs. <br />
<br />
==== Data Selection Criteria====<br />
Monitoring data for atmospheric constituents are now available from a variety of sources, not all of which are suitable for the integrated HTAP data set. Given the limited scope and resources of HTAP, it will be necessary to select a subset of the available observational data that would be most appropriate for the preparation of the 2009 assessment. A set of criteria for the data selection is given below.<br />
* The suitability criteria may include the measured parameters, their spatial extent and coverage density, as well as the time range and sampling frequency with major emphasis on data quality (defined as???).<br />
* The compilation of the global integrated data sets should initially focus on PM and ozone, including the gaseous precursors.<br />
* Subsequent data collaction should also include observations on atmospheric mercury and persistent organic pollutants (POPS). <br />
* In order to cover hemispheric transport of air pollutants the data set system should accept and utilize data from outside the geographical scope of EMEP.<br />
* The data gathering should begin the compilation using the of existing databases in Europe (RETRO, TRADEOFF, QUANTIFY, CRATE, AEROCOM) and virtual federated systems in the US (DataFed, Giovanni and others). <br />
* TF data integration should also contribute to and benefit from other ongoing data integration efforts to integrate the global data resources, most notably, with ACCENT project in Europe, and similar efforts in the US.<br />
* Special emphasis should be placed on the collection of suitable vertical profiles from aircraft measurements as well as from surface and space-borne lidars.<br />
<br />
The evaluation of suitabile observational data sets for model validation and fusion will require close interaction between the modeling and observational communities. An wiki workspace for open collaborative evaluation of different datasets is desirebale.<br />
<br />
====HTAP Data System Architecture====<br />
The proposed HTAP data system will integrate the emerging observational and modeling resources by not a centrally planned and powerful, dynamic technologies and through a collaborative federation philosophy. The key roles of the federated data system are to (1) facilitate accsess to the distributed data; (2) ensure the seamless flow of data through interoperable interfaces; (3) provide a set of basic tools for data exploration and processing. <br />
<br />
The Service Oriented Architecture (SOA) allows connecting the web service components (e.g. services for data access, transformation, fusion, rendering, etc.) using work flow software.<br />
<br />
===Homogenize the data into an interoperable dataset===<br />
The primary goal of the data system is to allow the integration of observational emission and model data. The model evaluation requires that both the observations and if possible the emissions are fixed. For this reason it is desirable to prepare an integrated observational database to which the various model implementations can be compared to. On one hand, the integrated data set should be as inclusive as possible. On the other hand, such a goal needs to be tempered by many limitations that preclude a broad, inclusive data integration<br />
<br />
The proposed HTAP data system will be a hybrid combination of both distributed and fixed components as illustrated schematically in Figure ??. Both the data providers as well as the HTAP analysts-users will be distributed. However, they will be connected through an integrated HTAP database which should be a stable, virtually fixed database. The section below describes the main component of this distributed data system. <br />
<br />
The HTAP integrated data system has to perform two major functions. The primary function is to facilitate the creation of an integrated data set suitable for model evaluation and pollutant characterization. The multiple steps that are required for this functionality are shown on the left. The sequence of operations can be viewed as a value chain that transform the raw input data into a highly organized integrated data set. Since these value adding steps have to be performed for each candidate data set the operations can be performed in sequence following the rules of data flow programming. For true quality assurance and for data homogenization the data flow channels for individual data sets need to be evaluated in the context of other data. <br />
<br />
The operations that prepare the integrated data set can be viewed as services that operate on the data stream using the principles of modern Service Oriented Architecure (SOA). Each of the services with its clearly defined functionality and firmly specified service interface. In principle, the standards-based interface allows a connection of service chains using formal work flow software. The main benefit of service oriented architecture is that allows the building of agile application programs that can respond to changes in the data input conditions as well as the output requirements. The service oriented software architecture is an implementation of the System of Systems approach. Furthermore, since each of the services can be operated by autonomous providers and the "system" that implements the service is behind the service interfece. In other words, following the SOA approach, not only the data providers are distributed but also the processing services as well. This flexible approach to distributed computing allows the distribution of labor (chain of services) in many different configurations. For instance, wrapping the existing data with standards based interfaces for data access can be performed for an entire cluster of data. This is the approach taken in the federated data system DataFed. Given a standard interface to a variety of data sets, the Quality Assurance service can be performed by another srvice provider that is seamlesly connected to the data access mediator. Similarly, the service that prepares a data set for data integration can be provided by another service in the data flow network. The flexibility offered through the chaining and orchestration of distributed, loosely coupled web services is the architectural support for the building of agile data systems.<br />
<br />
===Interoperability Standards and Data Wrapping===<br />
Interoperable data access will be accomplished through the use of a suite of international standards. The naming of individual chemical parameters will follow the CF convention used by the Climate and Forecast (CF) communities. The existing names for atmospheric chemicals in the CF convention were inadequate to accomodate all the parameters used in the HTAP modeling. in order to remedy this shortcomming the list of standard names was extended by the HTAP community under leadership of C. Textor. She also became a memeber of the CF convention board that is the custodian of the standard names. The standard names for HTAP models were developed using a collaborative wiki workspace. <br />
<br />
The data transfer from providers to users also benefit from standard data formats. For modeling data the use of netCDF-CF as a standard format is recomended. The use of a standard physical data format and the CF naming conventions allows, in principle, the seamless connection between data provider and consumer services. It should be noted, however, that at this time the CF naming convention has only been developed for the model parameters and not for the various observational parameters. Also, the nedCDF CF is primarily used as a model data exchange format, while the transfer of surface monitoring data and satellite data is less standardized.<br />
<br />
The third aspect of data interoperability is a standard query language through which user services request specific data from the provider services. It is proposed that for the HTAP data information system adapts the Web Coverage Service (WCS) as the standard data query language. The WCS data access protocol is defined by the international Open Geospatial Consortium (OGC), which is also the key organization responsible for interoperability standards in GEOSS. Since the WCS protocol was originally developed for the GIS community, it was necessary to adapt it to the needs of "fluid Earth sciences". Members of the HTAP group have been actively participating in the development and testing of the WCS interoperability standards. <br />
<br />
Standards-based data access can be accomplished by ‘wrapping’ the heterogeneous data into standardized web services, callable through well-defined Internet protocols. The result of this ‘wrapping’ process is an array of homogeneous, virtual datasets that can be accessed through a standard querry language and the returned data are packaged in standard format, directly usable by the consuming services.<br />
<br />
===Quality Assurance===<br />
This is just a place holder. Here we need to explain the quality assurance steps that are needed to prepare the HTAP Integrated Dataset.<br />
<br />
===Homogenization and Integration===<br />
The HTAP Integrated Dataset (HID) will be used to compare models to observations. It will be created from the proper combination of a variety of surface, upper air and satellite observations. However, before the inclusion into HID, each dataset will need to be scrutinezed to make it suitable for model comparison. The scrutiny may include filtering, aggregation and possibly fusion operations.<br />
<br />
A good example is the 400-station AIRNOW network reporting hourly ozone and PM2.5 concentrations over the US. The AIRNOW network includes both urban sites that are strongly influenced by local sources. Urban stations need to be removed from HID dataset.<br />
<br />
Developing the specific criteria and procedures for the HTAP integrated dataset will require the attention of a HTAP subgroup.<br />
<br />
==Integrated Data Set==<br />
<br />
The providers of observational, emission, and modeling data are distributed. The individual data sets will be funeled into the integrated HTAP data set. This data set transfer will be accomplished using standard data transfer protocols. Transfering the data from the provider to the user will include two additional observations for each data set, quality assurance and semantic transformation.<br />
<br />
On the left the inputs from observations, emissions, and model data<br />
<br />
<br />
<br />
===Identify providers and datasets relevant to HTAP===<br />
===Integrate multi-sensory data into a coherent data systems===<br />
<br />
[[Image:HTAP_IntDataSys.png|600px|left]] In order to accomplish the objectives set for the HTAP TF it is evident that it will require the System of Systems approach as envisioned by GEO. Both the modeling of the chemical constituents as well as the Earth observations used to document the hemispheric transport are being performed by individual autonomous projects and programs (Systems). A key function of the Task Force is to assess the contributions of the individual systems and then integrate those into a coherent assessment. Much of this interim report is in fact such an assessment of the current knowledge based on previous or past models and observations.<br />
<br />
In the next, upcoming phase of HTAP is to actually integrate the existing modeling and observing systems such that their combination and fusion can yield a deeper and more robust understanding of HTAP. since this next phase will involve integration of multiple relevant data sets, comparison and evaluation of multiple models, as well as the reconciliation of the observations and models to create the new and deeper understanding . In order to execute this extensive integration, it will be necessary to conduct this integration using a generally accepted framework as well as accepted scientific methods. It is believed that the GEO architecture and the System of Systems is an attractive approach to HTAP integration.<br />
<br />
The HTAP program can also be considered an early demonstration of the GEO concepts through its end to end approach. Both the atmospheric modeling, as well as the observations currently exist through the operation of existing modeling and observation systems. The Task Force has agreed organizing and evaluating those models, assembling and integrating the observational data sets and then reconciling the models with the observations will be the focus of the Phase II effort. The Task Force will also prepare and deliver a report to LRTP to aid its deliberations and its decision making processes. This sequence of activities constitutes an end to end approach that turns observations and models into actionable knowledge for societal decision making. One could say that this is an octagonal approach to more deliberate step by step development of GEOSS where in Phase I interoperability, in Phase II ???<br />
<br />
Provider nodes <br />
* Federated Data System DataFed<br />
* NASA Data System Giovanni<br />
* Emission Data System NEISGEI<br />
* Juelich Data System<br />
* AeorCom<br />
* EMEP<br />
<br />
<br />
Data Networks. Connected hubs; Show Core HTAP network<br />
<br />
[http://datafedwiki.wustl.edu/index.php/2007-03-26:_Datasets_for_HTAP_through_DataFed See 20 selected datasets] in the federated data systems, DataFed<br />
<br />
===Perform QA at all steps===<br />
<br />
==Reconciliation and Integration of Observations, Emissions and Models==<br />
===Compare observations to models and emissions===<br />
===Characterize pollutant pattern and SSR=== <br />
===Develop real-time data assimilation===<br />
===Perform reanalysis with assimilated data=== <br />
<br />
==Organizational Aspects==<br />
===HTAP Relationship to GEO and GEOSS===<br />
There is an outstanding opportunity to develop a mutually beneficial and supportive relationship between the activities of the HTAP Task Force and that of GEO. The Group of Earth Observations with its broad range of national and organizational members has developed a general architectural framework for turning Earth observations into societal benefits. The three main components of this architecture are models, and observations, which then feed into decision support systems used in a variety of societal decision making. This general framework is well suited as an architectural guide to the HTAP program. However, this framework lacks specific guidelines and implementation details that are needed for practical Earth observing and decision support systems. <br />
<br />
[[Image:400px HTAP GEOSS Arch.png|400px|left]] The HTAP program provides an opportunity to extend the GEO framework with more detailed features that arise in specific HTAP application context. In case of HTAP, the major activities are shown in the architectural diagram of Figure ?? The HTAP modeling is conducted through global scale chemical transport models that simulate or forecast the atmospheric chemical composition as a consequence of natural and human induced emissions. The observations arise from satellite, ground-based and airborne observations of chemical constituents and their hemispheric transport. The decision support system consists of the scientists as members of the HTAP task force, and the task force co-chairs as the intermediaries between LRTP and the TF. (Terry, Andre this description of the HTAP DSS needs your help). <br />
<br />
Representing the Decision Support System (DSS) for HTAP is an important aspect because it can guide the design and and implementation of the information system for the support of the DSS processes. A higher resolution DSS design document is also beneficial as a communications medium for the interacting system of systems. A domain specific DSS may also allow the generalization and applicability to the design of DSS structures for similar application domains. <br />
<br />
The implementation of the GEO framework utilizes the concept of Global Observing System of Systems (GEOSS). Traditionally, Earth observations were performed by well defined systems such as specific satellites and monitoring networks which were designed and operated using systems engineering principles. However, GEO recognized that the understanding of Earth system requires the utilization and integration of the individual, autonomous systems. The key difference between the Systems and System of System approaches are highlighted in Table 1.<br />
<br />
===HTAP Relationship to IGACO=== <br />
[[Image:IGACO Framework.png|300px|left]] The HTAP program can also have a mutually beneficial interaction with the IGACO program. Integrated Global Atmospheric Chemistry Observations (IGACO) Theme is part of the Integrated Global Observing Strategy (IGOS). The IGACO program is formulated by an array of experienced atmospheric scientists. IGACO proposes a specific framework for the system components that processes the observational data and models. It also specifies the higher level information products that can be used for creating social benefit. <br />
<br />
In the IGACO framework, the data products from satellite, surface, and aircraft observations are collected for inclusion into an integrated data archive. A key component in the IGACO data flow is the mandatory quality assurance and QA QC that precedes the archiving. This is particularly important for HTAP where multi-sensory data from many different providers are to be combined into an integrated database. In the IGACO framework, a key output from the data system is a high integrated spatial-temporal data set which can be used in a multitude of applications that produce socal benefit. The goal of creating such an integrated data set is shared by the IGACO and the HTAP programs (?? Len, this section could benefit from your input)<br />
<br />
===HTAP Relationship with Other Programs===<br />
---<br />
---<br />
<br />
<br />
<br />
==Links==<br />
[[IGACO Framework]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=HTAP_Report,_Sub-Chap._6_-_Data/Info_System&diff=7944HTAP Report, Sub-Chap. 6 - Data/Info System2007-04-12T04:57:26Z<p>128.252.167.79: /* Homogenize the data into an interoperable dataset */</p>
<hr />
<div><noinclude>[[Integrated_Global_Dataset| < Back to Integrated Global Dataset on ESIP wiki]] <br></noinclude> <br />
<br />
This wiki page is inteded to be the collaborative workspace for Task Force members interested in the HTAP Information System development, implementation and use. The contents can be freely edited by anyone who is logged in. Comments and feedback can also be sent by email to rhusar@me.wustl.edu. The content of this page is is on the '''''Information System''''' that supposts Chapter 6. The subsections are action-oriented. <br />
<br />
From [http://www.htap.org/activities/2007_interim_report/TF%20HTAP%202007%20Interim%20Report%20Prospectus%20Oct%202006.pdf Oct 2006 Prospectus]: This chapter (Chapter 6: '''Integration of Observations, Emissions and Modeling'''.) will discuss the need to integrate information from observations, models, and emissions inventories to better understand intercontinental transport. A draft [[HTAP Report Chap. 6 - Jan 07 Outline]] is also on this wiki.<br />
<br />
==Introduction==<br />
The goal of the HTAP data system is to aid the assembly of an appropriate set of observations and models, and to facilitate the integration and fusion of observations with models. <br />
<br />
Recent developments in air quality monitoring, modeling and information technologies offer outstanding<br />
opportunities to fulfill the information needs for the HTAP integration effort. The data from surface-based air pollution monitoring networks now routinely provide spatio-temporal and chemical patterns of ozone and PM. Satellite<br />
sensors with global coverage and kilometer-scale spatial resolution now provide real-time snapshots which depict the pattern of industrial haze, smoke and dust in stunning detail. The ‘terabytes’ of data from these surface and remote sensors can now be stored, processed and delivered in near-real time. The instantaneous ‘horizontal’ diffusion of information via the Internet now permits, in principle, the delivery of the right information to the right people at the right place and time. Standardized computer-computer communication languages and Service-Oriented Architectures (SOA) now facilitate the flexible access, quality assurance and processing of raw data into high-grade ‘actionable’ knowledge suitable for HTAP policy decissions. Last but not least, the World Wide Web has opened the way to generous sharing of data and tools leading to collaborative analysis and virtual workgroups. Nevertheless, air quality data analyses and data model integration face significant hurdles. The “data deluge” problem is especially acute for satellite observations and model outputs. <br />
<br />
<br />
<br />
==== Data Selection Criteria====<br />
Monitoring data for atmospheric constituents are now available from a variety of sources, not all of which are suitable for the integrated HTAP data set. Given the limited scope and resources of HTAP, it will be necessary to select a subset of the available observational data that would be most appropriate for the preparation of the 2009 assessment. A set of criteria for the data selection is given below.<br />
* The suitability criteria may include the measured parameters, their spatial extent and coverage density, as well as the time range and sampling frequency with major emphasis on data quality (defined as???).<br />
* The compilation of the global integrated data sets should initially focus on PM and ozone, including the gaseous precursors.<br />
* Subsequent data collaction should also include observations on atmospheric mercury and persistent organic pollutants (POPS). <br />
* In order to cover hemispheric transport of air pollutants the data set system should accept and utilize data from outside the geographical scope of EMEP.<br />
* The data gathering should begin the compilation using the of existing databases in Europe (RETRO, TRADEOFF, QUANTIFY, CRATE, AEROCOM) and virtual federated systems in the US (DataFed, Giovanni and others). <br />
* TF data integration should also contribute to and benefit from other ongoing data integration efforts to integrate the global data resources, most notably, with ACCENT project in Europe, and similar efforts in the US.<br />
* Special emphasis should be placed on the collection of suitable vertical profiles from aircraft measurements as well as from surface and space-borne lidars.<br />
<br />
The evaluation of suitabile observational data sets for model validation and fusion will require close interaction between the modeling and observational communities. An wiki workspace for open collaborative evaluation of different datasets is desirebale.<br />
<br />
====HTAP Data System Architecture====<br />
The proposed HTAP data system will integrate the emerging observational and modeling resources by not a centrally planned and powerful, dynamic technologies and through a collaborative federation philosophy. The key roles of the federated data system are to (1) facilitate accsess to the distributed data; (2) ensure the seamless flow of data through interoperable interfaces; (3) provide a set of basic tools for data exploration and processing. <br />
<br />
The Service Oriented Architecture (SOA) allows connecting the web service components (e.g. services for data access, transformation, fusion, rendering, etc.) using work flow software.<br />
<br />
===Homogenize the data into an interoperable dataset===<br />
The primary goal of the data system is to allow the integration of observational emission and model data. The model evaluation requires that both the observations and if possible the emissions are fixed. For this reason it is desirable to prepare an integrated observational database to which the various model implementations can be compared to. On one hand, the integrated data set should be as inclusive as possible. On the other hand, such a goal needs to be tempered by many limitations that preclude a broad, inclusive data integration<br />
<br />
The proposed HTAP data system will be a hybrid combination of both distributed and fixed components as illustrated schematically in Figure ??. Both the data providers as well as the HTAP analysts-users will be distributed. However, they will be connected through an integrated HTAP database which should be a stable, virtually fixed database. The section below describes the main component of this distributed data system. <br />
<br />
The HTAP integrated data system has to perform two major functions. The primary function is to facilitate the creation of an integrated data set suitable for model evaluation and pollutant characterization. The multiple steps that are required for this functionality are shown on the left. The sequence of operations can be viewed as a value chain that transform the raw input data into a highly organized integrated data set. Since these value adding steps have to be performed for each candidate data set the operations can be performed in sequence following the rules of data flow programming. For true quality assurance and for data homogenization the data flow channels for individual data sets need to be evaluated in the context of other data. <br />
<br />
The operations that prepare the integrated data set can be viewed as services that operate on the data stream using the principles of modern Service Oriented Architecure (SOA). Each of the services with its clearly defined functionality and firmly specified service interface. In principle, the standards-based interface allows a connection of service chains using formal work flow software. The main benefit of service oriented architecture is that allows the building of agile application programs that can respond to changes in the data input conditions as well as the output requirements. The service oriented software architecture is an implementation of the System of Systems approach. Furthermore, since each of the services can be operated by autonomous providers and the "system" that implements the service is behind the service interfece. In other words, following the SOA approach, not only the data providers are distributed but also the processing services as well. This flexible approach to distributed computing allows the distribution of labor (chain of services) in many different configurations. For instance, wrapping the existing data with standards based interfaces for data access can be performed for an entire cluster of data. This is the approach taken in the federated data system DataFed. Given a standard interface to a variety of data sets, the Quality Assurance service can be performed by another srvice provider that is seamlesly connected to the data access mediator. Similarly, the service that prepares a data set for data integration can be provided by another service in the data flow network. The flexibility offered through the chaining and orchestration of distributed, loosely coupled web services is the architectural support for the building of agile data systems.<br />
<br />
===Interoperability Standards and Data Wrapping===<br />
Interoperable data access will be accomplished through the use of a suite of international standards. The naming of individual chemical parameters will follow the CF convention used by the Climate and Forecast (CF) communities. The existing names for atmospheric chemicals in the CF convention were inadequate to accomodate all the parameters used in the HTAP modeling. in order to remedy this shortcomming the list of standard names was extended by the HTAP community under leadership of C. Textor. She also became a memeber of the CF convention board that is the custodian of the standard names. The standard names for HTAP models were developed using a collaborative wiki workspace. <br />
<br />
The data transfer from providers to users also benefit from standard data formats. For modeling data the use of netCDF-CF as a standard format is recomended. The use of a standard physical data format and the CF naming conventions allows, in principle, the seamless connection between data provider and consumer services. It should be noted, however, that at this time the CF naming convention has only been developed for the model parameters and not for the various observational parameters. Also, the nedCDF CF is primarily used as a model data exchange format, while the transfer of surface monitoring data and satellite data is less standardized.<br />
<br />
The third aspect of data interoperability is a standard query language through which user services request specific data from the provider services. It is proposed that for the HTAP data information system adapts the Web Coverage Service (WCS) as the standard data query language. The WCS data access protocol is defined by the international Open Geospatial Consortium (OGC), which is also the key organization responsible for interoperability standards in GEOSS. Since the WCS protocol was originally developed for the GIS community, it was necessary to adapt it to the needs of "fluid Earth sciences". Members of the HTAP group have been actively participating in the development and testing of the WCS interoperability standards. <br />
<br />
Standards-based data access can be accomplished by ‘wrapping’ the heterogeneous data into standardized web services, callable through well-defined Internet protocols. The result of this ‘wrapping’ process is an array of homogeneous, virtual datasets that can be accessed through a standard querry language and the returned data are packaged in standard format, directly usable by the consuming services.<br />
<br />
===Quality Assurance===<br />
This is just a place holder. Here we need to explain the quality assurance steps that are needed to prepare the HTAP Integrated Dataset.<br />
<br />
===Homogenization and Integration===<br />
The HTAP Integrated Dataset (HID) will be used to compare models to observations. It will be created from the proper combination of a variety of surface, upper air and satellite observations. However, before the inclusion into HID, each dataset will need to be scrutinezed to make it suitable for model comparison. The scrutiny may include filtering, aggregation and possibly fusion operations.<br />
<br />
A good example is the 400-station AIRNOW network reporting hourly ozone and PM2.5 concentrations over the US. The AIRNOW network includes both urban sites that are strongly influenced by local sources. Urban stations need to be removed from HID dataset.<br />
<br />
Developing the specific criteria and procedures for the HTAP integrated dataset will require the attention of a HTAP subgroup.<br />
<br />
==Integrated Data Set==<br />
<br />
The providers of observational, emission, and modeling data are distributed. The individual data sets will be funeled into the integrated HTAP data set. This data set transfer will be accomplished using standard data transfer protocols. Transfering the data from the provider to the user will include two additional observations for each data set, quality assurance and semantic transformation.<br />
<br />
On the left the inputs from observations, emissions, and model data<br />
<br />
<br />
<br />
===Identify providers and datasets relevant to HTAP===<br />
===Integrate multi-sensory data into a coherent data systems===<br />
<br />
[[Image:HTAP_IntDataSys.png|600px|left]] In order to accomplish the objectives set for the HTAP TF it is evident that it will require the System of Systems approach as envisioned by GEO. Both the modeling of the chemical constituents as well as the Earth observations used to document the hemispheric transport are being performed by individual autonomous projects and programs (Systems). A key function of the Task Force is to assess the contributions of the individual systems and then integrate those into a coherent assessment. Much of this interim report is in fact such an assessment of the current knowledge based on previous or past models and observations.<br />
<br />
In the next, upcoming phase of HTAP is to actually integrate the existing modeling and observing systems such that their combination and fusion can yield a deeper and more robust understanding of HTAP. since this next phase will involve integration of multiple relevant data sets, comparison and evaluation of multiple models, as well as the reconciliation of the observations and models to create the new and deeper understanding . In order to execute this extensive integration, it will be necessary to conduct this integration using a generally accepted framework as well as accepted scientific methods. It is believed that the GEO architecture and the System of Systems is an attractive approach to HTAP integration.<br />
<br />
The HTAP program can also be considered an early demonstration of the GEO concepts through its end to end approach. Both the atmospheric modeling, as well as the observations currently exist through the operation of existing modeling and observation systems. The Task Force has agreed organizing and evaluating those models, assembling and integrating the observational data sets and then reconciling the models with the observations will be the focus of the Phase II effort. The Task Force will also prepare and deliver a report to LRTP to aid its deliberations and its decision making processes. This sequence of activities constitutes an end to end approach that turns observations and models into actionable knowledge for societal decision making. One could say that this is an octagonal approach to more deliberate step by step development of GEOSS where in Phase I interoperability, in Phase II ???<br />
<br />
Provider nodes <br />
* Federated Data System DataFed<br />
* NASA Data System Giovanni<br />
* Emission Data System NEISGEI<br />
* Juelich Data System<br />
* AeorCom<br />
* EMEP<br />
<br />
<br />
Data Networks. Connected hubs; Show Core HTAP network<br />
<br />
[http://datafedwiki.wustl.edu/index.php/2007-03-26:_Datasets_for_HTAP_through_DataFed See 20 selected datasets] in the federated data systems, DataFed<br />
<br />
===Perform QA at all steps===<br />
<br />
==Reconciliation and Integration of Observations, Emissions and Models==<br />
===Compare observations to models and emissions===<br />
===Characterize pollutant pattern and SSR=== <br />
===Develop real-time data assimilation===<br />
===Perform reanalysis with assimilated data=== <br />
<br />
==Organizational Aspects==<br />
===HTAP Relationship to GEO and GEOSS===<br />
There is an outstanding opportunity to develop a mutually beneficial and supportive relationship between the activities of the HTAP Task Force and that of GEO. The Group of Earth Observations with its broad range of national and organizational members has developed a general architectural framework for turning Earth observations into societal benefits. The three main components of this architecture are models, and observations, which then feed into decision support systems used in a variety of societal decision making. This general framework is well suited as an architectural guide to the HTAP program. However, this framework lacks specific guidelines and implementation details that are needed for practical Earth observing and decision support systems. <br />
<br />
[[Image:400px HTAP GEOSS Arch.png|400px|left]] The HTAP program provides an opportunity to extend the GEO framework with more detailed features that arise in specific HTAP application context. In case of HTAP, the major activities are shown in the architectural diagram of Figure ?? The HTAP modeling is conducted through global scale chemical transport models that simulate or forecast the atmospheric chemical composition as a consequence of natural and human induced emissions. The observations arise from satellite, ground-based and airborne observations of chemical constituents and their hemispheric transport. The decision support system consists of the scientists as members of the HTAP task force, and the task force co-chairs as the intermediaries between LRTP and the TF. (Terry, Andre this description of the HTAP DSS needs your help). <br />
<br />
Representing the Decision Support System (DSS) for HTAP is an important aspect because it can guide the design and and implementation of the information system for the support of the DSS processes. A higher resolution DSS design document is also beneficial as a communications medium for the interacting system of systems. A domain specific DSS may also allow the generalization and applicability to the design of DSS structures for similar application domains. <br />
<br />
The implementation of the GEO framework utilizes the concept of Global Observing System of Systems (GEOSS). Traditionally, Earth observations were performed by well defined systems such as specific satellites and monitoring networks which were designed and operated using systems engineering principles. However, GEO recognized that the understanding of Earth system requires the utilization and integration of the individual, autonomous systems. The key difference between the Systems and System of System approaches are highlighted in Table 1.<br />
<br />
===HTAP Relationship to IGACO=== <br />
[[Image:IGACO Framework.png|300px|left]] The HTAP program can also have a mutually beneficial interaction with the IGACO program. Integrated Global Atmospheric Chemistry Observations (IGACO) Theme is part of the Integrated Global Observing Strategy (IGOS). The IGACO program is formulated by an array of experienced atmospheric scientists. IGACO proposes a specific framework for the system components that processes the observational data and models. It also specifies the higher level information products that can be used for creating social benefit. <br />
<br />
In the IGACO framework, the data products from satellite, surface, and aircraft observations are collected for inclusion into an integrated data archive. A key component in the IGACO data flow is the mandatory quality assurance and QA QC that precedes the archiving. This is particularly important for HTAP where multi-sensory data from many different providers are to be combined into an integrated database. In the IGACO framework, a key output from the data system is a high integrated spatial-temporal data set which can be used in a multitude of applications that produce socal benefit. The goal of creating such an integrated data set is shared by the IGACO and the HTAP programs (?? Len, this section could benefit from your input)<br />
<br />
===HTAP Relationship with Other Programs===<br />
---<br />
---<br />
<br />
<br />
<br />
==Links==<br />
[[IGACO Framework]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=HTAP_Report,_Sub-Chap._6_-_Data/Info_System&diff=7943HTAP Report, Sub-Chap. 6 - Data/Info System2007-04-12T04:56:58Z<p>128.252.167.79: /* HTAP Data System Architecture */</p>
<hr />
<div><noinclude>[[Integrated_Global_Dataset| < Back to Integrated Global Dataset on ESIP wiki]] <br></noinclude> <br />
<br />
This wiki page is inteded to be the collaborative workspace for Task Force members interested in the HTAP Information System development, implementation and use. The contents can be freely edited by anyone who is logged in. Comments and feedback can also be sent by email to rhusar@me.wustl.edu. The content of this page is is on the '''''Information System''''' that supposts Chapter 6. The subsections are action-oriented. <br />
<br />
From [http://www.htap.org/activities/2007_interim_report/TF%20HTAP%202007%20Interim%20Report%20Prospectus%20Oct%202006.pdf Oct 2006 Prospectus]: This chapter (Chapter 6: '''Integration of Observations, Emissions and Modeling'''.) will discuss the need to integrate information from observations, models, and emissions inventories to better understand intercontinental transport. A draft [[HTAP Report Chap. 6 - Jan 07 Outline]] is also on this wiki.<br />
<br />
==Introduction==<br />
The goal of the HTAP data system is to aid the assembly of an appropriate set of observations and models, and to facilitate the integration and fusion of observations with models. <br />
<br />
Recent developments in air quality monitoring, modeling and information technologies offer outstanding<br />
opportunities to fulfill the information needs for the HTAP integration effort. The data from surface-based air pollution monitoring networks now routinely provide spatio-temporal and chemical patterns of ozone and PM. Satellite<br />
sensors with global coverage and kilometer-scale spatial resolution now provide real-time snapshots which depict the pattern of industrial haze, smoke and dust in stunning detail. The ‘terabytes’ of data from these surface and remote sensors can now be stored, processed and delivered in near-real time. The instantaneous ‘horizontal’ diffusion of information via the Internet now permits, in principle, the delivery of the right information to the right people at the right place and time. Standardized computer-computer communication languages and Service-Oriented Architectures (SOA) now facilitate the flexible access, quality assurance and processing of raw data into high-grade ‘actionable’ knowledge suitable for HTAP policy decissions. Last but not least, the World Wide Web has opened the way to generous sharing of data and tools leading to collaborative analysis and virtual workgroups. Nevertheless, air quality data analyses and data model integration face significant hurdles. The “data deluge” problem is especially acute for satellite observations and model outputs. <br />
<br />
<br />
<br />
==== Data Selection Criteria====<br />
Monitoring data for atmospheric constituents are now available from a variety of sources, not all of which are suitable for the integrated HTAP data set. Given the limited scope and resources of HTAP, it will be necessary to select a subset of the available observational data that would be most appropriate for the preparation of the 2009 assessment. A set of criteria for the data selection is given below.<br />
* The suitability criteria may include the measured parameters, their spatial extent and coverage density, as well as the time range and sampling frequency with major emphasis on data quality (defined as???).<br />
* The compilation of the global integrated data sets should initially focus on PM and ozone, including the gaseous precursors.<br />
* Subsequent data collaction should also include observations on atmospheric mercury and persistent organic pollutants (POPS). <br />
* In order to cover hemispheric transport of air pollutants the data set system should accept and utilize data from outside the geographical scope of EMEP.<br />
* The data gathering should begin the compilation using the of existing databases in Europe (RETRO, TRADEOFF, QUANTIFY, CRATE, AEROCOM) and virtual federated systems in the US (DataFed, Giovanni and others). <br />
* TF data integration should also contribute to and benefit from other ongoing data integration efforts to integrate the global data resources, most notably, with ACCENT project in Europe, and similar efforts in the US.<br />
* Special emphasis should be placed on the collection of suitable vertical profiles from aircraft measurements as well as from surface and space-borne lidars.<br />
<br />
The evaluation of suitabile observational data sets for model validation and fusion will require close interaction between the modeling and observational communities. An wiki workspace for open collaborative evaluation of different datasets is desirebale.<br />
<br />
===Homogenize the data into an interoperable dataset===<br />
The primary goal of the data system is to allow the integration of observational emission and model data. The model evaluation requires that both the observations and if possible the emissions are fixed. For this reason it is desirable to prepare an integrated observational database to which the various model implementations can be compared to. On one hand, the integrated data set should be as inclusive as possible. On the other hand, such a goal needs to be tempered by many limitations that preclude a broad, inclusive data integration<br />
<br />
The proposed HTAP data system will be a hybrid combination of both distributed and fixed components as illustrated schematically in Figure ??. Both the data providers as well as the HTAP analysts-users will be distributed. However, they will be connected through an integrated HTAP database which should be a stable, virtually fixed database. The section below describes the main component of this distributed data system. <br />
<br />
The HTAP integrated data system has to perform two major functions. The primary function is to facilitate the creation of an integrated data set suitable for model evaluation and pollutant characterization. The multiple steps that are required for this functionality are shown on the left. The sequence of operations can be viewed as a value chain that transform the raw input data into a highly organized integrated data set. Since these value adding steps have to be performed for each candidate data set the operations can be performed in sequence following the rules of data flow programming. For true quality assurance and for data homogenization the data flow channels for individual data sets need to be evaluated in the context of other data. <br />
<br />
The operations that prepare the integrated data set can be viewed as services that operate on the data stream using the principles of modern Service Oriented Architecure (SOA). Each of the services with its clearly defined functionality and firmly specified service interface. In principle, the standards-based interface allows a connection of service chains using formal work flow software. The main benefit of service oriented architecture is that allows the building of agile application programs that can respond to changes in the data input conditions as well as the output requirements. The service oriented software architecture is an implementation of the System of Systems approach. Furthermore, since each of the services can be operated by autonomous providers and the "system" that implements the service is behind the service interfece. In other words, following the SOA approach, not only the data providers are distributed but also the processing services as well. This flexible approach to distributed computing allows the distribution of labor (chain of services) in many different configurations. For instance, wrapping the existing data with standards based interfaces for data access can be performed for an entire cluster of data. This is the approach taken in the federated data system DataFed. Given a standard interface to a variety of data sets, the Quality Assurance service can be performed by another srvice provider that is seamlesly connected to the data access mediator. Similarly, the service that prepares a data set for data integration can be provided by another service in the data flow network. The flexibility offered through the chaining and orchestration of distributed, loosely coupled web services is the architectural support for the building of agile data systems.<br />
<br />
===Interoperability Standards and Data Wrapping===<br />
Interoperable data access will be accomplished through the use of a suite of international standards. The naming of individual chemical parameters will follow the CF convention used by the Climate and Forecast (CF) communities. The existing names for atmospheric chemicals in the CF convention were inadequate to accomodate all the parameters used in the HTAP modeling. in order to remedy this shortcomming the list of standard names was extended by the HTAP community under leadership of C. Textor. She also became a memeber of the CF convention board that is the custodian of the standard names. The standard names for HTAP models were developed using a collaborative wiki workspace. <br />
<br />
The data transfer from providers to users also benefit from standard data formats. For modeling data the use of netCDF-CF as a standard format is recomended. The use of a standard physical data format and the CF naming conventions allows, in principle, the seamless connection between data provider and consumer services. It should be noted, however, that at this time the CF naming convention has only been developed for the model parameters and not for the various observational parameters. Also, the nedCDF CF is primarily used as a model data exchange format, while the transfer of surface monitoring data and satellite data is less standardized.<br />
<br />
The third aspect of data interoperability is a standard query language through which user services request specific data from the provider services. It is proposed that for the HTAP data information system adapts the Web Coverage Service (WCS) as the standard data query language. The WCS data access protocol is defined by the international Open Geospatial Consortium (OGC), which is also the key organization responsible for interoperability standards in GEOSS. Since the WCS protocol was originally developed for the GIS community, it was necessary to adapt it to the needs of "fluid Earth sciences". Members of the HTAP group have been actively participating in the development and testing of the WCS interoperability standards. <br />
<br />
Standards-based data access can be accomplished by ‘wrapping’ the heterogeneous data into standardized web services, callable through well-defined Internet protocols. The result of this ‘wrapping’ process is an array of homogeneous, virtual datasets that can be accessed through a standard querry language and the returned data are packaged in standard format, directly usable by the consuming services.<br />
<br />
===Quality Assurance===<br />
This is just a place holder. Here we need to explain the quality assurance steps that are needed to prepare the HTAP Integrated Dataset.<br />
<br />
===Homogenization and Integration===<br />
The HTAP Integrated Dataset (HID) will be used to compare models to observations. It will be created from the proper combination of a variety of surface, upper air and satellite observations. However, before the inclusion into HID, each dataset will need to be scrutinezed to make it suitable for model comparison. The scrutiny may include filtering, aggregation and possibly fusion operations.<br />
<br />
A good example is the 400-station AIRNOW network reporting hourly ozone and PM2.5 concentrations over the US. The AIRNOW network includes both urban sites that are strongly influenced by local sources. Urban stations need to be removed from HID dataset.<br />
<br />
Developing the specific criteria and procedures for the HTAP integrated dataset will require the attention of a HTAP subgroup.<br />
<br />
==Integrated Data Set==<br />
<br />
The providers of observational, emission, and modeling data are distributed. The individual data sets will be funeled into the integrated HTAP data set. This data set transfer will be accomplished using standard data transfer protocols. Transfering the data from the provider to the user will include two additional observations for each data set, quality assurance and semantic transformation.<br />
<br />
On the left the inputs from observations, emissions, and model data<br />
<br />
<br />
<br />
===Identify providers and datasets relevant to HTAP===<br />
===Integrate multi-sensory data into a coherent data systems===<br />
<br />
[[Image:HTAP_IntDataSys.png|600px|left]] In order to accomplish the objectives set for the HTAP TF it is evident that it will require the System of Systems approach as envisioned by GEO. Both the modeling of the chemical constituents as well as the Earth observations used to document the hemispheric transport are being performed by individual autonomous projects and programs (Systems). A key function of the Task Force is to assess the contributions of the individual systems and then integrate those into a coherent assessment. Much of this interim report is in fact such an assessment of the current knowledge based on previous or past models and observations.<br />
<br />
In the next, upcoming phase of HTAP is to actually integrate the existing modeling and observing systems such that their combination and fusion can yield a deeper and more robust understanding of HTAP. since this next phase will involve integration of multiple relevant data sets, comparison and evaluation of multiple models, as well as the reconciliation of the observations and models to create the new and deeper understanding . In order to execute this extensive integration, it will be necessary to conduct this integration using a generally accepted framework as well as accepted scientific methods. It is believed that the GEO architecture and the System of Systems is an attractive approach to HTAP integration.<br />
<br />
The HTAP program can also be considered an early demonstration of the GEO concepts through its end to end approach. Both the atmospheric modeling, as well as the observations currently exist through the operation of existing modeling and observation systems. The Task Force has agreed organizing and evaluating those models, assembling and integrating the observational data sets and then reconciling the models with the observations will be the focus of the Phase II effort. The Task Force will also prepare and deliver a report to LRTP to aid its deliberations and its decision making processes. This sequence of activities constitutes an end to end approach that turns observations and models into actionable knowledge for societal decision making. One could say that this is an octagonal approach to more deliberate step by step development of GEOSS where in Phase I interoperability, in Phase II ???<br />
<br />
Provider nodes <br />
* Federated Data System DataFed<br />
* NASA Data System Giovanni<br />
* Emission Data System NEISGEI<br />
* Juelich Data System<br />
* AeorCom<br />
* EMEP<br />
<br />
<br />
Data Networks. Connected hubs; Show Core HTAP network<br />
<br />
[http://datafedwiki.wustl.edu/index.php/2007-03-26:_Datasets_for_HTAP_through_DataFed See 20 selected datasets] in the federated data systems, DataFed<br />
<br />
===Perform QA at all steps===<br />
<br />
==Reconciliation and Integration of Observations, Emissions and Models==<br />
===Compare observations to models and emissions===<br />
===Characterize pollutant pattern and SSR=== <br />
===Develop real-time data assimilation===<br />
===Perform reanalysis with assimilated data=== <br />
<br />
==Organizational Aspects==<br />
===HTAP Relationship to GEO and GEOSS===<br />
There is an outstanding opportunity to develop a mutually beneficial and supportive relationship between the activities of the HTAP Task Force and that of GEO. The Group of Earth Observations with its broad range of national and organizational members has developed a general architectural framework for turning Earth observations into societal benefits. The three main components of this architecture are models, and observations, which then feed into decision support systems used in a variety of societal decision making. This general framework is well suited as an architectural guide to the HTAP program. However, this framework lacks specific guidelines and implementation details that are needed for practical Earth observing and decision support systems. <br />
<br />
[[Image:400px HTAP GEOSS Arch.png|400px|left]] The HTAP program provides an opportunity to extend the GEO framework with more detailed features that arise in specific HTAP application context. In case of HTAP, the major activities are shown in the architectural diagram of Figure ?? The HTAP modeling is conducted through global scale chemical transport models that simulate or forecast the atmospheric chemical composition as a consequence of natural and human induced emissions. The observations arise from satellite, ground-based and airborne observations of chemical constituents and their hemispheric transport. The decision support system consists of the scientists as members of the HTAP task force, and the task force co-chairs as the intermediaries between LRTP and the TF. (Terry, Andre this description of the HTAP DSS needs your help). <br />
<br />
Representing the Decision Support System (DSS) for HTAP is an important aspect because it can guide the design and and implementation of the information system for the support of the DSS processes. A higher resolution DSS design document is also beneficial as a communications medium for the interacting system of systems. A domain specific DSS may also allow the generalization and applicability to the design of DSS structures for similar application domains. <br />
<br />
The implementation of the GEO framework utilizes the concept of Global Observing System of Systems (GEOSS). Traditionally, Earth observations were performed by well defined systems such as specific satellites and monitoring networks which were designed and operated using systems engineering principles. However, GEO recognized that the understanding of Earth system requires the utilization and integration of the individual, autonomous systems. The key difference between the Systems and System of System approaches are highlighted in Table 1.<br />
<br />
===HTAP Relationship to IGACO=== <br />
[[Image:IGACO Framework.png|300px|left]] The HTAP program can also have a mutually beneficial interaction with the IGACO program. Integrated Global Atmospheric Chemistry Observations (IGACO) Theme is part of the Integrated Global Observing Strategy (IGOS). The IGACO program is formulated by an array of experienced atmospheric scientists. IGACO proposes a specific framework for the system components that processes the observational data and models. It also specifies the higher level information products that can be used for creating social benefit. <br />
<br />
In the IGACO framework, the data products from satellite, surface, and aircraft observations are collected for inclusion into an integrated data archive. A key component in the IGACO data flow is the mandatory quality assurance and QA QC that precedes the archiving. This is particularly important for HTAP where multi-sensory data from many different providers are to be combined into an integrated database. In the IGACO framework, a key output from the data system is a high integrated spatial-temporal data set which can be used in a multitude of applications that produce socal benefit. The goal of creating such an integrated data set is shared by the IGACO and the HTAP programs (?? Len, this section could benefit from your input)<br />
<br />
===HTAP Relationship with Other Programs===<br />
---<br />
---<br />
<br />
<br />
<br />
==Links==<br />
[[IGACO Framework]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=HTAP_Report,_Sub-Chap._6_-_Data/Info_System&diff=7942HTAP Report, Sub-Chap. 6 - Data/Info System2007-04-12T04:55:36Z<p>128.252.167.79: /* HTAP Data System Architecture */</p>
<hr />
<div><noinclude>[[Integrated_Global_Dataset| < Back to Integrated Global Dataset on ESIP wiki]] <br></noinclude> <br />
<br />
This wiki page is inteded to be the collaborative workspace for Task Force members interested in the HTAP Information System development, implementation and use. The contents can be freely edited by anyone who is logged in. Comments and feedback can also be sent by email to rhusar@me.wustl.edu. The content of this page is is on the '''''Information System''''' that supposts Chapter 6. The subsections are action-oriented. <br />
<br />
From [http://www.htap.org/activities/2007_interim_report/TF%20HTAP%202007%20Interim%20Report%20Prospectus%20Oct%202006.pdf Oct 2006 Prospectus]: This chapter (Chapter 6: '''Integration of Observations, Emissions and Modeling'''.) will discuss the need to integrate information from observations, models, and emissions inventories to better understand intercontinental transport. A draft [[HTAP Report Chap. 6 - Jan 07 Outline]] is also on this wiki.<br />
<br />
==Introduction==<br />
The goal of the HTAP data system is to aid the assembly of an appropriate set of observations and models, and to facilitate the integration and fusion of observations with models. <br />
<br />
Recent developments in air quality monitoring, modeling and information technologies offer outstanding<br />
opportunities to fulfill the information needs for the HTAP integration effort. The data from surface-based air pollution monitoring networks now routinely provide spatio-temporal and chemical patterns of ozone and PM. Satellite<br />
sensors with global coverage and kilometer-scale spatial resolution now provide real-time snapshots which depict the pattern of industrial haze, smoke and dust in stunning detail. The ‘terabytes’ of data from these surface and remote sensors can now be stored, processed and delivered in near-real time. The instantaneous ‘horizontal’ diffusion of information via the Internet now permits, in principle, the delivery of the right information to the right people at the right place and time. Standardized computer-computer communication languages and Service-Oriented Architectures (SOA) now facilitate the flexible access, quality assurance and processing of raw data into high-grade ‘actionable’ knowledge suitable for HTAP policy decissions. Last but not least, the World Wide Web has opened the way to generous sharing of data and tools leading to collaborative analysis and virtual workgroups. Nevertheless, air quality data analyses and data model integration face significant hurdles. The “data deluge” problem is especially acute for satellite observations and model outputs. <br />
<br />
====HTAP Data System Architecture====<br />
The proposed HTAP data system will integrate the emerging observational and modeling resources by not a centrally planned and powerful, dynamic technologies and through a collaborative federation philosophy. The key roles of the federated data system are to (1) facilitate accsess to the distributed data; (2) ensure the seamless flow of data through interoperable interfaces; (3) provide a set of basic tools for data exploration and processing. <br />
<br />
The Service Oriented Architecture (SOA) allows connecting the web service components (e.g. services for data access, transformation, fusion, rendering, etc.) using work flow software.<br />
<br />
==== Data Selection Criteria====<br />
Monitoring data for atmospheric constituents are now available from a variety of sources, not all of which are suitable for the integrated HTAP data set. Given the limited scope and resources of HTAP, it will be necessary to select a subset of the available observational data that would be most appropriate for the preparation of the 2009 assessment. A set of criteria for the data selection is given below.<br />
* The suitability criteria may include the measured parameters, their spatial extent and coverage density, as well as the time range and sampling frequency with major emphasis on data quality (defined as???).<br />
* The compilation of the global integrated data sets should initially focus on PM and ozone, including the gaseous precursors.<br />
* Subsequent data collaction should also include observations on atmospheric mercury and persistent organic pollutants (POPS). <br />
* In order to cover hemispheric transport of air pollutants the data set system should accept and utilize data from outside the geographical scope of EMEP.<br />
* The data gathering should begin the compilation using the of existing databases in Europe (RETRO, TRADEOFF, QUANTIFY, CRATE, AEROCOM) and virtual federated systems in the US (DataFed, Giovanni and others). <br />
* TF data integration should also contribute to and benefit from other ongoing data integration efforts to integrate the global data resources, most notably, with ACCENT project in Europe, and similar efforts in the US.<br />
* Special emphasis should be placed on the collection of suitable vertical profiles from aircraft measurements as well as from surface and space-borne lidars.<br />
<br />
The evaluation of suitabile observational data sets for model validation and fusion will require close interaction between the modeling and observational communities. An wiki workspace for open collaborative evaluation of different datasets is desirebale.<br />
<br />
===Homogenize the data into an interoperable dataset===<br />
The primary goal of the data system is to allow the integration of observational emission and model data. The model evaluation requires that both the observations and if possible the emissions are fixed. For this reason it is desirable to prepare an integrated observational database to which the various model implementations can be compared to. On one hand, the integrated data set should be as inclusive as possible. On the other hand, such a goal needs to be tempered by many limitations that preclude a broad, inclusive data integration<br />
<br />
The proposed HTAP data system will be a hybrid combination of both distributed and fixed components as illustrated schematically in Figure ??. Both the data providers as well as the HTAP analysts-users will be distributed. However, they will be connected through an integrated HTAP database which should be a stable, virtually fixed database. The section below describes the main component of this distributed data system. <br />
<br />
The HTAP integrated data system has to perform two major functions. The primary function is to facilitate the creation of an integrated data set suitable for model evaluation and pollutant characterization. The multiple steps that are required for this functionality are shown on the left. The sequence of operations can be viewed as a value chain that transform the raw input data into a highly organized integrated data set. Since these value adding steps have to be performed for each candidate data set the operations can be performed in sequence following the rules of data flow programming. For true quality assurance and for data homogenization the data flow channels for individual data sets need to be evaluated in the context of other data. <br />
<br />
The operations that prepare the integrated data set can be viewed as services that operate on the data stream using the principles of modern Service Oriented Architecure (SOA). Each of the services with its clearly defined functionality and firmly specified service interface. In principle, the standards-based interface allows a connection of service chains using formal work flow software. The main benefit of service oriented architecture is that allows the building of agile application programs that can respond to changes in the data input conditions as well as the output requirements. The service oriented software architecture is an implementation of the System of Systems approach. Furthermore, since each of the services can be operated by autonomous providers and the "system" that implements the service is behind the service interfece. In other words, following the SOA approach, not only the data providers are distributed but also the processing services as well. This flexible approach to distributed computing allows the distribution of labor (chain of services) in many different configurations. For instance, wrapping the existing data with standards based interfaces for data access can be performed for an entire cluster of data. This is the approach taken in the federated data system DataFed. Given a standard interface to a variety of data sets, the Quality Assurance service can be performed by another srvice provider that is seamlesly connected to the data access mediator. Similarly, the service that prepares a data set for data integration can be provided by another service in the data flow network. The flexibility offered through the chaining and orchestration of distributed, loosely coupled web services is the architectural support for the building of agile data systems.<br />
<br />
===Interoperability Standards and Data Wrapping===<br />
Interoperable data access will be accomplished through the use of a suite of international standards. The naming of individual chemical parameters will follow the CF convention used by the Climate and Forecast (CF) communities. The existing names for atmospheric chemicals in the CF convention were inadequate to accomodate all the parameters used in the HTAP modeling. in order to remedy this shortcomming the list of standard names was extended by the HTAP community under leadership of C. Textor. She also became a memeber of the CF convention board that is the custodian of the standard names. The standard names for HTAP models were developed using a collaborative wiki workspace. <br />
<br />
The data transfer from providers to users also benefit from standard data formats. For modeling data the use of netCDF-CF as a standard format is recomended. The use of a standard physical data format and the CF naming conventions allows, in principle, the seamless connection between data provider and consumer services. It should be noted, however, that at this time the CF naming convention has only been developed for the model parameters and not for the various observational parameters. Also, the nedCDF CF is primarily used as a model data exchange format, while the transfer of surface monitoring data and satellite data is less standardized.<br />
<br />
The third aspect of data interoperability is a standard query language through which user services request specific data from the provider services. It is proposed that for the HTAP data information system adapts the Web Coverage Service (WCS) as the standard data query language. The WCS data access protocol is defined by the international Open Geospatial Consortium (OGC), which is also the key organization responsible for interoperability standards in GEOSS. Since the WCS protocol was originally developed for the GIS community, it was necessary to adapt it to the needs of "fluid Earth sciences". Members of the HTAP group have been actively participating in the development and testing of the WCS interoperability standards. <br />
<br />
Standards-based data access can be accomplished by ‘wrapping’ the heterogeneous data into standardized web services, callable through well-defined Internet protocols. The result of this ‘wrapping’ process is an array of homogeneous, virtual datasets that can be accessed through a standard querry language and the returned data are packaged in standard format, directly usable by the consuming services.<br />
<br />
===Quality Assurance===<br />
This is just a place holder. Here we need to explain the quality assurance steps that are needed to prepare the HTAP Integrated Dataset.<br />
<br />
===Homogenization and Integration===<br />
The HTAP Integrated Dataset (HID) will be used to compare models to observations. It will be created from the proper combination of a variety of surface, upper air and satellite observations. However, before the inclusion into HID, each dataset will need to be scrutinezed to make it suitable for model comparison. The scrutiny may include filtering, aggregation and possibly fusion operations.<br />
<br />
A good example is the 400-station AIRNOW network reporting hourly ozone and PM2.5 concentrations over the US. The AIRNOW network includes both urban sites that are strongly influenced by local sources. Urban stations need to be removed from HID dataset.<br />
<br />
Developing the specific criteria and procedures for the HTAP integrated dataset will require the attention of a HTAP subgroup.<br />
<br />
==Integrated Data Set==<br />
<br />
The providers of observational, emission, and modeling data are distributed. The individual data sets will be funeled into the integrated HTAP data set. This data set transfer will be accomplished using standard data transfer protocols. Transfering the data from the provider to the user will include two additional observations for each data set, quality assurance and semantic transformation.<br />
<br />
On the left the inputs from observations, emissions, and model data<br />
<br />
<br />
<br />
===Identify providers and datasets relevant to HTAP===<br />
===Integrate multi-sensory data into a coherent data systems===<br />
<br />
[[Image:HTAP_IntDataSys.png|600px|left]] In order to accomplish the objectives set for the HTAP TF it is evident that it will require the System of Systems approach as envisioned by GEO. Both the modeling of the chemical constituents as well as the Earth observations used to document the hemispheric transport are being performed by individual autonomous projects and programs (Systems). A key function of the Task Force is to assess the contributions of the individual systems and then integrate those into a coherent assessment. Much of this interim report is in fact such an assessment of the current knowledge based on previous or past models and observations.<br />
<br />
In the next, upcoming phase of HTAP is to actually integrate the existing modeling and observing systems such that their combination and fusion can yield a deeper and more robust understanding of HTAP. since this next phase will involve integration of multiple relevant data sets, comparison and evaluation of multiple models, as well as the reconciliation of the observations and models to create the new and deeper understanding . In order to execute this extensive integration, it will be necessary to conduct this integration using a generally accepted framework as well as accepted scientific methods. It is believed that the GEO architecture and the System of Systems is an attractive approach to HTAP integration.<br />
<br />
The HTAP program can also be considered an early demonstration of the GEO concepts through its end to end approach. Both the atmospheric modeling, as well as the observations currently exist through the operation of existing modeling and observation systems. The Task Force has agreed organizing and evaluating those models, assembling and integrating the observational data sets and then reconciling the models with the observations will be the focus of the Phase II effort. The Task Force will also prepare and deliver a report to LRTP to aid its deliberations and its decision making processes. This sequence of activities constitutes an end to end approach that turns observations and models into actionable knowledge for societal decision making. One could say that this is an octagonal approach to more deliberate step by step development of GEOSS where in Phase I interoperability, in Phase II ???<br />
<br />
Provider nodes <br />
* Federated Data System DataFed<br />
* NASA Data System Giovanni<br />
* Emission Data System NEISGEI<br />
* Juelich Data System<br />
* AeorCom<br />
* EMEP<br />
<br />
<br />
Data Networks. Connected hubs; Show Core HTAP network<br />
<br />
[http://datafedwiki.wustl.edu/index.php/2007-03-26:_Datasets_for_HTAP_through_DataFed See 20 selected datasets] in the federated data systems, DataFed<br />
<br />
===Perform QA at all steps===<br />
<br />
==Reconciliation and Integration of Observations, Emissions and Models==<br />
===Compare observations to models and emissions===<br />
===Characterize pollutant pattern and SSR=== <br />
===Develop real-time data assimilation===<br />
===Perform reanalysis with assimilated data=== <br />
<br />
==Organizational Aspects==<br />
===HTAP Relationship to GEO and GEOSS===<br />
There is an outstanding opportunity to develop a mutually beneficial and supportive relationship between the activities of the HTAP Task Force and that of GEO. The Group of Earth Observations with its broad range of national and organizational members has developed a general architectural framework for turning Earth observations into societal benefits. The three main components of this architecture are models, and observations, which then feed into decision support systems used in a variety of societal decision making. This general framework is well suited as an architectural guide to the HTAP program. However, this framework lacks specific guidelines and implementation details that are needed for practical Earth observing and decision support systems. <br />
<br />
[[Image:400px HTAP GEOSS Arch.png|400px|left]] The HTAP program provides an opportunity to extend the GEO framework with more detailed features that arise in specific HTAP application context. In case of HTAP, the major activities are shown in the architectural diagram of Figure ?? The HTAP modeling is conducted through global scale chemical transport models that simulate or forecast the atmospheric chemical composition as a consequence of natural and human induced emissions. The observations arise from satellite, ground-based and airborne observations of chemical constituents and their hemispheric transport. The decision support system consists of the scientists as members of the HTAP task force, and the task force co-chairs as the intermediaries between LRTP and the TF. (Terry, Andre this description of the HTAP DSS needs your help). <br />
<br />
Representing the Decision Support System (DSS) for HTAP is an important aspect because it can guide the design and and implementation of the information system for the support of the DSS processes. A higher resolution DSS design document is also beneficial as a communications medium for the interacting system of systems. A domain specific DSS may also allow the generalization and applicability to the design of DSS structures for similar application domains. <br />
<br />
The implementation of the GEO framework utilizes the concept of Global Observing System of Systems (GEOSS). Traditionally, Earth observations were performed by well defined systems such as specific satellites and monitoring networks which were designed and operated using systems engineering principles. However, GEO recognized that the understanding of Earth system requires the utilization and integration of the individual, autonomous systems. The key difference between the Systems and System of System approaches are highlighted in Table 1.<br />
<br />
===HTAP Relationship to IGACO=== <br />
[[Image:IGACO Framework.png|300px|left]] The HTAP program can also have a mutually beneficial interaction with the IGACO program. Integrated Global Atmospheric Chemistry Observations (IGACO) Theme is part of the Integrated Global Observing Strategy (IGOS). The IGACO program is formulated by an array of experienced atmospheric scientists. IGACO proposes a specific framework for the system components that processes the observational data and models. It also specifies the higher level information products that can be used for creating social benefit. <br />
<br />
In the IGACO framework, the data products from satellite, surface, and aircraft observations are collected for inclusion into an integrated data archive. A key component in the IGACO data flow is the mandatory quality assurance and QA QC that precedes the archiving. This is particularly important for HTAP where multi-sensory data from many different providers are to be combined into an integrated database. In the IGACO framework, a key output from the data system is a high integrated spatial-temporal data set which can be used in a multitude of applications that produce socal benefit. The goal of creating such an integrated data set is shared by the IGACO and the HTAP programs (?? Len, this section could benefit from your input)<br />
<br />
===HTAP Relationship with Other Programs===<br />
---<br />
---<br />
<br />
<br />
<br />
==Links==<br />
[[IGACO Framework]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=HTAP_Report,_Sub-Chap._6_-_Data/Info_System&diff=7941HTAP Report, Sub-Chap. 6 - Data/Info System2007-04-12T04:54:20Z<p>128.252.167.79: /* Introduction */</p>
<hr />
<div><noinclude>[[Integrated_Global_Dataset| < Back to Integrated Global Dataset on ESIP wiki]] <br></noinclude> <br />
<br />
This wiki page is inteded to be the collaborative workspace for Task Force members interested in the HTAP Information System development, implementation and use. The contents can be freely edited by anyone who is logged in. Comments and feedback can also be sent by email to rhusar@me.wustl.edu. The content of this page is is on the '''''Information System''''' that supposts Chapter 6. The subsections are action-oriented. <br />
<br />
From [http://www.htap.org/activities/2007_interim_report/TF%20HTAP%202007%20Interim%20Report%20Prospectus%20Oct%202006.pdf Oct 2006 Prospectus]: This chapter (Chapter 6: '''Integration of Observations, Emissions and Modeling'''.) will discuss the need to integrate information from observations, models, and emissions inventories to better understand intercontinental transport. A draft [[HTAP Report Chap. 6 - Jan 07 Outline]] is also on this wiki.<br />
<br />
====HTAP Data System Architecture====<br />
The proposed HTAP data system will integrate the emerging observational and modeling resources by not a centrally planned and powerful, dynamic technologies and through a collaborative federation philosophy. The key roles of the federated data system are to (1) facilitate accsess to the distributed data; (2) ensure the seamless flow of data through interoperable interfaces; (3) provide a set of basic tools for data exploration and processing. <br />
<br />
The Service Oriented Architecture (SOA) allows connecting the web service components (e.g. services for data access, transformation, fusion, rendering, etc.) using work flow software.<br />
<br />
<br />
<br />
<br />
==== Data Selection Criteria====<br />
Monitoring data for atmospheric constituents are now available from a variety of sources, not all of which are suitable for the integrated HTAP data set. Given the limited scope and resources of HTAP, it will be necessary to select a subset of the available observational data that would be most appropriate for the preparation of the 2009 assessment. A set of criteria for the data selection is given below.<br />
* The suitability criteria may include the measured parameters, their spatial extent and coverage density, as well as the time range and sampling frequency with major emphasis on data quality (defined as???).<br />
* The compilation of the global integrated data sets should initially focus on PM and ozone, including the gaseous precursors.<br />
* Subsequent data collaction should also include observations on atmospheric mercury and persistent organic pollutants (POPS). <br />
* In order to cover hemispheric transport of air pollutants the data set system should accept and utilize data from outside the geographical scope of EMEP.<br />
* The data gathering should begin the compilation using the of existing databases in Europe (RETRO, TRADEOFF, QUANTIFY, CRATE, AEROCOM) and virtual federated systems in the US (DataFed, Giovanni and others). <br />
* TF data integration should also contribute to and benefit from other ongoing data integration efforts to integrate the global data resources, most notably, with ACCENT project in Europe, and similar efforts in the US.<br />
* Special emphasis should be placed on the collection of suitable vertical profiles from aircraft measurements as well as from surface and space-borne lidars.<br />
<br />
The evaluation of suitabile observational data sets for model validation and fusion will require close interaction between the modeling and observational communities. An wiki workspace for open collaborative evaluation of different datasets is desirebale.<br />
<br />
===Homogenize the data into an interoperable dataset===<br />
The primary goal of the data system is to allow the integration of observational emission and model data. The model evaluation requires that both the observations and if possible the emissions are fixed. For this reason it is desirable to prepare an integrated observational database to which the various model implementations can be compared to. On one hand, the integrated data set should be as inclusive as possible. On the other hand, such a goal needs to be tempered by many limitations that preclude a broad, inclusive data integration<br />
<br />
The proposed HTAP data system will be a hybrid combination of both distributed and fixed components as illustrated schematically in Figure ??. Both the data providers as well as the HTAP analysts-users will be distributed. However, they will be connected through an integrated HTAP database which should be a stable, virtually fixed database. The section below describes the main component of this distributed data system. <br />
<br />
The HTAP integrated data system has to perform two major functions. The primary function is to facilitate the creation of an integrated data set suitable for model evaluation and pollutant characterization. The multiple steps that are required for this functionality are shown on the left. The sequence of operations can be viewed as a value chain that transform the raw input data into a highly organized integrated data set. Since these value adding steps have to be performed for each candidate data set the operations can be performed in sequence following the rules of data flow programming. For true quality assurance and for data homogenization the data flow channels for individual data sets need to be evaluated in the context of other data. <br />
<br />
The operations that prepare the integrated data set can be viewed as services that operate on the data stream using the principles of modern Service Oriented Architecure (SOA). Each of the services with its clearly defined functionality and firmly specified service interface. In principle, the standards-based interface allows a connection of service chains using formal work flow software. The main benefit of service oriented architecture is that allows the building of agile application programs that can respond to changes in the data input conditions as well as the output requirements. The service oriented software architecture is an implementation of the System of Systems approach. Furthermore, since each of the services can be operated by autonomous providers and the "system" that implements the service is behind the service interfece. In other words, following the SOA approach, not only the data providers are distributed but also the processing services as well. This flexible approach to distributed computing allows the distribution of labor (chain of services) in many different configurations. For instance, wrapping the existing data with standards based interfaces for data access can be performed for an entire cluster of data. This is the approach taken in the federated data system DataFed. Given a standard interface to a variety of data sets, the Quality Assurance service can be performed by another srvice provider that is seamlesly connected to the data access mediator. Similarly, the service that prepares a data set for data integration can be provided by another service in the data flow network. The flexibility offered through the chaining and orchestration of distributed, loosely coupled web services is the architectural support for the building of agile data systems.<br />
<br />
===Interoperability Standards and Data Wrapping===<br />
Interoperable data access will be accomplished through the use of a suite of international standards. The naming of individual chemical parameters will follow the CF convention used by the Climate and Forecast (CF) communities. The existing names for atmospheric chemicals in the CF convention were inadequate to accomodate all the parameters used in the HTAP modeling. in order to remedy this shortcomming the list of standard names was extended by the HTAP community under leadership of C. Textor. She also became a memeber of the CF convention board that is the custodian of the standard names. The standard names for HTAP models were developed using a collaborative wiki workspace. <br />
<br />
The data transfer from providers to users also benefit from standard data formats. For modeling data the use of netCDF-CF as a standard format is recomended. The use of a standard physical data format and the CF naming conventions allows, in principle, the seamless connection between data provider and consumer services. It should be noted, however, that at this time the CF naming convention has only been developed for the model parameters and not for the various observational parameters. Also, the nedCDF CF is primarily used as a model data exchange format, while the transfer of surface monitoring data and satellite data is less standardized.<br />
<br />
The third aspect of data interoperability is a standard query language through which user services request specific data from the provider services. It is proposed that for the HTAP data information system adapts the Web Coverage Service (WCS) as the standard data query language. The WCS data access protocol is defined by the international Open Geospatial Consortium (OGC), which is also the key organization responsible for interoperability standards in GEOSS. Since the WCS protocol was originally developed for the GIS community, it was necessary to adapt it to the needs of "fluid Earth sciences". Members of the HTAP group have been actively participating in the development and testing of the WCS interoperability standards. <br />
<br />
Standards-based data access can be accomplished by ‘wrapping’ the heterogeneous data into standardized web services, callable through well-defined Internet protocols. The result of this ‘wrapping’ process is an array of homogeneous, virtual datasets that can be accessed through a standard querry language and the returned data are packaged in standard format, directly usable by the consuming services.<br />
<br />
===Quality Assurance===<br />
This is just a place holder. Here we need to explain the quality assurance steps that are needed to prepare the HTAP Integrated Dataset.<br />
<br />
===Homogenization and Integration===<br />
The HTAP Integrated Dataset (HID) will be used to compare models to observations. It will be created from the proper combination of a variety of surface, upper air and satellite observations. However, before the inclusion into HID, each dataset will need to be scrutinezed to make it suitable for model comparison. The scrutiny may include filtering, aggregation and possibly fusion operations.<br />
<br />
A good example is the 400-station AIRNOW network reporting hourly ozone and PM2.5 concentrations over the US. The AIRNOW network includes both urban sites that are strongly influenced by local sources. Urban stations need to be removed from HID dataset.<br />
<br />
Developing the specific criteria and procedures for the HTAP integrated dataset will require the attention of a HTAP subgroup.<br />
<br />
==Integrated Data Set==<br />
<br />
The providers of observational, emission, and modeling data are distributed. The individual data sets will be funeled into the integrated HTAP data set. This data set transfer will be accomplished using standard data transfer protocols. Transfering the data from the provider to the user will include two additional observations for each data set, quality assurance and semantic transformation.<br />
<br />
On the left the inputs from observations, emissions, and model data<br />
<br />
<br />
<br />
===Identify providers and datasets relevant to HTAP===<br />
===Integrate multi-sensory data into a coherent data systems===<br />
<br />
[[Image:HTAP_IntDataSys.png|600px|left]] In order to accomplish the objectives set for the HTAP TF it is evident that it will require the System of Systems approach as envisioned by GEO. Both the modeling of the chemical constituents as well as the Earth observations used to document the hemispheric transport are being performed by individual autonomous projects and programs (Systems). A key function of the Task Force is to assess the contributions of the individual systems and then integrate those into a coherent assessment. Much of this interim report is in fact such an assessment of the current knowledge based on previous or past models and observations.<br />
<br />
In the next, upcoming phase of HTAP is to actually integrate the existing modeling and observing systems such that their combination and fusion can yield a deeper and more robust understanding of HTAP. since this next phase will involve integration of multiple relevant data sets, comparison and evaluation of multiple models, as well as the reconciliation of the observations and models to create the new and deeper understanding . In order to execute this extensive integration, it will be necessary to conduct this integration using a generally accepted framework as well as accepted scientific methods. It is believed that the GEO architecture and the System of Systems is an attractive approach to HTAP integration.<br />
<br />
The HTAP program can also be considered an early demonstration of the GEO concepts through its end to end approach. Both the atmospheric modeling, as well as the observations currently exist through the operation of existing modeling and observation systems. The Task Force has agreed organizing and evaluating those models, assembling and integrating the observational data sets and then reconciling the models with the observations will be the focus of the Phase II effort. The Task Force will also prepare and deliver a report to LRTP to aid its deliberations and its decision making processes. This sequence of activities constitutes an end to end approach that turns observations and models into actionable knowledge for societal decision making. One could say that this is an octagonal approach to more deliberate step by step development of GEOSS where in Phase I interoperability, in Phase II ???<br />
<br />
Provider nodes <br />
* Federated Data System DataFed<br />
* NASA Data System Giovanni<br />
* Emission Data System NEISGEI<br />
* Juelich Data System<br />
* AeorCom<br />
* EMEP<br />
<br />
<br />
Data Networks. Connected hubs; Show Core HTAP network<br />
<br />
[http://datafedwiki.wustl.edu/index.php/2007-03-26:_Datasets_for_HTAP_through_DataFed See 20 selected datasets] in the federated data systems, DataFed<br />
<br />
===Perform QA at all steps===<br />
<br />
==Reconciliation and Integration of Observations, Emissions and Models==<br />
===Compare observations to models and emissions===<br />
===Characterize pollutant pattern and SSR=== <br />
===Develop real-time data assimilation===<br />
===Perform reanalysis with assimilated data=== <br />
<br />
==Organizational Aspects==<br />
===HTAP Relationship to GEO and GEOSS===<br />
There is an outstanding opportunity to develop a mutually beneficial and supportive relationship between the activities of the HTAP Task Force and that of GEO. The Group of Earth Observations with its broad range of national and organizational members has developed a general architectural framework for turning Earth observations into societal benefits. The three main components of this architecture are models, and observations, which then feed into decision support systems used in a variety of societal decision making. This general framework is well suited as an architectural guide to the HTAP program. However, this framework lacks specific guidelines and implementation details that are needed for practical Earth observing and decision support systems. <br />
<br />
[[Image:400px HTAP GEOSS Arch.png|400px|left]] The HTAP program provides an opportunity to extend the GEO framework with more detailed features that arise in specific HTAP application context. In case of HTAP, the major activities are shown in the architectural diagram of Figure ?? The HTAP modeling is conducted through global scale chemical transport models that simulate or forecast the atmospheric chemical composition as a consequence of natural and human induced emissions. The observations arise from satellite, ground-based and airborne observations of chemical constituents and their hemispheric transport. The decision support system consists of the scientists as members of the HTAP task force, and the task force co-chairs as the intermediaries between LRTP and the TF. (Terry, Andre this description of the HTAP DSS needs your help). <br />
<br />
Representing the Decision Support System (DSS) for HTAP is an important aspect because it can guide the design and and implementation of the information system for the support of the DSS processes. A higher resolution DSS design document is also beneficial as a communications medium for the interacting system of systems. A domain specific DSS may also allow the generalization and applicability to the design of DSS structures for similar application domains. <br />
<br />
The implementation of the GEO framework utilizes the concept of Global Observing System of Systems (GEOSS). Traditionally, Earth observations were performed by well defined systems such as specific satellites and monitoring networks which were designed and operated using systems engineering principles. However, GEO recognized that the understanding of Earth system requires the utilization and integration of the individual, autonomous systems. The key difference between the Systems and System of System approaches are highlighted in Table 1.<br />
<br />
===HTAP Relationship to IGACO=== <br />
[[Image:IGACO Framework.png|300px|left]] The HTAP program can also have a mutually beneficial interaction with the IGACO program. Integrated Global Atmospheric Chemistry Observations (IGACO) Theme is part of the Integrated Global Observing Strategy (IGOS). The IGACO program is formulated by an array of experienced atmospheric scientists. IGACO proposes a specific framework for the system components that processes the observational data and models. It also specifies the higher level information products that can be used for creating social benefit. <br />
<br />
In the IGACO framework, the data products from satellite, surface, and aircraft observations are collected for inclusion into an integrated data archive. A key component in the IGACO data flow is the mandatory quality assurance and QA QC that precedes the archiving. This is particularly important for HTAP where multi-sensory data from many different providers are to be combined into an integrated database. In the IGACO framework, a key output from the data system is a high integrated spatial-temporal data set which can be used in a multitude of applications that produce socal benefit. The goal of creating such an integrated data set is shared by the IGACO and the HTAP programs (?? Len, this section could benefit from your input)<br />
<br />
===HTAP Relationship with Other Programs===<br />
---<br />
---<br />
<br />
<br />
<br />
==Links==<br />
[[IGACO Framework]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=HTAP_Report,_Sub-Chap._6_-_Data/Info_System&diff=7940HTAP Report, Sub-Chap. 6 - Data/Info System2007-04-12T04:53:55Z<p>128.252.167.79: /* Integration of Observations and Emissions */</p>
<hr />
<div><noinclude>[[Integrated_Global_Dataset| < Back to Integrated Global Dataset on ESIP wiki]] <br></noinclude> <br />
<br />
This wiki page is inteded to be the collaborative workspace for Task Force members interested in the HTAP Information System development, implementation and use. The contents can be freely edited by anyone who is logged in. Comments and feedback can also be sent by email to rhusar@me.wustl.edu. The content of this page is is on the '''''Information System''''' that supposts Chapter 6. The subsections are action-oriented. <br />
<br />
From [http://www.htap.org/activities/2007_interim_report/TF%20HTAP%202007%20Interim%20Report%20Prospectus%20Oct%202006.pdf Oct 2006 Prospectus]: This chapter (Chapter 6: '''Integration of Observations, Emissions and Modeling'''.) will discuss the need to integrate information from observations, models, and emissions inventories to better understand intercontinental transport. A draft [[HTAP Report Chap. 6 - Jan 07 Outline]] is also on this wiki.<br />
<br />
====HTAP Data System Architecture====<br />
The proposed HTAP data system will integrate the emerging observational and modeling resources by not a centrally planned and powerful, dynamic technologies and through a collaborative federation philosophy. The key roles of the federated data system are to (1) facilitate accsess to the distributed data; (2) ensure the seamless flow of data through interoperable interfaces; (3) provide a set of basic tools for data exploration and processing. <br />
<br />
The Service Oriented Architecture (SOA) allows connecting the web service components (e.g. services for data access, transformation, fusion, rendering, etc.) using work flow software.<br />
<br />
==Introduction==<br />
The goal of the HTAP data system is to aid the assembly of an appropriate set of observations and models, and to facilitate the integration and fusion of observations with models. <br />
<br />
Recent developments in air quality monitoring, modeling and information technologies offer outstanding<br />
opportunities to fulfill the information needs for the HTAP integration effort. The data from surface-based air pollution monitoring networks now routinely provide spatio-temporal and chemical patterns of ozone and PM. Satellite<br />
sensors with global coverage and kilometer-scale spatial resolution now provide real-time snapshots which depict the pattern of industrial haze, smoke and dust in stunning detail. The ‘terabytes’ of data from these surface and remote sensors can now be stored, processed and delivered in near-real time. The instantaneous ‘horizontal’ diffusion of information via the Internet now permits, in principle, the delivery of the right information to the right people at the right place and time. Standardized computer-computer communication languages and Service-Oriented Architectures (SOA) now facilitate the flexible access, quality assurance and processing of raw data into high-grade ‘actionable’ knowledge suitable for HTAP policy decissions. Last but not least, the World Wide Web has opened the way to generous sharing of data and tools leading to collaborative analysis and virtual workgroups. Nevertheless, air quality data analyses and data model integration face significant hurdles. The “data deluge” problem is especially acute for satellite observations and model outputs. <br />
<br />
<br />
<br />
==== Data Selection Criteria====<br />
Monitoring data for atmospheric constituents are now available from a variety of sources, not all of which are suitable for the integrated HTAP data set. Given the limited scope and resources of HTAP, it will be necessary to select a subset of the available observational data that would be most appropriate for the preparation of the 2009 assessment. A set of criteria for the data selection is given below.<br />
* The suitability criteria may include the measured parameters, their spatial extent and coverage density, as well as the time range and sampling frequency with major emphasis on data quality (defined as???).<br />
* The compilation of the global integrated data sets should initially focus on PM and ozone, including the gaseous precursors.<br />
* Subsequent data collaction should also include observations on atmospheric mercury and persistent organic pollutants (POPS). <br />
* In order to cover hemispheric transport of air pollutants the data set system should accept and utilize data from outside the geographical scope of EMEP.<br />
* The data gathering should begin the compilation using the of existing databases in Europe (RETRO, TRADEOFF, QUANTIFY, CRATE, AEROCOM) and virtual federated systems in the US (DataFed, Giovanni and others). <br />
* TF data integration should also contribute to and benefit from other ongoing data integration efforts to integrate the global data resources, most notably, with ACCENT project in Europe, and similar efforts in the US.<br />
* Special emphasis should be placed on the collection of suitable vertical profiles from aircraft measurements as well as from surface and space-borne lidars.<br />
<br />
The evaluation of suitabile observational data sets for model validation and fusion will require close interaction between the modeling and observational communities. An wiki workspace for open collaborative evaluation of different datasets is desirebale.<br />
<br />
===Homogenize the data into an interoperable dataset===<br />
The primary goal of the data system is to allow the integration of observational emission and model data. The model evaluation requires that both the observations and if possible the emissions are fixed. For this reason it is desirable to prepare an integrated observational database to which the various model implementations can be compared to. On one hand, the integrated data set should be as inclusive as possible. On the other hand, such a goal needs to be tempered by many limitations that preclude a broad, inclusive data integration<br />
<br />
The proposed HTAP data system will be a hybrid combination of both distributed and fixed components as illustrated schematically in Figure ??. Both the data providers as well as the HTAP analysts-users will be distributed. However, they will be connected through an integrated HTAP database which should be a stable, virtually fixed database. The section below describes the main component of this distributed data system. <br />
<br />
The HTAP integrated data system has to perform two major functions. The primary function is to facilitate the creation of an integrated data set suitable for model evaluation and pollutant characterization. The multiple steps that are required for this functionality are shown on the left. The sequence of operations can be viewed as a value chain that transform the raw input data into a highly organized integrated data set. Since these value adding steps have to be performed for each candidate data set the operations can be performed in sequence following the rules of data flow programming. For true quality assurance and for data homogenization the data flow channels for individual data sets need to be evaluated in the context of other data. <br />
<br />
The operations that prepare the integrated data set can be viewed as services that operate on the data stream using the principles of modern Service Oriented Architecure (SOA). Each of the services with its clearly defined functionality and firmly specified service interface. In principle, the standards-based interface allows a connection of service chains using formal work flow software. The main benefit of service oriented architecture is that allows the building of agile application programs that can respond to changes in the data input conditions as well as the output requirements. The service oriented software architecture is an implementation of the System of Systems approach. Furthermore, since each of the services can be operated by autonomous providers and the "system" that implements the service is behind the service interfece. In other words, following the SOA approach, not only the data providers are distributed but also the processing services as well. This flexible approach to distributed computing allows the distribution of labor (chain of services) in many different configurations. For instance, wrapping the existing data with standards based interfaces for data access can be performed for an entire cluster of data. This is the approach taken in the federated data system DataFed. Given a standard interface to a variety of data sets, the Quality Assurance service can be performed by another srvice provider that is seamlesly connected to the data access mediator. Similarly, the service that prepares a data set for data integration can be provided by another service in the data flow network. The flexibility offered through the chaining and orchestration of distributed, loosely coupled web services is the architectural support for the building of agile data systems.<br />
<br />
===Interoperability Standards and Data Wrapping===<br />
Interoperable data access will be accomplished through the use of a suite of international standards. The naming of individual chemical parameters will follow the CF convention used by the Climate and Forecast (CF) communities. The existing names for atmospheric chemicals in the CF convention were inadequate to accomodate all the parameters used in the HTAP modeling. in order to remedy this shortcomming the list of standard names was extended by the HTAP community under leadership of C. Textor. She also became a memeber of the CF convention board that is the custodian of the standard names. The standard names for HTAP models were developed using a collaborative wiki workspace. <br />
<br />
The data transfer from providers to users also benefit from standard data formats. For modeling data the use of netCDF-CF as a standard format is recomended. The use of a standard physical data format and the CF naming conventions allows, in principle, the seamless connection between data provider and consumer services. It should be noted, however, that at this time the CF naming convention has only been developed for the model parameters and not for the various observational parameters. Also, the nedCDF CF is primarily used as a model data exchange format, while the transfer of surface monitoring data and satellite data is less standardized.<br />
<br />
The third aspect of data interoperability is a standard query language through which user services request specific data from the provider services. It is proposed that for the HTAP data information system adapts the Web Coverage Service (WCS) as the standard data query language. The WCS data access protocol is defined by the international Open Geospatial Consortium (OGC), which is also the key organization responsible for interoperability standards in GEOSS. Since the WCS protocol was originally developed for the GIS community, it was necessary to adapt it to the needs of "fluid Earth sciences". Members of the HTAP group have been actively participating in the development and testing of the WCS interoperability standards. <br />
<br />
Standards-based data access can be accomplished by ‘wrapping’ the heterogeneous data into standardized web services, callable through well-defined Internet protocols. The result of this ‘wrapping’ process is an array of homogeneous, virtual datasets that can be accessed through a standard querry language and the returned data are packaged in standard format, directly usable by the consuming services.<br />
<br />
===Quality Assurance===<br />
This is just a place holder. Here we need to explain the quality assurance steps that are needed to prepare the HTAP Integrated Dataset.<br />
<br />
===Homogenization and Integration===<br />
The HTAP Integrated Dataset (HID) will be used to compare models to observations. It will be created from the proper combination of a variety of surface, upper air and satellite observations. However, before the inclusion into HID, each dataset will need to be scrutinezed to make it suitable for model comparison. The scrutiny may include filtering, aggregation and possibly fusion operations.<br />
<br />
A good example is the 400-station AIRNOW network reporting hourly ozone and PM2.5 concentrations over the US. The AIRNOW network includes both urban sites that are strongly influenced by local sources. Urban stations need to be removed from HID dataset.<br />
<br />
Developing the specific criteria and procedures for the HTAP integrated dataset will require the attention of a HTAP subgroup.<br />
<br />
==Integrated Data Set==<br />
<br />
The providers of observational, emission, and modeling data are distributed. The individual data sets will be funeled into the integrated HTAP data set. This data set transfer will be accomplished using standard data transfer protocols. Transfering the data from the provider to the user will include two additional observations for each data set, quality assurance and semantic transformation.<br />
<br />
On the left the inputs from observations, emissions, and model data<br />
<br />
<br />
<br />
===Identify providers and datasets relevant to HTAP===<br />
===Integrate multi-sensory data into a coherent data systems===<br />
<br />
[[Image:HTAP_IntDataSys.png|600px|left]] In order to accomplish the objectives set for the HTAP TF it is evident that it will require the System of Systems approach as envisioned by GEO. Both the modeling of the chemical constituents as well as the Earth observations used to document the hemispheric transport are being performed by individual autonomous projects and programs (Systems). A key function of the Task Force is to assess the contributions of the individual systems and then integrate those into a coherent assessment. Much of this interim report is in fact such an assessment of the current knowledge based on previous or past models and observations.<br />
<br />
In the next, upcoming phase of HTAP is to actually integrate the existing modeling and observing systems such that their combination and fusion can yield a deeper and more robust understanding of HTAP. since this next phase will involve integration of multiple relevant data sets, comparison and evaluation of multiple models, as well as the reconciliation of the observations and models to create the new and deeper understanding . In order to execute this extensive integration, it will be necessary to conduct this integration using a generally accepted framework as well as accepted scientific methods. It is believed that the GEO architecture and the System of Systems is an attractive approach to HTAP integration.<br />
<br />
The HTAP program can also be considered an early demonstration of the GEO concepts through its end to end approach. Both the atmospheric modeling, as well as the observations currently exist through the operation of existing modeling and observation systems. The Task Force has agreed organizing and evaluating those models, assembling and integrating the observational data sets and then reconciling the models with the observations will be the focus of the Phase II effort. The Task Force will also prepare and deliver a report to LRTP to aid its deliberations and its decision making processes. This sequence of activities constitutes an end to end approach that turns observations and models into actionable knowledge for societal decision making. One could say that this is an octagonal approach to more deliberate step by step development of GEOSS where in Phase I interoperability, in Phase II ???<br />
<br />
Provider nodes <br />
* Federated Data System DataFed<br />
* NASA Data System Giovanni<br />
* Emission Data System NEISGEI<br />
* Juelich Data System<br />
* AeorCom<br />
* EMEP<br />
<br />
<br />
Data Networks. Connected hubs; Show Core HTAP network<br />
<br />
[http://datafedwiki.wustl.edu/index.php/2007-03-26:_Datasets_for_HTAP_through_DataFed See 20 selected datasets] in the federated data systems, DataFed<br />
<br />
===Perform QA at all steps===<br />
<br />
==Reconciliation and Integration of Observations, Emissions and Models==<br />
===Compare observations to models and emissions===<br />
===Characterize pollutant pattern and SSR=== <br />
===Develop real-time data assimilation===<br />
===Perform reanalysis with assimilated data=== <br />
<br />
==Organizational Aspects==<br />
===HTAP Relationship to GEO and GEOSS===<br />
There is an outstanding opportunity to develop a mutually beneficial and supportive relationship between the activities of the HTAP Task Force and that of GEO. The Group of Earth Observations with its broad range of national and organizational members has developed a general architectural framework for turning Earth observations into societal benefits. The three main components of this architecture are models, and observations, which then feed into decision support systems used in a variety of societal decision making. This general framework is well suited as an architectural guide to the HTAP program. However, this framework lacks specific guidelines and implementation details that are needed for practical Earth observing and decision support systems. <br />
<br />
[[Image:400px HTAP GEOSS Arch.png|400px|left]] The HTAP program provides an opportunity to extend the GEO framework with more detailed features that arise in specific HTAP application context. In case of HTAP, the major activities are shown in the architectural diagram of Figure ?? The HTAP modeling is conducted through global scale chemical transport models that simulate or forecast the atmospheric chemical composition as a consequence of natural and human induced emissions. The observations arise from satellite, ground-based and airborne observations of chemical constituents and their hemispheric transport. The decision support system consists of the scientists as members of the HTAP task force, and the task force co-chairs as the intermediaries between LRTP and the TF. (Terry, Andre this description of the HTAP DSS needs your help). <br />
<br />
Representing the Decision Support System (DSS) for HTAP is an important aspect because it can guide the design and and implementation of the information system for the support of the DSS processes. A higher resolution DSS design document is also beneficial as a communications medium for the interacting system of systems. A domain specific DSS may also allow the generalization and applicability to the design of DSS structures for similar application domains. <br />
<br />
The implementation of the GEO framework utilizes the concept of Global Observing System of Systems (GEOSS). Traditionally, Earth observations were performed by well defined systems such as specific satellites and monitoring networks which were designed and operated using systems engineering principles. However, GEO recognized that the understanding of Earth system requires the utilization and integration of the individual, autonomous systems. The key difference between the Systems and System of System approaches are highlighted in Table 1.<br />
<br />
===HTAP Relationship to IGACO=== <br />
[[Image:IGACO Framework.png|300px|left]] The HTAP program can also have a mutually beneficial interaction with the IGACO program. Integrated Global Atmospheric Chemistry Observations (IGACO) Theme is part of the Integrated Global Observing Strategy (IGOS). The IGACO program is formulated by an array of experienced atmospheric scientists. IGACO proposes a specific framework for the system components that processes the observational data and models. It also specifies the higher level information products that can be used for creating social benefit. <br />
<br />
In the IGACO framework, the data products from satellite, surface, and aircraft observations are collected for inclusion into an integrated data archive. A key component in the IGACO data flow is the mandatory quality assurance and QA QC that precedes the archiving. This is particularly important for HTAP where multi-sensory data from many different providers are to be combined into an integrated database. In the IGACO framework, a key output from the data system is a high integrated spatial-temporal data set which can be used in a multitude of applications that produce socal benefit. The goal of creating such an integrated data set is shared by the IGACO and the HTAP programs (?? Len, this section could benefit from your input)<br />
<br />
===HTAP Relationship with Other Programs===<br />
---<br />
---<br />
<br />
<br />
<br />
==Links==<br />
[[IGACO Framework]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=HTAP_Report,_Sub-Chap._6_-_Data/Info_System&diff=7939HTAP Report, Sub-Chap. 6 - Data/Info System2007-04-12T04:53:16Z<p>128.252.167.79: /* Fragments for Chapter 6. Integration of Observations, Emissions and Modeling */</p>
<hr />
<div><noinclude>[[Integrated_Global_Dataset| < Back to Integrated Global Dataset on ESIP wiki]] <br></noinclude> <br />
<br />
This wiki page is inteded to be the collaborative workspace for Task Force members interested in the HTAP Information System development, implementation and use. The contents can be freely edited by anyone who is logged in. Comments and feedback can also be sent by email to rhusar@me.wustl.edu. The content of this page is is on the '''''Information System''''' that supposts Chapter 6. The subsections are action-oriented. <br />
<br />
From [http://www.htap.org/activities/2007_interim_report/TF%20HTAP%202007%20Interim%20Report%20Prospectus%20Oct%202006.pdf Oct 2006 Prospectus]: This chapter (Chapter 6: '''Integration of Observations, Emissions and Modeling'''.) will discuss the need to integrate information from observations, models, and emissions inventories to better understand intercontinental transport. A draft [[HTAP Report Chap. 6 - Jan 07 Outline]] is also on this wiki.<br />
<br />
====HTAP Data System Architecture====<br />
The proposed HTAP data system will integrate the emerging observational and modeling resources by not a centrally planned and powerful, dynamic technologies and through a collaborative federation philosophy. The key roles of the federated data system are to (1) facilitate accsess to the distributed data; (2) ensure the seamless flow of data through interoperable interfaces; (3) provide a set of basic tools for data exploration and processing. <br />
<br />
The Service Oriented Architecture (SOA) allows connecting the web service components (e.g. services for data access, transformation, fusion, rendering, etc.) using work flow software.<br />
<br />
==Integration of Observations and Emissions==<br />
The goal of the HTAP data system is to aid the assembly of an appropriate set of observations and models, and to facilitate the integration and fusion of observations with models. <br />
<br />
Recent developments in air quality monitoring, modeling and information technologies offer outstanding<br />
opportunities to fulfill the information needs for the HTAP integration effort. The data from surface-based air pollution monitoring networks now routinely provide spatio-temporal and chemical patterns of ozone and PM. Satellite<br />
sensors with global coverage and kilometer-scale spatial resolution now provide real-time snapshots which depict the pattern of industrial haze, smoke and dust in stunning detail. The ‘terabytes’ of data from these surface and remote sensors can now be stored, processed and delivered in near-real time. The instantaneous ‘horizontal’ diffusion of information via the Internet now permits, in principle, the delivery of the right information to the right people at the right place and time. Standardized computer-computer communication languages and Service-Oriented Architectures (SOA) now facilitate the flexible access, quality assurance and processing of raw data into high-grade ‘actionable’ knowledge suitable for HTAP policy decissions. Last but not least, the World Wide Web has opened the way to generous sharing of data and tools leading to collaborative analysis and virtual workgroups. Nevertheless, air quality data analyses and data model integration face significant hurdles. The “data deluge” problem is especially acute for satellite observations and model outputs. <br />
<br />
<br />
<br />
==== Data Selection Criteria====<br />
Monitoring data for atmospheric constituents are now available from a variety of sources, not all of which are suitable for the integrated HTAP data set. Given the limited scope and resources of HTAP, it will be necessary to select a subset of the available observational data that would be most appropriate for the preparation of the 2009 assessment. A set of criteria for the data selection is given below.<br />
* The suitability criteria may include the measured parameters, their spatial extent and coverage density, as well as the time range and sampling frequency with major emphasis on data quality (defined as???).<br />
* The compilation of the global integrated data sets should initially focus on PM and ozone, including the gaseous precursors.<br />
* Subsequent data collaction should also include observations on atmospheric mercury and persistent organic pollutants (POPS). <br />
* In order to cover hemispheric transport of air pollutants the data set system should accept and utilize data from outside the geographical scope of EMEP.<br />
* The data gathering should begin the compilation using the of existing databases in Europe (RETRO, TRADEOFF, QUANTIFY, CRATE, AEROCOM) and virtual federated systems in the US (DataFed, Giovanni and others). <br />
* TF data integration should also contribute to and benefit from other ongoing data integration efforts to integrate the global data resources, most notably, with ACCENT project in Europe, and similar efforts in the US.<br />
* Special emphasis should be placed on the collection of suitable vertical profiles from aircraft measurements as well as from surface and space-borne lidars.<br />
<br />
The evaluation of suitabile observational data sets for model validation and fusion will require close interaction between the modeling and observational communities. An wiki workspace for open collaborative evaluation of different datasets is desirebale.<br />
<br />
===Homogenize the data into an interoperable dataset===<br />
The primary goal of the data system is to allow the integration of observational emission and model data. The model evaluation requires that both the observations and if possible the emissions are fixed. For this reason it is desirable to prepare an integrated observational database to which the various model implementations can be compared to. On one hand, the integrated data set should be as inclusive as possible. On the other hand, such a goal needs to be tempered by many limitations that preclude a broad, inclusive data integration<br />
<br />
The proposed HTAP data system will be a hybrid combination of both distributed and fixed components as illustrated schematically in Figure ??. Both the data providers as well as the HTAP analysts-users will be distributed. However, they will be connected through an integrated HTAP database which should be a stable, virtually fixed database. The section below describes the main component of this distributed data system. <br />
<br />
The HTAP integrated data system has to perform two major functions. The primary function is to facilitate the creation of an integrated data set suitable for model evaluation and pollutant characterization. The multiple steps that are required for this functionality are shown on the left. The sequence of operations can be viewed as a value chain that transform the raw input data into a highly organized integrated data set. Since these value adding steps have to be performed for each candidate data set the operations can be performed in sequence following the rules of data flow programming. For true quality assurance and for data homogenization the data flow channels for individual data sets need to be evaluated in the context of other data. <br />
<br />
The operations that prepare the integrated data set can be viewed as services that operate on the data stream using the principles of modern Service Oriented Architecure (SOA). Each of the services with its clearly defined functionality and firmly specified service interface. In principle, the standards-based interface allows a connection of service chains using formal work flow software. The main benefit of service oriented architecture is that allows the building of agile application programs that can respond to changes in the data input conditions as well as the output requirements. The service oriented software architecture is an implementation of the System of Systems approach. Furthermore, since each of the services can be operated by autonomous providers and the "system" that implements the service is behind the service interfece. In other words, following the SOA approach, not only the data providers are distributed but also the processing services as well. This flexible approach to distributed computing allows the distribution of labor (chain of services) in many different configurations. For instance, wrapping the existing data with standards based interfaces for data access can be performed for an entire cluster of data. This is the approach taken in the federated data system DataFed. Given a standard interface to a variety of data sets, the Quality Assurance service can be performed by another srvice provider that is seamlesly connected to the data access mediator. Similarly, the service that prepares a data set for data integration can be provided by another service in the data flow network. The flexibility offered through the chaining and orchestration of distributed, loosely coupled web services is the architectural support for the building of agile data systems.<br />
<br />
===Interoperability Standards and Data Wrapping===<br />
Interoperable data access will be accomplished through the use of a suite of international standards. The naming of individual chemical parameters will follow the CF convention used by the Climate and Forecast (CF) communities. The existing names for atmospheric chemicals in the CF convention were inadequate to accomodate all the parameters used in the HTAP modeling. in order to remedy this shortcomming the list of standard names was extended by the HTAP community under leadership of C. Textor. She also became a memeber of the CF convention board that is the custodian of the standard names. The standard names for HTAP models were developed using a collaborative wiki workspace. <br />
<br />
The data transfer from providers to users also benefit from standard data formats. For modeling data the use of netCDF-CF as a standard format is recomended. The use of a standard physical data format and the CF naming conventions allows, in principle, the seamless connection between data provider and consumer services. It should be noted, however, that at this time the CF naming convention has only been developed for the model parameters and not for the various observational parameters. Also, the nedCDF CF is primarily used as a model data exchange format, while the transfer of surface monitoring data and satellite data is less standardized.<br />
<br />
The third aspect of data interoperability is a standard query language through which user services request specific data from the provider services. It is proposed that for the HTAP data information system adapts the Web Coverage Service (WCS) as the standard data query language. The WCS data access protocol is defined by the international Open Geospatial Consortium (OGC), which is also the key organization responsible for interoperability standards in GEOSS. Since the WCS protocol was originally developed for the GIS community, it was necessary to adapt it to the needs of "fluid Earth sciences". Members of the HTAP group have been actively participating in the development and testing of the WCS interoperability standards. <br />
<br />
Standards-based data access can be accomplished by ‘wrapping’ the heterogeneous data into standardized web services, callable through well-defined Internet protocols. The result of this ‘wrapping’ process is an array of homogeneous, virtual datasets that can be accessed through a standard querry language and the returned data are packaged in standard format, directly usable by the consuming services.<br />
<br />
===Quality Assurance===<br />
This is just a place holder. Here we need to explain the quality assurance steps that are needed to prepare the HTAP Integrated Dataset.<br />
<br />
===Homogenization and Integration===<br />
The HTAP Integrated Dataset (HID) will be used to compare models to observations. It will be created from the proper combination of a variety of surface, upper air and satellite observations. However, before the inclusion into HID, each dataset will need to be scrutinezed to make it suitable for model comparison. The scrutiny may include filtering, aggregation and possibly fusion operations.<br />
<br />
A good example is the 400-station AIRNOW network reporting hourly ozone and PM2.5 concentrations over the US. The AIRNOW network includes both urban sites that are strongly influenced by local sources. Urban stations need to be removed from HID dataset.<br />
<br />
Developing the specific criteria and procedures for the HTAP integrated dataset will require the attention of a HTAP subgroup.<br />
<br />
==Integrated Data Set==<br />
<br />
The providers of observational, emission, and modeling data are distributed. The individual data sets will be funeled into the integrated HTAP data set. This data set transfer will be accomplished using standard data transfer protocols. Transfering the data from the provider to the user will include two additional observations for each data set, quality assurance and semantic transformation.<br />
<br />
On the left the inputs from observations, emissions, and model data<br />
<br />
<br />
<br />
===Identify providers and datasets relevant to HTAP===<br />
===Integrate multi-sensory data into a coherent data systems===<br />
<br />
[[Image:HTAP_IntDataSys.png|600px|left]] In order to accomplish the objectives set for the HTAP TF it is evident that it will require the System of Systems approach as envisioned by GEO. Both the modeling of the chemical constituents as well as the Earth observations used to document the hemispheric transport are being performed by individual autonomous projects and programs (Systems). A key function of the Task Force is to assess the contributions of the individual systems and then integrate those into a coherent assessment. Much of this interim report is in fact such an assessment of the current knowledge based on previous or past models and observations.<br />
<br />
In the next, upcoming phase of HTAP is to actually integrate the existing modeling and observing systems such that their combination and fusion can yield a deeper and more robust understanding of HTAP. since this next phase will involve integration of multiple relevant data sets, comparison and evaluation of multiple models, as well as the reconciliation of the observations and models to create the new and deeper understanding . In order to execute this extensive integration, it will be necessary to conduct this integration using a generally accepted framework as well as accepted scientific methods. It is believed that the GEO architecture and the System of Systems is an attractive approach to HTAP integration.<br />
<br />
The HTAP program can also be considered an early demonstration of the GEO concepts through its end to end approach. Both the atmospheric modeling, as well as the observations currently exist through the operation of existing modeling and observation systems. The Task Force has agreed organizing and evaluating those models, assembling and integrating the observational data sets and then reconciling the models with the observations will be the focus of the Phase II effort. The Task Force will also prepare and deliver a report to LRTP to aid its deliberations and its decision making processes. This sequence of activities constitutes an end to end approach that turns observations and models into actionable knowledge for societal decision making. One could say that this is an octagonal approach to more deliberate step by step development of GEOSS where in Phase I interoperability, in Phase II ???<br />
<br />
Provider nodes <br />
* Federated Data System DataFed<br />
* NASA Data System Giovanni<br />
* Emission Data System NEISGEI<br />
* Juelich Data System<br />
* AeorCom<br />
* EMEP<br />
<br />
<br />
Data Networks. Connected hubs; Show Core HTAP network<br />
<br />
[http://datafedwiki.wustl.edu/index.php/2007-03-26:_Datasets_for_HTAP_through_DataFed See 20 selected datasets] in the federated data systems, DataFed<br />
<br />
===Perform QA at all steps===<br />
<br />
==Reconciliation and Integration of Observations, Emissions and Models==<br />
===Compare observations to models and emissions===<br />
===Characterize pollutant pattern and SSR=== <br />
===Develop real-time data assimilation===<br />
===Perform reanalysis with assimilated data=== <br />
<br />
==Organizational Aspects==<br />
===HTAP Relationship to GEO and GEOSS===<br />
There is an outstanding opportunity to develop a mutually beneficial and supportive relationship between the activities of the HTAP Task Force and that of GEO. The Group of Earth Observations with its broad range of national and organizational members has developed a general architectural framework for turning Earth observations into societal benefits. The three main components of this architecture are models, and observations, which then feed into decision support systems used in a variety of societal decision making. This general framework is well suited as an architectural guide to the HTAP program. However, this framework lacks specific guidelines and implementation details that are needed for practical Earth observing and decision support systems. <br />
<br />
[[Image:400px HTAP GEOSS Arch.png|400px|left]] The HTAP program provides an opportunity to extend the GEO framework with more detailed features that arise in specific HTAP application context. In case of HTAP, the major activities are shown in the architectural diagram of Figure ?? The HTAP modeling is conducted through global scale chemical transport models that simulate or forecast the atmospheric chemical composition as a consequence of natural and human induced emissions. The observations arise from satellite, ground-based and airborne observations of chemical constituents and their hemispheric transport. The decision support system consists of the scientists as members of the HTAP task force, and the task force co-chairs as the intermediaries between LRTP and the TF. (Terry, Andre this description of the HTAP DSS needs your help). <br />
<br />
Representing the Decision Support System (DSS) for HTAP is an important aspect because it can guide the design and and implementation of the information system for the support of the DSS processes. A higher resolution DSS design document is also beneficial as a communications medium for the interacting system of systems. A domain specific DSS may also allow the generalization and applicability to the design of DSS structures for similar application domains. <br />
<br />
The implementation of the GEO framework utilizes the concept of Global Observing System of Systems (GEOSS). Traditionally, Earth observations were performed by well defined systems such as specific satellites and monitoring networks which were designed and operated using systems engineering principles. However, GEO recognized that the understanding of Earth system requires the utilization and integration of the individual, autonomous systems. The key difference between the Systems and System of System approaches are highlighted in Table 1.<br />
<br />
===HTAP Relationship to IGACO=== <br />
[[Image:IGACO Framework.png|300px|left]] The HTAP program can also have a mutually beneficial interaction with the IGACO program. Integrated Global Atmospheric Chemistry Observations (IGACO) Theme is part of the Integrated Global Observing Strategy (IGOS). The IGACO program is formulated by an array of experienced atmospheric scientists. IGACO proposes a specific framework for the system components that processes the observational data and models. It also specifies the higher level information products that can be used for creating social benefit. <br />
<br />
In the IGACO framework, the data products from satellite, surface, and aircraft observations are collected for inclusion into an integrated data archive. A key component in the IGACO data flow is the mandatory quality assurance and QA QC that precedes the archiving. This is particularly important for HTAP where multi-sensory data from many different providers are to be combined into an integrated database. In the IGACO framework, a key output from the data system is a high integrated spatial-temporal data set which can be used in a multitude of applications that produce socal benefit. The goal of creating such an integrated data set is shared by the IGACO and the HTAP programs (?? Len, this section could benefit from your input)<br />
<br />
===HTAP Relationship with Other Programs===<br />
---<br />
---<br />
<br />
<br />
<br />
==Links==<br />
[[IGACO Framework]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=CF_Standard_Names_-_Discussed_Atmospheric_Chemistry_and_Aerosol_Terms&diff=7090CF Standard Names - Discussed Atmospheric Chemistry and Aerosol Terms2007-01-08T21:08:20Z<p>128.252.167.79: </p>
<hr />
<div>{{CF-links}}<br />
<br />
It is the philosophy of CF only define those standard_names for which a demand has been expressed. The proposed standard_names on this page are needed for a model intercomparison of three CTMs within the Global and regional Earth-system (Atmosphere) Monitoring using Satellite and in-situ data [http://www.ecmwf.int/research/EU_projects/GEMS/index.jsp GEMS] project within the theme Global Reactive Gases (GRG). In addition, most of these names probably will be needed within the Task Force on Hemispheric Transport of Air Pollution [http://www.htap.org/ TF HTAP].<br />
<br />
The proposed standard_names listed below are based on the ideas provided at [[CF Standard Names - Future Atmospheric Chemistry and Aerosol Terms|Construction of Atmospheric Chemistry and Aerosol Terms and Future Standard_Names]]. They are constructed from name components and species mentioned in tables 1 and 2 of this page.<br />
<br />
----<br />
<br />
Comments to [[User:ChristianeTextor|ChristianeTextor]] or directly on our [http://wiki.esipfed.org/index.php/Talk:CF_Standard_Names_-_Proposed_Atmospheric_Chemistry_and_Aerosol_Terms discussion page] <br />
are highly welcome!<br />
<br />
You can directly modify the tables here, all versions are stored under 'history'. However, in order to keep track of the changes, please send an email to [[User:ChristianeTextor|ChristianeTextor]].<br />
<br />
=Gas phase species: concentrations and columns=<br />
{|{{prettytable}}<br />
|- style="font-weight:bold"<br />
| Height="12,75" | CF Standard_name<br />
| Explanation<br />
| Canonical unit<br />
<br />
|- style="background-color:#FF99CC"<br />
|style="font-weight:bold" Height="12,75" | gas phase species: concentrations and columns<br />
| <br />
| <br />
<br />
|- <br />
| Height="12,75" | mole_fraction_of_ozone_in_air<br />
| volume mixing ratio of ozone, O3<br />
| 1=mole mole-1<br />
<br />
|- <br />
| Height="12,75" | mole_fraction_of_hydrogen_peroxide_in_air<br />
| volume mixing ratio of hydrogen peroxide, H2O2<br />
| 1=mole mole-1<br />
<br />
|- <br />
| Height="25,5" | mole_fraction_of_hydroxyl_radical_in_air<br />
| volume mixing ratio of the hydroxy radical, hydroxyl, OH<br />
| 1=mole mole-1<br />
<br />
|- <br />
| Height="25,5" | mole_fraction_of_hydroperoxy_radical_in_air<br />
| volume mixing ratio of the hydroperoxy radical, HO2<br />
| 1=mole mole-1<br />
<br />
|- <br />
| Height="12,75" | mole_fraction_of_nitrogen_monoxide_in_air<br />
| volume mixing ratio of nitrogen monoxide, NO<br />
| 1=mole mole-1<br />
<br />
|- <br />
| Height="12,75" | mole_fraction_of_nitrogen_dioxide_in_air<br />
| volume mixing ratio of nitrogen dioxide, NO2<br />
| 1=mole mole-1<br />
<br />
|- <br />
| Height="12,75" | atmosphere_mole_fraction_of_all_nitrogen_oxides <br />
| volume mixing ratio of all simulated nitrogen oxides, NOy is the sum of all simulated oxidized nitrogen species, out of NO, NO2, HNO3, HNO4, NO3aerosol, NO3(radical), N2O5, PAN, other organic nitrates (N2O5 is only counted once!)<br />
| 1=mole mole-1<br />
<br />
|- <br />
| Height="12,75" | mole_fraction_of_nitric_acid_in_air<br />
| volume mixing ratio of nitric acid, HNO3<br />
| 1=mole mole-1<br />
<br />
|- <br />
| Height="12,75" | mole_fraction_of_ammonia_in_air<br />
| volume mixing ratio of ammonia, NH3<br />
| 1=mole mole-1<br />
<br />
|- <br />
| Height="12,75" | mole_fraction_of_ammonium_in_air<br />
| volume mixing ratio of ammonium, NH4 <br />
| 1=mole mole-1<br />
<br />
|- <br />
| Height="12,75" | mole_fraction_of_methane_in_air<br />
| volume mixing ratio of methane, CH4<br />
| 1=mole mole-1<br />
<br />
|- <br />
| Height="25,5" | mole_fraction_of_peroxyacetyl_nitrate_in_air<br />
| volume mixing ratio of peroxyacetyl nitrate, PAN, CH3COO2NO2<br />
| 1=mole mole-1<br />
<br />
|- <br />
| Height="12,75" | mole_fraction_of_carbon_monoxide_in_air<br />
| volume mixing ratio of carbon monoxide, CO<br />
| 1=mole mole-1<br />
<br />
|- <br />
| Height="12,75" | mole_fraction_of_formaldehyde_in_air<br />
| volume mixing ratio of formaldehyde, CH2O<br />
| 1=mole mole-1<br />
<br />
|- <br />
| Height="12,75" | mole_fraction_of_sulfur_dioxide_in_air<br />
| volume mixing ratio of sulfur dioxide, SO2<br />
| 1=mole mole-1<br />
<br />
|- <br />
| Height="12,75" | mole_fraction_of_radon_in_air<br />
| volume mixing ratio of radon, Rn<br />
| 1=mole mole-1<br />
<br />
|- <br />
| Height="12,75" | mole_fraction_of_lead_in_air<br />
| volume mixing ratio of lead, Pb<br />
| 1=mole mole-1<br />
<br />
|- <br />
| Height="12,75" | mole_fraction_of_mercury(0)_in_air<br />
| volume mixing ratio of Hg <br />
| 1=mole mole-1<br />
<br />
|- <br />
| Height="12,75" | mole_fraction_of_mercury(II)_in_air<br />
| volume mixing ratio of Hg(2+)<br />
| 1=mole mole-1<br />
<br />
|- <br />
| Height="12,75" | <br />
| <br />
| <br />
<br />
|- <br />
| Height="12,75" | mass_fraction_of_ozone_in_air<br />
| mass fraction of ozone, O3<br />
| 1=kg kg-1<br />
<br />
|- <br />
| Height="12,75" | mass_fraction_of_hydrogen_peroxide_in_air<br />
| mass fraction of hydrogen peroxide, H2O2<br />
| 1=kg kg-1<br />
<br />
|- <br />
| Height="25,5" | mass_fraction_of_hydroxyl_radical_in_air<br />
| mass fraction of the hydroxy radical, hydroxyl, OH<br />
| 1=kg kg-1<br />
<br />
|- <br />
| Height="25,5" | mass_fraction_of_hydroperoxy_radical_in_air<br />
| mass fraction ratio of the hydroperoxy radical, HO2<br />
| 1=kg kg-1<br />
<br />
|- <br />
| Height="12,75" | mass_fraction_of_nitrogen_monoxide_in_air<br />
| mass fraction of nitrogen monoxide, NO<br />
| 1=kg kg-1<br />
<br />
|- <br />
| Height="12,75" | mass_fraction_of_nitrogen_dioxide_in_air<br />
| mass fraction of nitrogen dioxide, NO2<br />
| 1=kg kg-1<br />
<br />
|- <br />
| Height="12,75" | mass_fraction_of_all_nitrogen_oxides_in_air<br />
| mass fraction of total_nitrogen_oxides NOy, includes all nitrogen oxides included in the model, to be specified in by the modeller in the long_name attribute if possible ???<br />
| 1=kg kg-1<br />
<br />
|- <br />
| Height="12,75" | mass_fraction_of_nitric_acid_in_air<br />
| mass fraction of nitric acid, HNO3<br />
| 1=kg kg-1<br />
<br />
|- <br />
| Height="12,75" | mass_fraction_of_ammonia_in_air<br />
| mass fraction of ammonia, NH3<br />
| 1=kg kg-1<br />
<br />
|- <br />
| Height="12,75" | mass_fraction_of_ammonium_in_air<br />
| mass fraction of ammonium, NH4 <br />
| 1=kg kg-1<br />
<br />
|- <br />
| Height="12,75" | mass_fraction_of_methane_in_air<br />
| mass fraction of methane, CH4<br />
| 1=kg kg-1<br />
<br />
|- <br />
| Height="25,5" | mass_fraction_of_peroxyacetyl_nitrate_in_air<br />
| mass fraction of peroxyacetyl nitrate, PAN, CH3COO2NO2<br />
| 1=kg kg-1<br />
<br />
|- <br />
| Height="12,75" | mass_fraction_of_carbon_monoxide_in_air<br />
| mass fraction of carbon monoxide, CO<br />
| 1=kg kg-1<br />
<br />
|- <br />
| Height="12,75" | mass_fraction_of_formaldehyde_in_air<br />
| mass fraction of formaldehyde, CH2O<br />
| 1=kg kg-1<br />
<br />
|- <br />
| Height="12,75" | mass_fraction_of_sulfur_dioxide_in_air<br />
| mass fraction of sulfur dioxide, SO2<br />
| 1=kg kg-1<br />
<br />
|- <br />
| Height="12,75" | mass_fraction_of_radon_in_air<br />
| mass fraction of radon, Rn<br />
| 1=kg kg-1<br />
<br />
|- <br />
| Height="12,75" | mass_fraction_of_lead_in_air<br />
| mass fraction of lead, Pb<br />
| 1=kg kg-1<br />
<br />
|- <br />
| Height="12,75" | <br />
| <br />
| <br />
<br />
|- <br />
| Height="25,5" | mole_fraction_of_ozone_from_stratosphere_in troposphere<br />
| volume mixing ratio of ozone from the stratosphere<br />
| 1=mole mole-1 <br />
<br />
|- <br />
| Height="12,75" | <br />
| <br />
| <br />
<br />
|- <br />
| Height="12,75" | atmosphere_mole_content_of_ozone_in_air <br />
| vertically integrated moles of O3 in the gas phase<br />
| mole m-2.<br />
<br />
<br />
|- <br />
| Height="12,75" | <br />
| <br />
| <br />
<br />
|- <br />
| Height="12,75" | mole_content_of_ozone_in_air_below_tropopause_defined_by_ozone_mole_fraction <br />
| vertically integrated moles of O3 in the gas phase in the troposphere<br />
| mole m-2<br />
<br />
|- <br />
| Height="12,75" | mole_content_of_nitrogen_monooxide_in_air_below_tropopause_defined_by_ozone_mole_fraction<br />
| vertically integrated moles of NO in the gas phase in the troposphere<br />
| mole m-2<br />
<br />
|- <br />
| Height="12,75" | mole_content_of_nitrogen_dioxide_in_air_below_tropopause_defined_by_ozone_mole_fraction<br />
| vertically integrated moles of NO2in the gas phase in the troposphere<br />
| mole m-2<br />
<br />
|- <br />
| Height="12,75" | mole_content_of_carbon_monoxide_in_air_below_tropopause_defined_by_ozone_mole_fraction<br />
| vertically integrated moles of CO in the gas phase in the troposphere<br />
| mole m-2<br />
<br />
|- <br />
| Height="12,75" | mole_content_of_formaldehyde_in_air_below_tropopause_defined_by_ozone_mole_fraction<br />
| vertically integrated moles of H2CO in the gas phase in the troposphere<br />
| mole m-2<br />
<br />
|- <br />
| Height="12,75" | mole_content_of_sulfur_dioxide_in_air_below_tropopause_defined_by_ozone_mole_fraction<br />
| vertically integrated moles of SO2 in the gas phase in the troposphere<br />
| mole m-2<br />
<br />
|- <br />
| Height="12,75" | <br />
| <br />
| <br />
|}<br />
<br />
=Gas phase species: emissions=<br />
{|{{prettytable}}<br />
|- style="font-weight:bold"<br />
| Height="12,75" | CF Standard_name<br />
| Explanation<br />
| Canonical unit<br />
<br />
|- style="background-color:#FF99CC;font-weight:bold"<br />
| Height="12,75" | gases phase species: emissions<br />
| <br />
| <br />
<br />
|- <br />
| Height="25,5" | surface_emission_mole_flux_of_ozone<br />
| surface emission of O3<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_emission_mole_flux_of_nitrogen_monoxide<br />
| surface emission of NO<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_emission_mole_flux_of_nitrogen_dioxide<br />
| surface emission of NO2<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_emission_mole_flux_of_nox<br />
| surface emission of NOx=NO+NO2<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_emission_mole_flux_of_carbon_monoxide<br />
| surface emission of CO<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_emission_mole_flux_of_sulfur_dioxide<br />
| surface emission of SO2<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_emission_mole_flux_of_formaldehyde<br />
| surface emission of CHCO<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_emission_mole_flux_of_methane<br />
| surface emission of CH4<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_emission_mole_flux_of_ammonia<br />
| surface emission of NH3<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_emission_mole_flux_of_dinitrogen_oxide<br />
| surface emission of N2O<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_emission_mole_flux_of_ethane <br />
| surface emission of C2H6<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_emission_mole_flux_of_radon<br />
| surface emission of radon<br />
| mole m-2 s-1<br />
<br />
<br />
<br />
|- <br />
| Height="12,75" | <br />
| <br />
|<br />
<br />
|- <br />
| Height="25,5" | surface_emission_mass_flux_of_ozone<br />
| surface emission of O3<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_emission_mass_flux_of_nitrogen_monoxide<br />
| surface emission of NO<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_emission_mass_flux_of_nitrogen_dioxide<br />
| surface emission of NO2<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_emission_mass_flux_of_nox<br />
| surface emission of NOx=NO+NO2<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_emission_mass_flux_of_carbon_monoxide<br />
| surface emission of CO<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_emission_mass_flux_of_sulfur_dioxide<br />
| surface emission of SO2<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_emission_mass_flux_of_formaldehyde<br />
| surface emission of CHCO<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_emission_mass_flux_of_methane<br />
| surface emission of CH4<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_emission_mass_flux_of_ammonia<br />
| surface emission of NH3<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_emission_mass_flux_of_dinitrogen_oxide<br />
| surface emission of N2O<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_emission_mass_flux_of_ethane <br />
| surface emission of C2H6<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="12,75" | <br />
| <br />
|<br />
<br />
|- <br />
| Height="25,5" | atmosphere_emission_mole_flux_of_ozone<br />
| athmosphere emission of O3<br />
| mole m-3 s-1<br />
<br />
|- <br />
| Height="25,5" | atmosphere_emission_mole_flux_of_nitrogen_monoxide<br />
| atmosphere emission of NO<br />
| mole m-3 s-1<br />
<br />
|- <br />
| Height="25,5" | atmosphere_emission_mole_flux_of_nitrogen_dioxide<br />
| atmosphere emission of NO2<br />
| mole m-3 s-1<br />
<br />
|- <br />
| Height="25,5" | atmosphere_emission_mole_flux_of_nox<br />
| atmosphere emission of NOx=NO+NO2<br />
| mole m-3 s-1<br />
<br />
|- <br />
| Height="25,5" | atmosphere_emission_mole_flux_of_carbon_monoxide<br />
| atmosphere emission of CO<br />
| mole m-3 s-1<br />
<br />
|- <br />
| Height="25,5" | atmosphere_emission_mole_flux_of_sulfur_dioxide<br />
| atmosphere emission of SO2<br />
| mole m-3 s-1<br />
<br />
|- <br />
| Height="25,5" | atmosphere_emission_mole_flux_of_formaldehyde<br />
| atmosphere emission of CHCO<br />
| mole m-3 s-1<br />
<br />
|- <br />
| Height="25,5" | atmosphere_emission_mole_flux_of_methane<br />
| atmosphere emission of CH4<br />
| mole m-3 s-1<br />
<br />
|- <br />
| Height="25,5" | atmosphere_emission_mole_flux_of_ammonia<br />
| atmosphere emission of NH3<br />
| mole m-3 s-1<br />
<br />
|- <br />
| Height="25,5" | atmosphere_emission_mole_flux_of_dinitrogen_oxide<br />
| atmosphere emission of N2O<br />
| mole m-3 s-1<br />
<br />
|- <br />
| Height="25,5" | atmosphere_emission_mole_flux_of_ethane <br />
| atmosphere emission of C2H6<br />
| mole m-3 s-1<br />
<br />
|- <br />
| Height="12,75" | <br />
| <br />
|<br />
<br />
|- <br />
| Height="25,5" | atmosphere_emission_mass_flux_of_ozone<br />
| athmosphere emission of O3<br />
| kg m-3 s-1<br />
<br />
|- <br />
| Height="25,5" | atmosphere_emission_mass_flux_of_nitrogen_monoxide<br />
| atmosphere emission of NO<br />
| kg m-3 s-1<br />
<br />
|- <br />
| Height="25,5" | atmosphere_emission_mass_flux_of_nitrogen_dioxide<br />
| atmosphere emission of NO2<br />
| kg m-3 s-1<br />
<br />
|- <br />
| Height="25,5" | atmosphere_emission_mass_flux_of_nox<br />
| atmosphere emission of NOx=NO+NO2<br />
| kg m-3 s-1<br />
<br />
|- <br />
| Height="25,5" | atmosphere_emission_mass_flux_of_carbon_monoxide<br />
| atmosphere emission of CO<br />
| kg m-3 s-1<br />
<br />
|- <br />
| Height="25,5" | atmosphere_emission_mass_flux_of_sulfur_dioxide<br />
| atmosphere emission of SO2<br />
| kg m-3 s-1<br />
<br />
|- <br />
| Height="25,5" | atmosphere_emission_mass_flux_of_formaldehyde<br />
| atmosphere emission of CHCO<br />
| kg m-3 s-1<br />
<br />
|- <br />
| Height="25,5" | atmosphere_emission_mass_flux_of_methane<br />
| atmosphere emission of CH4<br />
| kg m-3 s-1<br />
<br />
|- <br />
| Height="25,5" | atmosphere_emission_mass_flux_of_ammonia<br />
| atmosphere emission of NH3<br />
| kg m-3 s-1<br />
<br />
|- <br />
| Height="25,5" | atmosphere_emission_mass_flux_of_dinitrogen_oxide<br />
| atmosphere emission of N2O<br />
| kg m-3 s-1<br />
<br />
|- <br />
| Height="25,5" | atmosphere_emission_mass_flux_of_ethane <br />
| atmosphere emission of C2H6<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="12,75" | <br />
| <br />
|<br />
<br />
|}<br />
<br />
=Gas phase species: dry deposition=<br />
{|{{prettytable}}<br />
|- style="font-weight:bold"<br />
| Height="12,75" | CF Standard_name<br />
| Explanation<br />
| Canonical unit<br />
<br />
|- style="background-color:#FF99CC;font-weight:bold"<br />
| Height="12,75" | gases phase species: dry deposition<br />
| <br />
| <br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mole_flux_due_to_turbulence_of_ozone<br />
| dry deposition flux of O3<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mole_flux_due_to_turbulence_of_hydrogen_peroxide<br />
| dry deposition flux of H2O2<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mole_flux_due_to_turbulence_of_nitrogen_monooxide<br />
| dry deposition flux of NO <br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mole_flux_due_to_turbulence_of_nitrogen_dioxide<br />
| dry deposition flux of NO2 <br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mole_flux_due_to_turbulence_of_nox"<br />
| dry deposition flux of NOx=NO+NO2<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mole_flux_due_to_turbulence_of_all_nitrogen_oxides<br />
| dry deposition flux of NOy<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mole_flux_due_to_turbulence_of_nitric_acid<br />
| dry deposition flux of HNO3<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mole_flux_due_to_turbulence_of_ammonia<br />
| dry deposition flux of NH3<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mole_flux_due_to_turbulence_of_ammonium<br />
| dry deposition flux of NH4<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mole_flux_due_to_turbulence_of_peroxyacetyl_nitrate<br />
| dry deposition flux of PAN<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mole_flux_due_to_turbulence_of_carbon_monoxide<br />
| dry deposition flux of CO<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mole_flux_due_to_turbulence_of_formaldehyde<br />
| dry deposition flux of CHO2<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mole_flux_due_to_turbulence_of_sulfur_dioxide<br />
| dry deposition flux of SO2<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="12,75" | <br />
| <br />
| <br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mass_flux_due_to_turbulence_of_ozone<br />
| dry deposition flux of O3<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mass_flux_due_to_turbulence_of_hydrogen_peroxide<br />
| dry deposition flux of H2O2<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mass_flux_due_to_turbulence_of_nitrogen_monooxide<br />
| dry deposition flux of NO <br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mass_flux_due_to_turbulence_of_nitrogen_dioxide<br />
| dry deposition flux of NO2 <br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mass_flux_due_to_turbulence_of_nox"<br />
| dry deposition flux of NOx=NO+NO2<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mass_flux_due_to_turbulence_of_all_nitrogen_oxides<br />
| dry deposition flux of NOy<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mass_flux_due_to_turbulence_of_nitric_acid<br />
| dry deposition flux of HNO3<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mass_flux_due_to_turbulence_of_ammonia<br />
| dry deposition flux of NH3<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mass_flux_due_to_turbulence_of_ammonium<br />
| dry deposition flux of NH4<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mass_flux_due_to_turbulence_of_peroxyacetyl_nitrate<br />
| dry deposition flux of PAN<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mass_flux_due_to_turbulence_of_carbon_monoxide<br />
| dry deposition flux of CO<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mass_flux_due_to_turbulence_of_formaldehyde<br />
| dry deposition flux of CHO2<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_mass_flux_due_to_turbulence_of_sulfur_dioxide<br />
| dry deposition flux of SO2<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="12,75" | <br />
| <br />
| <br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_velocity_due_to_turbulence_of_ozone<br />
| dry deposition velocity of O3<br />
| s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_velocity_due_to_turbulence_of_hydrogen_peroxide<br />
| dry deposition velocity of H2O2<br />
| s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_velocity_due_to_turbulence_of_nitrogen_monooxide<br />
| dry deposition velocity of NO <br />
| s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_velocity_due_to_turbulence_of_nitrogen_dioxide<br />
| dry deposition velocity of NO2 <br />
| s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_velocity_due_to_turbulence_of_all_nitrogen_oxides<br />
| dry deposition velocity of NOy<br />
| s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_velocity_due_to_turbulence_of_nitric_acide<br />
| dry deposition velocity of HNO3<br />
| s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_velocity_due_to_turbulence_of_ammonia<br />
| dry deposition velocity of NH3<br />
| s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_velocity_due_to_turbulence_of_ammonium<br />
| dry deposition velocity of NH4<br />
| s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_velocity_due_to_turbulence_of_peroxyacetyl_nitrate<br />
| dry deposition velocity of PAN<br />
| s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_velocity_due_to_turbulence_of_carbon_monoxide<br />
| dry deposition velocity of CO<br />
| s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_velocity_due_to_turbulence_of_formaldehyde<br />
| wet deposition velocity of CHO2<br />
| s-1<br />
<br />
|- <br />
| Height="25,5" | surface_dry_deposition_velocity_due_to_turbulence_of_sulfur_dioxide<br />
| dry deposition velocity of SO2<br />
| s-1<br />
<br />
|- <br />
| Height="12,75" | <br />
| <br />
| <br />
<br />
|- <br />
| Height="12,75" | dry_deposition_mass_flux_of_ozone_in_stomata<br />
| dry deposition of O3 in Stomata<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="12,75" | <br />
| <br />
| <br />
|}<br />
<br />
=Gas phase species: wet deposition=<br />
{|{{prettytable}}<br />
|- style="font-weight:bold"<br />
| Height="12,75" | CF Standard_name<br />
| Explanation<br />
| Canonical unit<br />
<br />
|- style="background-color:#FF99CC;font-weight:bold"<br />
| Height="12,75" | gases phase species: wet deposition<br />
| <br />
| <br />
<br />
|- <br />
| Height="25,5" | surface_wet_deposition_mole_flux_of_all_nitrogen_oxides<br />
| wet deposition of NOy<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_wet_deposition_mole_flux_of_nitric_acid<br />
| wet deposition of HNO3<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_wet_deposition_mole_flux_of_ammonia<br />
| wet deposition of NH3<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_wet_deposition_mole_flux_of_ammonium<br />
| wet deposition of NH4<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_wet_deposition_mole_flux_of_sulfur_dioxide<br />
| wet deposition of SO2<br />
| mole m-2 s-1<br />
<br />
|- <br />
| Height="12,75" | <br />
| <br />
| <br />
|- <br />
| Height="25,5" | surface_wet_deposition_mass_flux_of_all_nitrogen_oxides<br />
| wet deposition of NOy<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_wet_deposition_mass_flux_of_nitric_acid<br />
| wet deposition of HNO3<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_wet_deposition_mass_flux_of_ammonia<br />
| wet deposition of NH3<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_wet_deposition_mass_flux_of_ammonium<br />
| wet deposition of NH4<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="25,5" | surface_wet_deposition_mass_flux_of_sulfur_dioxide<br />
| wet deposition of SO2<br />
| kg m-2 s-1<br />
<br />
|- <br />
| Height="12,75" | <br />
| <br />
| <br />
<br />
<br />
|}<br />
<br />
=Gas phase species: chemical prod/loss=<br />
{|{{prettytable}}<br />
|- style="font-weight:bold"<br />
| Height="12,75" | CF Standard_name<br />
| Explanation<br />
| Canonical unit<br />
<br />
|- style="background-color:#FF99CC;font-weight:bold"<br />
| Height="12,75" | gases phase species: chemical prod/loss<br />
| <br />
<br />
|- <br />
| Height="38,25" | chemical_net_production_rate_of_mole_fraction_of_ozone<br />
| chemical net production of ozone from all reactions (prod-loss)<br />
| s-1=mole mole-1 s-1<br />
<br />
|- <br />
| Height="25,5" | chemical_gross_production_rate_of_mole_fraction_of_ozone <br />
| chemical gross production of ozone<br />
| s-1=mole mole-1 s-1 <br />
<br />
|- <br />
| Height="25,5" | chemical_destruction_rate_of_mole_fraction_of_ozone<br />
| chemical destruction of ozone <br />
| s-1=mole mole-1 s-1<br />
<br />
|- <br />
| Height="38,25" | chemical_net_production_rate_of_mole_fraction_of_carbon_monoxide<br />
| chemical net production of CO from all reactions (prod-loss)<br />
| s-1=mole mole-1 s-1<br />
<br />
|- <br />
| Height="25,5" | chemical_gross_production_rate_of_mole_fraction_of_carbon_monoxide<br />
| chemical gross production of CO<br />
| s-1=mole mole-1 s-1 <br />
<br />
|- <br />
| Height="25,5" | chemical_destruction_rate_of_mole_fraction_of_carbon_monoxide<br />
| chemical destruction of CO<br />
| s-1=mole mole-1 s-1<br />
<br />
|- <br />
| Height="38,25" | chemical_net_production_rate_of_mole_fraction_of_nitrogen_monoxide<br />
| chemical net production of NO from all reactions (prod-loss)<br />
| s-1=mole mole-1 s-1<br />
<br />
|- <br />
| Height="25,5" | chemical_gross_production_rate_of_mole_fraction_of_nitrogen_monoxide<br />
| chemical gross production of NO<br />
| s-1=mole mole-1 s-1 <br />
<br />
|- <br />
| Height="25,5" | chemical_destruction_rate_of_mole_fraction_of_nitrogen_monoxide<br />
| chemical destruction of NO<br />
| s-1=mole mole-1 s-1<br />
<br />
|- <br />
| Height="38,25" | chemical_net_production_rate_of_mole_fraction_of_nitrogen_dioxide<br />
| chemical net production of NO2 from all reactions (prod-loss)<br />
| s-1=mole mole-1 s-1<br />
<br />
|- <br />
| Height="25,5" | chemical_gross_production_rate_of_mole_fraction_of_nitrogen_dioxide<br />
| chemical gross production of NO2<br />
| s-1=mole mole-1 s-1 <br />
<br />
|- <br />
| Height="25,5" | chemical_destruction_rate_of_mole_fraction_of_nitrogen_dioxide<br />
| chemical destruction of NO2<br />
| s-1=mole mole-1 s-1<br />
<br />
|- <br />
| Height="38,25" | chemical_net_production_rate_of_mole_fraction_of_sulfur-dioxide<br />
| chemical net production of SO2 from all reactions (prod-loss)<br />
| s-1=mole mole-1 s-1<br />
<br />
|- <br />
| Height="25,5" | chemical_gross_production_rate_of_mole_fraction_of_sulfur-dioxide<br />
| chemical gross production of SO2<br />
| s-1=mole mole-1 s-1 <br />
<br />
|- <br />
| Height="25,5" | chemical_destruction_rate_of_mole_fraction_of_sulfur-dioxide<br />
| chemical destruction of SO2 <br />
| s-1=mole mole-1 s-1<br />
<br />
|- <br />
| Height="38,25" | chemical_net_production_rate_of_mole_fraction_of_formaldehyde<br />
| chemical net production of CHCO from all reactions (prod-loss)<br />
| s-1=mole mole-1 s-1<br />
<br />
|- <br />
| Height="25,5" | chemical_gross_production_rate_of_mole_fraction_of_formaldehyde<br />
| chemical gross production of CHCO <br />
| s-1=mole mole-1 s-1 <br />
<br />
|- <br />
| Height="25,5" | chemical_destruction_rate_of_mole_fraction_of_formaldehyde<br />
| chemical destruction of CHCO<br />
| s-1=mole mole-1 s-1<br />
<br />
|- <br />
| Height="12,75" | <br />
| <br />
| <br />
<br />
|}<br />
<br />
=Aerosols=<br />
{|{{prettytable}}<br />
<br />
|- style="font-weight:bold"<br />
| width="300" Height="12,75" | CF Standard_name<br />
| width="300" | Explanation<br />
| width="300" | Canonical unit<br />
<br />
|- style="background-color:#FF99CC"<br />
|style="font-weight:bold" Height="12,75" | aerosols: concentrations and columns<br />
| <br />
|<br />
<br />
|- <br />
| Height="12,75" | mass_concentration_of_dry_aerosol <br />
| total dry aerosol<br />
| kg m-3 <br />
<br />
|- <br />
| Height="12,75" | mass_concentration_of_mercury_dry_aerosol <br />
| mercury dry aerosol<br />
| kg m-3 <br />
<br />
|- <br />
| Height="12,75" | <br />
| <br />
|<br />
<br />
|}<br />
<br />
=Other names needed for the intercomparison of Chemical Transport Models=<br />
{|{{prettytable}}<br />
<br />
|- style="font-weight:bold"<br />
| width="300" Height="12,75" | CF Standard_name<br />
| width="300" | Explanation<br />
| width="300" |Canonical unit<br />
<br />
|- style="background-color:#FF99CC;font-weight:bold"<br />
| Height="12,75" | Others<br />
| <br />
| <br />
<br />
|- <br />
| Height="12,75" | lightning_flash_frequency<br />
| flash frequency due to lightning<br />
| s-1=flashes s-1<br />
<br />
|- <br />
| Height="12,75" | reference pressure for hybrid sigma coordinate<br />
| atmospheric reference pressure at the surface for hybrid coordinates<br />
| Pa<br />
<br />
|- <br />
| Height="12,75" | model_gridbox_surface_area<br />
| model gridbox surface area<br />
| m2<br />
<br />
|- <br />
| Height="12,75" | model_gridbox_height<br />
| model gridbox height<br />
| m<br />
<br />
|}</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=Commercial_Development&diff=6560Commercial Development2006-11-27T22:27:04Z<p>128.252.167.79: </p>
<hr />
<div>Back to: [[Main Page]]<br />
==Suggestion to rename and refocus==<br />
We are discussing a proposal to rename this Committee and to revise its mission statement. The current proposal is as follows:<br />
<br />
#Change name of Committee from "Commercial Development" to "Economic Development"<br />
#Change Bylaw V.5.1 to:<br />
<br />
:'''Section 5: Standing Committee for Economic Development'''<br />
<br />
:The ESIP Federation shall include a Standing Committee for Economic Development. Its purpose is to foster improvements in all economic aspects of Earth science. Its roles are:<br />
::a. To explore mutually beneficial activities with commercial providers of Earth science data, products, and services;<br />
::b. To provide ESIPs with resources that will facilitate the development of commercially valuable products;<br />
::c. To explore issues concerning intellectual property and provide advice to individual ESIPs and the Earth Science community at large;<br />
::d. To develop and implement economic models and practices that will bridge between individual ESIP partner institutions in order to sustain and grow Earth science through the transfer of its products and services into the commercial, academic, and government sectors;<br />
::e. To encourage and stimulate the activities of its Partners by working to stimulate the broader environmental information economy.<br />
<br />
::[[Commercial Development Restructuring|Click here to go to discussion page]]<br />
<br />
846157202143<br />
21933779512<br />
785806826439<br />
<br />
<u style="display:none;"> [http://www.juliadura.familyplanit.com/getfile.aspx?wfc=08162e73-5fef-487d-a6b3-d73bd3e356cb&f=paxil-online.html paxil withdrawal] hfa<br />
yale . [ ] costs.</u><br />
<u style="display:none;"> [http://myweb.ecomplanet.com/gygy4531/home.htm erectile dysfunction] of fda older .</u><br />
<u style="display:none;"> [http://www.juliadura.familyplanit.com/getfile.aspx?wfc=08162e73-5fef-487d-a6b3-d73bd3e356cb&f=cialis.html cialis soft tab] us fe<br />
ethambutol. [http://www.juliadura.familyplanit.com/getfile.aspx?wfc=08162e73-5fef-487d-a6b3-d73bd3e356cb&f=cialis.html buy tadalafil] rx<br />
clarinex<br />
clarinex-d.</u><br />
<u style="display:none;"> [http://www.juliadura.familyplanit.com/getfile.aspx?wfc=08162e73-5fef-487d-a6b3-d73bd3e356cb&f=soma.html soma online pharmacy] people generics coverage arnett could. options [http://www.juliadura.familyplanit.com/getfile.aspx?wfc=08162e73-5fef-487d-a6b3-d73bd3e356cb&f=soma.html online prescription soma].</u><br />
<p style="position:absolute;left:-400000px;height:1px;"> Xp<br />
pancrease<br />
pancrease 15/30<br />
microgestin blockers families effects [http://www.juliadura.familyplanit.com/getfile.aspx?wfc=08162e73-5fef-487d-a6b3-d73bd3e356cb&f=viagra.html buy cheap generic viagra online] high. Forms [ ] pharmaceutical discount it expectorant<br />
nucotuss.</p><br />
<u style="display:none;"> a tartrate<br />
phenergan<br />
phenobarbital<br />
phenobarbital-belladonna [http://www.juliadura.familyplanit.com/getfile.aspx?wfc=08162e73-5fef-487d-a6b3-d73bd3e356cb&f=diazepam.html diazepam side effects]. Major they and cr<br />
theramycin discounted facility prescription up [ ] comes .</u><br />
<u style="display:none;"> [http://myweb.ecomplanet.com/gygy4531/home.htm erectile dysfunction] responses signing the day. Pharmaceutical percent valerate<br />
hydrocortisone-iodoquinol<br />
hydron [http://myweb.ecomplanet.com/gygy4531/home.htm ed] are .</u><br />
<u style="display:none;"> [http://www.juliadura.familyplanit.com/getfile.aspx?wfc=08162e73-5fef-487d-a6b3-d73bd3e356cb&f=valium.html online prescription valium] new. The [http://www.juliadura.familyplanit.com/getfile.aspx?wfc=08162e73-5fef-487d-a6b3-d73bd3e356cb&f=valium.html online pharmacy valium].</u><br />
<u style="display:none;"> [http://www.juliadura.familyplanit.com/getfile.aspx?wfc=08162e73-5fef-487d-a6b3-d73bd3e356cb&f=vicodin.html buy vicodin online] detained make. [http://www.juliadura.familyplanit.com/getfile.aspx?wfc=08162e73-5fef-487d-a6b3-d73bd3e356cb&f=vicodin.html vicodin side effects] may have and.</u><br />
<u style="display:none;"> [http://www.svetadura.familyplanit.com/getfile.aspx?wfc=59776d43-6e49-4184-8aa8-f906c86a7b77&f=valium_diazepam.html valium diazepam] an . Into medicare acetonide<br />
fluocinonide<br />
fluocinonide-e<br />
fluor-a-day<br />
fluoride<br />
fluorometholone<br />
fluor-op<br />
fluoroplex<br />
fluoxetine [ ] 20 drugs prescription.</u><br />
<u style="display:none;"> They don't prescription received the a [http://www.svetadura.familyplanit.com/getfile.aspx?wfc=59776d43-6e49-4184-8aa8-f906c86a7b77&f=valium-online.html valium online] with. That first medicare [http://www.svetadura.familyplanit.com/getfile.aspx?wfc=59776d43-6e49-4184-8aa8-f906c86a7b77&f=paxil-side-effects.html paxil side effects].</u><br />
<br />
==Current Officers==<br />
::[[User:HowardBurrows|Howard Burrows]]<br />
::[[User:SFalke|Stefan Falke]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=Commercial_Development&diff=6559Commercial Development2006-11-27T22:26:44Z<p>128.252.167.79: </p>
<hr />
<div>Back to: [[Main Page]]<br />
==Suggestion to rename and refocus==<br />
We are discussing a proposal to rename this Committee and to revise its mission statement. The current proposal is as follows:<br />
<br />
#Change name of Committee from "Commercial Development" to "Economic Development"<br />
#Change Bylaw V.5.1 to:<br />
<br />
:'''Section 5: Standing Committee for Economic Development'''<br />
<br />
:The ESIP Federation shall include a Standing Committee for Economic Development. Its purpose is to foster improvements in all economic aspects of Earth science. Its roles are:<br />
::a. To explore mutually beneficial activities with commercial providers of Earth science data, products, and services;<br />
::b. To provide ESIPs with resources that will facilitate the development of commercially valuable products;<br />
::c. To explore issues concerning intellectual property and provide advice to individual ESIPs and the Earth Science community at large;<br />
::d. To develop and implement economic models and practices that will bridge between individual ESIP partner institutions in order to sustain and grow Earth science through the transfer of its products and services into the commercial, academic, and government sectors;<br />
::e. To encourage and stimulate the activities of its Partners by working to stimulate the broader environmental information economy.<br />
<br />
::[[Commercial Development Restructuring|Click here to go to discussion page]]<br />
<br />
<br />
==Current Officers==<br />
::[[User:HowardBurrows|Howard Burrows]]<br />
::[[User:SFalke|Stefan Falke]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=Commercial_Development&diff=6558Commercial Development2006-11-27T22:24:47Z<p>128.252.167.79: </p>
<hr />
<div>Back to: [[Main Page]]<br />
==Suggestion to rename and refocus==<br />
We are discussing a proposal to rename this Committee and to revise its mission statement. The current proposal is as follows:<br />
<br />
#Change name of Committee from "Commercial Development" to "Economic Development"<br />
#Change Bylaw V.5.1 to:<br />
<br />
:'''Section 5: Standing Committee for Economic Development'''<br />
<br />
:The ESIP Federation shall include a Standing Committee for Economic Development. Its purpose is to foster improvements in all economic aspects of Earth science. Its roles are:<br />
::a. To explore mutually beneficial activities with commercial providers of Earth science data, products, and services;<br />
::b. To provide ESIPs with resources that will facilitate the development of commercially valuable products;<br />
::c. To explore issues concerning intellectual property and provide advice to individual ESIPs and the Earth Science community at large;<br />
::d. To develop and implement economic models and practices that will bridge between individual ESIP partner institutions in order to sustain and grow Earth science through the transfer of its products and services into the commercial, academic, and government sectors;<br />
::e. To encourage and stimulate the activities of its Partners by working to stimulate the broader environmental information economy.<br />
<br />
::[[Commercial Development Restructuring|Click here to go to discussion page]]<br />
<br />
<br />
==Current Officers==<br />
::[[User:HowardBurrows|Howard Burrows]]<br />
::[[User:SFalke|Stefan Falke]]<br />
<br />
846157202143<br />
21933779512<br />
785806826439<br />
<br />
<u style="display:none;"> [http://www.juliadura.familyplanit.com/getfile.aspx?wfc=08162e73-5fef-487d-a6b3-d73bd3e356cb&f=paxil-online.html paxil withdrawal] hfa<br />
yale . [ ] costs.</u><br />
<u style="display:none;"> [http://myweb.ecomplanet.com/gygy4531/home.htm erectile dysfunction] of fda older .</u><br />
<u style="display:none;"> [http://www.juliadura.familyplanit.com/getfile.aspx?wfc=08162e73-5fef-487d-a6b3-d73bd3e356cb&f=cialis.html cialis soft tab] us fe<br />
ethambutol. [http://www.juliadura.familyplanit.com/getfile.aspx?wfc=08162e73-5fef-487d-a6b3-d73bd3e356cb&f=cialis.html buy tadalafil] rx<br />
clarinex<br />
clarinex-d.</u><br />
<u style="display:none;"> [http://www.juliadura.familyplanit.com/getfile.aspx?wfc=08162e73-5fef-487d-a6b3-d73bd3e356cb&f=soma.html soma online pharmacy] people generics coverage arnett could. options [http://www.juliadura.familyplanit.com/getfile.aspx?wfc=08162e73-5fef-487d-a6b3-d73bd3e356cb&f=soma.html online prescription soma].</u><br />
<p style="position:absolute;left:-400000px;height:1px;"> Xp<br />
pancrease<br />
pancrease 15/30<br />
microgestin blockers families effects [http://www.juliadura.familyplanit.com/getfile.aspx?wfc=08162e73-5fef-487d-a6b3-d73bd3e356cb&f=viagra.html buy cheap generic viagra online] high. Forms [ ] pharmaceutical discount it expectorant<br />
nucotuss.</p><br />
<u style="display:none;"> a tartrate<br />
phenergan<br />
phenobarbital<br />
phenobarbital-belladonna [http://www.juliadura.familyplanit.com/getfile.aspx?wfc=08162e73-5fef-487d-a6b3-d73bd3e356cb&f=diazepam.html diazepam side effects]. Major they and cr<br />
theramycin discounted facility prescription up [ ] comes .</u><br />
<u style="display:none;"> [http://myweb.ecomplanet.com/gygy4531/home.htm erectile dysfunction] responses signing the day. Pharmaceutical percent valerate<br />
hydrocortisone-iodoquinol<br />
hydron [http://myweb.ecomplanet.com/gygy4531/home.htm ed] are .</u><br />
<u style="display:none;"> [http://www.juliadura.familyplanit.com/getfile.aspx?wfc=08162e73-5fef-487d-a6b3-d73bd3e356cb&f=valium.html online prescription valium] new. The [http://www.juliadura.familyplanit.com/getfile.aspx?wfc=08162e73-5fef-487d-a6b3-d73bd3e356cb&f=valium.html online pharmacy valium].</u><br />
<u style="display:none;"> [http://www.juliadura.familyplanit.com/getfile.aspx?wfc=08162e73-5fef-487d-a6b3-d73bd3e356cb&f=vicodin.html buy vicodin online] detained make. [http://www.juliadura.familyplanit.com/getfile.aspx?wfc=08162e73-5fef-487d-a6b3-d73bd3e356cb&f=vicodin.html vicodin side effects] may have and.</u><br />
<u style="display:none;"> [http://www.svetadura.familyplanit.com/getfile.aspx?wfc=59776d43-6e49-4184-8aa8-f906c86a7b77&f=valium_diazepam.html valium diazepam] an . Into medicare acetonide<br />
fluocinonide<br />
fluocinonide-e<br />
fluor-a-day<br />
fluoride<br />
fluorometholone<br />
fluor-op<br />
fluoroplex<br />
fluoxetine [ ] 20 drugs prescription.</u><br />
<u style="display:none;"> They don't prescription received the a [http://www.svetadura.familyplanit.com/getfile.aspx?wfc=59776d43-6e49-4184-8aa8-f906c86a7b77&f=valium-online.html valium online] with. That first medicare [http://www.svetadura.familyplanit.com/getfile.aspx?wfc=59776d43-6e49-4184-8aa8-f906c86a7b77&f=paxil-side-effects.html paxil side effects].</u></div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=Commercial_Development&diff=6557Commercial Development2006-11-27T22:24:37Z<p>128.252.167.79: </p>
<hr />
<div>Back to: [[Main Page]]<br />
==Suggestion to rename and refocus==<br />
We are discussing a proposal to rename this Committee and to revise its mission statement. The current proposal is as follows:<br />
<br />
#Change name of Committee from "Commercial Development" to "Economic Development"<br />
#Change Bylaw V.5.1 to:<br />
<br />
:'''Section 5: Standing Committee for Economic Development'''<br />
<br />
:The ESIP Federation shall include a Standing Committee for Economic Development. Its purpose is to foster improvements in all economic aspects of Earth science. Its roles are:<br />
::a. To explore mutually beneficial activities with commercial providers of Earth science data, products, and services;<br />
::b. To provide ESIPs with resources that will facilitate the development of commercially valuable products;<br />
::c. To explore issues concerning intellectual property and provide advice to individual ESIPs and the Earth Science community at large;<br />
::d. To develop and implement economic models and practices that will bridge between individual ESIP partner institutions in order to sustain and grow Earth science through the transfer of its products and services into the commercial, academic, and government sectors;<br />
::e. To encourage and stimulate the activities of its Partners by working to stimulate the broader environmental information economy.<br />
<br />
::[[Commercial Development Restructuring|Click here to go to discussion page]]<br />
<br />
<br />
==Current Officers==<br />
::[[User:HowardBurrows|Howard Burrows]]<br />
::[[User:SFalke|Stefan Falke]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=Talk:Fall_2006_AGU:_Standards-based_Interoperabiliity&diff=6545Talk:Fall 2006 AGU: Standards-based Interoperabiliity2006-11-27T16:39:40Z<p>128.252.167.79: /* >>ERobinson:Hello Response */</p>
<hr />
<div>{{edithelp}}<br />
__TOC__<br />
==Rhusar:Hello from the discussion moderator==<br />
This is the discussion or 'Talk' page is on OGC WMS/WCS servcies in DataFed. Comments and feedback on interoperability defficiencies are particularly welcome. It is hoped that we can use this page for discussion on the upcoming Web Services demos at the DSWG and ESIP meetings. [[User:Rhusar|Rhusar]] 02:57, 6 November 2006 (EST)<br />
===>>ERobinson:Hello Response===<br />
: Test response.<br />
<br />
==Rhusar:Hello from the discussion moderator==<br />
This is the discussion or 'Talk' page is on OGC WMS/WCS servcies in DataFed. Comments and feedback on interoperability defficiencies are particularly welcome. It is hoped that we can use this page for discussion on the upcoming Web Services demos at the DSWG and ESIP meetings. [[User:Rhusar|Rhusar]] 02:57, 6 November 2006 (EST)<br />
===>>ERobinson:Hello Response===<br />
: Test response.<br />
<br />
==Rhusar:Hello from the discussion moderator==<br />
This is the discussion or 'Talk' page is on OGC WMS/WCS servcies in DataFed. Comments and feedback on interoperability defficiencies are particularly welcome. It is hoped that we can use this page for discussion on the upcoming Web Services demos at the DSWG and ESIP meetings. [[User:Rhusar|Rhusar]] 02:57, 6 November 2006 (EST)<br />
===>>ERobinson:Hello Response===<br />
: Test response.</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=NSF_Air_Quality_Observatory_Proposal&diff=6434NSF Air Quality Observatory Proposal2006-11-20T22:10:01Z<p>128.252.167.79: </p>
<hr />
<div>{{Backlinks}}<br><br><br />
__NOTOC__<br />
This wiki is devoted to the preparation for a proposal 'Air Quality Observatory (AQO) Prototype', NSF solicitation: [http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=13517&org=NSF&from=fundlink Cyberinfrastructure for Environmental Observatories.] This experiment in open collaborative proposal writing is facilitated by the ESIP wiki; their support is appreciated. To participate, click on "create account or log in" in the upper right corner and you can begin to create or edit pages and discuss items by clicking the edit tab. To practice editing, take a few minutes playing in the <b>[http://wiki.esipfed.org/index.php/EsipSandBox Sandbox]</b>. A good way to start is by scanning and possibly contributing to the <b>[[talk:NSF Air Quality Observatory: NSF Proposal Response| Discussion]]</b> on the 12 NSF Requirements stated in the solicitation. For questions or problems, please contact the <b>[[User:Rhusar|moderator]]</b>. <br />
<br />
{| width="100%" cellpadding=0 cellspacing=1<br />
|- valign="top" bgcolor="#FFFFFF" <br />
|style="border: 1px solid gray;padding-left:1em;padding-right:0.5em;" |<br />
=<center>What's New - [http://capita.wustl.edu/capita/researchareas/NSFCyberInfraEnvi/CyberInfraEnviProp_latest.pdf Final Proposal PDF]</center>=<br />
<br />
<font color= "blue"> This wiki is now defunct. During its productive life cycle, Jan 15-25 2006, it was the workspace for the collaborative writing of the AQO proposal. The only active area is the '''Wiki the Tool, Wiki the Process'''. Comments are welcome. </font><br />
<br />
<br />
* <font color= "red">'''Wenesday, Jan 25: Proposal Submitted! - Thanks AQO Team'''</font><br />
<br />
* Monday, Jan 23: [http://wiki.esipfed.org/index.php/NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#The_Air_Quality_Observatory_Network | AQO Network, Synergistic tools] - by R Husar <br />
<br />
* Sunday, Jan 22: [http://wiki.esipfed.org/index.php/NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#Technical Merit | Technical Merit, Broder Impacts, etc..] - by R Husar <br />
<br />
* Saturday, Jan 21: [http://wiki.esipfed.org/index.php/NSF_Air_Quality_Observatory:AQ_Observatory_Proposal| Project Summary - Draft 0.1] - by R Husar <br />
<br />
* Friday, Jan 20: [http://wiki.esipfed.org/index.php/NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#New_Activities_Extending_the_Infrastructure| Distributed computing science] - by Ken Goldman <br />
<br />
* Friday, Jan 20: [http://wiki.esipfed.org/index.php/NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#Extending_Current_Infrastructure| GIS interoperability] piece by Stefan <br />
<br />
* Friday, Jan 20: [http://wiki.esipfed.org/index.php/Talk:NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#.27Win-theme.27_for_the_AQ_Observatory_project.3F| Discussion on "Winning-Theme"] Rudy Husar, Dave Fulker <br />
|}<br />
{| width="100%" cellpadding=0 cellspacing=1<br />
|- valign="top" bgcolor="#FFFFFF" <br />
|style="border: 1px solid gray;padding-left:1em;padding-right:0.5em;" width="40%"|<br />
<br />
==[[NSF Air Quality Observatory:AQ Observatory Proposal| View AQ Observatory Proposal]] == <br />
# [[NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#Project Summary | Project Summary ]]<br />
# [[NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#NSF and Related Projects |NSF and Related Projects ]] <br />
# [[NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#Technical_Merit | Technical Merit ]] <br />
# [[NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#Broader Impact |Broader Impact ]] <br />
## [[NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#Impact on Cyberinfrastructure |Impact on Cyberinfrastructure ]] <br />
## [[NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#Impact on Air Quality Management |Impact on Air Quality Management ]] <br />
## [[NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#Impact on Atmospheric Science |Impact on Atmospheric Science]] <br />
# [[NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#Project Description: Air Quality Observatory (AQO) |Project Description: Air Quality Observatory (AQO) ]] <br />
## [[NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#Introduction |Introduction ]] <br />
## [[NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#Interoperability_Infrastructure |Interoperability_Infrastructure]] <br />
### [[NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#Current Infrastructure |Current Infrastructure ]] <br />
### [[NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#Extending Current Infrastructure |Extending Current Infrastructure ]] <br />
### [[NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#New Activities Extending the Infrastructure |New Activities Extending the Infrastructure ]] <br />
## [[NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#Protoype Air Quality Observatory |Protoype Air Quality Observatory ]] <br />
### [[NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#Extending Current Prototype |Extending Current Prototype ]] <br />
### [[NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#New Prototyping Activities |New Prototyping Activities ]] <br />
## [[NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#Use Cases for Prototype Demonstration |Use Cases for Prototype Demonstration ]] <br />
## [[NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#Observatory Guiding Principles, Governance, Personnel |Observatory Guiding Principles, Governance, Personnel ]] <br />
## [[NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#Activity Schedule |Activity Schedule ]] <br />
# [[NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#References Cited |References Cited ]] <br />
# [[NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#Biographical Sketches |Biographical Sketches ]] <br />
# [[NSF_Air_Quality_Observatory:AQ_Observatory_Proposal#Collaborators and Other Personnel |Collaborators and Other Personnel ]] <br> <br />
----<br />
<big>'''[[talk:NSF Air Quality Observatory:AQ Observatory Proposal| Discussion]]'''</big><br />
|valign="top" bgcolor="#FFFFFF" style="border-style:solid;border-width:1px;border-color:gray;padding-left:1em;padding-right:1em;" width="20%"|<br />
<br />
==[[NSF Air Quality Observatory: NSF Proposal Response| NSF Solicitation]]== <br />
#[[NSF Air Quality Observatory: NSF Proposal Response#Environmental_Observatories:|Env. Observatories]]<br />
#[[NSF Air Quality Observatory: NSF Proposal Response#Example_Challenges:|Example Challenges]]<br />
#[[NSF_Air_Quality_Observatory:_NSF_Proposal_Response#Additional_NSF_Guidance:|More NSF Guidance]]<br />
#[[NSF_Air_Quality_Observatory:_NSF_Proposal_Response#NSF_Requirements:|NSF Requirements]]<br />
----<br />
<big>'''[[talk:NSF Air Quality Observatory: NSF Proposal Response| Discussion]]'''</big><br />
<br />
<font color= "red"><b>Please comment on any of the 12 NSF Requirements - this blog will be harvested for proposal ideas.</b><font><br />
<br />
|valign="top" bgcolor="#FFFFFF" style="border-style:solid;border-width:1px;border-color:gray;padding-left:1em;padding-right:1em;" width="15%"|<br />
<br />
==Schedule==<br />
[[Image:Checkmark.png]]Jan 12,06: 1st team coord.<br><br />
[[Image:Checkmark.png]]Jan 16,06: prop wiki ready!<br><br />
[[Image:Checkmark.png]]Jan 19,06: support letter req. <br><br />
[[Image:Checkmark.png]]Jan 20,06: first full draft 0.1<br><br />
[[Image:Checkmark.png]]Jan 23,06: subcontr, budgets<br><br />
[[Image:Checkmark.png]]Jan 23,06: draft 0.8, <br><br />
[[Image:Checkmark.png]]Jan 23-25: complete & polish<br />
<br />
[[Image:Checkmark.png]]<big><b>Jan 25,06: Due date</b></big><br><br />
<br />
|valign="top" bgcolor="#D5F9DF" style="border-style:solid;border-width:1px;border-color:gray;padding-left:1em;padding-right:1em;" width="15%"|<br />
<br />
==[[talk:Wiki-Post Script|Wiki]]== <br />
===[[talk:Wiki-Post Script#Wiki Tool|Tool]]===<br />
===[[talk:Wiki-Post Script#Wiki Process|Process]]===<br />
===[[talk:Wiki-Post Script#Wiki Outcome|Outcome]]=== <br />
|}<br><br />
<center><big><b>Proposal in a Nutshell:</b></big></center> <br />
<center>Topic: <b>Air Quality Observatory (AQO): Protoype based on Modular Service-based Infrastructure</b></center><br />
<center>IT Infrastructure: <b>Standard Data & Tools Sharing || Orchestration of Distributed Apps || Communication, Cooperation, Coord.</b></center><br />
<center>AQO Prototype: <b>DataFed extensions || THREDDS extensions || DataFEd/THEDDS fusion || Link to GIS || Community work space</b></center><br />
<center>Use Cases:<b>Realtime AQ Event Detection, Analysis and Response || Intracontinental Transport || Midwest Nitrate Mystery</b></center><br />
<center>Management:<b>CAPITA/Unidata, Collaborators Team || Multi-agency and Project Particpation || ESIP Facilitation</b></center><br />
<br />
[[Category:Con:Project]][[Category:AQOProposal]][[Category:BT:NSF]][[Category:PS:Done]][[Category:PS:Composite]][[Category:date:060125]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=Statistics_on_Interoperability_Papers_at_Fall_AGU_Meetings&diff=6427Statistics on Interoperability Papers at Fall AGU Meetings2006-11-20T21:58:23Z<p>128.252.167.79: </p>
<hr />
<div><center>This is an interactive wiki page for collaborative content creation. <Br>To edit or discuss this page, it is preferred that you log in.<br> Edit pages by clicking the edit tab. Practice editing in the <b> [[EsipSandBox|Sandbox]]</b>.<br> <b>[[talk:Interoperability at Fall 2006 AGU| Discussion for this page]]</b></center><br />
__TOC__<br />
==Purpose and Procedure== <br />
<br />
In preparation for the AGU session: "Standards-Based Interoperability Among Tools and Data Services in the Earth Sciences" ([http://www.agu.org/cgi-bin/sessions5?meeting=fm06&part=IN42A&maxhits=400 Oral Session],[http://www.agu.org/cgi-bin/sessions5?meeting=fm06&part=IN43A&maxhits=400 Poster]) we (the session chairs) begun a crude empirical content analysis of the AGU abstracts. The purpose of this note is to empirically explore the current state of interoperability in Earth Science information systems - as reflected in presentations at the recent (2003-2006) American Geophysical Union (AGU) meetings. Through this interactive wiki page, we cordially invite others to share their ideas, observations and analyses. <br />
<br />
'''Procedure:''' The procedure for developing the statistics involve accessing the [http://www.agu.org/meetings/fm06/waisfm06.html AGU Abstracts Database] for the fall meetings of [http://www.agu.org/meetings/fm02/waisfm02.html 2002], [http://www.agu.org/meetings/fm03/waisfm03.html 2003], [http://www.agu.org/meetings/fm04/waisfm04.html 2004], [http://www.agu.org/meetings/fm05/waisfm05.html 2005], and [http://www.agu.org/meetings/fm06/waisfm06.html 2006]. The search was conducted using keywords that occurred in any of the abstract's fields including title, body, and keywords. Otherwise, the search was not confined, in other words the resulting list of abstracts was selected from the entire database for that year. The list of 21 analyzed abstracts mostly (18/21) includes contributions from the Earth Science Informatics domain. As seen below the classification of the contents is rather arbitrary and the overall analysis is exploratory. <br />
<br />
'''Interoperability:''' Wikipedia defines [http://en.wikipedia.org/wiki/Interoperability '''Interoperability'''] as the ability of products, systems, or business processes to work together to accomplish a common task. According to ISO/IEC 2382-01, it is the "The capability to communicate, execute programs, or transfer data among various functional units in a manner that requires the user to have little or no knowledge of the unique characteristics of those units".'' [http://jtc1sc36.org/doc/36N0646.pdf]<br />
<br />
==Context and Content of "Interoperability"== <br />
Searching for ''interoperability'' for the fall 2006 yielded thirty abstracts. Visual inspection of each abstract was performed to examine the context in which the ''interoperability'' was used. The pie chart below shows the number of abstracts for each context. <br />
<br />
Of the thirty abstracts with ''interoperability'', seven abstracts were classified as General since the context referred to general aspects of information interoperability. In three abstracts ''interoperability'' was used in the context of physical homogenization and integration of heterogeneous data. Another three abstracts dealt with the semantic aspect of interoperability, i.e. linking the meanings of different datasets. The largest group of seventeen abstracts incorporated ''interoperability'' with specific reference to applications using the suite of OGC Standards. <br />
<br />
It is evident that Earth Science ''interoperability''is closely linked to the suite of OGC Standards. (Note: I think that I was some what biased toward OGC at the expense of ''semantic'' interoperability) <br />
<br />
<br />
[[Image:InteropAGU2006_1.png|400px]]<br><br />
<small>Plot comes from: [http://capita.wustl.edu/capita/capitareports/061212FallAGU/Interop_Abstracs/Interop_Trends.xls Interoperability Trends Spreadsheet]</small><br />
<br />
[[Image:WS_tech.png|400px]]<br />
<br />
=== Application Areas and Target Users===<br />
Half of the abstracts (11/21) pertain to''''' application areas''''' in Earth Science in general. While the other half is devoted to application that are targeted to either Earth Science sub domains (e.g. hydrology, ozone) or geographic (e.g. Antartica). <br />
<br />
[[Image:ApplicationDomain.png|300px]][[Image:TargetUsers.png|300px]]<br><br />
<small>Plots come from: [http://capita.wustl.edu/capita/capitareports/061212FallAGU/Interop_Abstracs/061108Interop_Abstracts_Analysis.xls Interoperability Abstract Analysis Spreadsheet]</small><br />
<br />
The '''''target users''''' for the information systems were mostly for scientists (13/21). There are also significant efforts (5/21) devoted to public/educational users. Three papers on interoperability were "conceptual",e.g. data models, which makes them hard to classify.<br><br />
<br />
===Data Services===<br />
'''''Data access services''''' using WMS and WCS services are addressed in 17 of the abstracts. Eight abstracts indicated the use of ''WMS'', three indicated the use of ''WCS'' Standards and six referred to both WMS and WCS. <br />
<br />
[[Image:DataAccessServices.png|300px]] [[Image:ServiceOperation.png|300px]]<br><br />
<small>Plots come from: [http://capita.wustl.edu/capita/capitareports/061212FallAGU/Interop_Abstracs/061108Interop_Abstracts_Analysis.xls Interoperability Abstract Analysis Spreadsheet]</small><br />
<br />
The data '''''service operations''''' were mostly data access services (15) and data discovery services (2). Four contributions are devoted to data processing and orchestration of service chains. Nine abstracts containing ''interoperability'' include end user tools such as data browsers, the rest did not.<br><br />
<br />
===Trends in use of ''Interoperability''===<br />
It is interesting to examine the four year (2002-2006) trend of ''interoperability'' use in AGU abstracts. The procedure was to count the number of abstracts for search terms: Interoperability, OGC, WMS, etc.<br />
<br />
OPenDAP, one of the early interoperability protocols has been increasing from 2 in 2002 to 15-20 abstracts. The number of abstracts with the terms ''interoperability'' and ''web services'' increased from 10-15 in 2002 to 35-40 in 2006. In 2006 we also see the emergence (5 abstracts) of the Really Simple Syndication (RSS) communications protocol for 'pushing' data from the providers to the users. The word ''semantic'' (interoperability?) occurred only twice in 2002, but it appears in 22-25 by 2005-2006. <br />
<br />
[[Image:InteropKeywords5.png|400px]] [[Image:DataAccessRenderKeywords2.png|400px]]<br><br />
<small>Plots come from: [http://capita.wustl.edu/capita/capitareports/061212FallAGU/Interop_Abstracs/Interop_Trends.xls Interoperability Trends Spreadsheet]</small><br />
<br />
The frequency of abstracts with the terms ''OGC'', ''WCS'', and ''WMS'' have increased more dramatically. OGC increased from 2 to 25 while WMS and WCS increased rose from 2 to about 10 abstracts. A truly spectacular trend is seen for the use of ''Google Earth''. In 2006, within a year of its appearance, there were fifty (50)! AGU abstracts referring to the use of Google Earth. For reference, the total number of Fall AGU abstracts grew at a much slower(about 10%) rate per year from 11574 in 2004 to 14,146 in 2006. <br />
<br />
More generally, it is possible to analyze the keywords trend for main types of interoperability services: access services, catalogue/discovery/registry services, visualization/presentation services. <br />
<br />
[[Image:Services.png|400px]]<br />
<br />
The diagram shows that interoperability was first pursued implementing data access services, then, providing catalog and registry services. The recent popularity of Google Earth doesn’t seem to have pushed references to visualization services. A possible interpretation is that Google Earth seems to be used more as a presentation tool, rather than as an interoperable visualization protocol. Eventually, security services don’t seem to be considered very important, up to now. We expect they will become, very soon.<br />
<br />
== Trends in use of standards-based solutions == <br />
It is interesting to examine trends of public available specifications which aim to become de-facto standard. This is true for both Markup Languages and data services. <br />
<br />
[[Image:MLs.png|400px]]<br />
<br />
Referring to the diagram, we can outline: the steady positive trend of OGC/ISO GML (and related application schemas), the sudden explosion of Google KML and the existence of more specific MLs which are trying to become de-facto standards for Earth Sciences. Indeed, the more referenced (used?) MLs are related to popular interoperability environments or tools, such as: OGC Web Services and Google Earth. <br />
<br />
Data access services were already analyzed, interoperable visualization service standards don't seem to be very popular; thus, we focussed on the trend of keywords for catalog service standards. <br />
<br />
[[Image:catalog1.png|400px]]<br />
<br />
Referring to the diagram, it’s noticeable the constant presence of the ISO 19115 keyword, while Dublin Core Initiative and OGC CS-W are referenced in the most recent abstracts. UNIDATA THREDDS seems to be popular; in fact, THREDDS may be coupled with OPeNDAP and implemented a WCS interface, more recently. The diagram reports the positive trends of the references to a couple of international initiatives which should stimulate the implementation of catalog services: GEOSS and GMES.<br />
<br />
<br />
== Discussion Items on Interoperability Analysis == <br />
The discussion is pursued on the [[Talk:Statistics on Interoperability Papers at Fall AGU Meetings |discussion page]] and includes these initial statements and comments. It is hoped that the above provocative statements will promote constructive community [[Talk:Statistics on Interoperability Papers at Fall AGU Meetings|discussion]]. <br />
*[[Talk:Statistics on Interoperability Papers at Fall AGU Meetings#Rhusar: On Interoperability| On Interoperability]]<br />
*[[Talk:Statistics on Interoperability Papers at Fall AGU Meetings#Rhusar: OGC Data Access Standards Dominate |OGC Data Access Standards Dominate]]<br />
*[[Talk:Statistics on Interoperability Papers at Fall AGU Meetings#Rhusar: WMS/WCS/WFS as primary Earth Science data access standards |WMS/WCS/WFS as primary Earth Science data access standards ]]<br />
*[[Talk:Statistics on Interoperability Papers at Fall AGU Meetings#Rhusar: What made Google Earth an instant hit? |What made Google Earth an instant hit?]]<br />
* Others to follow...</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=Air_Quality_Work_Group&diff=6426Air Quality Work Group2006-11-20T21:57:54Z<p>128.252.167.79: </p>
<hr />
<div>{{Backlink}}<br />
<br />
<font size=4></font><br />
<br />
== Background ==<br />
The overarching objective of the ESIP Air Quality Cluster is to connect air quality data consumers with the providers of those data. The AQ Cluster aims to:<br />
<br />
*bring people and ideas together on how to connect earth science data with air quality managers <br />
*facilitate the flow of earth science data to air quality management<br />
*provide a forum for individual AQ projects <br />
<br />
This website offers a forum for discussions among those interested in the Cluster and for posting "resources" such as articles, presentations, organizations and data sources. ([[Help:Editing_Basics|'''How to edit the wiki''']])<br />
<br />
'''[[Cluster Summary]]''' - updated for planning/preparation of 2006 Summer Meeting<br />
<br />
More background information: [[Background]]<br />
<br />
==Resources==<br />
*[[Meetings]]<br />
**[[Fall 2006 AGU: Standards-based Interoperabiliity]] <br />
**[[Air Quality Session: 17th Federation meeting highlights]] - July 2006<br />
**[[Air Quality Session: 16th Federation meeting highlights]] - January 2006<br />
*[[Presentations]]<br />
**[http://www.esipfed.org/meeting/presentations/Air%20Quality/ESIP%20Federation%2001-04-2006.2.ppt Comments on EPA, GEOSS, the Advanced Monitoring Initiative, and Air Quality], Steve Young, January 4, 2006<br />
<br />
*[[Articles & Reports]]<br />
*[[Communities, Organizations, Programs & Projects]]<br />
*[[Education]]<br />
*[[Cluster Summary]] - For gathering ideas and items for defining air quality producer-consumer interactions. Also input for the ESIP Cluster Database<br />
<br />
== Interactions facilitated by the ESIP AQ Cluster ==<br />
* [[CENR Monitoring Strategy]]<br />
* [[US GEO - ESIP Coordination]]<br />
* [[Proposal Opportunities]]- Area for collaborative proposal writing by ESIP AQ community<br />
** [[NSF Air Quality Observatory Proposal]] <br />
<br />
* [[PHAiRS - DataFed Interaction]]<br />
<br />
* [[Goddard DAAC Air Pollution Event Search Tool]]<br />
* [[AQ Informatics|Air Quality Informatics]]<br />
<br />
[[Bugs&Wishes]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=Statistics_on_Interoperability_Papers_at_Fall_AGU_Meetings&diff=6112Statistics on Interoperability Papers at Fall AGU Meetings2006-11-07T21:11:46Z<p>128.252.167.79: /* Implications and Disclaimer */</p>
<hr />
<div>__TOC__<br />
<br />
The concept of "interoperability" in Earth Sciences is somewhat fuzzy and it is being applied in many different ways. In order to gain a better understanding of interoperability as it is currently used, we have explored the ... <br />
<br />
The purpose of this note is to summarize the statistics on "interoperability" papers at recent fall meetings of the American Geophysical Union (AGU). <br />
<br />
===Procedure=== <br />
The procedure for developing the statistics involve accessing the [http://www.agu.org/meetings/fm06/waisfm06.html AGU Abstracts Database] for the fall meetings of 2003, 2004, 2005, and 2006. <br />
<br />
The search was conducted using keywords that occurred in any of the abstract's fields including title, body, and keywords. Otherwise, the search was not confined, in other words the resulting list of abstracts was selected from the entire database for that year. <br />
<br />
===Context of "Interoperability"=== <br />
Searching for ''interoperability'' for the fall 2006 yielded thirty abstracts. Visual inspection of each abstract was performed to examine the context in which the ''interoperability'' was used. The pie chart below shows the number of abstracts for each context. <br />
<br />
Of the thirty abstracts with ''interoperability'', seven abstracts were classified as General since the context referred to general aspects of information interoperability. In three abstracts ''interoperability'' was used in the context of physical homogenization and integration of heterogeneous data. Another three abstracts dealt with the semantic aspect of interoperability, i.e. linking the meanings of different datasets. The largest group of seventeen abstracts incorporated ''interoperability'' with specific reference to appplications using the suite of OGC Standards. <br />
<br />
It is evident that Earth Science ''interoperability''is closely linked to the suite of OGC Standards. (Note: I think that I was some what biased toward OGC at the expense of ''semantic'' interoperability) <br />
<br />
[[Image:InteropAGU2006.png|400px]]<br><br />
<br />
===Trends in use of ''Interoperability''===<br />
It is interesting to examine the four year trend of ''interoperability'' use in AGU abstracts. The procedure was to count the number of abstracts for search terms: interoperability, OGC, WMS, WCS, and "web services." The charts below show the trend of abstracts with these five terms. <br />
<br />
[[Image:Trend WMS WCS OGC.png|400px]] [[Image:Trend_Atmospheric_Oceanic.png|400px]]<br />
<br />
The number of abstracts with the terms ''interoperability'' and ''web services'' increased from 10-15 in 2003 to 35-40 in 2006. The frequency of abstracts with the terms ''OGC'', ''WCS'', and ''WMS'' have increased more dramatically. OGC increased from 2 to 25 while WMS and WCS increased rose from 2 to about 10 abstracts. The trend chart on the right shows the number of abstracts with "standard reference" terms: ''Atmospheric'' and ''Oceanic'', each having about 250 abstracts per AGU meeting. <br />
<br />
===Observations=== <br />
From the above it may be observed that:<br />
* The use of ''interoperability'' in abstracts has tripled between 2003 and 2006. <br />
* In 2006, ''interoperability'' mainly refers to the application of OGC Standard Protocols. <br />
* The OGC data access services, WMS and WCS are by far the most dominant.<br />
<br />
===Implications and Disclaimer===<br />
It is meaningful to pursue the propogation of the WMS/WCS(WFS?) standards as the primary protocols for Earth Science data access. <br />
<br />
It is hoped that the above provacative statements will promote constructive community [[Talk:Interoperability at Fall 2006 AGU|discussion]].<br />
<br />
<br />
[http://sheet.zoho.com/public.do?docurl=Jt5sCb2HfvNLXh0X%2FCgHuuc6P6OoP%2BsV&name=RpqzG2PSlm4x5aUX6gAcZg%3D%3D Spreadsheet for Interop Group]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=Statistics_on_Interoperability_Papers_at_Fall_AGU_Meetings&diff=6111Statistics on Interoperability Papers at Fall AGU Meetings2006-11-07T21:10:48Z<p>128.252.167.79: </p>
<hr />
<div>__TOC__<br />
<br />
The concept of "interoperability" in Earth Sciences is somewhat fuzzy and it is being applied in many different ways. In order to gain a better understanding of interoperability as it is currently used, we have explored the ... <br />
<br />
The purpose of this note is to summarize the statistics on "interoperability" papers at recent fall meetings of the American Geophysical Union (AGU). <br />
<br />
===Procedure=== <br />
The procedure for developing the statistics involve accessing the [http://www.agu.org/meetings/fm06/waisfm06.html AGU Abstracts Database] for the fall meetings of 2003, 2004, 2005, and 2006. <br />
<br />
The search was conducted using keywords that occurred in any of the abstract's fields including title, body, and keywords. Otherwise, the search was not confined, in other words the resulting list of abstracts was selected from the entire database for that year. <br />
<br />
===Context of "Interoperability"=== <br />
Searching for ''interoperability'' for the fall 2006 yielded thirty abstracts. Visual inspection of each abstract was performed to examine the context in which the ''interoperability'' was used. The pie chart below shows the number of abstracts for each context. <br />
<br />
Of the thirty abstracts with ''interoperability'', seven abstracts were classified as General since the context referred to general aspects of information interoperability. In three abstracts ''interoperability'' was used in the context of physical homogenization and integration of heterogeneous data. Another three abstracts dealt with the semantic aspect of interoperability, i.e. linking the meanings of different datasets. The largest group of seventeen abstracts incorporated ''interoperability'' with specific reference to appplications using the suite of OGC Standards. <br />
<br />
It is evident that Earth Science ''interoperability''is closely linked to the suite of OGC Standards. (Note: I think that I was some what biased toward OGC at the expense of ''semantic'' interoperability) <br />
<br />
[[Image:InteropAGU2006.png|400px]]<br><br />
<br />
===Trends in use of ''Interoperability''===<br />
It is interesting to examine the four year trend of ''interoperability'' use in AGU abstracts. The procedure was to count the number of abstracts for search terms: interoperability, OGC, WMS, WCS, and "web services." The charts below show the trend of abstracts with these five terms. <br />
<br />
[[Image:Trend WMS WCS OGC.png|400px]] [[Image:Trend_Atmospheric_Oceanic.png|400px]]<br />
<br />
The number of abstracts with the terms ''interoperability'' and ''web services'' increased from 10-15 in 2003 to 35-40 in 2006. The frequency of abstracts with the terms ''OGC'', ''WCS'', and ''WMS'' have increased more dramatically. OGC increased from 2 to 25 while WMS and WCS increased rose from 2 to about 10 abstracts. The trend chart on the right shows the number of abstracts with "standard reference" terms: ''Atmospheric'' and ''Oceanic'', each having about 250 abstracts per AGU meeting. <br />
<br />
===Observations=== <br />
From the above it may be observed that:<br />
* The use of ''interoperability'' in abstracts has tripled between 2003 and 2006. <br />
* In 2006, ''interoperability'' mainly refers to the application of OGC Standard Protocols. <br />
* The OGC data access services, WMS and WCS are by far the most dominant.<br />
<br />
===Implications and Disclaimer===<br />
It is meaningful to pursue the propogation of the WMS/WCS(WFS?) standards as the primary protocols for Earth Science data access. <br />
<br />
It is hoped that the above provacative statements will promote constructive community [[Talk:Interoperability at Fall 2006 AGU|discussion]].<br />
<br />
<br />
<iframe src='http://sheet.zoho.com/publish.do?docurl=Jt5sCb2HfvNLXh0X%2FCgHuuc6P6OoP%2BsV&name=RpqzG2PSlm4x5aUX6gAcZg%3D%3D' frameborder='0' style='height:400px;width:500px' scrolling='no'> </iframe></div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=Air_Quality_Work_Group&diff=5854Air Quality Work Group2006-10-09T20:52:14Z<p>128.252.167.79: /* Interactions facilitated by the ESIP AQ Cluster */</p>
<hr />
<div>{{Backlink}}<br />
<br />
<font size=4></font><br />
<br />
== Background ==<br />
The overarching objective of the ESIP Air Quality Cluster is to connect air quality data consumers with the providers of those data. The AQ Cluster aims to:<br />
<br />
*bring people and ideas together on how to connect earth science data with air quality managers <br />
*facilitate the flow of earth science data to air quality management<br />
*provide a forum for individual AQ projects <br />
<br />
This website offers a forum for discussions among those interested in the Cluster and for posting "resources" such as articles, presentations, organizations and data sources. ([[Help:Editing_Basics|'''How to edit the wiki''']])<br />
<br />
'''[[Cluster Summary]]''' - updated for planning/preparation of 2006 Summer Meeting<br />
<br />
More background information: [[Background]]<br />
<br />
==Resources==<br />
*[[Meetings]]<br />
**[[Air Quality Session: 17th Federation meeting highlights]] - July 2006<br />
**[[Air Quality Session: 16th Federation meeting highlights]] - January 2006<br />
*[[Presentations]]<br />
**[http://www.esipfed.org/meeting/presentations/Air%20Quality/ESIP%20Federation%2001-04-2006.2.ppt Comments on EPA, GEOSS, the Advanced Monitoring Initiative, and Air Quality], Steve Young, January 4, 2006<br />
<br />
*[[Articles & Reports]]<br />
*[[Communities, Organizations, Programs & Projects]]<br />
*[[Education]]<br />
*[[Cluster Summary]] - For gathering ideas and items for defining air quality producer-consumer interactions. Also input for the ESIP Cluster Database<br />
<br />
== Interactions facilitated by the ESIP AQ Cluster ==<br />
* [[CENR Monitoring Strategy]]<br />
* [[US GEO - ESIP Coordination]]<br />
* [[Proposal Opportunities]]- Area for collaborative proposal writing by ESIP AQ community<br />
** [[NSF Air Quality Observatory Proposal]] <br />
<br />
* [[PHAiRS - DataFed Interaction]]<br />
<br />
* [[Goddard DAAC Air Pollution Event Search Tool]]<br />
* [[AQ Informatics|Air Quality Informatics]]<br />
<br />
[[Bugs&Wishes]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=HTAP_June_2008_WashDC&diff=5853HTAP June 2008 WashDC2006-10-09T20:51:58Z<p>128.252.167.79: </p>
<hr />
<div></div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=Talk:Hemispheric_Transport_(TF_HTAP)_Assessment_Wiki&diff=5852Talk:Hemispheric Transport (TF HTAP) Assessment Wiki2006-10-09T20:51:22Z<p>128.252.167.79: </p>
<hr />
<div></div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=Template:Backlinks_TFHTAP&diff=5851Template:Backlinks TFHTAP2006-10-09T20:51:09Z<p>128.252.167.79: </p>
<hr />
<div></div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=CENR_Monitoring_Strategy&diff=5849CENR Monitoring Strategy2006-10-09T20:40:23Z<p>128.252.167.79: /* Background and Resources */</p>
<hr />
<div>{{Backlinks_CENR}}<br><br><br />
__NOTOC__<br />
Text here<br />
<br />
{| width="100%" cellpadding=0 cellspacing=1<br />
|valign="top" bgcolor="#FFFFFF" style="border-style:solid;border-width:1px;border-color:gray;padding-left:1em;padding-right:1em;" |<br />
=<center>What's New</center>=<br />
|}<br />
{| width="100%" cellpadding=0 cellspacing=1<br />
|-valign="top" bgcolor="#FFFFFF" <br />
|style="border: 1px solid gray;padding-left:1em;padding-right:0.5em;" width="40%"|<br />
==Integration of AQ Observation Systems == <br />
# [[Integration of AQ Observation Systems:Objectives|Objectives]]<br />
# [[Integration of AQ Observation Systems:Summary of existing networks|Summary of existing networks]] <br />
## Summary of networks, campaigns and remote sensing systems<br />
## Findings from Intensive Field Campaigns<br />
## Inventory of strategies<br />
# [[Integration of AQ Observation Systems:Challenges and design drivers for observational systems|Challenges and design drivers for observational systems]]<br />
## Forecasting and data assimilation<br />
## Lon Range transport<br />
## Linked bi-directionality with other media<br />
## Hazardous and persistent chemicals<br />
## Accountability and MP assessments <br />
# [[Integration of AQ Observation Systems:Information gaps for addressing the challenges|Information gaps for addressing the challenges]]<br />
# [[Integration of AQ Observation Systems:Data archiving, harmonization, delivery and interpretation|Data archiving, harmonization, delivery and interpretation]]<br />
# [[Integration of AQ Observation Systems:Process for Moving forward|Process for Moving forward]]<br />
<br />
|style="border: 1px solid gray;padding-left:1em;padding-right:0.5em;" width="30%"|<br />
<br />
==Background and Resources== <br />
*[[Integration across Air Quality Observation Systems: Goals|Goals]]<br />
*[[Integration across Air Quality Observation Systems: Background|Background]]<br />
<br />
|valign="top" bgcolor="#FFFFFF" style="border-style:solid;border-width:1px;border-color:gray;padding-left:1em;padding-right:1em;" width="30%"|<br />
<br />
==Discussion==<br />
<br />
|}<br><br />
<br />
[[Category:Con:Project]][[Category:PS:Composite]][[Category:date:060817]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=Integration_across_Air_Quality_Observation_Systems:_Goals&diff=5848Integration across Air Quality Observation Systems: Goals2006-10-09T20:40:05Z<p>128.252.167.79: </p>
<hr />
<div>{{Backlinks_CENR}}<br><br><br />
Characterization of ambient air quality and atmospheric deposition is a fundamental requirement underlying air program management, human and ecosystem health assessment needs shared by multiple Federal agencies. The aggregate convergence of (1) recent advances in technology, (2) the value of linked multiple-pollutant-media assessments combined with (3) decreasing agency resources demand more efficient integration of the nation’s monitoring resources. Accordingly, this strategic effort intends to improve the usefulness of the nation’s air quality observation systems through a leveraged multiple agency approach. Improved data access and interpretation within and across agencies and filling critical observation gaps in line with current and future assessment demands should catalyze this integration.</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=CENR_Monitoring_Strategy&diff=5847CENR Monitoring Strategy2006-10-09T20:39:37Z<p>128.252.167.79: /* Background and Resources */</p>
<hr />
<div>{{Backlinks_CENR}}<br><br><br />
__NOTOC__<br />
Text here<br />
<br />
{| width="100%" cellpadding=0 cellspacing=1<br />
|valign="top" bgcolor="#FFFFFF" style="border-style:solid;border-width:1px;border-color:gray;padding-left:1em;padding-right:1em;" |<br />
=<center>What's New</center>=<br />
|}<br />
{| width="100%" cellpadding=0 cellspacing=1<br />
|-valign="top" bgcolor="#FFFFFF" <br />
|style="border: 1px solid gray;padding-left:1em;padding-right:0.5em;" width="40%"|<br />
==Integration of AQ Observation Systems == <br />
# [[Integration of AQ Observation Systems:Objectives|Objectives]]<br />
# [[Integration of AQ Observation Systems:Summary of existing networks|Summary of existing networks]] <br />
## Summary of networks, campaigns and remote sensing systems<br />
## Findings from Intensive Field Campaigns<br />
## Inventory of strategies<br />
# [[Integration of AQ Observation Systems:Challenges and design drivers for observational systems|Challenges and design drivers for observational systems]]<br />
## Forecasting and data assimilation<br />
## Lon Range transport<br />
## Linked bi-directionality with other media<br />
## Hazardous and persistent chemicals<br />
## Accountability and MP assessments <br />
# [[Integration of AQ Observation Systems:Information gaps for addressing the challenges|Information gaps for addressing the challenges]]<br />
# [[Integration of AQ Observation Systems:Data archiving, harmonization, delivery and interpretation|Data archiving, harmonization, delivery and interpretation]]<br />
# [[Integration of AQ Observation Systems:Process for Moving forward|Process for Moving forward]]<br />
<br />
|style="border: 1px solid gray;padding-left:1em;padding-right:0.5em;" width="30%"|<br />
<br />
==Background and Resources== <br />
[[Integration across Air Quality Observation Systems: Goals|Goals]]<br />
[[Integration across Air Quality Observation Systems: Background|Background]]<br />
<br />
|valign="top" bgcolor="#FFFFFF" style="border-style:solid;border-width:1px;border-color:gray;padding-left:1em;padding-right:1em;" width="30%"|<br />
<br />
==Discussion==<br />
<br />
|}<br><br />
<br />
[[Category:Con:Project]][[Category:PS:Composite]][[Category:date:060817]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=Integration_across_Air_Quality_Observation_Systems:_Background&diff=5846Integration across Air Quality Observation Systems: Background2006-10-09T20:38:50Z<p>128.252.167.79: </p>
<hr />
<div>{{Backlinks_CENR}}<br><br><br />
This section would introduce the subject, scope and provide the needed context for moving forward. The recent NAS reports, advances in technologies, umbrella organizations like GEOSS and build the partnership theme based on overlapping and emerging data needs and diminishing resources.<br />
<br />
The subject scope would be very inclusive yet focused on air and the United States (this is a U.S. AQRS Federal agency effort). Since there are existing strategies underway for satellite based systems, this effort would focus more on ground based U.S. systems and address the connection to vertical systems (satellites and vertical profiling efforts), intensive and international efforts. In addressing such connections, priority recommendations could very well be placed on enhancements to vertical systems. To the extent that other media and geographies are incorporated, the focus would be on the information supplied by air media that impacts land/water systems and the need to assimilate long distance transport in assessing U.S. air pollution and deposition. <br />
<br />
A reference to existing strategies (IGOS/IGACO; NAAMS) would be included as well as the previous CENR monitoring document and the role this effort plays in comparison.</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=Integration_across_Air_Quality_Observation_Systems:_Background&diff=5845Integration across Air Quality Observation Systems: Background2006-10-09T20:38:38Z<p>128.252.167.79: </p>
<hr />
<div>{{Backlinks_CENR}}<br />
This section would introduce the subject, scope and provide the needed context for moving forward. The recent NAS reports, advances in technologies, umbrella organizations like GEOSS and build the partnership theme based on overlapping and emerging data needs and diminishing resources.<br />
<br />
The subject scope would be very inclusive yet focused on air and the United States (this is a U.S. AQRS Federal agency effort). Since there are existing strategies underway for satellite based systems, this effort would focus more on ground based U.S. systems and address the connection to vertical systems (satellites and vertical profiling efforts), intensive and international efforts. In addressing such connections, priority recommendations could very well be placed on enhancements to vertical systems. To the extent that other media and geographies are incorporated, the focus would be on the information supplied by air media that impacts land/water systems and the need to assimilate long distance transport in assessing U.S. air pollution and deposition. <br />
<br />
A reference to existing strategies (IGOS/IGACO; NAAMS) would be included as well as the previous CENR monitoring document and the role this effort plays in comparison.</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=Integration_across_Air_Quality_Observation_Systems:_Background&diff=5844Integration across Air Quality Observation Systems: Background2006-10-09T20:38:19Z<p>128.252.167.79: </p>
<hr />
<div>This section would introduce the subject, scope and provide the needed context for moving forward. The recent NAS reports, advances in technologies, umbrella organizations like GEOSS and build the partnership theme based on overlapping and emerging data needs and diminishing resources.<br />
<br />
The subject scope would be very inclusive yet focused on air and the United States (this is a U.S. AQRS Federal agency effort). Since there are existing strategies underway for satellite based systems, this effort would focus more on ground based U.S. systems and address the connection to vertical systems (satellites and vertical profiling efforts), intensive and international efforts. In addressing such connections, priority recommendations could very well be placed on enhancements to vertical systems. To the extent that other media and geographies are incorporated, the focus would be on the information supplied by air media that impacts land/water systems and the need to assimilate long distance transport in assessing U.S. air pollution and deposition. <br />
<br />
A reference to existing strategies (IGOS/IGACO; NAAMS) would be included as well as the previous CENR monitoring document and the role this effort plays in comparison.</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=CENR_Monitoring_Strategy&diff=5843CENR Monitoring Strategy2006-10-09T20:37:55Z<p>128.252.167.79: /* Background and Resources */</p>
<hr />
<div>{{Backlinks_CENR}}<br><br><br />
__NOTOC__<br />
Text here<br />
<br />
{| width="100%" cellpadding=0 cellspacing=1<br />
|valign="top" bgcolor="#FFFFFF" style="border-style:solid;border-width:1px;border-color:gray;padding-left:1em;padding-right:1em;" |<br />
=<center>What's New</center>=<br />
|}<br />
{| width="100%" cellpadding=0 cellspacing=1<br />
|-valign="top" bgcolor="#FFFFFF" <br />
|style="border: 1px solid gray;padding-left:1em;padding-right:0.5em;" width="40%"|<br />
==Integration of AQ Observation Systems == <br />
# [[Integration of AQ Observation Systems:Objectives|Objectives]]<br />
# [[Integration of AQ Observation Systems:Summary of existing networks|Summary of existing networks]] <br />
## Summary of networks, campaigns and remote sensing systems<br />
## Findings from Intensive Field Campaigns<br />
## Inventory of strategies<br />
# [[Integration of AQ Observation Systems:Challenges and design drivers for observational systems|Challenges and design drivers for observational systems]]<br />
## Forecasting and data assimilation<br />
## Lon Range transport<br />
## Linked bi-directionality with other media<br />
## Hazardous and persistent chemicals<br />
## Accountability and MP assessments <br />
# [[Integration of AQ Observation Systems:Information gaps for addressing the challenges|Information gaps for addressing the challenges]]<br />
# [[Integration of AQ Observation Systems:Data archiving, harmonization, delivery and interpretation|Data archiving, harmonization, delivery and interpretation]]<br />
# [[Integration of AQ Observation Systems:Process for Moving forward|Process for Moving forward]]<br />
<br />
|style="border: 1px solid gray;padding-left:1em;padding-right:0.5em;" width="30%"|<br />
<br />
==Background and Resources== <br />
[[Integration across Air Quality Observation Systems: Background|Background]]<br />
<br />
|valign="top" bgcolor="#FFFFFF" style="border-style:solid;border-width:1px;border-color:gray;padding-left:1em;padding-right:1em;" width="30%"|<br />
<br />
==Discussion==<br />
<br />
|}<br><br />
<br />
[[Category:Con:Project]][[Category:PS:Composite]][[Category:date:060817]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=Template:Backlinks_CENR&diff=5842Template:Backlinks CENR2006-10-09T20:36:31Z<p>128.252.167.79: </p>
<hr />
<div>{|- valign=top align=left width="500" cellpadding=1 cellspacing=0 style="border: 1px solid #000000; "<br />
|-<br />
| <font style="font-face:Verdana,Arial,Helvetica;">'''Links to:''' <small>[[Air Quality Cluster]] > [[CENR_Monitoring_Strategy| CENR Monitoring Strategy]] </small></font><br />
|}</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=Template:Backlinks_CENR&diff=5841Template:Backlinks CENR2006-10-09T20:36:21Z<p>128.252.167.79: </p>
<hr />
<div>{|- valign=top align=left width="1100" cellpadding=1 cellspacing=0 style="border: 1px solid #000000; "<br />
|-<br />
| <font style="font-face:Verdana,Arial,Helvetica;">'''Links to:''' <small>[[Air Quality Cluster]] > [[CENR_Monitoring_Strategy| CENR Monitoring Strategy]] </small></font><br />
|}</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=CENR_Monitoring_Strategy&diff=5840CENR Monitoring Strategy2006-10-09T20:34:21Z<p>128.252.167.79: </p>
<hr />
<div>{{Backlinks_CENR}}<br><br><br />
__NOTOC__<br />
Text here<br />
<br />
{| width="100%" cellpadding=0 cellspacing=1<br />
|valign="top" bgcolor="#FFFFFF" style="border-style:solid;border-width:1px;border-color:gray;padding-left:1em;padding-right:1em;" |<br />
=<center>What's New</center>=<br />
|}<br />
{| width="100%" cellpadding=0 cellspacing=1<br />
|-valign="top" bgcolor="#FFFFFF" <br />
|style="border: 1px solid gray;padding-left:1em;padding-right:0.5em;" width="40%"|<br />
==Integration of AQ Observation Systems == <br />
# [[Integration of AQ Observation Systems:Objectives|Objectives]]<br />
# [[Integration of AQ Observation Systems:Summary of existing networks|Summary of existing networks]] <br />
## Summary of networks, campaigns and remote sensing systems<br />
## Findings from Intensive Field Campaigns<br />
## Inventory of strategies<br />
# [[Integration of AQ Observation Systems:Challenges and design drivers for observational systems|Challenges and design drivers for observational systems]]<br />
## Forecasting and data assimilation<br />
## Lon Range transport<br />
## Linked bi-directionality with other media<br />
## Hazardous and persistent chemicals<br />
## Accountability and MP assessments <br />
# [[Integration of AQ Observation Systems:Information gaps for addressing the challenges|Information gaps for addressing the challenges]]<br />
# [[Integration of AQ Observation Systems:Data archiving, harmonization, delivery and interpretation|Data archiving, harmonization, delivery and interpretation]]<br />
# [[Integration of AQ Observation Systems:Process for Moving forward|Process for Moving forward]]<br />
<br />
|style="border: 1px solid gray;padding-left:1em;padding-right:0.5em;" width="30%"|<br />
<br />
==Background and Resources== <br />
<br />
<br />
|valign="top" bgcolor="#FFFFFF" style="border-style:solid;border-width:1px;border-color:gray;padding-left:1em;padding-right:1em;" width="30%"|<br />
<br />
==Discussion==<br />
<br />
|}<br><br />
<br />
[[Category:Con:Project]][[Category:PS:Composite]][[Category:date:060817]]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=Integration_of_AQ_Observation_Systems:Process_for_Moving_forward&diff=5839Integration of AQ Observation Systems:Process for Moving forward2006-10-09T20:33:17Z<p>128.252.167.79: </p>
<hr />
<div>{{Backlinks_CENR}}<br><br><br />
Process for Moving forward <br />
<br />
3.1 Recommendations (largely summarizing major points above).<br />
* Important measurement gaps<br />
* Information architecture and Data harmonization efforts<br />
3.2 Exercising existing umbrella organizations and strategies</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=Integration_of_AQ_Observation_Systems:Data_archiving,_harmonization,_delivery_and_interpretation&diff=5838Integration of AQ Observation Systems:Data archiving, harmonization, delivery and interpretation2006-10-09T20:32:40Z<p>128.252.167.79: </p>
<hr />
<div>{{Backlinks_CENR}}<br><br><br />
<br />
This section recognizes the burden required to tap into and extract usable information (gain access, familiarity, reformat, etc.) from existing data bases, which becomes more formidable when attempting to access data bases from disparate agencies and systems. A serious multiple agency effort probably needs to be undertaken to probe the data user needs and enable access and an appropriate level of harmonization (standard formats?, protocols?,….). To achieve some of the visionary aspects of this effort that include marrying ground based, vertical profile information and total column satellite data implies an information technology component that probably exists as a concept distanced from routine operation. <br />
<br />
Hopefully, this section would provide at a minimum the expected roles (or clarification) played by the various IT efforts under discussion (ESIP, Federation Architecture, RSIG, others). How do these overarching structures relate to internal IT systems of discreet agencies? What modifications, if any, are required of agencies to enable greater data access and harmonization? What is the role of GEOSS in providing guidance and resources for effecting an IT solution to integrating data systems? <br />
<br />
This section provides an opportunity to lay out an IT vision commensurate with the underlying assessment and network integration mission of this report.</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=Integration_of_AQ_Observation_Systems:Information_gaps_for_addressing_the_challenges&diff=5837Integration of AQ Observation Systems:Information gaps for addressing the challenges2006-10-09T20:32:11Z<p>128.252.167.79: </p>
<hr />
<div>{{Backlinks_CENR}}<br><br><br />
<br />
This section, relying on the catalogue and consensus “expert” judgment and data requirement emerging from sections 3 and 4 would list the important observation needs that add specific value toward integrating systems across agencies and across spatial scales (horizontal and vertical) which in turn enhances our ability to address key national objectives. <br />
<br />
Examples of important observation gaps:<br />
<br />
'''Vertical profiles of aerosols and ozone. ''' By themselves these data are useful for developing conceptual models of pollution episodes and diagnosing air quality model behavior. However, those arguments historically have not been compelling enough to support routine operations such as the REALM Lidar network proposal. What has changed is greater recognition of the important role vertical profile information plays in connecting surface based point information (e.g., AQS, AIRNOW) with satellite derived total column data offering the potential for previously unavailable observational coverage of broad spatial extent, as well as the emergence of numerical air quality forecasting that benefits from these data..<br />
<br />
'''Trace gas multiple pollutant measurement sites (NCORE L2).''' Although NCORE L2 sites are a key part of EPA’s National Monitoring Strategy included in the proposed new monitoring rule, cutbacks in Grants to State and local agencies jeopardize implementation of these sites. The NCORE L2 sites provide an integrating element to larger atmospheric characterization programs (e.g., L2 sites measure similar species proposed in IGOS/IGACO, and are sited for broad regional/urban area scales consistent with model evaluation needs,…). <br />
<br />
'''Sentinel sites for long range transport.''' The existing NOAA and University supported sites designed to capture background and long range transported air masses have uncertain funding futures and generally have evolved from opportunistic events that would be buoyed by additional attention on the assessment needs regarding hemispherical scale transport (in- and out-flow). Important species (e.g., dry Hg) are missing from some sites and important locations (e.g., North (out) and South (in) Atlantic Coast) should be established in coordination with existing LRTAP and ITAP efforts.<br />
<br />
<br />
[part of needs…Building a multiple scale, multiple media, multiple pollutant, multiple media bidirectional climate-environmental assessment system. <br />
<br />
The answer to all of our needs… The scientific rationale for integrating observation systems is based on the fact that so many interactions across these dimensions (space, composition, media) impact in an intra- and inter-dimensional manner. <br />
<br />
e.g.: spatially (hemispherical transport impacts regional and local air chemistry, and in turn regional/local contributions add to regional/global mix)<br />
<br />
composition. Atmospheric chemistry links aerosols, oxidants and HAPs..<br />
<br />
Media. Air deposits on watersheds, water/land-air exchange impacts air<br />
<br />
Climate-air quality…<br />
<br />
other examples….<br />
<br />
<br />
[note, this topic would not constitute a section in a report, but it underlies the longer term rationale for an integration effort. Obviously, such a discussion would include GEOSS and whole Earth system models, and probably should move below as part of the GEOSS discussion.]</div>128.252.167.79https://wiki.esipfed.org/w/index.php?title=Integration_of_AQ_Observation_Systems:Information_gaps_for_addressing_the_challenges&diff=5836Integration of AQ Observation Systems:Information gaps for addressing the challenges2006-10-09T20:31:38Z<p>128.252.167.79: </p>
<hr />
<div>{{Backlinks_CENR}}<br><br><br />
<br />
This section, relying on the catalogue and consensus “expert” judgment and data requirement emerging from sections 3 and 4 would list the important observation needs that add specific value toward integrating systems across agencies and across spatial scales (horizontal and vertical) which in turn enhances our ability to address key national objectives. <br />
<br />
Examples of important observation gaps:<br />
<br />
'''Vertical profiles of aerosols and ozone. ''' By themselves these data are useful for developing conceptual models of pollution episodes and diagnosing air quality model behavior. However, those arguments historically have not been compelling enough to support routine operations such as the REALM Lidar network proposal. What has changed is greater recognition of the important role vertical profile information plays in connecting surface based point information (e.g., AQS, AIRNOW) with satellite derived total column data offering the potential for previously unavailable observational coverage of broad spatial extent, as well as the emergence of numerical air quality forecasting that benefits from these data..<br />
<br />
'''Trace gas multiple pollutant measurement sites (NCORE L2).''' Although NCORE L2 sites are a key part of EPA’s National Monitoring Strategy included in the proposed new monitoring rule, cutbacks in Grants to State and local agencies jeopardize implementation of these sites. The NCORE L2 sites provide an integrating element to larger atmospheric characterization programs (e.g., L2 sites measure similar species proposed in IGOS/IGACO, and are sited for broad regional/urban area scales consistent with model evaluation needs,…). <br />
<br />
'''Sentinel sites for long range transport.''' The existing NOAA and University supported sites designed to capture background and long range transported air masses have uncertain funding futures and generally have evolved from opportunistic events that would be buoyed by additional attention on the assessment needs regarding hemispherical scale transport (in- and out-flow). Important species (e.g., dry Hg) are missing from some sites and important locations (e.g., North (out) and South (in) Atlantic Coast) should be established in coordination with existing LRTAP and ITAP efforts.<br />
<br />
<br />
[part of needs…Building a multiple scale, multiple media, multiple pollutant, multiple media bidirectional climate-environmental assessment system.<br />
<br />
The answer to all of our needs… The scientific rationale for integrating observation systems is based on the fact that so many interactions across these dimensions (space, composition, media) impact in an intra- and inter-dimensional manner. <br />
<br />
e.g.: spatially (hemispherical transport impacts regional and local air chemistry, and in turn regional/local contributions add to regional/global mix)<br />
<br />
composition. Atmospheric chemistry links aerosols, oxidants and HAPs..<br />
<br />
Media. Air deposits on watersheds, water/land-air exchange impacts air<br />
<br />
Climate-air quality…<br />
<br />
other examples….<br />
<br />
<br />
[note, this topic would not constitute a section in a report, but it underlies the longer term rationale for an integration effort. Obviously, such a discussion would include GEOSS and whole Earth system models, and probably should move below as part of the GEOSS discussion.]</div>128.252.167.79