Difference between revisions of "Sensor Data Management Middleware"
Line 43: | Line 43: | ||
<br/> | <br/> | ||
Some of the applications and platforms listed above are often identified as a software of choice for many different organizations. More details about each of these components are provided in the next section of this document. | Some of the applications and platforms listed above are often identified as a software of choice for many different organizations. More details about each of these components are provided in the next section of this document. | ||
+ | |||
+ | {| class="wikitable sortable" | ||
+ | |- | ||
+ | ! Program !! Licensing !! Cost !! Input data format !! Export data format !! Needed programming expertise | ||
+ | |- | ||
+ | | Antelope Orb || Proprietary || Pay || ASCII, Binary || ASCII, Binary || Advanced | ||
+ | |- | ||
+ | | Aquarius || Proprietary || Pay || || || Advanced | ||
+ | |- | ||
+ | | ArcGIS || Proprietary || Pay || ASCII, shapefiles || ASCII, shapefiles || Advanced | ||
+ | |- | ||
+ | | B3 || Open source || Free || ASCII || ASCII || None to Basic | ||
+ | |- | ||
+ | | BigSense and LtSense || Open source || Free || Binary || CSV, JSON, TXT, XML || Advanced | ||
+ | |- | ||
+ | | Cosm || || || || || | ||
+ | |- | ||
+ | | CUAHSI HIS || Open source || Free || ASCII || XML, WaterML || Intermediate | ||
+ | |- | ||
+ | | DataTurbine || Open source || Free || ASCII, Binary || ASCII, Binary || Advanced | ||
+ | |- | ||
+ | | EddyPro || Proprietary || Pay || Binary || ASCII, Binary || Intermediate | ||
+ | |- | ||
+ | | GCE Toolbox || Matlab is proprietary, Toolbox is open source || Matlab is pay, Toolbox is free || ASCII, Binary(?), database || ASCII, Binary(?), MAT, database || Intermediate to advanced | ||
+ | |- | ||
+ | | Hobolink (Onset) || Proprietary || Free || Proprietary || ASCII, Proprietary || None | ||
+ | |- | ||
+ | | Hoboware (Onset) || Proprietary || Pay || Proprietary || ASCII, Proprietary || None | ||
+ | |- | ||
+ | | Kepler || Open source || Free || ASCII, Binary || ASCII, Binary || Basic to Advanced | ||
+ | |- | ||
+ | | Lake Analyzer || Proprietary/Open source || Free || ASCII || ASCII || Basic | ||
+ | |- | ||
+ | | LoggerNet (Campbell) || Proprietary || Pay || Proprietary || ASCII, database || Intermediate | ||
+ | |- | ||
+ | | Nexsen's Technology || Proprietary || Pay || || || | ||
+ | |- | ||
+ | | Pegasus || || || || || | ||
+ | |- | ||
+ | | R || Open source || Free || ASCII, Binary, database || ASCII, Binary, database || Intermediate to Advanced | ||
+ | |- | ||
+ | | SAS || Proprietary || Pay || ASCII, Binary, database || ASCII, Binary, database || Intermediate to Advanced | ||
+ | |- | ||
+ | | Taverna || Open source || Free || || || Intermediate to Advanced | ||
+ | |- | ||
+ | | Vista Data Vision || Proprietary || Pay || ASCII || ASCII || Basic? | ||
+ | |- | ||
+ | | VizTrails || Open source || Free || ASCII || ASCII || Basic to Advanced | ||
+ | |- | ||
+ | | WaterML support || || || || || | ||
+ | |- | ||
+ | | WISKI || Proprietary || Pay || ASCII || ASCII || Advanced | ||
+ | |- | ||
+ | | YSI EcoNet || Proprietary || Pay || || || | ||
+ | |} | ||
+ | |||
+ | <br/> | ||
+ | |||
+ | {| class="wikitable sortable" | ||
+ | |- | ||
+ | ! Program !! Licensing !! Cost !! Input data format !! Export data format !! Level of expertise | ||
+ | |- | ||
+ | | Antelope Orb || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | Aquarius || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | ArcGIS || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | B3 || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | BigSense and LtSense || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | Cosm || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | CUAHSI HIS || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | DataTurbine || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | EddyPro || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | GCE Matlab || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | Onset Hobolink || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | Onset Hoboware || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | Kepler || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | Lake Analyzer || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | Campbell LoggerNet || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | Nexsen's Technology || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | Pegasus || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | R || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | SAS || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | Taverna || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | Vista Data Vision || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | VizTrails || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | WaterML support || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | WISKI || Example || Example || Example || Example || Example | ||
+ | |- | ||
+ | | YSI EcoNet || Example || Example || Example || Example || Example | ||
+ | |} |
Revision as of 17:34, March 24, 2014
back to EnviroSensing Cluster main page
Overview
Middleware are software packages and procedures that reside virtually between data collectors, such as automated sensors, and data ‘consumers’, such as data repositories, websites, or other software applications. Middleware can be used to perform tasks such as streaming data from data loggers to servers, archiving data, analyzing data, or generating visualizations.
Many middleware packages are available for developing a comprehensive, reliable, and cost-effective environmental information management system. Each middleware option can have a unique set of requirements or capabilities, and costs can vary widely. A single middleware package may be used if it includes all of the user requirements, or multiple middleware may be bundled into a data management system if they are compatible or interoperable with each other and the rest of the data collection and management system.
This section describes multiple middleware packages that are currently available, and provides examples of how different software and procedures are being used to collect, analyze, visualize, and disseminate sensor-supplied environmental data.
Introduction
There are multiple factors that may affect the choice, use, and performance of middleware. These factors may be classified according to a group’s research agenda, technological requirements, and personnel skill sets.
- Research Agenda: The research agenda of a group is a major determinant of the type of middleware system needed. A group focused on only one or a few narrowly focused research questions may need fewer types of sensors and consequently, fewer software modules may be adequate to streamline data processing from collection to the end goal. A team that investigates multiple questions spanning multiple research domains is likely to use more diverse and/or larger sets of sensors. There may not be a single middleware package that can meet all of the needs of a research group. In this case, multiple packages will need to be linked into a workflow.
- Technological Requirements: The technological requirements of a research program may vary from simple to complex. If the research can be done with sensors from a single, well-managed company, the proprietary software packaged with the purchased sensor network may be adequate for at least a major portion of the information management system. For example, for Campbell Scientific dataloggers, their “LoggerNet” software integrates communication, data download, display and graphics functions. However, some dataloggers and sensors (particularly innovative ones, custom-built), may need custom-written software. It is important to plan time and budgets for required software upgrades, licensing, additional packages, support, and maintenance. Systems that cost less in the outset may not always be cheaper over the long run. It is also important to consider how to best meet infrastructure and bandwidth requirements, while deploying middleware on a variety of servers or laptop computers in the field or lab setting. Depending on the data and hardware infrastructure characteristics, each middleware option can introduce benefits or drawbacks to the overall system functionality.
- Personnel Skills: Another key factor to consider is the skill set of the personnel. A complex data management system may require multiple people, each with a unique skill set such as database design, system architecture, web programming, etc. It is important to correctly identify each person’s skill set and role in data management tasks. It may also be necessary to plan for additional hires or job-training to addresses various scenarios and solutions, to identify appropriate salaries, and to budget enough time for software development and system administration. More details about the personnel roles and skills can be found in the “Roles and required skill sets“ section.
Methods
Middleware can be classified with respect to the functionality they provide, such as:
- Controlling instrumentation and data collection: Modules may be used to control sampling intervals, manage the event-triggered (burst) or continuous sampling regimes, communicate and transfer data between the instrumentation and other system components.
- Data monitoring, processing, and analysis: Modules may provide alarm management, perform automated QA/QC on data streams, or run derivative calculations including averages, aggregation and accumulation, data shifting and transformation, filtering of time series records with respect to the dates, value range, location, station/variable type, or other criteria.
- Export and publishing of data: Modules may provide functionality to export sensor data to different formats (e.g., ASCII, binary, or xml), different archives, make data discoverable through geospatial catalogues, or publish the data through web services.
- Data visualization: Modules may provide visualization (e.g., tables, graphs, sonograms) of geospatial and/or time series data from sensor arrays or workflow structures.
- Documentation: Modules may be used to document field events through paperless collection of field data, integrate sensor data and documentation (see sensor tracking & documentation section), or handle sensor calibration records.
- Other supported functionality: Modules may be used to provide access to external data (e.g., ODBC, JDBC, OLE DB), to connect or chain other middleware components, or to implement mobile applications.
Middleware can also be classified by software proprietary rights and whether they are considered applications or platforms. Accordingly, we can identify different groups of middleware:
- Proprietary data management applications and platforms
- Proprietary research applications
- Limited open source applications (free packages that can be used with proprietary solutions)
- Open source data management applications and platforms
- Open source research applications and programming languages
Some of the applications and platforms listed above are often identified as a software of choice for many different organizations. More details about each of these components are provided in the next section of this document.
Program | Licensing | Cost | Input data format | Export data format | Needed programming expertise |
---|---|---|---|---|---|
Antelope Orb | Proprietary | Pay | ASCII, Binary | ASCII, Binary | Advanced |
Aquarius | Proprietary | Pay | Advanced | ||
ArcGIS | Proprietary | Pay | ASCII, shapefiles | ASCII, shapefiles | Advanced |
B3 | Open source | Free | ASCII | ASCII | None to Basic |
BigSense and LtSense | Open source | Free | Binary | CSV, JSON, TXT, XML | Advanced |
Cosm | |||||
CUAHSI HIS | Open source | Free | ASCII | XML, WaterML | Intermediate |
DataTurbine | Open source | Free | ASCII, Binary | ASCII, Binary | Advanced |
EddyPro | Proprietary | Pay | Binary | ASCII, Binary | Intermediate |
GCE Toolbox | Matlab is proprietary, Toolbox is open source | Matlab is pay, Toolbox is free | ASCII, Binary(?), database | ASCII, Binary(?), MAT, database | Intermediate to advanced |
Hobolink (Onset) | Proprietary | Free | Proprietary | ASCII, Proprietary | None |
Hoboware (Onset) | Proprietary | Pay | Proprietary | ASCII, Proprietary | None |
Kepler | Open source | Free | ASCII, Binary | ASCII, Binary | Basic to Advanced |
Lake Analyzer | Proprietary/Open source | Free | ASCII | ASCII | Basic |
LoggerNet (Campbell) | Proprietary | Pay | Proprietary | ASCII, database | Intermediate |
Nexsen's Technology | Proprietary | Pay | |||
Pegasus | |||||
R | Open source | Free | ASCII, Binary, database | ASCII, Binary, database | Intermediate to Advanced |
SAS | Proprietary | Pay | ASCII, Binary, database | ASCII, Binary, database | Intermediate to Advanced |
Taverna | Open source | Free | Intermediate to Advanced | ||
Vista Data Vision | Proprietary | Pay | ASCII | ASCII | Basic? |
VizTrails | Open source | Free | ASCII | ASCII | Basic to Advanced |
WaterML support | |||||
WISKI | Proprietary | Pay | ASCII | ASCII | Advanced |
YSI EcoNet | Proprietary | Pay |
Program | Licensing | Cost | Input data format | Export data format | Level of expertise |
---|---|---|---|---|---|
Antelope Orb | Example | Example | Example | Example | Example |
Aquarius | Example | Example | Example | Example | Example |
ArcGIS | Example | Example | Example | Example | Example |
B3 | Example | Example | Example | Example | Example |
BigSense and LtSense | Example | Example | Example | Example | Example |
Cosm | Example | Example | Example | Example | Example |
CUAHSI HIS | Example | Example | Example | Example | Example |
DataTurbine | Example | Example | Example | Example | Example |
EddyPro | Example | Example | Example | Example | Example |
GCE Matlab | Example | Example | Example | Example | Example |
Onset Hobolink | Example | Example | Example | Example | Example |
Onset Hoboware | Example | Example | Example | Example | Example |
Kepler | Example | Example | Example | Example | Example |
Lake Analyzer | Example | Example | Example | Example | Example |
Campbell LoggerNet | Example | Example | Example | Example | Example |
Nexsen's Technology | Example | Example | Example | Example | Example |
Pegasus | Example | Example | Example | Example | Example |
R | Example | Example | Example | Example | Example |
SAS | Example | Example | Example | Example | Example |
Taverna | Example | Example | Example | Example | Example |
Vista Data Vision | Example | Example | Example | Example | Example |
VizTrails | Example | Example | Example | Example | Example |
WaterML support | Example | Example | Example | Example | Example |
WISKI | Example | Example | Example | Example | Example |
YSI EcoNet | Example | Example | Example | Example | Example |