Sensor Data Acquisition

back to EnviroSensing Cluster main page

Overview

Traditionally, environmental sensor data from remote field sites were manually retrieved during infrequent site visits. However, with today's technology, these data can now be acquired in real-time. Indeed, there are several methods of automating data acquisition from remote sites, but there is insufficient knowledge among the environmental sensor community about their availability and functionality. Moreover, there are several factors that should be taken into consideration when choosing a remote data acquisition method, including desired data collection frequency, bandwidth requirements, hardware and network protocols, line-of-sight, power consumption, security, reliability and redundancy, expertise, and budget. Here, we provide an overview of these methods and recommend best practices for their implementation.

Introduction

The classic method of acquiring environmental sensor data from remote field sites involves routine technician site visits, in which s/he connects a laptop to a datalogger, an electronic device that records sensor data over time, and manually downloads data recorded since the last site visit. Once the technician returns to the lab, s/he is then responsible for manually uploading these data to a server for later processing and archival.

While manual acquisition methods are generally effective, there are many reasons to automate environmental sensor data acquisition. For instance, if the site is not visited frequently enough, the datalogger memory can become full and depending on how the datalogger is programmed, sensor data will either overwrite itself or stop recording entirely. This situation often occurs at remote sites that become periodically inaccessible due to environmental conditions, such as heavy winter snow pack. Second, the burden of responsibility for not only the successful retrieval of the sensor data, but also the subsequent upload to a server for safekeeping, lies solely on the technician. Moreover, with any instrumented site, there is the inherent potential for sensor or power failure. Automated data acquisition systems allow technicians to learn of such issues prior to visiting the field site, reducing the potential for data loss. Finally, automated data acquisition methods save hundreds of person hours and vehicle miles that would have otherwise been spent manually acquiring data or troubleshooting unanticipated problems, thus improving the overall quality of the data.

Bidirectional communication methods have the additional advantages of allowing technicians to remotely change system settings, test configurations, and troubleshoot problems. These methods also open the field to a wide variety of devices that may be deployed at a remote field site, such as controllable cameras, on-site wireless hotspots, and IP-enabled control or automation equipment.

Considerations

The decision of which sensor data acquisition method to use at a given site requires the careful consideration of many factors, for which we provide an overview here.

Collection Frequency

What is the desired collection frequency? How important is real-time accessibility? For instance, the data could be retrieved in near real-time (every few minutes to every few hours) or just once or twice per day. High frequency datasets or images should be collected more frequently.

Bandwidth

Bandwidth can be an important consideration, particularly when high frequency data are being collected. Will cameras be utilized at the site? Where is broadband point of presence (POP) located? Does equipment work with required bandwidth? More frequent collection intervals require less bandwidth per transmission are are recommended for high frequency datasets or for images.

Protocols

Hardware

Many dataloggers only have serial (RS232) ports, therefore requiring a serial-to-ethernet converter to interface with automated acquisition instrumentation. USB.

Network

Private vs public IP networks.

Line-of-sight

Evaluation of environment, topography, and vegetation. Can be initially determined using LOS calculators, which use DEM models, but must be ground truthed. Often requires a repeater infrastructure. Choosing repeater locations involves many of the same considerations for choosing site selection. Distance to repeater is a factor. Automated sensor data acquisition methods require many of the same site selection considerations discussed in Sensor Site and Platform Selection, particularly when selecting repeater sites.

Power

How important is real-time accessibility? (e.g., what is desired collection frequency?). What are the transmission type power requirements, onsite buffer size. Redundancy is preferred, especially in very remote sites. If power is disrupted, will system resume operations?

Security

Physical Security

see Sensor, Site, and Platform Selection

Network Security

Encryption keys, VPN

Reliability and Redundancy

of transmission mode and of equipment

Expertise

Plug-n-play vs.

Budget

Costs of implementing a data acquisition and transmission method depend on existing infrastructure, initial setup costs including personnel, personnel costs, specifically technician maintenance, and recurring costs, such as monthly recurring costs with cellular transmission.

Methods

There are three general categories of remote sensor data acquisition methods: manual, unidirectional telemetry, and bidirectional telemetry. Each has advantages and disadvantages in terms of infrastructure, cost, reliability, required expertise, and power consumption.

Manual

This method involves scheduled visits to the site by a field technician, who uses a serial-to-computer connection and/or flash memory transfer of environmental sensor data to their laptop or similar device. Upon returning from the field, the technician is responsible for manually uploading these data to a server. This acquisition method is simple and may be the only option when site instrumentation generates large data files. However, this method provides no real-time data access and therefore, no knowledge of instrumentation failures. Moreover, the reliability of this method is completely dependent on the technician.

Unidirectional

Unidirectional sensor data acquisition methods involve regularly scheduled wireless data transmission from a remote site to a server, with no offsite ability to control or change sensor settings. These include...

Geostationary Operational Environmental Satellite (GOES)

This method is preferred in very remote and potentially rugged areas where other automated transmission methods would not work. While it does not require line-of-site to a repeater like most other transmission methods, it does require a view to the southern sky. Additionally, the GOES method has a low power requirement. However, GOES has several disadvantages, including a high initial investment (<$5K) and requires training and licensing. Moreover, less than 100 values can be transferred per hour, making it disadvantageous for sites that sample at high frequencies.

Data transfer speed for GOES systems is typically limited to 1200 bits per second with 10 second transfer assignments occurring once every hour. During each 10 second period, one can transfer up to 1500 bytes of data (12,000 bits / 8) including the 53 byte GOES header string. In other words, maximum 1447 bytes with time stamps and measured values can be transferred to the satellite during one transmission interval. Most often, GOES messages are organized in a time ordered format similar to the following example:

 0105E59013190131824G30+1NN196WXW00517
 0 13:00:00 23.7,43,5,245,-55.1,5,245,23.7,23.7,12.8
 1 12:30:00 23.7,43,-55.1,204,1011.09,0.000,0.0,24.7,0.270,-0.456,-0.997,-0.416,-2.687,23.5,0.00,214.81,0.00,5,245
 1 12:45:00 23.7,43,-55.1,204,1011.11,0.000,0.0,24.7,0.249,-0.468,-0.994,-0.436,-2.650,23.5,0.00,214.82,0.00,5,245

Here, first line represents the GOES header string that includes the address, date and UTC time of the transfer (13:18:24), signal information, satellite information, message length and some other characters. In the example above, the lines that follow carry the time stamp and value information from the sensor sets 0 and 1. As the length of each character in the sensor set string is 1 byte, we can see that our GOES message has approximately 280 bytes used from 1447 bytes that are a theoretical maximum for the transfer. However, in order to accommodate the possible differences between the station sending time, decoders, and scheduled reception time, we never want to reach this value.

Prospective users of the GOES system must fill out the System Use Agreement (SUA) form and, upon approval, receive and sign the Memorandum of Agreement (MOA) from the NOAA's Satellite and Information Service (NESDIS). After the MOA is approved, NESDIS will issue a channel assignment and an ID address code to the applying organization. Non-U.S. government and research organizations must be sponsored by a U.S. government agency in order to apply for this permission. Upon approval, all users must purchase equipment that has been certified to be compatible with the GOES Data Collection System. As of May 2013, GOES transmitters must conform to the certification standards version 2 (also known as CS2). This change was implemented to double the number of GOES channels on the same bandwidth. As a result, old GOES transmitters that are only compatible with the CS1 standard cannot be used for new NESDIS assignments. For assignments obtained prior to May 2012, CS1 transmitters will be supported until 2023. If you consider buying the used equipment for GEOS transmission, make sure the transmitters are compliant with the CS2 standard.

Meteor-burst radio

Like GOES, this method does not require line-of-sight and has a low power requirement. However, it requires a large antenna, arrangement of service, and has a very slow transmission rate.

Iridium Satellite service

Iridium provides the only complete global satellite coverage. The new Iridium Pilot is available until 2016. The next generation of Iridium is expected to be implemented around that time frame. The Pilot is very easy to install and maintain with a waterproof body and USB interface. With this simple interface a laptop can be connected and surfing the web within minutes. Recently, the cost has become more affordable with a per data usage cost structure. Since Iridium operates in the L band it is nearly impervious to weather. Iridium is used primarily for marine communication.

Bidirectional

This method involves bidirectional (and typically wireless) transmission of data from a remote site to a server, with the ability to modify datalogger programs and/or sensor settings remotely. These methods generally require line-of-sight and security considerations (both network and physical). Sometimes, can be purchased from an Internet Service Provider (ISP) if there is commercial coverage in the area, or can be manually installed in remote areas. Often, connectivity can be extended to computers onsite. Combination of several methods may be required in certain situations.

ISM band radio network

(unlicensed, 900 MHz, 2.4 GHz, 5.x GHz): The ISM band radios are commonly referred to as Wi-Fi radios and come in a variety of frequencies. This method has many advantages in that it is nonproprietary, has no recurring costs, uses public radio frequencies, allows transmission of large datasets, utilizes inexpensive hardware, is not restricted to a single vendor or device type, and has increasing compatibility with many devices. However, it requires line-of-site, a network interface on loggers and devices, and basic to advanced network administration skills. These radios are also subject to interference, particularly in more populated areas, and have higher power requirements than most other transmission methods.

Cellular

This method has prolific coverage and minimal ongoing maintenance. However, it requires a reliable cellular network be present and comes with monthly recurring costs. Occasionally a contract may be required unless can be negotiated through university or organization.

Vendor-specific radio network

For example, Freewave…

Satellite internet

This method is similar to GOES in many respects. It has slow uplink speeds and high latency, requires a subscription, and on-site vendor setup is required.

Licensed radio

This method is expensive.

Mesh Networks

To be completed…

Wired

While all methods discussed utilize wireless transmission protocols, wired bidirectional transmission may be possible via in-ground or aerial copper or fiber optics.

Best Practices

• Think about data acquisition as part of site design. It is more expensive to add telemetry to a preexisting site than to integrate with initial site construction. Make sure to include acquisition method power consumption in the site power budget, or a separate power system will be required.
• Use software tools with radio or a handheld spectrum analyzer to survey RF conditions on-site. For instance, urban areas are typically noisier with respect to RF interference, and for Wi-Fi transmission methods, 5 Ghz frequencies are preferred.
• Use a bidirectional transmission method to provide more control and flexibility.
• Over-engineer power system, especially when powering repeaters and other sites in hard to reach areas.
• Use equipment that can conserve power (sleep mode)
• Provide adequate local storage for disrupted transmissions. Adequate “off logger” local storage is recommended to avoid losing data when/if logger is reset.
• Provide redundancy, such that when one link goes down, the site is still remotely accessible. This is related to network architecture planning - multiple geographic/hardware paths along backhaul routes to field hubs is highly desirable. Examples include: parallel backhauls, multiple internet points of access, "failover" paths. Having a "back door" into the network, even over reduced speed links, can allow a tech to remotely troubleshoot problems on the main links.
• Standardize transmission protocol across all sites to provide easier network management.
• Match radio band, power, antenna, and bandwidth to application. For instance, when a site generates high frequency sensor data, high bandwidth and high data collection frequency are recommended.
• Use a narrow bandwidth for your RF devices/coordinate frequencies between radio systems
• Thoroughly document all site coordinates, IP addresses, maps, radio azimuth, zenith.
• When using an IP based acquisition method, use public IP addresses for easier remote management of devices.