Sensor Data Quality

• Sensor Quality Assurance (QA) (preventative)
  • Maintenance and calibration schedule
  • Sensor redundancy
  • Continuous monitoring and evaluating
• Near real-time processing and Quality Control (QC) on data streams
  • Timestamp integrity - sequential, fixed intervals
  • Range checks - extreme values based on sensor specifications, or based on historical seasonal ranges
• Other methods for near real-time or post-streaming QC
  • Manual "eyes-on" checks
    • graphics software for displaying near real time data streams
  • Plausibility checks, e.g., TMAX-TMIN>0
  • Variance checks - outlier detection, e.g., Sigma (standard deviation) or Delta/step (difference of subsequent pairs)
  • Sensor drift checks - integration of several data streams (correlations or conditioning with redundant or nearby sensors)
  • Persistence checks - repeating values
• Documentation of methods applied to data
  • Identify streaming QC methods, thresholds, assumptions
  • If no QC, that should be made clear too
• Data qualifiers - data flags
  • Many vocabularies in use - standardize or cross-walk?
  • Rich vocabulary of fine-grained flags for streaming data – intended to guide local review
  • Simpler vocabulary of flags for “final” data for public consumption
  • Define new data columns in lieu of data qualifiers?
    • Method shifts, sensor changes
    • Place key documentation as close to data value as possible