Example CF-NetCDF for Satellite

Back to WCS Access to netCDF Files

A Real life example how to download and store satellite data.

Total Ozone Mapping Spectrometer (TOMS) satellite

This data has been collected by three different satellites from 1978 to present. It is downloadable in text files, as a linear rectangular grid.

Nimbus satellite:

• Data download site. ftp://toms.gsfc.nasa.gov/pub/omi/data/aerosol/
• First 1978-11-01: ftp://toms.gsfc.nasa.gov/pub/nimbus7/data/aerosol/Y1978/L3_aersl_n7t_19781101.txt
• Last 1993-05-06: ftp://toms.gsfc.nasa.gov/pub/nimbus7/data/aerosol/Y1993/L3_aersl_n7t_19930506.txt

Grid size 288 * 180

EPTOMS satellite:

• Data download site ftp://toms.gsfc.nasa.gov/pub/eptoms/data/aerosol/
• First 1996-07-22: ftp://toms.gsfc.nasa.gov/pub/eptoms/data/aerosol/Y1996/L3_aersl_ept_19960722.txt
• Last 2007-12-31: ftp://toms.gsfc.nasa.gov/pub/eptoms/data/aerosol/Y2005/L3_aersl_ept_20051231.txt

Grid size 288 * 180

As you can see, there is a gap between 1993-05-06 and 1996-07-22.

OMI satellite:

• data download site: ftp://toms.gsfc.nasa.gov/pub/omi/data/aerosol/
• First 2004-10-01: ftp://toms.gsfc.nasa.gov/pub/omi/data/aerosol/Y2004/L3_aersl_omi_20041001.txt
• Year 2011, growing: ftp://toms.gsfc.nasa.gov/pub/omi/data/aerosol/Y2011

Grid size 360 * 180

The longitude size went from 288 to 360

Creating an Empty CF-NetCDF File

Creating NetCDF files programmatically is doable, but harder than it should be. It's much easier to create a high level, text version of the file and use a tool to turn it into a real binary NetCDF file. We use NetCDF Markup Langunage, NCML. The NCML language is similar to the CDL language. Since NCML is XML based, it's more verbose than the domain-specific CDL. But there are few tools, that understand CDL, whereas every programming language has an XML package. Therefore NCML is used.

Contents of the Definition File

A NetCDF file with CF convention contains following things:

• Global Attributes
• Dimensions
• Dimension Variables
• Data Variables

CF Conventions are not enforced by NCML. A person who designs an empty CF-NetCDF must know the basics of the conventions.

The real AerosolIndex.ncml contains the following declarations.

Global Attributes

   Conventions = "CF-1.0"
   title = "NASA TOMS Project"
   comment = "NASA Total Ozone Mapping Spectrometer Project"
   keywords = "TimeRes:Day, DataSet:TOMS_AI_G"

The file follows CF conventions version 1.0. Attributes title, comment and keywords are published by the WCS as coverage title, description and keywords. Other attributes can exists without limits.

Dimensions and Dimension Variables

The AerosolIndex is a three-dimensional variable. Two of the dimensions are fixed: Latitude and Longitude.

The NetCDF dimensions contain only the dimension length. That's why CF convention associates a dimensional variable with each.

For latitude, the dimension and variable name is lat for both. The data in the variable [-89.5, -88.5 ... 88.5, 89.5] enumerates latitude of each index in the dimension. The variable needs also standard attributes:

   standard_name = "latitude"
   long_name = "latitude"
   units = "degrees_north"
   axis = "Y"

Similar for longitude dimension.

Time dimension is a little different. It is unlimited, meaning that new time slices can be added to the data. The time dimension is initially length 0 and the time variable is an integer, counting days from the first data measurement time:

   standard_name = "time"
   long_name = "time"
   units = "days since 1978-11-01"
   axis = "T"

So for 1978-11-01 the time variable value is "0" and for 1978-11-02 it is "1".

It is advisable to use integers as data type. If you have hourly data, don't use "days since 2000-01-01" and float datatype, since days = hours/24 does not have a nice decimal digit representation. You'll get rounding errors etc.

Data Variables

The data variable is type float, dimensions (time, lat, lon). Time must be the first dimension, since it is unlimited. Order (time, lat, lon) is recommended by CF conventions.

   long_name = "Aerosol Index"
   units = "fraction"
   _FillValue = NaN
   missing_value = NaN

Missing data is most naturally presented by Not a Number, NaN. Using values like -999 is possible, but carries the danger of causing errors in calculations.

Creating NetCDF from NCML

By running the following python program AI_create.py the empty netcdf cube is done. Any other NCML tool could be used.

Downloading the Data Files

It's possible to download a data file and directly append it into the netcdf cube, without storing any temporary files. This this approach has the drawback, that if anything goes wrong, you have to download everything again. Maybe you want to do some data processing, and now redoing the whole process is inconvenient due to long download times. With current disk space prices, it's better to first download the files and store them locally. Since these are just text files, it would be possible to just use your browser to download them. While this works great when the dataset is updated max four times a year. If the data is updated daily, it's very nice to have a script you can call at will to download a file.

The module AI_ftp.py does just that. It stores the file locally, and retrieves only new files. It can be also used as a library, it allows downloading a file at will.

Compile Data Files into CF-NetCDF

Compile the Text File into an rectangular array

This is the part that requires most of the programming in the whole system. Since the data is in text files, the script needs to read the text, parse the numbers and assign into the time slice array. You cannot create one text file reading routine that can read any format. So this code cannot be directly applied to anything but these AerosolIndex files, you have to write your own reader for your data.

In many cases, if the data is in daily netcdf files, this parsing can be omitted since reading arrays from netcdf files is trivial.

The data parser can serve as sample code.

Append the time slice

This is again the easy part: since everything is now standardized, the library interface can be easy to use. AI_update.py is short and straightforward.

Client-side browser view of NASA TOMS WCS

Map Query for large area and single time instance: Identifier=AerosolIndex RangeSubset=AI TimeSequence=2011-01-01 BoundingBox=-179.5,-89.5,179.5,89.5] . The actual WCS getCoverage call is:

http://data1.datafed.net:8080/NASA_TOMS?Service=WCS&Version=1.1.2&Request=GetCoverage&Identifier=AerosolIndex&Format=image/netcdf&Store=true&TimeSequence=2011-01-01&RangeSubset=AI&BoundingBox=-179.5,-89.5,179.5,89.5,urn:ogc:def:crs:OGC:2:84

Time Series Query for a time range and a single location: Identifier=AerosolIndex RangeSubset=AI TimeSequence=2010-01-01/2011-07-05/P1D . The actual WCS getCoverage call is:

http://data1.datafed.net:8080/NASA_TOMS?Service=WCS&Version=1.1.2&Request=GetCoverage&Identifier=AerosolIndex&Format=image/netcdf&Store=true&TimeSequence=2010-01-01/2011-07-05&RangeSubset=AI&BoundingBox=-10.5,4.5,-10.5,4.5,urn:ogc:def:crs:OGC:2:84

TimeSequence=2005-06-01/2011-09-01/PT1H has time_min/time_max/periodicity. ISO 8601 definition, PT1H is hourly, P1D is daily.

Final Product

Core Catalog

Browse Data

Service Page