Note: This page was originally sourced from our Blogger page: http://ncarrda.blogspot.com/2016/01/ds7350-ncep-gdas-satellite-data-extended.html

New and improved, with more satellites!

Visualization of DMSP F17 WV courtesy of REMSS.

A chance encounter with members of the WRF Data Assimilation (WRFDA) team led me down a rabbit hole to improve the usefulness of ds735.0 for users of WRFDA.

In 2009, RDA began archiving satellite data products that were ingested into GDAS.  (We backfilled the data to 2004 or 2005.)  Since then, the number and types of satellite data products ingested into GDAS has grown.  It was time for ds735.0 to keep up.


Beginning in October 2015, the number of types of satellite data products archived daily in ds735.0 expanded from 4 to 24.  We've renamed ds735.0 from NCEP GDAS Satellite Radiances to NCEP GDAS Satellite Data to better reflect the contents.  Check out the new full data product list for ds735.0.

In addition to data encoded in NCEP BUFR format, we've added a few plain text files, including TCVITALS (tropical cyclone center coordinates); and ABIAS and ABIAS_PC bias correction parameters.  This will help users of Community Gridpoint Statistical Interpolation System (GSI) and WRF Four-Dimensional Data Assimilation (FDDA).

Researchers who perform data denial and Observation System Simulation Experiments (OSSE) or study areas where conventional data is scarce may also find ds735.0 useful.

Satellites, and the sensors on them, come and go.  Not every type of data is available every day.

Some file descriptions found on the internet are out-dated and no longer reflect the current parameter units. When NCEP switched from assimilating TB to radiances, the GDAS file contents changed accordingly.   Documentation on the web may not have been updated to reflect the change.  Thus, trust the contents of each file, not old information around the web.

(Current versions of GSI and WRFDA software will read the BUFR codes and process the data correctly, regardless of the units.)

What if you need satellite data prior to late 2015, need a consistent satellite data retrieval algorithm for a longer time series, or are doing a retrospective study?

Use ds099.0 NCEP Climate Forecast System Initial Conditions, Full Ingest Data (1979-ongoing). This covers the entire modern satellite era, but is updated irregularly, with a time lag.  These satellite data products are processed with consistent software over the time period and with more precise satellite orbit information.

If climate accuracy is important, use ds099.0.

If you don't need timeliness, then use ds099.0 (which only goes up through 2014 for now).

If you are running WRD DA, WRF FDDA, or GSI in near real-time, then ds735.0 is fine.

Aside about data curation:

There is more to expanding or adding to a dataset than just writing a script to start collecting data off the NCEP servers; data curation is much more than collecting buckets of data.

In addition to the automated metadata collection software that RDA runs on EVERY file we archive, I opened up and manually examined the contents for a few files of each type. In that process, I discovered that NCAR CISL needed to update to the new version of NCEP BUFRLIB so that we could accurately read newer types of NCEP satellite data products.  (Old versions will give erroneous values.)

While trying to understand each data file, I corresponded with, or met in person with, NCEP staff to go over each field in each file that I didn't understand.  This required reading a great deal of documentation so that I can answer RDA user questions accurately and quickly.

The RDA assigns data sets to data specialists by expertise.  Oceanographic datasets are assigned to oceanographers.  Meteorological datasets are assigned to meteorologists.  Prior to working at the RDA,  I (Grace Peng) worked in Satellite Calibration and Validation (Cal/Val) for DMSP and other meteorological satellite programs.  Prior to that, I earned a PhD in Chemical Physics, where I wrote models and compared model output with physical measurements.

If you want to just grab the data and run your models, that's fine.  If you need expert data consulting, we can do that, too. rdahelp@ucar.edu