Where's my data?
July 7, 2015
Posted by: RDA Team
Note: This page was originally sourced from our Blogger page: http://ncarrda.blogspot.com/2015/07/wheres-my-data.htmlThe RDA considers the ecosystem that data resides in when deciding what to archive. For instance, we archive both NCEP GFS and FNL analyses*; they are part of the NCEP Global Data Assimilation System (GDAS) system.
- List of RDA datasets related to GDAS
- ds351.0 NCEP ADP Global Upper Air Observational Weather Data, October 1999 - continuing
- ds461.0 NCEP ADP Global Surface Observational Weather Data, October 1999 - continuing
- ds735.0 NCEP GDAS Satellite Radiance Data
We pull data from national centers such as NCEP and monitor the data flow. In late June and early July, 2015, we experienced a drop in ds735.0 expected file sizes of the data. After some investigation, we traced it to planned network maintenance at NCEP that resulted in a reduction of satellite data that were ingested in the GDAS update cycle.
The investigation turned up some insights that I would like to share. Do try this at home.
- View the ds735.0 Web Server Holdings for the AIRS subset. Notice that the files are smaller than expected for June 22 and July 1, 2015 (all dates and times UTC).
- Click on the smaller tar files to download them one at a time. Examination of their contents shows an absence of the 06Z file and a short 12Z file:
- View the NCEP Real Time Data Monitoring System page.
- Scroll down to 'Model Data Dump Tables' and select 06z under GDS.
- You should see a table labeled 'GDS Dump Data Counts Time Series Plots'. Select airsev and you should see this plot showing the data drops.
- Now go back and try 12z and see the missing days for this cycle.
- Now select acars (Aircraft Communications Addressing and Reporting System), which is part of ds335.0. Those with NCAR internal access accounts can view the inventory of corresponding big_endian/gdas.aircar.tHHz.YYYYMMDD.bufr.be files.
What's causing these data drops? Leave a comment with your answer. The first correct answer will get a NCAR photo postcard autographed by the RDA data specialist team.
We have a winner in less than 2 hours. That must have been too easy. Now, I will send out another NCAR postcard to the first person to explain the periodicity in the Rapid Update ACARS data ingest.