1. Development and Futures of Research Data Archives
Steven Worley
Data Support Section, SCD
31 January 2006
1/1/2019

2. Dataset Development @ SCD TIGGE
Topics
Data Access Points
Reanalyses
Output Resources
Dataset Development @ SCD
TIGGE
1/1/2019

3. Data Access Points for Research Data Archives (RDA)
MSS
What
All RDA data and backups
Access
Everyone with SCD computing account
What
Much RDA data, all metadata (complete)
Access
Public, traditional service (web,ftp,delayed mode)
RDA
Server
What
Some RDA data, complete RDA catalog
Many other UCAR/NCAR catalogs and data
Access
Public, interactive and interoperable services
CDP
Server
1/1/2019

4. Reanalyses - output resources
Major Extant Reanalyses
NCEP/NCAR,
28 sigma, 11 theta, 17 plvl, 2.5º, 1.875º Gaussian, 6hrly
NCEP/DOE AMIP-II,
28 sigma, 17 plvl, 2.5º, 1.875º Gaussian, 6hrly
NCEP North America Regional,
Eta/NAM, NOAH, 45 sigma, 29 plvl, 32km, 3hrly
ERA-40,
60 sigma, T159, 2.5º, 22 plvl, 15 theta, 125km Gaussian, 6hrly
Focused efforts
1/1/2019

5. NARR Domain
1/1/2019

6. Reanalyses - output resources
Future Reanalyses
JRA-25 by JMA,
40L, T106
completed, negotiating to get HR copy
MERRA by NASA, current date
1º, hydrological cycle focus
Low resolution testing - now, start July 2006
ERA-Interim, 1989  CDAS
T255, 91L, 4DVAR, just beginning now
NOAA ESRL 20th Century
in 2006, then
Based on SLP only
1/1/2019

7. Reanalyses - Dataset Development @ SCD
Feedback records from ERA-40
AMS IIPS P2.12, Doug Schuster
Observed data + QC metadata + variational analysis system (VAS) metadata
E.g observation - model field departures
Uses
Observations with added QC
Deeper understanding of reanalysis by examining departures
What’s been done
Convert BUFR to ASCII and separate surface and UA archives
All data online + software + documentation + web/ftp download
Temporal and spatial subset extraction by web interface request
1/1/2019

8. Reanalyses - Dataset Development @ SCD
1/1/2019

9. Reanalyses - Dataset Development @ SCD
ERA-40 Derived Products
Note in GRIB: T85 = 4 TB, T106 = 6 TB
T85 Mlvl vertical coordinate error in netCDF
T85 6-hourly 23.plvl has subsetting interface on RDA Server
T85 Monthly 23.plvl, surface and single level has subsetting and files on CDP
1/1/2019

10. Reanalyses - Dataset Development @ SCD
ERA-40 Services from RDA Server
24 ERA-40 standard products
(check mark = online, RDA web and FTP file level access)
Model Resolution Daily (4x) Analysis Fields
Model Resolution Daily (4x) Forecast Fields
Model Resolution Monthly Mean Fields
Model Resolution Monthly Mean Forecast Fields
2.5 degree Daily (4x) Analysis Fields
2.5 degree Monthly Mean Fields
Ocean Wave Fields (anal. & forecasts)
1/1/2019

11. Reanalyses - Dataset Development @ SCD
ERA-40 Services from RDA Server
Subsetting Service (variable, level, and time)
By or web form request (delayed mode)
Pressure level, surface and single levels
6-hourly, 2.5°
6-hourly, model resolution
By automated processing
6-hourly, T85, 23 pressure levels
Metadatabase to get HP access to GRIB files
Output formats, GRIB or CF-conforming netCDF
Spatial constraints to be added in the future
1/1/2019

12. Reanalyses - Dataset Development @ SCD
ERA-40 Services from CDP Server
Motivation: Provide an additional data access channel by exploiting netCDF formatted data in an advanced portal environment
Data Product - Monthly, 2.5°, 23 pressure level, surface and single level
Features:
Real-time .nc file download
Variable selection (multiple,OK)
Spatial selection
Time selection
Level selection
Output - netCDF
1/1/2019

13. Improving observational data resources
Objective:
Compile and provide access to the most complete and highest quality data collections.
What we seek?
Historical data not yet in the archives
Current day data streams to extend the archives
How?
Ask questions, investigate leads, request copies, and encourage free and open data sharing using the RDA resources as exchange collateral
Quality check the new data received, always preserve the original archive
Compare with extant archives to gauge potential improvement
Why?
Better data assets to support direct atmospheric research, and future reanalyses
1/1/2019

14. Improving observational data resources, Examples
29K profiles
Stewardship - needs ship track check, integrated with other UA data
Little used - not NNR, yes ERA-40
1/1/2019

15. Enhancements relative to GTS data receipts
Increased coverage
10-20 years more data
New data in remote areas
1/1/2019

16. Improving observational data resources, Examples
ICOADS, global surface marine observations
New Release, 2.2, coverage
New Publication Available (Worley et al, 2005, IJC)
Collaborating partners, NOAA ESRL and NOAA NCDC
Access
Full data set online
User subset request interface
select time, space, variables, and filtering
1/1/2019

17. TIGGE (THORPEX Interactive Grand Global Ensemble) THORPEX (WMO World Weather Research Program, THe Observing system Research and Predictability Experiment)
Objective: Create of a database of ensemble forecasts from world-wide NWP centers
NWP Participants
NCEP, CMC, BMRC, ECMWF, JMA, CMA, CPTEC, UKMO and KMA
TIGGE Archive Centers
NCAR (SCD), ECMWF and CMA
Project Features
GRIB2 data format
@ NCAR, user access through CDP and the MSS
Receive HR native grids
Up to 200 GB/day
1/1/2019

18. TIGGE IDD/LDM transfer rates, 17-19 January 2006
TIGGE - Status
Very early stages
NCAR and ECMWF hope to initiate operational exchange March-April 2006
ECMWF/CPTEC/NCAR/Unidata tuned TIGGE IDD/LDM, 100 GB in 12 hours
Optimized the TCP/IP window size
Enlarged the LDM receive and send buffer size
Red = from NCAR
Blue = from ECMWF
Green = from CPTEC
TIGGE IDD/LDM transfer rates, January 2006
After tuning, +8 GB/hr
MB/hr
Before tuning, ~3 GB/hr
1/1/2019

19. We have and will further expand data access
Wrap Up
We have and will further expand data access
Especially on RDA and CDP Servers
Strongly linked - no need to know our infrastructure
Reanalyses
Continue to develop what we have
More on the horizon
Big on data curation and stewardship
TIGGE - new resource for those interested in ensemble weather forecasting
1/1/2019

20. Thanks RDA Server CDP Server http://dss.ucar.edu/
1/1/2019