1. TIGGE Data Archive and Access System at NCAR
Doug Schuster
Dave Stepaniak
Hannah Wilcox

2. Topic Outline Archive Summary Data Management System
NCAR’s portal and access

3. Archive Summary Ingest ~245 GB/day from 11 data providers. Online Data
Period, most recent two weeks
~ 3.5 TB , public products
~ 2.5 TB, data preparation, sub-setting, TIGGE database
Offline Data
Full period of record
~ 180 TB, NCAR Mass Storage System
Provide access to both

4. TIGGE Archive Centers and Data Providers
UKMO
CMC
CMA
ECMWF
MeteoFrance
NCAR
NCEP
JMA
NCDC
KMA
IDD/LDM
HTTP
FTP
Archive Centre
CPTEC
Current Data Provider
BoM
IDD/LDM
Internet Data Distribution / Local Data Manager
Commodity internet application to send and receive data

5. Data Management Challenges
Ingest 245 GB/day from 11 data providers.
Ensure data completeness.
Build archive forecast files from individual GRIB-2 messages.
Collect archive forecast file and GRIB-2 message level metadata.
Reformat and distribute 15 GB/day of NCEP data to partner archive centers.
Service resend requests for missing NCEP fields.
Service archive resend requests for full NCEP forecast cycles from SAN and MSS.
Ensure all data copied from SAN to MSS and registered in RDADB.
Provide data distribution through TIGGE portal and RDA server.
24 x 7 operations.

6. NCAR TIGGE Data Management System
Data Ingest
Data Subsetting and Distribution
Metadata Collection

7. Data Distribution Challenges
Anomalies between centers
Native horizontal resolution
Number of ensemble members
Number of forecast initialization times (1, 2, 4x daily)
Forecast length
Number of fields provided
Internal file compression (e.g. jpg) was not specified
Personally, I’ve gained an appreciation for this.
Influence on Research
User interface must inform users - commonalities constrain research

8. Get Forecast Data Two User Interfaces NCAR online file archive
Selection options (Portal or RDA)
Center(s)
Date
File type (sl, pl, etc)
Initialization time
Forecast length
User customized files
Selection options (Portal)
Same as for files, plus
Parameter Subsets
Grid Interpolation
Spatial subsets
Formats, GRIB2, NetCDF
Real Time
Delayed Mode
Download Options
Point and click using browser, one file at a time
Script to run on local machine
User and password encrypted ‘wget’ commands
background process to access all files

9. Summary of Data Providers, Oct ‘08