TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox

Background on TIGGE WMO World Weather Research Programme THORPEX  THe Observing system Research and Predictability EXperiment  THORPEX Interactive Global Grand Ensemble (TIGGE) Archive supports research  Grand Ensemble = multiple NWP centers ensembles are combined (an ensemble of ensembles)  10 international NWP Centers contributing to TIGGE

Background on TIGGE Three mirrored archive centers  NCAR  ECMWF  CMA {Shared System Development!}  Daily Data Flow Metrics  245 GB  1.6 Million gridded fields as separate data packets

Data Receipt Archive Centre Current Data Provider NCAR NCEP CMC UKMO ECMWF MeteoFrance JMA KMA CMA BoM CPTEC IDD/LDM HTTP FTP Unidata IDD/LDM Internet Data Distribution / Local Data Manager Commodity internet application to send and receive data NCDC

Major Challenges Insure data receipt, build complete archive  Collate data fields into different files types  Exchange manifest files as part of IDD/LDM data transmission between Archive centers  Verify send, receive  Automated resend requests for missing fields  Harvest and hold metadata in MySQL DB’s  Identify location of every field in file set  Updated often  Critical for users interface and background data processing

Major Challenges  Limited online storage – 4 TB, ≅ 2.0 weeks temporal coverage  Full archive on NCAR Mass Storage System  User registration and metrics required  Accept data policy; for research and education only  48 hour delay from forecast initialization time

Major Challenges  Access system must accurately display what data are available as users make selections  Driven by multi-center research (Grand Ensemble)  E.g., want to experiment with NCEP and JMA forecasts for 500 hPa temperatures  Maximum common data is determine by JMA  12Z forecast initialization and 9-day forecast

Differences between centers

User access demonstration Animation, what you will see  Multiple centers  (ECMWF, UKMO, NCEP, CMA, CMC, KMA)  Fields/Parameters  (Geopotential Height, 2m Temperature)  Levels  (500 hPa, Single Level)  Spatial and temporal ranges  (Global, 3-days, 12Z initialization, 48 hour forecasts)  Regridding to common spatial resolution  (1.5°)  Output format  (netCDF)

Sample Data Request for an Event

Retrieve Completed Subset

Subset Request Animation

Gustav/Hannah Animation

Lessons Learned  Manifest files and automated resend are critical for a complete archive  The impact of different contributions from the NWP centers across archive cannot be under estimated  There are important design considerations to insure prompt browser interactions  Caching data from the DB

Lessons Learned  Computational resource requirements ramp up quickly with multi-dimensional problems  D’s, center, ensemble member, parameter, forecast length, etc.  Archive file structure choices greatly impact subsetting ability  TIGGE currently based on synoptic order  Time-series by parameter could be better?

End

Future  Consider adding storage resources to keep a longer period of data online  Integrate more computation resources to reduce time to fill requests  Create a way to re-stage MSS data to online for internet users

Outline  Brief Background on TIGGE Archive  Data Receipt  Challenges  Access  Lessons Learn

User registration and metrics required  Conditions for use  Research and education only  48-hour delay on availability

Limited online storage – 5 TB, ≅ 2.5 weeks  1-2 TB additional storage for user request preparation  Supplement Access  Direct Archive File download, though interface or Research Data Archive (RDA)  All data (200 TB) available through RDA from NCAR Mass Storage System  TC data?