Data for Climate and Energy Studies Steven Worley Computational and Information Systems Laboratory NCAR

Topics Scope of the NCAR Research Data Archive (RDA) Discovery and Access Highlights User ranked popular datasets Examples Near-term service improvements 7 May NCAR-CSM Symposium on Climate and Energy

Scope of the NCAR Research Data Archive (RDA) Focus on atmospheric, oceanographic, and related geo-sciences observational data and derived analyses.  Some weather forecast data  Do not specialize in climate prediction datasets 7 May NCAR-CSM Symposium on Climate and Energy Active stewardship program to maintain and grow the RDA for 40+ years.  Large variety, 600+ datasets, ~ 400 TB, 4M files

Discovery and Access Highlights 7 May NCAR-CSM Symposium on Climate and Energy Primary design feature for web portal Data Discovery – Find Data!

Discovery and Access Highlights 7 May NCAR-CSM Symposium on Climate and Energy Multiple Methods - simple to interoperable 1.Find the files in our lists and download Through your browser – limit 2GB We create a ‘wget’ script for you – run in background on your machine – no limit 2.You select temporal, spatial, parameter domains We build a file list for you Download options as in 1 3.Data is not online to the web – but, is on archive storage We automatically stage data to online, then download 4.You select temporal, spatial, parameter domains - we build CURL commands - you get only the grids you select About CURL Client URL Library functions Readily available on Linux OS We use HTPPS protocols – others are available Applies well to WMO GRIB data format Users modify the CURL commands and script them to perform routine data extractions from RDA

User ranked popular datasets 7 May NCAR-CSM Symposium on Climate and Energy Unique users FY09datasetsTitles 2878ds082.0, ds083.2, ds083.0NCEP FNL Operational Model Global Tropospheric Analyses 924ds090.0NCEP/NCAR Global Reanalysis Products 510ds758.0, ds759.3, ds759.2NGDC Global 2' and 5' Elevations, USGS 30 ARC-second 477 ds461.0, ds351.0 ds337.0, ds464.0,ds353.4NCEP ADP/PREPBUFR Global Surface and Upper Air Observations 358ds608.0NCEP North American Regional Reanalysis (NARR) 264ds609.2GCIP NCEP ETA model output 262ds540.1, ds540.0International Comprehensive Ocean-Atmosphere Data Set (ICOADS) 190ds744.4QSCAT/NCEP Blended Ocean Winds 173ds277.0NCEP V2.0 OI Global SST, V3.0 Extended Reconstructed Analyses 153ds335.0, ds336.0Unidata (IDD) Observations and Model Data 106ds091.0NCEP/DOE Reanalysis II 106ds552.1, ds552.0, ds556.0River Discharge Data 91ds277.3Hadley Centre Global Sea Ice and Sea Surface Temperature (HadISST) 89ds824.1, ds330.3Global Tropical Cyclone "Best Track" Position and Intensity Data, TIGGE Cyclone Tracks 72ds570.0World Monthly Surface Station Climatology 69ds314.0Global Meteorological Forcing Dataset for Land Surface Modeling 68ds900.0U.S. AFGWC Station (Surface and Upper Air) Library 61ds260.3NOCS Surface Flux Dataset v2.0 58ds285.3Japanese Subsurface Temperature And Salinity Analyses V6.7 56ds512.0CPC Global Summary of Day/Month Observations 56ds625.0Japanese 25-year Reanalysis Project 55ds578.1, ds485.0China Monthly Station Precipitation and Temperature, Daily Precip. and Monthly Soil Temperature 53ds285.0World Ocean Database and World Ocean Atlas 47ds770.0GISS Soil and Surface Slope 45ds215.0 Global Monthly Surface Temperature Anomalies ( ), Precipitation ( ), and Sea Level Pressure ( ) from the University of East Anglia Climatic Research Unit 42ds277.7NOAA OI 1/4 Degree Daily SST Analysis 42ds330.2TIGGE Near Real-time 40ds472.0TDL U.S. and Canada Surface Hourly Observations 36ds232.2Scatterometer Climatology of Ocean Winds 32ds131.1, ds131.0NOAA-CIRES Twentieth Century Global Reanalysis Version I and II 30ds260.2CORE.2 Global Air-Sea Flux Dataset 27ds885.1NCDC TD9640 U.S. Palmer Drought Indices 25ds627.0ERA-Interim Project 25ds510.0NCDC TD3200 U.S. Cooperative Summary of Day 24ds564.0Global Historical Climatology Network (GHCN) Temperature, Precipitation, Pressure 5921All DatasetsAll DSS datasets Top 30 datasets/groups FY09 ~ 6000 Unique Users Annually

One example Final Global Analysis from NOAA/NCEP  4x Daily  Updated in the RDA 1x/day  1° horizontal resolution  26 vertical pressure levels, plus surface  Series starts in 1999  Over 55 parameter fields 7 May NCAR-CSM Symposium on Climate and Energy

One example 7 May NCAR-CSM Symposium on Climate and Energy

Re-analyses 7 May 2010 NCAR-CSM Symposium on Climate and Energy 9 Table 1: Global atmospheric and oceanographic re-analyses are one of many valuable data resources provided by external organizations that employ the expertise of RDA consultants and are the most recent major reanalyses available in the Research Data Archive. Most time periods are ongoing, that is, providers continue to produce the products gong forward in time. In general, all reanalyses also have lower temporal and horizontal resolutions than those shown above. Most reanalyses also have variables on vertical model coordinate levels, as well as large numbers of surface specific fields, and vertically integrate values.

Near-term service improvements  Current and soon-to-be workflow 7 May NCAR-CSM Symposium on Climate and Energy

Complete User Community Advantages: Fast access to online data – limited part of RDA Access to all RDA content metadata Access to RDA data processing services Complete User Community Disadvantages: Slow access to MSS data – delayed mode Have to create a separate RDA account and log in Data processing requests take a long time to finish Slow download speeds for some users HPC User Community Advantages: Access to full RDA Fast computing No login required HPC User Community Disadvantages: No access to online data Use MSS as a file server No direct access to RDA metadata No direct access to RDA data processing services Require separate account to access RDA web server

HPC User Community Improvements: Fast access to full RDA Access to all RDA content metadata Access to RDA data processing services Single CISL account Single “first point of contact” Complete User Community Improvements: Fast access to full RDA Expanded data processing services available Single CISL account - no separate RDA account Faster download speeds – grid-based tools, e.g. GRIDFTP Single “first point of contact” for user support Resolved all the disadvantages New Challenges: GPFS and HPSS don’t have generic file use logging Need for metrics & services HPSS doesn’t have sophisticated file access control Some RDA assets have limited access policies Abandon a functional RDA registration system – retool a 20K+ user DB Of course, there will be more! Big transition while maintaining RDA content building and services

End  Scope of the NCAR Research Data Archive (RDA)  Discovery and Access Highlights  User ranked popular datasets  Examples  Near-term service improvements 7 May NCAR-CSM Symposium on Climate and Energy