Ensemble Handling in GrADS Jennifer M. Adams Brian Doty IGES/COLA

What is GrADS? GrADS is an interactive tool that integrates data access, analysis, and visualization Handles many data formats: binary, NetCDF, HDF, GRIB1&2, BUFR Two data models for gridded and in situ data Expression handling is flexible, compact, recursive Programmable interface for scripting Written in C; code is open source (GPL)

A GrADS Graphics Example A nice graphic we trot out often for show and tell. Illustrates several of the graphics output types: contours, colorized vectors, wind barbs, colorized station marks Data are from a variety of sources: gridded satellite obs in HDF, gridded model output in GRIB, station obs in BUFR

What is the GrADS Data Server? GDS is a stable, secure, OPeNDAP data server that provides subsetting and server-side analysis services over the internet GDS can serve any GrADS-readable dataset, and unifies all data formats into a NetCDF framework Open a data set with http://servername/filename instead of /disk/filename GrADS and the GDS are a coupled software system. Where GrADS goes, GDS must follow.

News from GrADS/GDS Team GrADS has a 5th grid dimension for ensembles GrADS has a GRIB2 interface GDS can serve any GrADS data set GrADS is a client for all OPeNDAP data sets GrADS will support GIS-compatible outputs GrADS 2.0 and GDS 2.0 were both released in 2008.

The New Ensemble Dimension in GrADS A 5th grid dimension for ensemble members ‘set X, Y, Z, T, or E’ or ‘set lon, lat, lev, time, or ens’ A virtual dimension for forecast time offset ‘display temp(ft=2)’ ‘display temp(ftime=24hr)’

GrADS Metadata Requirements for Ensemble Members Unique name / number Initial time Length If GRIB2, some additional octet values One time axis spans all members All members must have common X, Y, Z axes GrADS does not constrain ensemble members to have the same initial time -- or even the same length. The hard requirements are that one time axis must span all members and the lat/lon/lev axes must be the same for all members.

GrADS GRIB2 Descriptor File Wesley’s g2ctl works very well, but doesn’t handle EDEF (yet). DSET /gens/prod/gefs.%iy4%im2%id2/%ih2/pgrb2a/ge%e.t%ih2z.pgrb2af%f2 TDEF 17 linear 00z09oct2008 6hr EDEF 23 avg 17 00z09oct2008 0 spr 17 00z09oct2008 2 c00 17 00z09oct2008 1,0 p01 17 00z09oct2008 3,1 p02 17 00z09oct2008 3,2 p03 17 00z09oct2008 3,3 p04 17 00z09oct2008 3,4 . . . p19 17 00z09oct2008 3,19 p20 17 00z09oct2008 3,20 ENDEDEF @ ens String avg Unweighted mean of all members @ ens String spr Standard deviation with respect to ensemble mean @ ens String c00 Control forecast @ ens String p01 Positively perturbed forecast The GRIB2 codes are octets 35 and 36 from Section 4 (PDT # 1, 2, 11, and 12) The output from grib2scan and gribmap provides some guidance for creating the EDEF statement.

Examples of Ensemble Data Sets NCEP GFS Ensembles (GENS) NCEP Climate Forecast System (CFS) NCEP Short Range Ensemble Forecasts (SREF) ESRL MRF Reforecasting Experiment WCRP CMIP3 Multi-Model Data (IPCC AR4) TIGGE Now that the ensemble handing is working for all data types including GRIB2, GrADS is ready to handle a lot of important 5D data sets.

Ensemble Data Sets Behind GDS Data become more usable and accessible Subsets over all dimensions Server-side analysis File aggregation Format translation Ensemble metadata standards: Putting 5-D data sets behind GDS makes the data even more usable and accessible for a variety of reasons (listed above). Developing standards for ensemble metadata has opened a can of worms. Our approach is to present metadata in such a way so that GrADS will understand it, and so that it doesn’t interfere with other clients.

Ensemble Forecast Time Series (Longitude, Latitude, and Level are fixed) Consider a forecast time series from 21 ensemble members, each drawn with a different color. The members all agree for the first 2-3 days, then begin to diverge in the 3-7 day period, and after that there is little coherence to the forecast. Forecast Time --->

Ensemble Forecast Grid (Longitude, Latitude, and Level are fixed) Ensemble Member Here is the same data drawn as a 2-D Time v. Ensemble plot Each row in the grid represents one of the colored lines drawn in the previous plot. Pixels are colored according to the data values. I can draw it this way because the ensembles are handled as the 5th dimension in my gridded forecast data set. The well-defined purple stripe is the event all members agree on at the beginning of the forecast, and the image gets noisier as the forecast evolves. Forecast Time --->

Ten Ensemble Forecasts (Longitude, Latitude, and Level are fixed) Ensemble Member Now we expand our ensemble set to include the same data from ten previous forecasts initialized at 12 hour intervals. Note the time axis has expanded to accommodate the shift in the temporal coverage of additional ensemble forecasts. You can see how the purple stripe feature has gradually coalesced over the five-day period from something incoherent and noisy into a well-defined event. Forecast Time --->

CFS Daily Hindcast (Longitude, Latitude, and Level are fixed) Ensemble Member This is the first two months of a daily CFS hindcast. Note the staggered start times and different lengths for ensemble members. Time Axis ---->

Ensemble Forecast Time Series (Longitude, Latitude, and Level are fixed) Back to a plot I showed earlier. If the data were behind GDS, it could take a long time to draw this because you would need to download 21 time series. This spaghetti-style drawing can be improved by exploiting GrADS analysis capabilities in the E dimension … Forecast Time --->

Ensemble Mean = tloop(ave(Z,e=2,e=23)) Ensemble Min/Max = tloop(min(Z,ens=c00,ens=p20)) +/- StdDev of Ensemble Mean = tloop(sqrt(ave(pow(Z-Zave,2),e=1,e=21))) You can get an even better display by doing 4 relatively simple calculations: the ensemble mean, the standard deviation of the ensemble mean, and the min/max of all the members. For GDS data sets, these derived quantities are calculated on the server side. 17 Mb operated on, 1 Kb downloaded: data requirements reduced by 5 orders of magnitude. Forecast Time --->

TIGGE Data Behind GDS at NCAR Perfect testbed for ensemble handling and GRIB2 interface Boost to usage of TIGGE data Forecasts sorted by date and by provider Time series of analyses Nearly unbearable load on old hardware 48-hour data embargo Int’l agreement requires password protection At dataportal.ucar.edu: ~250 Gb/day 5 Tb online 2-3 week window

TIGGE Multi-Member Multi-Model Ensemble 500mb Geopotential Height valid August 30, 2008 7-day Lead 5-day Lead 3-day Lead 1-day Lead Here’s an example of what can be done with TIGGE data behind GDS -- Each color is an ensemble mean forecast of 500mb Height from 8 different TIGGE providers. The white contours show the multi-model ensemble average (198 members). The four panels show the forecasts with the same valid time but with lead times of 1, 3, 5, and 7 days. In this example, we operated on 6.0 Gb of grib2 data, downloaded 3.3 Mb.

TIGGE MME Forecast Error and Ensemble Spread 500mb Geopotential Height valid August 30, 2008 7-day Lead 5-day Lead The gray-shaded background field is the standard deviation of the multi-model ensemble mean. The white blobs are areas where the models disagree, I.e., where the colored contours in the previous panel diverge. Note how these “clouds of variance” dissipate as lead time shrinks. The colored contours show the absolute value of the forecast error, which is the multi-model ensemble mean minus the analysis (the mean of the 00hr forecasts from the control run of each model). 3-day Lead 1-day Lead

TIGGE Forecasts of Hurricane Ike valid: 12z 9 Sep - 00z 13 Sep init: 00z 8 Sep init: 12z 8 Sep init: 00z 9 Sep These are predicted tracks for hurricane Ike from four different TIGGE models (one color per model). The tracks are created by connecting the dots that mark the location of the sea level pressure minimum at each time step within the valid date range (a 3.5-day period). Multiple tracks in each panel are from individual ensemble members. Each row represents a different initialization date, with later dates towards the bottom. Models: RED = China CMA ORANGE = Canada CMC GREEN = ECMWF BLUE = NCEP init: 12z 9 Sep