Advanced Study Program Research Review Tom Hopson April 6, 2007

Overview: I. Operational Bangladesh Flood Forecasting 1. Background of project 2. Precipitation inputs: ECMWF ensemble forecasts and satellite rainfall corrections 3. Seasonal Forecasts 4. Brahmaputra Pilot programs II. Verifying the Relationship between Ensemble Forecast Spread and Skill first part of talk about combining two distinctly different hydrologic modeling techniques to improve forecast error for each forecast lead-time; approach similar to ‘super-ensemble’ method. second part covers how we “dress” discharge ensembles derived from EPS precipitation with historical modeling errors to produce discharge ensembles and exceedance probabilities that take into account most sources of error and thus generate more complete forecast probabilities.

Operational Flood Forecasting for Bangladesh: Tom Hopson Peter Webster GT A. R. Subbiah and R. Selvaraju, ADPC Climate Forecast Applications for Bangladesh (CFAB): NCAR/USAID-OFDA/GT/ADPC/ECMWF Bangladesh Stakeholders: Bangladesh Meteorological Department, Flood Forecasting and Warning Center, Bangladesh Water Development Board, Department of Agriculture Extension, Disaster Management Bureau, Institute of Water Modeling, Center for Environmental and Geographic Information Services, CARE-Bangladesh

Bangladesh background About 1/3 of land area floods the monsoon rainy season Size: slightly smaller than Iowa Border countries: Burma (193 km), India (4,053 km) Population: 140 million 36% of population below poverty line Within the top 5 of: poorest and most densely populated in the world Sample of Flood History: 1988: 3/4 of country inundated, 1300 people killed, 30 million homeless, $1 billion in property loss 1998: 60% of country inundated for 3 months, 1000 killed, 40 million homeless 2004: flooding in Brahmaputra basin killed 500 people, displaced 30 million for 3 weeks, 40% of capitol city Dhaka (10 million people) under water

(World Food Program)

The Climate Forecast Applications Project CFAB - Combined Meghna/Brahmaputra/Ganges peak discharges 100,000-150,000 m^3/s => Large river systems -- Limited or no advanced warning of upstream severe flood-stage discharges; only know once the flows reach the Indian/Bangladesh border -- good working relationhship with Bangladesh’s Flood Forecasting Warning Centre -- send us near-real time border discharge measurements; capitalize on this data source by incorporating into our forecasting and error-correction schemes -- Goal: utilizing ECMWF’s 1 to 10 day 51 member EPS forecasts; discharge forecast schemes went “operational” in 2003; will continue to provide operational 1 to 10 day, multi-week, and seasonal border discharge forecasts in 2004 Bangladesh at confluence of Brahmaputra and Ganges Rivers Limited warning of upstream river discharges CFAB’s GOAL: Provide operational upper catchment flood-stage discharge and precipitation forecasts at differing time-scales => Utilize good quality daily border discharge measurements

Three-Tier Overlapping Forecast System Developed for Bangladesh SEASONAL OUTLOOK: “Broad brush” probabilistic forecast of rainfall and river discharge. Updated each month. Produced out to 6 months based on ECMWF’s seasonal 40 member ensemble forecasts. Currently most useful skill out 3 months. Information for strategic planning for agriculture and allied sectors and also for disaster preparedness. 20-25 DAY FORECAST: Forecast of average 5-day rainfall and river discharge 3-4 weeks in advance. Updated every 5 days. Strategic and tactical decisions in the agricultural, water resources and disaster management sectors, particularly for the management of floods and drought. 1-10 DAY FORECAST: Forecast of rainfall and precipitation in probabilistic form updated every day. Based on ECMWF’s 51 member ensemble weather forecasts. Skillful out 7-10 days. Provide probability of flood level exceedance at the entry point of the Ganges & Brahmaputra. Useful for emergency planning, and selective planting or harvesting to reduce potential crop losses at the beginning or end of the cropping cycle.

Daily Operational Flood Forecasting Sequence

ECMWF Ensemble Precipitation Forecast Adjustments -- mapping forecasts from “model-” to “observational-”space Brahmaputra Catchment-avg Forecasts Hydrology model initial conditions driven by near-real-time GPCP / CMORPH / Raingage precipitation Ideally, observations would be statistically “just another ensemble member” Approach: calculate historical NWP-climatology PDF and observation-climatology PDF for each grid using a “kernel” method For each forecast ensemble, determine its quantile in model-space and extract equivalent quantile in observation-space Discharge forecasts employ a blend of precipitation forecasts (ECMWF) and near-real-time “observations” (GPCP or CMORPH) Precipitation “observations” used to initialize discharge model conditions (soil-moisture, lagged subcatchment discharges, etc.) before the forecast period. GPCP and CMORPHprecipitation “observations” based on IR and microwave satellite data, but distinctly different approaches. (Note: potential systematic error in GPCP in 2003 data shown here) Adjust ECMWFcatchment-average ensemble forecasts to satisfy similar distribution as the “observations” using the criterion of the uniformity of the verification rank; method similar to that proposed method of Hamill and Colucci, 1997

Quantile to Quantile Mapping Model Climatology “Observed” Climatology Pmax Pmax Precipitation Pfcst Padj 25th 50th 75th 100th 25th 50th 75th 100th Quantile Quantile

Mean-Square-Error of the Ensemble-Mean shows skill out to 7-8 days ECMWF Ensemble Precipitation Forecast Adjustments -- mapping forecasts from “model-” to “observational-”space Brahmaputra Adjusted Forecasts Benefits: --Gridded “realistic” forecast values --spatial- and temporal covariances preserved Drawbacks: --limited sample set for model-space PDF (2 yrs) --rank histograms show “under-variance” Mean-Square-Error of the Ensemble-Mean shows skill out to 7-8 days

Quantile Regression approach:maintaining skill no worse than “persistence” for non-Gaussian PDF’s (ECMWF Brahmaputra catchment Precipitation) 1 day 4 day “Multi-model” statistical approach applied to RAL’s ATEC mesoscale ensemble forecasts (Josh Hacker) 7 day 10 day

Precipitation Estimates Rain gauge estimates: NOAA CPC and WMO GTS 0.5 X 0.5 spatial resolution; 24h temporal resolution approximately 100 gauges reporting over combined catchment 24hr reporting delay Satellite-derived estimates: Global Precipitation Climatology Project (GPCP) 0.25X0.25 spatial resolution; 3hr temporal resolution 6hr reporting delay geostationary infrared “cold cloud top” estimates calibrated from SSM/I and TMI microwave instruments 3) Satellite-derived estimates: NOAA CPC “CMORPH” 18hr reporting delay precipitation rain rates derived from microwave instruments (SSM/I, TMI, AMSU-B), but “cloud tracking” done using infrared satellites => New Project (Dave Gochis, Gyuwon Lee): optimally blend products together along with uncertainty estimates Incorporate under a now-casting framework

2004 Discharge Forecast Results Brahmaputra Discharge Ensembles Confidence Intervals 2 day Critical Q black dash Observed Q black dot Ensemble Members in color 50% 95% 7 day 8 day 7 day 8 day 3 day 3 day 4 day 4 day These figures show the 1 to 5 day discharge forecasts for the Brahmaputra and Ganges basins for 2003. We have randomly selected 51 new ensemble members from the combined model error/precipitation forecast uncertainty PDF discussed in the previous slide 5 day 5 day 9 day 10 day 9 day 10 day

2004 Danger Level Probabilities Brahmaputra 7-10 day Forecasts Ganges 7-10 day Forecasts

Brahmaputra 7-10 day Forecasts 2006 Ensemble Forecasts Brahmaputra 7-10 day Forecasts Ganges 7-10 day Forecasts

2006 Danger Level Probabilities Brahmaputra 7-10 day Forecasts Ganges 7-10 day Forecasts

Forecasts Improvements Quantile regression approach to improve hydrologic multi-model and final error correction algorithm Automated “seamless” daily to seasonal discharge forecasts merging ECMWF weather and seasonal forecasts, updated daily

Five Pilot Sites chosen in 2006 consultation workshops based on biophysical, social criteria: Rajpur Union -- 16 sq km -- 16,000 pop. Uria Union -- 23 sq km -- 14,000 pop. Kaijuri Union -- 45 sq km -- 53,000 pop. Gazirtek Union -- 32 sq km -- 23,000 pop. Bhekra Union -- 11 sq km -- 9,000 pop. (annual income: 30,000 Tk; US$400)

Livelihoods What can be done with useful forecasts?

Conclusions 2003: Daily operational probabilistic discharge forecasts “experimentally” disseminated 2004: -- Multi-model approach operational -- Forecasts fully-automated -- CFAB became an institutionalized entity of the Bangladesh federal government 2006: -- USAID-OFDA/CARE 4-year funding commitment -- Forecasts incorporated into Bangladesh flood warning program 2007: 5 pilot studies implemented for 1-10day forecasts along the Brahmaputra In 2003, our 1 to 10 day discharge forecasts for the Ganges and Brahmaputra basins were incorporated operationally into the Bangadesh’s flood forecast warning operations Our forecasts are based on ECMWF’s 51-member EPS forecasts, along with near-real-time satellite-derived precipitation estimates of the Global Precipitation Climatology Project and CMORPH Discharge forecasts show good skill out to 5 to 7 days, and “useful” skill out to 10 days. Because of the 1 to 2 day travel time of flow from the border to the major populations centers, this translates into good skill 7 to 9 day forecasts. Future work: combine ECMWF EPS precipitation forecasts with our good quality multi-weekly statistically derived precipitation schemes to generate a “seam-less” discharge forecast

Overview: I. Bangladesh Flood Forecasting Project 1. Background of project 2. Precipitation inputs: ECMWF ensemble forecasts and satellite rainfall corrections 3. Seasonal Forecasts 4. Brahmaputra Pilot programs II. Verifying the Relationship between Ensemble Forecast Spread and Skill first part of talk about combining two distinctly different hydrologic modeling techniques to improve forecast error for each forecast lead-time; approach similar to ‘super-ensemble’ method. second part covers how we “dress” discharge ensembles derived from EPS precipitation with historical modeling errors to produce discharge ensembles and exceedance probabilities that take into account most sources of error and thus generate more complete forecast probabilities.

Motivation for generating ensemble forecasts: Greater accuracy of ensemble mean forecast (half the error variance of single forecast) Likelihood of extremes Non-Gaussian forecast PDF’s Ensemble spread as a representation of forecast uncertainty

Ensemble “Spread” or “Dispersion” Forecast “Skill” or “Error” Probability “dispersion” or “spread” Rainfall [mm/day] “skill” or “error”

ECMWF Brahmaputra catchment Precipitation Forecasts vs TRMM/CMORPH/CDC-GTS Rain gauge Estimates 1 day 4 day Points: -- ensemble dispersion increases with forecast lead-time -- dispersion variability within each lead-time -- Provide information about forecast certainty? How to Verify? -- rank histogram? No. (Hamill, 2001) -- ensemble spread- forecast error correlation? 7 day 10 day

Overview -- Useful Ways to Measure Ensemble Forecast System’s Spread-Skill Relationship: Spread-Skill Correlation misleading (Houtekamer, 1993; Whitaker and Loughe, 1998) Propose 3 alternative scores 1) “normalized” spread-skill correlation 2) “binned” spread-skill correlation 3) “binned” rank histogram Considerations: -- sufficient variance of the forecast spread? (outperforms ensemble mean forecast dressed with error climatology?) -- outperform heteroscedastic error model? -- account for observation uncertainty and under-sampling

Naturally Paired Spread-skill measures: Set I (L1 measures): Error measures: absolute error of the ensemble mean forecast absolute error of a single ensemble member Spread measures: ensemble standard deviation mean absolute difference of the ensembles about the ensemble mean Set II (squared moments; L2 measures): square error of the ensemble mean forecast square error of a single ensemble member ensemble variance

Spread-Skill Correlation … 1 day ECMWF r = 0.33 “Perfect” r = 0.68 4 day ECMWF r = 0.41 “Perfect” r = 0.56 ECMWF spread-skill (black) correlation << 1 Even “perfect model” (blue) correlation << 1 and varies with forecast lead-time 7 day 10 day ECMWF r = 0.39 “Perfect” r = 0.53 ECMWF r = 0.36 “Perfect” r = 0.49

Limits on the spread-skill Correlation for a “Perfect” Model Governing ratio, g: (s = ensemble spread: variance, standard deviation, etc.) Limits: Set I Set II What’s the Point? -- correlation depends on how spread-skill defined -- depends on stability properties of the system being modeled -- even in “perfect” conditions, correlation much less than 1.0

One option … Assign dispersion bins, then: 2) Average the error values in each bin, then correlate 3) Calculate individual rank histograms for each bin, convert to a scalar measure

Option 2: “binned” Spread-skill Correlation “perfect model” (blue) approaches perfect correlation “no-skill” model (red) has expected under-dispersive “U-shape” ECMWF forecasts (black) generally under-dispersive, improving with lead-time Heteroscedastic model (green) slightly better(worse) than ECMWF forecasts for short(long) lead-times 1 day 4 day 7 day 10 day

Option 2: PDF’s of “binned” spread-skill correlations -- accounting for sampling and verification uncertainty “perfect model” (blue) PDF peaked near 1.0 for all lead-times “no-skill” model (red) PDF has broad range of values ECMWF forecast PDF (black) overlaps both “perfect” and “no-skill” PDF’s Heteroscedastic model (green) slightly better(worse) than ECMWF forecasts for short(long) lead-times 4 day 1 day 7 day 10 day

Conclusions Spread-skill correlation can be misleading measure of utility of ensemble dispersion Dependent on “stability” properties of environmental system 3 alternatives: 1) “normalized” (skill-score) spread-skill correlation 2) “binned” spread-skill correlation 3) “binned” rank histogram ratio of moments of “spread” distribution also indicates utility -- if ratio --> 1.0, fixed “climatological” error distribution may provide a far cheaper estimate of forecast error Truer test of utility of forecast dispersion is a comparison with a heteroscedastic error model => a statistical error model may be superior (and cheaper) Important to account for observation and sampling uncertainties when doing a verification

Thank You!