1. Advanced Study Program
Research Review
Tom Hopson
April 6, 2007

2. Overview: I. Operational Bangladesh Flood Forecasting
1. Background of project
2. Precipitation inputs: ECMWF ensemble forecasts and satellite rainfall corrections
3. Seasonal Forecasts
4. Brahmaputra Pilot programs
II. Verifying the Relationship between Ensemble Forecast Spread and Skill
first part of talk about combining two distinctly different hydrologic modeling techniques to improve forecast error for each forecast lead-time; approach similar to ‘super-ensemble’ method.
second part covers how we “dress” discharge ensembles derived from EPS precipitation with historical modeling errors to produce discharge ensembles and exceedance probabilities that take into account most sources of error and thus generate more complete forecast probabilities.

3. Operational Flood Forecasting for Bangladesh:
Tom Hopson
Peter Webster GT
A. R. Subbiah and R. Selvaraju, ADPC
Climate Forecast Applications for Bangladesh (CFAB):
NCAR/USAID-OFDA/GT/ADPC/ECMWF
Bangladesh Stakeholders: Bangladesh Meteorological Department, Flood Forecasting and Warning Center, Bangladesh Water Development Board, Department of Agriculture Extension, Disaster Management Bureau, Institute of Water Modeling, Center for Environmental and Geographic Information Services, CARE-Bangladesh

5. Bangladesh background
About 1/3 of land area floods the monsoon rainy season
Size: slightly smaller than Iowa
Border countries: Burma (193 km), India (4,053 km)
Population: 140 million
36% of population below poverty line
Within the top 5 of: poorest and most densely populated in the world
Sample of Flood History:
1988: 3/4 of country inundated, 1300 people killed, 30 million homeless, $1 billion in property loss
1998: 60% of country inundated for 3 months, 1000 killed, 40 million homeless
2004: flooding in Brahmaputra basin killed 500 people, displaced 30 million for 3 weeks, 40% of capitol city Dhaka (10 million people) under water

6. (World Food Program)

10. The Climate Forecast Applications Project CFAB
- Combined Meghna/Brahmaputra/Ganges peak discharges 100, ,000 m^3/s => Large river systems
-- Limited or no advanced warning of upstream severe flood-stage discharges; only know once the flows reach the Indian/Bangladesh border
-- good working relationhship with Bangladesh’s Flood Forecasting Warning Centre -- send us near-real time border discharge measurements; capitalize on this data source by incorporating into our forecasting and error-correction schemes
-- Goal: utilizing ECMWF’s 1 to 10 day 51 member EPS forecasts; discharge forecast schemes went “operational” in 2003; will continue to provide operational 1 to 10 day, multi-week, and seasonal border discharge forecasts in 2004
Bangladesh at confluence of Brahmaputra and Ganges Rivers
Limited warning of upstream river discharges
CFAB’s GOAL: Provide operational upper catchment flood-stage discharge and precipitation forecasts at differing time-scales
=> Utilize good quality daily border discharge measurements

11. Three-Tier Overlapping Forecast System Developed for Bangladesh
SEASONAL OUTLOOK: “Broad brush” probabilistic forecast of rainfall and river discharge. Updated each month. Produced out to 6 months based on ECMWF’s seasonal 40 member ensemble forecasts. Currently most useful skill out 3 months. Information for strategic planning for agriculture and allied sectors and also for disaster preparedness.
20-25 DAY FORECAST: Forecast of average 5-day rainfall and river discharge 3-4 weeks in advance. Updated every 5 days. Strategic and tactical decisions in the agricultural, water resources and disaster management sectors, particularly for the management of floods and drought.
1-10 DAY FORECAST: Forecast of rainfall and precipitation in probabilistic form updated every day. Based on ECMWF’s 51 member ensemble weather forecasts. Skillful out 7-10 days. Provide probability of flood level exceedance at the entry point of the Ganges & Brahmaputra. Useful for emergency planning, and selective planting or harvesting to reduce potential crop losses
at the beginning or end of the cropping cycle.

13. Daily Operational Flood Forecasting Sequence

14. ECMWF Ensemble Precipitation Forecast Adjustments -- mapping forecasts from “model-” to “observational-”space
Brahmaputra Catchment-avg Forecasts
Hydrology model initial conditions driven by near-real-time GPCP / CMORPH / Raingage precipitation
Ideally, observations would be statistically “just another ensemble member”
Approach: calculate historical NWP-climatology PDF and observation-climatology PDF for each grid using a “kernel” method
For each forecast ensemble, determine its quantile in model-space and extract equivalent quantile in observation-space
Discharge forecasts employ a blend of precipitation forecasts (ECMWF) and near-real-time “observations” (GPCP or CMORPH)
Precipitation “observations” used to initialize discharge model conditions (soil-moisture, lagged subcatchment discharges, etc.) before the forecast period.
GPCP and CMORPHprecipitation “observations” based on IR and microwave satellite data, but distinctly different approaches. (Note: potential systematic error in GPCP in 2003 data shown here)
Adjust ECMWFcatchment-average ensemble forecasts to satisfy similar distribution as the “observations” using the criterion of the uniformity of the verification rank; method similar to that proposed method of Hamill and Colucci, 1997

15. Quantile to Quantile Mapping
Model Climatology
“Observed” Climatology
Pmax
Pmax
Precipitation
Pfcst
Padj
25th
50th
75th
100th
25th
50th
75th
100th
Quantile
Quantile

16. Mean-Square-Error of the Ensemble-Mean shows skill out to 7-8 days
ECMWF Ensemble Precipitation Forecast Adjustments -- mapping forecasts from “model-” to “observational-”space
Brahmaputra Adjusted Forecasts
Benefits:
--Gridded “realistic” forecast values
--spatial- and temporal covariances preserved
Drawbacks:
--limited sample set for model-space PDF (2 yrs)
--rank histograms show “under-variance”
Mean-Square-Error of the Ensemble-Mean shows skill out to 7-8 days

17. Quantile Regression approach:maintaining skill no
worse than “persistence” for non-Gaussian PDF’s
(ECMWF Brahmaputra catchment Precipitation)
1 day
4 day
“Multi-model” statistical approach applied to RAL’s ATEC mesoscale ensemble forecasts (Josh Hacker)
7 day
10 day

18. Precipitation Estimates
Rain gauge estimates: NOAA CPC and WMO GTS
0.5 X 0.5 spatial resolution; 24h temporal resolution
approximately 100 gauges reporting over combined catchment
24hr reporting delay
Satellite-derived estimates: Global Precipitation Climatology Project (GPCP)
0.25X0.25 spatial resolution; 3hr temporal resolution
6hr reporting delay
geostationary infrared “cold cloud top” estimates calibrated from SSM/I and TMI microwave instruments
3) Satellite-derived estimates: NOAA CPC “CMORPH”
18hr reporting delay
precipitation rain rates derived from microwave instruments (SSM/I, TMI, AMSU-B), but “cloud tracking” done using infrared satellites
=> New Project (Dave Gochis, Gyuwon Lee):
optimally blend products together along with uncertainty estimates
Incorporate under a now-casting framework

19. 2004 Discharge Forecast Results
Brahmaputra Discharge Ensembles
Confidence Intervals
2 day
Critical Q black dash
Observed Q black dot
Ensemble Members in color
50%
95%
7 day
8 day
7 day
8 day
3 day
3 day
4 day
4 day
These figures show the 1 to 5 day discharge forecasts for the Brahmaputra and Ganges basins for We have randomly selected 51 new ensemble members from the combined model error/precipitation forecast uncertainty PDF discussed in the previous slide
5 day
5 day
9 day
10 day
9 day
10 day

20. 2004 Danger Level Probabilities
Brahmaputra 7-10 day Forecasts
Ganges 7-10 day Forecasts

21. Brahmaputra 7-10 day Forecasts
2006 Ensemble Forecasts
Brahmaputra 7-10 day Forecasts
Ganges 7-10 day Forecasts

22. 2006 Danger Level Probabilities
Brahmaputra 7-10 day Forecasts
Ganges 7-10 day Forecasts

25. Forecasts Improvements
Quantile regression approach to improve hydrologic multi-model and final error correction algorithm
Automated “seamless” daily to seasonal discharge forecasts merging ECMWF weather and seasonal forecasts, updated daily

26. Five Pilot Sites chosen in 2006 consultation workshops based on biophysical, social criteria:
Rajpur Union
-- 16 sq km
-- 16,000 pop.
Uria Union
-- 23 sq km
-- 14,000 pop.
Kaijuri Union
-- 45 sq km
-- 53,000 pop.
Gazirtek Union
-- 32 sq km
-- 23,000 pop.
Bhekra Union
-- 11 sq km
-- 9,000 pop.
(annual income:
30,000 Tk; US$400)

27. Livelihoods
What can be done with useful forecasts?

28. Conclusions
2003: Daily operational probabilistic discharge forecasts “experimentally” disseminated
2004:
-- Multi-model approach operational
-- Forecasts fully-automated
-- CFAB became an institutionalized entity of the Bangladesh federal government
2006:
-- USAID-OFDA/CARE 4-year funding commitment
-- Forecasts incorporated into Bangladesh flood warning program
2007: 5 pilot studies implemented for 1-10day forecasts along the Brahmaputra
In 2003, our 1 to 10 day discharge forecasts for the Ganges and Brahmaputra basins were incorporated operationally into the Bangadesh’s flood forecast warning operations
Our forecasts are based on ECMWF’s 51-member EPS forecasts, along with near-real-time satellite-derived precipitation estimates of the Global Precipitation Climatology Project and CMORPH
Discharge forecasts show good skill out to 5 to 7 days, and “useful” skill out to 10 days.
Because of the 1 to 2 day travel time of flow from the border to the major populations centers, this translates into good skill 7 to 9 day forecasts.
Future work: combine ECMWF EPS precipitation forecasts with our good quality multi-weekly statistically derived precipitation schemes to generate a “seam-less” discharge forecast

29. Overview: I. Bangladesh Flood Forecasting Project
1. Background of project
2. Precipitation inputs: ECMWF ensemble forecasts and satellite rainfall corrections
3. Seasonal Forecasts
4. Brahmaputra Pilot programs
II. Verifying the Relationship between Ensemble Forecast Spread and Skill
first part of talk about combining two distinctly different hydrologic modeling techniques to improve forecast error for each forecast lead-time; approach similar to ‘super-ensemble’ method.
second part covers how we “dress” discharge ensembles derived from EPS precipitation with historical modeling errors to produce discharge ensembles and exceedance probabilities that take into account most sources of error and thus generate more complete forecast probabilities.

30. Motivation for generating ensemble forecasts:
Greater accuracy of ensemble mean forecast (half the error variance of single forecast)
Likelihood of extremes
Non-Gaussian forecast PDF’s
Ensemble spread as a representation of forecast uncertainty

31. Ensemble “Spread” or “Dispersion” Forecast “Skill” or “Error”
Probability
“dispersion” or “spread”
Rainfall [mm/day]
“skill” or “error”

32. ECMWF Brahmaputra catchment Precipitation Forecasts
vs TRMM/CMORPH/CDC-GTS Rain gauge Estimates
1 day
4 day
Points:
-- ensemble dispersion
increases with forecast
lead-time
-- dispersion variability
within each lead-time
-- Provide information
about forecast certainty?
How to Verify?
-- rank histogram?
No. (Hamill, 2001)
-- ensemble spread-
forecast error
correlation?
7 day
10 day

33. Overview -- Useful Ways to Measure Ensemble Forecast System’s Spread-Skill Relationship:
Spread-Skill Correlation misleading (Houtekamer, 1993; Whitaker and Loughe, 1998)
Propose 3 alternative scores
1) “normalized” spread-skill correlation
2) “binned” spread-skill correlation
3) “binned” rank histogram
Considerations:
-- sufficient variance of the forecast spread? (outperforms ensemble mean forecast dressed with error climatology?)
-- outperform heteroscedastic error model?
-- account for observation uncertainty and under-sampling

34. Naturally Paired Spread-skill measures:
Set I (L1 measures):
Error measures:
absolute error of the ensemble mean forecast
absolute error of a single ensemble member
Spread measures:
ensemble standard deviation
mean absolute difference of the ensembles about the ensemble mean
Set II (squared moments; L2 measures):
square error of the ensemble mean forecast
square error of a single ensemble member
ensemble variance

35. Spread-Skill Correlation …
1 day
ECMWF
r = 0.33
“Perfect”
r = 0.68
4 day
ECMWF
r = 0.41
“Perfect”
r = 0.56
ECMWF spread-skill (black) correlation << 1
Even “perfect model” (blue) correlation << 1 and varies with forecast lead-time
7 day
10 day
ECMWF
r = 0.39
“Perfect”
r = 0.53
ECMWF
r = 0.36
“Perfect”
r = 0.49

36. Limits on the spread-skill Correlation for a “Perfect” Model
Governing ratio, g:
(s = ensemble spread: variance, standard deviation, etc.)
Limits:
Set I
Set II
What’s the Point?
-- correlation depends on
how spread-skill defined
-- depends on stability properties
of the system being modeled
-- even in “perfect” conditions,
correlation much less than 1.0

37. One option … Assign dispersion bins, then:
2) Average the error values in each bin, then correlate
3) Calculate individual rank histograms for each bin, convert to a scalar measure

38. Option 2: “binned” Spread-skill Correlation
“perfect model” (blue) approaches perfect correlation
“no-skill” model (red) has expected under-dispersive “U-shape”
ECMWF forecasts (black) generally under-dispersive, improving with lead-time
Heteroscedastic model (green) slightly better(worse) than ECMWF forecasts for short(long) lead-times
1 day
4 day
7 day
10 day

39. Option 2: PDF’s of “binned” spread-skill correlations -- accounting for sampling and verification uncertainty
“perfect model” (blue) PDF peaked near 1.0 for all lead-times
“no-skill” model (red) PDF has broad range of values
ECMWF forecast PDF (black) overlaps both “perfect” and “no-skill” PDF’s
Heteroscedastic model (green) slightly better(worse) than ECMWF forecasts for short(long) lead-times
4 day
1 day
7 day
10 day

40. Conclusions
Spread-skill correlation can be misleading measure of utility of ensemble dispersion
Dependent on “stability” properties of environmental system
3 alternatives:
1) “normalized” (skill-score) spread-skill correlation
2) “binned” spread-skill correlation
3) “binned” rank histogram
ratio of moments of “spread” distribution also indicates utility
-- if ratio --> 1.0, fixed “climatological” error distribution may provide a far cheaper estimate of forecast error
Truer test of utility of forecast dispersion is a comparison with a heteroscedastic error model => a statistical error model may be superior (and cheaper)
Important to account for observation and sampling uncertainties when doing a verification

41. Thank You!