Page 1© Crown copyright Scale selective verification of precipitation forecasts Nigel Roberts and Humphrey Lean

Page 2© Crown copyright Background - What do we want to know? 1km model – should improve precipitation forecasts …. But, smaller scales are less predictable (Zhang et al 2003) – yet higher expectation Can a 1-km model provide more accurate and useful forecasts of rainfall events on the scales of river catchments? On what scales should the output be presented? What does data assimilation do to different scales?

Page 3© Crown copyright Verification approach Verify over different spatial scales using a conceptually simple approach. Fractions/probabilities from nearest neighbouring points. (also Ebert) Verify against radar – good spatial coverage. Stable network over UK. Verify accumulations - smooth out temporal noise. Use accumulation exceedance thresholds e.g. > 4 mm, > 8 mm ….

Page 4© Crown copyright Radar12 km forecast1 km forecast mm The problem we face 0100 km Six hour accumulations 10 to 16 UTC 13th May 2003

Page 5© Crown copyright Schematic example - different scales

Page 6© Crown copyright 1-km forecastRadar mm Six hour accumulations 10 to 16 UTC 13th May 2003

Page 7© Crown copyright 4 mm threshold, Fractions at grid scale (1 or 0) ModelRadar > 4 mm Fraction

Page 8© Crown copyright ModelRadar > 4 mm Fraction 4 mm threshold, Fractions within 35x35 km squares

Page 9© Crown copyright ModelRadar Fraction 4 mm threshold, Fractions within 75x75 km squares

Page 10© Crown copyright ModelRadar Fraction 4 mm threshold, Fractions within 105x105 km squares

Page 11© Crown copyright Brier score for comparing fractions Skill score for fractions/probabilities - Fractions Skill Score (FSS) A score for comparing fractions with fractions

Page 12© Crown copyright Graphical behaviour of the Fractions Skill Score

Page 13© Crown copyright Evaluation of precipitation forecasts from a 1km NWP forecast model 11 convective cases (Summer 2003/2004) 44 forecasts out to T+7 12km with data assimilation (3-hour cycle) Concentrating on 1km spun up from 12km fields at T+1 (no additional data assimilation at 1km)

Page 14© Crown copyright Results for 6-hour accumulations

Page 15© Crown copyright Results for 6-hour accumulations

Page 16© Crown copyright Results for 1-hour accumulations

Page 17© Crown copyright Scale selective evaluation of operational 12km model precipitation forecasts during 2003 hourly accumulations

Page 18© Crown copyright Conclusions about the approach  Verification approach is useful  the concept is intuitively easy to understand  gives information about spatial accuracy  links output products to verification  agreement with subjective assessment  Issues ….  choice and interpretation of scores, dealing with a bias  radar is regarded as ‘truth’!  variability from case to case  should be compared with other approaches – e.g. Barbara Casati

Page 19© Crown copyright Conclusions about the results  Forecasts of the spatial distribution of rainfall are more accurate at 1km than 12km at scales > ~20-30km.  The scale at which satisfactory skill is achieved is improved at 1km. e.g. from 70km (12km model) to 55km (1km model) for a 10% threshold  The more isolated the events being predicted the poorer the forecast skill, however the 1km model gave the greatest improvement for the most isolated events.  The impact of resolution is greater for hourly than for 6-hourly accumulations. Skill at 1km is maintained after the spin-up period.  Operational 12km model - Skill drops off most quickly after the analysis time at the smallest scales (~ <100 km).

Page 20© Crown copyright Caution - bias

Page 21© Crown copyright Scale selective evaluation of operational 12km model precipitation forecasts during 2003 widespread localised

Page 22© Crown copyright Variation between forecasts

Page 23© Crown copyright Scale selective evaluation of operational 12km model precipitation forecasts during 2003