Tom Hopson, Jason Knievel, Yubao Liu, Gregory Roux, Wanli Wu Ensemble-4DWX update: focus on guidance for configuring ensemble members Tom Hopson, Jason Knievel, Yubao Liu, Gregory Roux, Wanli Wu National Security Applications Program Research Applications Laboratory ATEC-4DWX IPR, April 20-21 2010

Reminder of status in Aug 2009 Improving scripting to manage the computational resources Increased calibration stations from 26 to 34 (regional) Calibration hindcast increased from roughly 600 to over 1000 points Increased hindcast size allows calibration of 2 additional lead-times (36hr, 42hr) Included new skill measure for ensemble skill-spread relationship Corrected over-dispersion of calibration process Automated calibrating 2-m T Testing automated calibration of 2-m dew point T National Security Applications Program Research Applications Laboratory ATEC-4DWX IPR, April 20-21 2010

Progress since August 2010 Adding new graphics for DPG’s easier access to post-processed results Continue to improve scripting and potential use of new computational resources to improve post-processing times Wind speed calibration development Developed tool for testing utility of ensemble number to optimize DPG’s reduced model configuration over potential new locations Work continuing on scientific papers Cad - clear air damming study National Security Applications Program Research Applications Laboratory ATEC-4DWX IPR, April 20-21 2010

Recap on Calibration and verification tools Ensemble calibration to correct predicted distribution Calibration is needed for users capable of decision making with probabilistic guidance. Will be needed for foreseeable future Verification of different ensemble characteristics is easily completed when performing calibration National Security Applications Program Research Applications Laboratory ATEC-4DWX IPR, April 20-21 2010

Calibration: review and recap “bias” obs Forecast PDF Probability Probability Forecast PDF obs “spread” or “dispersion” calibration Temperature [K] Temperature [K] Calibration (and verification) is now fully-automated Utilizes “persistence” if available 34 sites in and near DPG Full calibration for all sites ~ 1X per week for each weather variable Using lookup tables ~ every hour (8 hrs, was 1hr) National Security Applications Program Research Applications Laboratory ATEC-4DWX IPR, April 20-21 2010

Probability/°K T [K] Probability/°K T [K] Step 2: For each quan, use forward step-wise cross-validation to select best regress set Selection requires: a) min QR cost function, b) binomial distrib at 95% confidence If requirements not met, retain climatological “prior” Step I: Determine climatological quantiles Probability/°K climatological PDF 1. Regressor set: 1. reforecast ens 2. ens mean 3. ens stdev 4. persistence 5. LR quantile (not shown) 3. T [K] 2. 4. Temperature [K] observed forecasts Time Step 3: segregate forecasts based on ens dispersion; refit models (Step 2) for each range Final result: “sharper” posterior PDF represented by interpolated quans forecasts posterior Forecast PDF I. II. III. II. I. Probability/°K prior T [K] Temperature [K] Time National Security Applications Program Research Applications Laboratory ATEC-4DWX IPR, April 20-21 2010

3-hr dewpoint time series Station DPG S01 Before Calibration After Calibration National Security Applications Program Research Applications Laboratory ATEC-4DWX IPR, April 20-21 2010

42-hr dewpoint time series Station DPG S01 Before Calibration After Calibration National Security Applications Program Research Applications Laboratory ATEC-4DWX IPR, April 20-21 2010

PDFs: raw vs. calibrated Blue is “raw” ensemble Black is calibrated ensemble Red is the observed value Notice: significant change in both “bias” and dispersion of final PDF (also notice PDF asymmetries) obs ATEC-4DWX IPR, April 20-21 2010

Troubled rank histograms 0 10 20 30 Counts 0 10 20 30 Counts 1 2 3 4 5 6 7 8 9 10 Ensemble # 1 2 3 4 5 6 7 8 9 10 Ensemble # ATEC-4DWX IPR, April 20-21 2010

3-hr dewpoint rank histograms Station DPG S01 National Security Applications Program Research Applications Laboratory ATEC-4DWX IPR, April 20-21 2010

42-hr dewpoint rank histograms Station DPG S01 National Security Applications Program Research Applications Laboratory ATEC-4DWX IPR, April 20-21 2010

RMSE of ensemble members Station DPG S01 3hr Lead-time 42hr Lead-time National Security Applications Program Research Applications Laboratory ATEC-4DWX IPR, April 20-21 2010

Significant calibration regressors Station DPG S01 3hr Lead-time 42hr Lead-time National Security Applications Program Research Applications Laboratory ATEC-4DWX IPR, April 20-21 2010

Assessing ensemble utility Goal: provide guidance in ensemble member selection Caveat: results only applying for T and Td surface variables – potential different ensemble usage for different variables and levels Approach: Starting with complete ensemble set, rank model ensembles in terms of their usage in the calibration procedure by assessing over all weather variables, locations, and lead-times Iteratively remove least-used ensembles from calibration, and use verification statistics to determine the cost of reduction What is the “optimal size” that balances forecast skill and computational expense? National Security Applications Program Research Applications Laboratory ATEC-4DWX IPR, April 20-21 2010

Single value to summarize performance. Skill Scores Single value to summarize performance. Reference forecast - best naive guess; persistence, climatology A perfect forecast implies that the object can be perfectly observed Positively oriented – Positive is good National Security Applications Program Research Applications Laboratory ATEC-4DWX IPR, Aug 11-12 2009 ATEC-4DWX IPR, April 20-21 2010

Preliminary Results … Brier referenced to Persistence RMSE referenced to Persistence National Security Applications Program Research Applications Laboratory ATEC-4DWX IPR, April 20-21 2010

… more theoretically Probability/°K Temperature [K] Probability/°K 42hr Lead-time Uncorrelated ensembles Probability/°K Temperature [K] Correlated ensembles Probability/°K More ensembles usefully define mean of PDF if error growth less than linear *and* uncorrelated For perfectly-correlated ensembles, addition of *any* ensemble degrades skill Temperature [K] National Security Applications Program Research Applications Laboratory ATEC-4DWX IPR, April 20-21 2010

Future plans Continue work on refining calibration algorithm to decrease computation time (required as new weather variables are added) Implement for wind speed and wind direction Continue development of guidance tool for optimal ensemble selection for operational use Develop scheme for model points without surface observations over whole model domain National Security Applications Program Research Applications Laboratory ATEC-4DWX IPR, April 20-21 2010