1. Evaluation of REMSAD- BRAVO Simulations Using Tracer Data and Synthesized Modeling Michael Barna Cooperative Institute for Research in the Atmosphere Colorado State University, Fort Collins, CO Bret Schichtel, Kristi Gebhart and William Malm Air Resources Division National Park Service, Fort Collins, CO PM Model Performance Workshop RTP, NC 10-11 February 2004

2. Acknowledgements Assistance for the REMSAD simulations conducted at CIRA/CSU –Betty Pun, Shiang-Yuh Wu and Christian Seigneur (AER): initial assistance with REMSAD and met data processing –Hampden Kuhns (DRI) and Jeff Vukovich (MCNC): emissions inventory –Eladio Knipping and Naresh Kumar (EPRI): sulfur concentrations from GOCART –Nelson Seaman (PSU): MM5 simulations –Sharon Douglas, Tom Myers (ICF) and Tom Braverman (EPA): useful discussions on model evaluation

3. BRAVO: a study designed to understand haze at Big Bend National Park Big Bend NP is located in remote southwestern Texas, along the Texas/Mexico border Haze has increased in recent years – a rarity for a western park BRAVO (Big Bend Regional Aerosol and Visibility Observational Study) investigates the pollution sources that are contributing to this haze –Field program: July-October 1999 –Many participants: EPANPSNOAA EPRICSUDRI TCEQAEREt al.

4. Flight Over BBNP Area (5 November 2003)

5. Who is contributing sulfate to BBNP? Sulfate is the main constituent of visibility- impairing PM at BBNP Who is contributing? –the Carbon I/II power plant just over the border? –sources in eastern Texas? –sources in the eastern US? –how large is the influence of the boundary concentrations?

6. BRAVO’s “weight of evidence” approach to determine sulfate attributions Don’t rely on one analytical method or model; rather, use “weight of evidence” approach: Source-oriented models: Receptor-oriented models: Hybrid models: REMSADTrMB“Synthesized REMSAD” CMAQFMB“Synthesized CMAQ”

7. This talk will look at three ways to evaluate the BRAVO air quality simulations Simulation of conserved tracers –Important but somewhat dull (Barna) Simulation of sulfate with base emissions –Important but somewhat dull (Barna) Identifying model biases using “synthesis inversion analysis” –Exciting! (Schichtel)

8. Evaluating the REMSAD BRAVO sims Simulation of conserved tracer –examine transport and dispersion of conservative tracers –if model can’t simulate transport and dispersion there’s not point in continuing Simulation of sulfate with base emissions –time series analysis of predicted sulfate against BRAVO and CASTNET monitors –evaluate different periods to identify potential temporal biases –evaluate different monitors to identify potential spatial biases –evaluate at spatial patterns of interpolated observations and predictions – do the match?

9. Evaluating the REMSAD BRAVO sims (cont’d) Use “synthesized inversion modeling” to identify biases with respect to different source regions –A hybrid approach that starts with attribution results from REMSAD (or CMAQ or any model) –Use a statistical approach to identify multiplicative terms for each source region that would result in a best fit to the measurement data –If REMSAD attributions for that source region are perfect: scaling coef = 1 underestimated: scaling coef > 1 (i.e., need to increase) overestimated: scaling coef < 1 (i.e., need to decrease)

10. Simulation of conserved tracers

11. Predicting transport is the most important aspect of air quality modeling No other modeled process, e.g., emissions, deposition, chemical transformation, has as big an impact on model results as transport transport = advection + turbulent diffusion A tracer experiment is the most robust method for evaluating transport –Halocarbon tracer is conserved – negligible transformation and deposition –Detectable at very low concentrations –We know release rates – can check skill of receptor models for determining attribution –expensive

12. BRAVO tracer source and receptor sites Tracer release sites: Eagle Pass San Antonio Big Brown PP Parish PP Tracer receptors at BBNP: Persimmon Gap K-Bar San Vicente Example tracer plumes from REMSAD:

13. Observed and predicted tracer time series Eagle Pass Tracer NE Texas Tracer San Antonio TracerHouston Tracer observed predicted

14. Performance (or lack thereof?) statistics Eagle PassNE TexasHouston San Antonio Average Observed (ppqV)0.210.000.060.52 Average Predicted (ppqV)0.390.020.030.33 R:0.470.340.310.52 Normalized Gross Error:412%130%74%70% Normalized Bias:380%65%-71%-24% What do we expect for “good performance”? Expecting perfection is naïve…. –Grid models aren’t ideal for simulating plumes – the “real” plumes likely have very strong concentration gradients that won’t be represented by model –Complex terrain is complex…and will not be resolved at 36 km

15. Problems with this time series analysis Tracer concentrations at two of the four sites are too low for meaningful time series analysis (negative concentrations!), but there is still useful information here Looking at the preceding time series, your eye tells you that the model clearly has some skill (e.g., timing of Eagle Pass tracer), but this is not reflected in the bias or error statistics

16. Comparing interpolated spatial patterns Need to move beyond simple time series analysis to something more comprehensivie –How to assess patterns? –Magnitude –Concentration gradients –Spatial shifts (e.g., tomorrow’s predicted pattern matches today’s observed pattern) Observed sulfate spatial patterns: Predicted sulfate spatial patterns:

17. Simulation of sulfate using the base emissions inventory

18. REMSAD SO2 and SO4 plumes Predicted SO2 Before using REMSAD to assign sulfate source attributions, need to evaluate the “base case” Predicted SO4

19. How much skill does REMSAD have in predicting sulfate? (BRAVO sites)

20. July 1999: Sept 1999: Oct 1999: Aug 1999:

21. Performance statistics: 37 BRAVO sites OverallJul-99Aug-99Sep-99Oct-99 Observed Average (ug/m3)3.12.13.5 2.8 Predicted Average (ug/m3)3.31.12.83.84.6 R0.610.400.750.630.60 Normalized Error62%51%53%43%98% Normalized Bias1%-41%-43%2%78% Data Completeness98%88%100%

22. How much skill does REMSAD have in predicting sulfate? (CASTNET sites) July 1999: Sept 1999: Oct 1999: Aug 1999:

23. OverallJul-99Aug-99Sep-99Oct-99 Observed Average (ug/m3)4.55.85.64.12.6 Predicted Average (ug/m3)5.05.66.24.53.6 R0.900.920.910.880.87 Normalized Error45%36% 43%65% Normalized Bias21%3%12%21%50% Data Completeness97%99%97%96%97% Performance statistics: 67 CASTNET sites

24. Monthly spatial patterns of bias

25. Observed and predicted spatial patterns Observed sulfate Predicted sulfate Need to develop a quantitative metric that describes the agreement between two spatial patterns!

26. Synthesized inversion modeling

27. Using models for sulfate source apportionment in BRAVO Models can be used for “source attributions”, i.e., “who is causing the pollution at a receptor” Example: remove SO2 emissions from Texas and re-run the model. How do sulfate concentrations at a receptor site change. How this was done for BRAVO: remove SO2 from a source region and re-run REMSAD

28. Sulfate contributions for each region from REMSAD – “unscramble the sulfate egg” Base Case Sulfate = Texas Sulfate + E. US Sulfate + Mexico Sulfate + W. US Sulfate + Boundary Sulfate

29. REMSAD daily attributions for sulfate at Big Bend NP for the major source regions need to add mass here…....but which sources need to increased or decreased? and reduce mass here….

30. Use synthesis inversion modeling to address biases when determining attributions Synthesis inversion modeling – a technique for identifying model biases by combining observations with model results c i =vector of sulfate observations G ij =matrix of the source attribution from each source region/time pair to each observation s j = source attribution scaling coefficients m i = modeled concentration values  i = errors in c i

31. Apply scaling factors to original predictions to get “synthesized REMSAD”

32. New sulfate attributions at Big Bend NP for the BRAVO period

33. Conclusions REMSAD is one tool among many used in BRAVO for developing sulfate source attributions….but we need to try and understand model errors and biases Unfortunately, model evaluation is often ambiguous, difficult and incomplete We often can’t determine why certain model results arise – it is too hard to analyze the individual processes that drive the results –“Cloud processing” of SO2 - are clouds in the right place? Rainout? –Are Mexican emission rates known? –Are predicted oxidant concentrations correct? –And lots more conjecture….

34. Conclusions (cont’d) Tracer experiments provide the minimum bar that the model should get over – if transport can’t be simulated then everything else is suspect Longer simulations (months) are necessary to elucidate temporal biases Larger domains (continental) are necessary to elucidate spatial biases We need better tools than the “standard issue” time series analyses –Synthesized inversion to merge observations with model predictions to identify –Develop a metric that describes the agreement between spatial patterns

35. Conclusions (cont’d) Don’t trust one model; rather, examine results from both receptor models and regional models Questions: –barna@cira.colostate.edu –schichtel@cira.colostate.edu (synthesis inversion)