1. 1 Update on Enterprise Strategic Implementation Plan for Generating and Communicating Forecast Uncertainty Paul Hirschberg and Elliot Abrams, Co-Chairs August 11, 2009 AMS Summer Community Meeting AMS Ad Hoc Committee on Uncertainty in (ACUF) Uncertainty in Forecasts (ACUF)

2. 2 Outline BackgroundBackground Draft PlanDraft Plan DiscussionDiscussion Next StepsNext Steps

3. 3 Background ACUF Charge ACUF formed by AMS Commission on Weather and Climate Enterprise, Board on Enterprise CommunicationACUF formed by AMS Commission on Weather and Climate Enterprise, Board on Enterprise Communication Charged to formulate a cross-Enterprise plan to provide forecast uncertainty information to NationCharged to formulate a cross-Enterprise plan to provide forecast uncertainty information to Nation 2006 NRC Report - Completing the Forecast2006 NRC Report - Completing the Forecast “… confluence of compelling reasons for Enterprise to transition to a new paradigm … in which uncertainty is integral and essential component of all forecasts.” 2008 AMS Statement - Enhancing Weather Information with Probability Forecasts2008 AMS Statement - Enhancing Weather Information with Probability Forecasts “Ideally, all weather forecasts would include information that accurately quantifies their uncertainty.” Purpose of ACUF Plan – guide for Enterprise partnership (following NRC Fair Weather Report) – not prescriptive program planPurpose of ACUF Plan – guide for Enterprise partnership (following NRC Fair Weather Report) – not prescriptive program plan

4. 4 Background ACUF Demographics and Organization >60 volunteers from across Enterprise>60 volunteers from across Enterprise Organized into Topic Workgroups along plan componentsOrganized into Topic Workgroups along plan components Integration Co-Leads E. Abrams (AccuWeather) P. Hirschberg (NOAA) Needs/Benefits Leads B. Morrow (FSU) J. Gaynor (NOAA) Strategic Goals Lead D. Hilderbrand (NWS) Solutions Leads R. Hoffman (AER) T. Hamill (OAR) B. Bua (UCAR/COMET) Roles & Resp. Leads B. Philips (UMASS) J. Sokich (NWS) Roadmap Lead N. Stuart (NOAA)

5. 5 Background Not Starting from Scratch NRC ReportNRC Report AMS StatementAMS Statement WMO GuidelinesWMO Guidelines THORPEX results and plansTHORPEX results and plans Related literature -- physical science, socio-economic, behavioral, …Related literature -- physical science, socio-economic, behavioral, … Best Practices and PlansBest Practices and Plans –National / International hydromet services –Industry –Academia and Labs Ten-day Temperature and Precipitation Probability Forecast for Trondheim, Norway from Norwegian Meteorological Institute Temperature Spread Precipitation Spread Mean + Spread (50%+30%)

6. 6 Draft Plan Organization Use and Benefits of Forecast Uncertainty InformationUse and Benefits of Forecast Uncertainty Information Vision and Strategic GoalsVision and Strategic Goals Enterprise Partners’ Roles and ResponsibilitiesEnterprise Partners’ Roles and Responsibilities Implementation RoadmapImplementation Roadmap –Objectives –Performance Measures –Tasks Monitoring ProgressMonitoring Progress Tasks Performance Measures & Targets Objectives Roles & Responsibilities Vision Strategic Goals Benefits

7. 7 Societal evidence that decisions can be improved with uncertainty information and knowledge of risk (cost of doing something vs. cost of doing nothing)Societal evidence that decisions can be improved with uncertainty information and knowledge of risk (cost of doing something vs. cost of doing nothing) –Insurance –Medical –Investments Evidence of needs and benefits of hydrometeorological forecast uncertainty information in literatureEvidence of needs and benefits of hydrometeorological forecast uncertainty information in literature ACUF Plan includes scenarios illustrating how uncertainty information can improve decisions and outcomes in various areas, which if extrapolated nationally, could lead to potentially large benefitsACUF Plan includes scenarios illustrating how uncertainty information can improve decisions and outcomes in various areas, which if extrapolated nationally, could lead to potentially large benefits Draft Plan Uses and Benefits of Forecast Uncertainty

8. 8 Currently, weather impacts cause 70% of all air traffic delays ~$28 billion/yearCurrently, weather impacts cause 70% of all air traffic delays ~$28 billion/year ~2/3 of these delays could be avoided with better weather information and use~2/3 of these delays could be avoided with better weather information and use Improving National Airspace System (NAS) Efficiency Current: Air traffic managers utilize radar and deterministic forecast products to divert aircraft around severe weather using set “play book routes” Future: Predicted probabilistic available flow capacity ratio based on probability of weather hazards will be used to route aircraft through severe weather 300 km route reduction Through holes in the “Reds”

9. 9 Probability of exceeding severity thresholds for surface winds and seas estimated from ensembles and used to populate Risk MatrixProbability of exceeding severity thresholds for surface winds and seas estimated from ensembles and used to populate Risk Matrix Based on pattern of risk, decision makers can take action:Based on pattern of risk, decision makers can take action: –moving naval assets to areas that are favorable for piracy activity –providing divert recommendations to shipping Draft Plan Uses and Benefits of Forecast Uncertainty Piracy around Horn of Africa costs U.S. maritime industry up to $16 billion/yearPiracy around Horn of Africa costs U.S. maritime industry up to $16 billion/year U.S. Navy developing capability to use “Operational Risk Management” techniques to translate objective weather uncertainty guidance to piracy riskU.S. Navy developing capability to use “Operational Risk Management” techniques to translate objective weather uncertainty guidance to piracy risk Enhancing Security* Risk surface for an 84-h forecast Meteorological Risk to Pirates Low Risk High Risk Gulf of Amen *Courtesy Jim Hansen, Naval Research Laboratory

10. 10 Draft Plan Vision Enterprise partnership generates & communicates forecast uncertainty information meeting Nation’s needs for informed decisions-making protecting life and property,protecting life and property, supporting national defense and homeland security,supporting national defense and homeland security, enhancing the economy, andenhancing the economy, and meeting specific needs of partners, users, and customers.meeting specific needs of partners, users, and customers.

11. 11 Draft Plan Current Enterprise Capabilities (sample) North American Ensemble Forecast SystemNorth American Ensemble Forecast System Ensemble Model Output StatisticsEnsemble Model Output Statistics Short-range ensemble forecast systemShort-range ensemble forecast system HFIP hurricane ensembleHFIP hurricane ensemble Haz. Wx. Testbed / Spring ProgramHaz. Wx. Testbed / Spring Program Storm-scale ensemblesStorm-scale ensembles University/Lab ensembles and productsUniversity/Lab ensembles and products Storm Prediction Center ProductsStorm Prediction Center Products Commercial sector tailoring of specific solutions and advice for customersCommercial sector tailoring of specific solutions and advice for customers … Storm Prediction Center Convective Outlooks

12. 12 Draft Plan Key Enterprise Gaps Generating uncertainty information: Ensemble forecast systems do not yet provide reliable probabilistic forecasts in all circumstances owing to:Generating uncertainty information: Ensemble forecast systems do not yet provide reliable probabilistic forecasts in all circumstances owing to: –Systematic model errors –Model resolution –Parameterizations of sub-grid scale physical processes –Model initializations not properly representing analysis uncertainty Communicating Uncertainty Information: Limited knowledge on best ways to communicating, manage, and apply uncertainty informationCommunicating Uncertainty Information: Limited knowledge on best ways to communicating, manage, and apply uncertainty information –Forecast uncertainty viewed negatively by public –Lack of service provider and user knowledge about uncertainty –Inconsistent nomenclature, graphics –Lack of user requirements –Unclear how to best communicate to and assist users

13. 13 http://www.spc.noaa.gov/exper/sref/ Ensemble forecast systems may be biased and/or deficient in spread resulting in misestimated probabilitiesEnsemble forecast systems may be biased and/or deficient in spread resulting in misestimated probabilities Heavy rain fell in area where none of the ensemble members predicted it Draft Plan Key Enterprise Gaps - Generating

14. 14 Draft Plan Key Enterprise Gaps – Communicating Sample Product: Day 4-8 Convective Outlook from Storm Prediction CenterSample Product: Day 4-8 Convective Outlook from Storm Prediction Center What does “Predictability Too Low” Mean? a)Situation not predictable b)Severe Storms possible, threat < 30% c)Cannot beat climatology d)Severe Storms possible, threat < 5% e)Situation not clear

15. 15 Understanding nature of forecast uncertainty -- how to quantify it; how societal and human factors influence communication and use of uncertainty informationUnderstanding nature of forecast uncertainty -- how to quantify it; how societal and human factors influence communication and use of uncertainty information Generating a reliable foundational suite of uncertainty data, products, services, and information needed by user communities;Generating a reliable foundational suite of uncertainty data, products, services, and information needed by user communities; Communicating uncertainty information effectively, and Assisting users in interpreting and applying information in decision makingCommunicating uncertainty information effectively, and Assisting users in interpreting and applying information in decision making Enabling development, acquisition, and operation of forecast uncertainty processing and communication systems with necessary computational, telecommunications, and other types of infrastructure.Enabling development, acquisition, and operation of forecast uncertainty processing and communication systems with necessary computational, telecommunications, and other types of infrastructure. Enterprise must build capabilities in four key, interrelated strategic areas to meet challenges of increasing focus on probabilistic forecasting Draft Plan Strategic Goals

16. 16 Draft Plan Recommended Enterprise Partners’ Roles and Responsibilities Enterprise consists of four primary sectors:Enterprise consists of four primary sectors: 1) Government (local, state, and federal governments) 2) Commercial (companies and media) 3) Academia (universities and associated research institutions) 4) Non-Government Organizations (NGOs) Distinct leadership roles for each partner group to fillDistinct leadership roles for each partner group to fill But, expertise and resources of all sectors need to be leveraged to bring uncertainty into mainstream of decision makingBut, expertise and resources of all sectors need to be leveraged to bring uncertainty into mainstream of decision making

17. 17 Draft Plan Recommended Enterprise Partners’ Roles and Responsibilities Government Generate and sustain a foundational (baseline) suite of forecast uncertainty data, products, services, and information in response to National needsGenerate and sustain a foundational (baseline) suite of forecast uncertainty data, products, services, and information in response to National needs –Ensure availability to all Develop and maintain Nation’s baseline probabilistic forecast machineryDevelop and maintain Nation’s baseline probabilistic forecast machinery –High-performance supercomputing –Infrastructure to ensure open sharing of large amounts of data. –Basic set of interpretative material Provide education, interpretation, and decision assistance to “.gov” decision makers -- for protection of life and propertyProvide education, interpretation, and decision assistance to “.gov” decision makers -- for protection of life and property Establish/exploit test beds to bring latest uncertainty-focused research activities into operations and applicationsEstablish/exploit test beds to bring latest uncertainty-focused research activities into operations and applications Help define new educational skill sets needed for evolving focus on uncertainty, e.g., mathematics, science, communication skillsHelp define new educational skill sets needed for evolving focus on uncertainty, e.g., mathematics, science, communication skills

18. 18 Draft Plan Recommended Enterprise Partners’ Roles and Responsibilities Commercial Weather and Climate Companies Extend and tailor information provided by Government to meet their clients’ specific needsExtend and tailor information provided by Government to meet their clients’ specific needs Provide education, interpretation, and decision assistance to clientsProvide education, interpretation, and decision assistance to clientsMedia Provide forecast uncertainty interpretive information to publicProvide forecast uncertainty interpretive information to public Help determine what forecast uncertainty products and services public wants (e.g. through surveys)Help determine what forecast uncertainty products and services public wants (e.g. through surveys) Help educate public about how to use uncertainty products and how to incorporate uncertainty into their daily decisions and planningHelp educate public about how to use uncertainty products and how to incorporate uncertainty into their daily decisions and planning

19. 19 Draft Plan Recommended Enterprise Partners’ Roles and Responsibilities Academic Broaden basic education in hydrometeorologyBroaden basic education in hydrometeorology –Probabilistic forecasting concepts, including statistics and chaos theory –Applying social sciences to integrate uncertainty information into decision-making –Applying social sciences to integrate uncertainty information into decision-making Perform basic research to advance understanding and knowledgePerform basic research to advance understanding and knowledge –Predictability, advanced ensemble and statistical techniques –Socio-economic value of forecast uncertainty information –How to best communicate and apply uncertainty in decision making Develop leading-edge, high-risk productsDevelop leading-edge, high-risk products Transfer knowledge to operations and/or end-user applicationsTransfer knowledge to operations and/or end-user applications –Participate in testbeds

20. 20 Draft Plan Recommended Enterprise Partners’ Roles and Responsibilities Non Government Organizations Build consensus and unite Enterprise behind vision and goalsBuild consensus and unite Enterprise behind vision and goals Facilitate communication and advocacyFacilitate communication and advocacy Help define educational requirements and skillsHelp define educational requirements and skills Educate membersEducate members Offer training and education to usersOffer training and education to users

21. 21 Roadmap contains set of objectives to accomplish four strategic goalsRoadmap contains set of objectives to accomplish four strategic goals Each objective includes:Each objective includes: –Background –Need –Current capabilities and gaps –Performance measures and targets –Proposed solution strategy, specific tasks and sector leads. Tasks include specific actions for short-, mid-, and long-term periods over next decadeTasks include specific actions for short-, mid-, and long-term periods over next decade Draft Plan Roadmap Overview

22. 22 Draft Plan Roadmap Objectives Strategic Goal 1: Understand Forecast Uncertainty Predictability Theoretical basis and optimal design of uncertainty prediction systems Best methods for communicating uncertainty Predictability Theoretical basis and optimal design of uncertainty prediction systems Best methods for communicating uncertainty Strategic Goal 2: Generate a Foundational Set of Uncertainty Data and Information Ensemble prediction systems Probabilistic nowcasting systems Statistical and not-statistical post-processing techniques Probabilistic forecast preparation and management systems Forecaster and other service provider training Probabilistic verification systems Digital probabilistic forecasts in Weather Information Database Ensemble prediction systems Probabilistic nowcasting systems Statistical and not-statistical post-processing techniques Probabilistic forecast preparation and management systems Forecaster and other service provider training Probabilistic verification systems Digital probabilistic forecasts in Weather Information Database

23. 23 Draft Plan Roadmap Objectives Strategic Goal 1: Understand Forecast Uncertainty Predictability Theoretical basis and optimal design of uncertainty prediction systems Best methods for communicating uncertainty Predictability Theoretical basis and optimal design of uncertainty prediction systems Best methods for communicating uncertainty Strategic Goal 2: Generate a Foundational Set of Uncertainty Data and Information Ensemble prediction systems Probabilistic nowcasting systems Statistical and not-statistical post-processing techniques Probabilistic forecast preparation and management systems Forecaster and other service provider training Probabilistic verification systems Digital probabilistic forecasts in Weather Information Database Ensemble prediction systems Probabilistic nowcasting systems Statistical and not-statistical post-processing techniques Probabilistic forecast preparation and management systems Forecaster and other service provider training Probabilistic verification systems Digital probabilistic forecasts in Weather Information Database

24. 24 Draft Plan Roadmap Objectives, (cont.) Strategic Goal 3: Communicate / Assist Users Interpret and Apply Uncertainty Information Strategic Goal 4: Enable with Supporting Infrastructure High performance computing Adequate telecommunications Comprehensive archive and data access Research-to-operations and applications policies and processes and test beds User infrastructure needs High performance computing Adequate telecommunications Comprehensive archive and data access Research-to-operations and applications policies and processes and test beds User infrastructure needs K-12, undergraduate and graduate education User education and outreach Presentation uncertainty forecast products Decision assistance tools and services User evaluation and feedback K-12, undergraduate and graduate education User education and outreach Presentation uncertainty forecast products Decision assistance tools and services User evaluation and feedback

25. 25 Draft Plan Roadmap Sample Tasks Strategic Goal 1: Understand Forecast Uncertainty Synthesize results of previous studies on limits of predictability (Lead: Aca.) Host special fora for social science and related research community to interact with hydrometeorological community (NGO) Synthesize results of previous studies on limits of predictability (Lead: Aca.) Host special fora for social science and related research community to interact with hydrometeorological community (NGO) Strategic Goal 2: Generate a Foundational Set of Uncertainty Data and Information Ensemble prediction systems (Gov., Aca.) Probabilistic nowcasting systems (Gov., Aca.) Statistical and not-statistical post- processing techniques (Gov., Aca.) Probabilistic forecast preparation and management systems (Lead: Gov., Aca.) Forecaster and other service provider training (Gov., Aca., Com.) Probabilistic verification systems (Gov.,Aca.) Digital probabilistic forecasts in Weather Information Database (Gov.) Ensemble prediction systems (Gov., Aca.) Probabilistic nowcasting systems (Gov., Aca.) Statistical and not-statistical post- processing techniques (Gov., Aca.) Probabilistic forecast preparation and management systems (Lead: Gov., Aca.) Forecaster and other service provider training (Gov., Aca., Com.) Probabilistic verification systems (Gov.,Aca.) Digital probabilistic forecasts in Weather Information Database (Gov.)

26. 26 Draft Plan Roadmap Sample Tasks, (cont.) Strategic Goal 3: Communicate / Assist Users Interpret and Apply Uncertainty Information Strategic Goal 4: Enable with Supporting Infrastructure Determine CPU cycles necessary to run ensemble systems, as well as calibration techniques (Gov.) Advocate, procure, and install necessary high-performance computers (Gov./All) Extend NOAA Operational Model Archive and distribution System (NOMADS) to accommodate raw/post- processed ensemble output (Gov.) Determine CPU cycles necessary to run ensemble systems, as well as calibration techniques (Gov.) Advocate, procure, and install necessary high-performance computers (Gov./All) Extend NOAA Operational Model Archive and distribution System (NOMADS) to accommodate raw/post- processed ensemble output (Gov.) Build web site(s) for educators to share resources (Lead: Aca.) Assimilate new uncertainty material into undergraduate courses (Aca.) Incorporate forecast uncertainty into media (Lead: Com.) Apply forecast uncertainty to economic sectors (Com.) Build web site(s) for educators to share resources (Lead: Aca.) Assimilate new uncertainty material into undergraduate courses (Aca.) Incorporate forecast uncertainty into media (Lead: Com.) Apply forecast uncertainty to economic sectors (Com.)

27. 27 Discussion Next Steps Complete Version 0.5 of Plan (Aug. 09)Complete Version 0.5 of Plan (Aug. 09) Full Committee Review – Ver. 1.0 (Sept. 09)Full Committee Review – Ver. 1.0 (Sept. 09) Stakeholder Review – Ver. 2.0 (Nov. 09)Stakeholder Review – Ver. 2.0 (Nov. 09) AMS Town Hall (Jan. 10)AMS Town Hall (Jan. 10) Final draft (Feb. 10)Final draft (Feb. 10) AMS Leadership Review (Mar. 10)AMS Leadership Review (Mar. 10) Publication (Apr. 10)Publication (Apr. 10)

28. 28 Discussion Summary and Monitoring Progress ACUF vision:ACUF vision: –Enterprise partnership generates & communicates comprehensive, skillful, and reliable forecast uncertainty information meeting Nation’s decision-making needs Emerging Strategic Implementation Plan:Emerging Strategic Implementation Plan: –Recommends strategic goals, objectives, and implementation tasks for Enterprise partners over next decade –Is not programmatic – does not define cost, schedule, and performance information. This is purview of decision makers throughout Enterprise. Committee recommends:Committee recommends: –Plan be used by program managers/decision makers to guide activities and partnerships –AMS Commission Steering Committee (CSC) monitor progress

29. 29 To Get More Involved Please contact Andrea Bleistein andrea.bleistein@noaa.gov andrea.bleistein@noaa.gov

30. 30 Backup

31. 31 Draft Plan Uses and Benefits of Forecast Uncertainty Typical year brings 6 hurricanes, 1200 tornadoes, 5000 floods, 10,000 violent thunderstorms, … causing 500 deaths, 5000 injuries, and $14 billion in losses Lengthening “warning” lead times is one key to saving more lives and property Protecting Life and Protecting Life and Property Current “Warn on Detection” paradigm provides up to ~20m warning lead time based on observations Incipient Storm 1-hour later Future “Warn on Probability” paradigm may provide up to ~1h warning lead time based on probabilistic thresholds

32. 32 Roadmap Sample Tasks Synthesize results of previous studies on limits of predictability and how it is dependant on phenomena, scale, and metrics (Lead: Aca.) Sustain current funding of ensemble and predictability research through programs such as THORPEX. (Lead: Aca., under Gov. funding) Host special fora for social science and related research community to interact with hydrometeorological community (Lead: NGO) Facilitate building public/private consortia for funding research on forecast uncertainty communication (Lead: NGO/Aca.) Test EnKF and explore hybridization of variational and EnKF (Lead: Gov.) Work w extramural researchers and lab partners to test experimental ensemble forecast systems in operational environments (Lead: Gov.) Increase SREF resolution 3-fold (to 10-20 km) (Lead: Gov.) Continue to test algorithms to correct/calibrate ensembles including Ensemble Kernel Density MOS (Lead: Gov.) Input weather into a hydrologic ensemble prediction system (Lead: Gov./ Aca.)

33. 33 Roadmap Sample Tasks, (cont.) Identify existing web-based training on uncertainty and probabilistic NWSP Develop materials for in-person and on-line training. Develop prototype probabilistic verification packages for particular applications, such as aviation forecasts Form team to identify a standard set of methodologies for evaluation of uncertainty products, evaluate new metrics and publish a verification manuel Meet all probabilistic NextGen requirements for 4d-Wx Cube Develop sample problems illustrating concepts of probability and statistics in hydrometeorological applications and put into DLESE?? (Lead: NGO) Build web site(s) for educators to share resources (Lead: Aca.) Assimilate new unceratinty material into undergraduate cources (Lead: Aca.) Incorporate forecast uncertainty into media (Lead: Com.) Determine CPU cycles necessary to run global, regional, and extreme-event ensemble systems, as well as calibration techniques (Lead: Gov.) Advocate, procure, and install necessry high-performance computers (Lead: Gov/All) Extend NOAA Operatioanl Model Archive and distribution System (NOMADS) to accommodate raw/postprocessed ensemble output Use the Developmental Testbed and other testbeds to test experimental ensemble techniques (Lead: Gov)