G-131 & E/L-131: Exercise Evaluation and Improvement Planning Exercise Evaluation Overview Kenneth L. Wierman Jr. DHS/FEMA Headquarters 2010 National REP Conference

FEMA Directive 123-15 16 January 2009 Course Goal To familiarize individuals with exercise planning or evaluation responsibilities with the Homeland Security Exercise Evaluation Program’s (HSEEP) exercise evaluation and improvement planning process, and to provide those students opportunities to practice key exercise evaluation skills. FEMA Directive 123-15 16 January 2009 2010 National REP Conference

2010 National REP Conference Target Audience Individuals who manage, participate in, review, or sponsor exercise evaluation and/or improvement planning Federal, State, local, Tribal, Territorial government Private sector Non-governmental 2010 National REP Conference

2010 National REP Conference Course Prerequisites IS-120.A: Introduction to Exercises IS-130: Exercise Evaluation and Improvement Planning HSEEP Training Course Radiological Emergency Preparedness Program (REP) and Planning Course (Recommended) 2010 National REP Conference

Course Objectives (1 of 2) Describe the need for a systematic approach to exercise evaluation List the eight steps of the exercise evaluation and improvement planning process Identify pre-exercise activities necessary for a successful evaluation Explain the function of Exercise Evaluation Guides (EEGs) and their relationship to the Target Capabilities List (TCL)/Universal Task List (UTL) 2010 National REP Conference

Course Objectives (2 of 2) Practice evaluation observation, data analysis, and report writing skills Describe post-exercise analysis activities Explain the purpose and format of the After Action Report/Improvement Plan (AAR/IP) Describe how a Corrective Action Program (CAP) translates exercise outcomes into continual improvements to preparedness 2010 National REP Conference

2010 National REP Conference Course Structure (1 of 2) Unit 1: Exercise Evaluation Overview Unit 2: Exercise Evaluation Process Unit 3: Planning and Organizing the Evaluation Unit 4: Observing the Exercise and Collecting Data Unit 5: Analyzing Data Unit 6: After Action Report and After Action Conference Unit 7: The Corrective Action Program Course Summary (End of Two-Day Course) 2010 National REP Conference

2010 National REP Conference Course Structure (2 of 2) Unit 8: Introduction to Evaluator Skill Practice Unit 9: Customizing EEGs from TCLs and Objectives Unit 10: Exercise Evaluation: Skill Practice #1 Unit 11: Exercise Evaluation: Skill Practice #2 Unit 12: Exercise Evaluation: Skill Practice #3 Unit 13: Exercise Evaluation: Skill Practice #4 Unit 14: Course Summary 2010 National REP Conference

2010 National REP Conference Course Background Original course introduced by FEMA in 1992 In 2002, DHS developed and introduced the Homeland Security Exercise and Evaluation Program (HSEEP) All-hazards Performance- and capabilities-based All levels of government DHS and FEMA have worked collaboratively to standardize the language and concepts used in the exercise evaluation process. This course updated to reflect HSEEP guidance 2010 National REP Conference

Activity: Defining an Exercise What is an Exercise? Why do we Exercise? 2010 National REP Conference

2010 National REP Conference Defining an Exercise A focused practice activity that places players in a simulated situation Conducted to support an overall assessment of a capability Primary benefits: Individual and team training System improvement Critical component of preparedness cycle 2010 National REP Conference

Activity: Defining Exercise Evaluation What is Exercise Evaluation? 2010 National REP Conference

Defining Exercise Evaluation Exercise evaluation is the act of: Reviewing or observing and recording exercise activity or conduct Assessing/analyzing behaviors and activities against exercise objectives/standards Noting strengths, weaknesses, deficiencies, or other observations 2010 National REP Conference

Activity: Importance of Exercise Evaluation & Improvement Planning Why is Exercise Evaluation and Improvement Planning Important? 2010 National REP Conference

Importance of Exercise Evaluation & Improvement Planning (1 of 2) Assesses a jurisdiction or organization's performance by identifying strengths and areas for improvement Improvement planning Leverages the outputs of the evaluation process by developing Improvement Plans (IPs) Provides a disciplined process for implementing corrective actions and socializing best practices 2010 National REP Conference

Importance of Exercise Evaluation & Improvement Planning (2 of 2) Exercise evaluation and improvement planning are critical components of the preparedness cycle Implementation of corrective actions is the mechanism by which exercises can inform and improve other components of the preparedness cycle 2010 National REP Conference

Activity: Customizing Exercise Evaluation Guides (EEGs) Plan Ref Task Metric Yes No Time [Name, Pg/Sec#] Order transportation services for evacuation. [From plan] Coordinate with supporting agencies and prearranged providers to obtain appropriate means of transportation for those requiring transportation assistance (e.g., buses, ambulances, handicap-assisted vans). Implement plans for providing alternative means of transport for immobilized individuals or those needing other special assistance in transit. 2010 National REP Conference

Activity: Customizing Exercise Evaluation Guides (EEGs) Activity Analysis Observations (Each bullet will need a completed After Action Report Input form.) Strengths Areas for Improvement Root Cause Additional Observations: 2010 National REP Conference

2010 National REP Conference Activity: Customizing Exercise Evaluation Guides (EEGs) – AAR Input Form Observation [Insert a short, complete sentence that describes the general observation. Then select from the lists below whether the observation is a strength or an area for improvement. A strength is an observed action, behavior, procedure, and/or practice that is worthy of recognition and special notice. Areas for improvement are those areas in which the evaluator observed that a necessary task was not performed or that a task was performed with notable problems. Lastly, identify the capability element that this observation falls under.] Noted Strength Area for Improvement Capability Element Equipment Organization Personnel Planning Process Training References (Standards, Policies, or Plans) [List relevant plans, policies, procedures, laws, and/or regulations, or sections of these plans, policies, procedures, laws, and/or regulations. If no references apply to the observation, it is acceptable to list “N/A” or “Not Applicable.”] Analysis [Include a description of the behavior or actions at the core of the observation, as well as a brief description of what happened and the consequence(s) (positive or negative) of the action or behavior. If an action was performed successfully, include any relevant innovative approaches utilized by the exercise participants. If an action was not performed adequately, the root causes contributing to the shortcoming must be identified.] Recommendations [Insert recommendations to address identified areas for improvement, based on the judgment and experience of the evaluation team. If the observation was identified as a strength without corresponding recommendations, insert “None.”] 2010 National REP Conference

October 2009, February 2010 and April 2010 Pilot Courses October 2009, February 2010 and April 2010 30 students Federal Agencies State and Local FEMA Chemical Stockpile Emergency Preparedness Program (CSEPP) National Exercise Division (NED) Radiological Emergency Preparedness Program (REPP) 2010 National REP Conference

Upcoming HSEEP Exercises Columbia Generating Station – 30 August 2010 Indian Point – 20 September 2010 Susquehanna Steam Electric Station – 5 October 2010 Pilgrim and Quad Cities – 15 November 2010 2010 National REP Conference

2010 National REP Conference Summary What are the objectives of this course? What is exercise evaluation and improvement planning? Why is exercise evaluation and improvement planning important? 2010 National REP Conference

2010 National REP Conference