1. G-131 & E/L-131: Exercise Evaluation and Improvement Planning
Exercise Evaluation Overview
Kenneth L. Wierman Jr.
DHS/FEMA Headquarters
2010 National REP Conference

2. FEMA Directive 123-15 16 January 2009
Course Goal
To familiarize individuals with exercise planning or evaluation responsibilities with the Homeland Security Exercise Evaluation Program’s (HSEEP) exercise evaluation and improvement planning process, and to provide those students opportunities to practice key exercise evaluation skills.
FEMA Directive January 2009
2010 National REP Conference

3. 2010 National REP Conference
Target Audience
Individuals who manage, participate in, review, or sponsor exercise evaluation and/or improvement planning
Federal, State, local, Tribal, Territorial government
Private sector
Non-governmental
2010 National REP Conference

4. 2010 National REP Conference
Course Prerequisites
IS-120.A: Introduction to Exercises
IS-130: Exercise Evaluation and Improvement Planning
HSEEP Training Course
Radiological Emergency Preparedness Program (REP) and Planning Course (Recommended)
2010 National REP Conference

5. Course Objectives (1 of 2)
Describe the need for a systematic approach to exercise evaluation
List the eight steps of the exercise evaluation and improvement planning process
Identify pre-exercise activities necessary for a successful evaluation
Explain the function of Exercise Evaluation Guides (EEGs) and their relationship to the Target Capabilities List (TCL)/Universal Task List (UTL)
2010 National REP Conference

6. Course Objectives (2 of 2)
Practice evaluation observation, data analysis, and report writing skills
Describe post-exercise analysis activities
Explain the purpose and format of the After Action Report/Improvement Plan (AAR/IP)
Describe how a Corrective Action Program (CAP) translates exercise outcomes into continual improvements to preparedness
2010 National REP Conference

7. 2010 National REP Conference
Course Structure (1 of 2)
Unit 1: Exercise Evaluation Overview
Unit 2: Exercise Evaluation Process
Unit 3: Planning and Organizing the Evaluation
Unit 4: Observing the Exercise and Collecting Data
Unit 5: Analyzing Data
Unit 6: After Action Report and After Action Conference
Unit 7: The Corrective Action Program
Course Summary (End of Two-Day Course)
2010 National REP Conference

8. 2010 National REP Conference
Course Structure (2 of 2)
Unit 8: Introduction to Evaluator Skill Practice
Unit 9: Customizing EEGs from TCLs and Objectives
Unit 10: Exercise Evaluation: Skill Practice #1
Unit 11: Exercise Evaluation: Skill Practice #2
Unit 12: Exercise Evaluation: Skill Practice #3
Unit 13: Exercise Evaluation: Skill Practice #4
Unit 14: Course Summary
2010 National REP Conference

9. 2010 National REP Conference
Course Background
Original course introduced by FEMA in 1992
In 2002, DHS developed and introduced the Homeland Security Exercise and Evaluation Program (HSEEP)
All-hazards
Performance- and capabilities-based
All levels of government
DHS and FEMA have worked collaboratively to standardize the language and concepts used in the exercise evaluation process.
This course updated to reflect HSEEP guidance
2010 National REP Conference

10. Activity: Defining an Exercise
What is an Exercise?
Why do we Exercise?
2010 National REP Conference

11. 2010 National REP Conference
Defining an Exercise
A focused practice activity that places players in a simulated situation
Conducted to support an overall assessment of a capability
Primary benefits:
Individual and team training
System improvement
Critical component of preparedness cycle
2010 National REP Conference

12. Activity: Defining Exercise Evaluation
What is Exercise Evaluation?
2010 National REP Conference

13. Defining Exercise Evaluation
Exercise evaluation is the act of:
Reviewing or observing and recording exercise activity or conduct
Assessing/analyzing behaviors and activities against exercise objectives/standards
Noting strengths, weaknesses, deficiencies, or other observations
2010 National REP Conference

14. Activity: Importance of Exercise Evaluation & Improvement Planning
Why is Exercise Evaluation and Improvement Planning Important?
2010 National REP Conference

15. Importance of Exercise Evaluation & Improvement Planning (1 of 2)
Assesses a jurisdiction or organization's performance by identifying strengths and areas for improvement
Improvement planning
Leverages the outputs of the evaluation process by developing Improvement Plans (IPs)
Provides a disciplined process for implementing corrective actions and socializing best practices
2010 National REP Conference

16. Importance of Exercise Evaluation & Improvement Planning (2 of 2)
Exercise evaluation and improvement planning are critical components of the preparedness cycle
Implementation of corrective actions is the mechanism by which exercises can inform and improve other components of the preparedness cycle
2010 National REP Conference

17. Activity: Customizing Exercise Evaluation Guides (EEGs)
Plan Ref
Task
Metric
Yes No
Time
[Name, Pg/Sec#]
Order transportation services for evacuation.
[From plan]
Coordinate with supporting agencies and prearranged providers to obtain appropriate means of transportation for those requiring transportation assistance (e.g., buses, ambulances, handicap-assisted vans).
Implement plans for providing alternative means of transport for immobilized individuals or those needing other special assistance in transit.
2010 National REP Conference

18. Activity: Customizing Exercise Evaluation Guides (EEGs)
Activity Analysis
Observations (Each bullet will need a completed After Action Report Input form.)
Strengths
Areas for Improvement
Root Cause
Additional Observations:
2010 National REP Conference

19. 2010 National REP Conference
Activity: Customizing Exercise Evaluation Guides (EEGs) – AAR Input Form
Observation
[Insert a short, complete sentence that describes the general observation. Then select from the lists below whether the observation is a strength or an area for improvement. A strength is an observed action, behavior, procedure, and/or practice that is worthy of recognition and special notice. Areas for improvement are those areas in which the evaluator observed that a necessary task was not performed or that a task was performed with notable problems. Lastly, identify the capability element that this observation falls under.]
Noted Strength
Area for Improvement
Capability Element
Equipment
Organization
Personnel
Planning
Process
Training
References (Standards, Policies, or Plans)
[List relevant plans, policies, procedures, laws, and/or regulations, or sections of these plans, policies, procedures, laws, and/or regulations. If no references apply to the observation, it is acceptable to list “N/A” or “Not Applicable.”]
Analysis
[Include a description of the behavior or actions at the core of the observation, as well as a brief description of what happened and the consequence(s) (positive or negative) of the action or behavior. If an action was performed successfully, include any relevant innovative approaches utilized by the exercise participants. If an action was not performed adequately, the root causes contributing to the shortcoming must be identified.]
Recommendations
[Insert recommendations to address identified areas for improvement, based on the judgment and experience of the evaluation team. If the observation was identified as a strength without corresponding recommendations, insert “None.”]
2010 National REP Conference

20. October 2009, February 2010 and April 2010
Pilot Courses
October 2009, February 2010 and April 2010
30 students
Federal Agencies
State and Local
FEMA
Chemical Stockpile Emergency Preparedness Program (CSEPP)
National Exercise Division (NED)
Radiological Emergency Preparedness Program (REPP)
2010 National REP Conference

21. Upcoming HSEEP Exercises
Columbia Generating Station – 30 August 2010
Indian Point – 20 September 2010
Susquehanna Steam Electric Station – 5 October 2010
Pilgrim and Quad Cities – 15 November 2010
2010 National REP Conference

22. 2010 National REP Conference
Summary
What are the objectives of this course?
What is exercise evaluation and improvement planning?
Why is exercise evaluation and improvement planning important?
2010 National REP Conference

23. 2010 National REP Conference