1. Evaluating a Firearms Qualification Program EAC 584 North Carolina State University Team 4 Ashlea Anderson Selby Bass Taylor Francis Stephanie Goins Ashleigh Watts

2. Overview Wilson, NC Police Department Annual In- Service Firearms Training Level 1, 2, and 3 measures incorporated – Analysis of levels 1 and 2 Primary conclusions and suggestions for improvement in future courses

3. Program Course mandated by the State of North Carolina for all law enforcement officers – One day, 10-hour course with three parts – 4 hours of classroom academics with final exam – Day and night qualifying on firing range – Obstacle Course Eighteen participants – Approximately 9% of total force – Approximately 20% of total annual qualifiers

4. Methodology Level 1 – Reaction survey created by Team 4 – 7 five-point Likert scale questions, 3 open-ended questions Level 2 – Observational checklist created by Team 4 and completed by course instructor (SME) – Exam created and mandated by North Carolina Justice Academy (NCJA) Level 3 – Obstacle course simulating physical and mental stress and fatigue of possible real-world experience – Requirements and pass/fail criteria developed by NCJA

5. Level 1

6. Results and Data Level 1 – Three types of analysis for Likert scale questions Average response per respondent Average response per question Variance – Open-ended answers analysis Answers categorized by themes Overall summary of themes

7. Level 1 – Likert Scale

11. Level 1 – Open-ended List two things that you would change or improve: CategoryNumber of CommentsExample of Comments More shooting/combat drills11 I would like to see more time on the range shooting Increased training on Firearm maintenance and mechanical knowledge 4 More training on the various parts of the firearm Different types of firearms3 I would like to get experience with different weapons Increased instructor feedback 3 Instructors watch each person shoot and critique them Improved shooting range2Better, more up to date range

12. Level 1 – Open-ended CategoryNumber of CommentsExample of Comments Shooting opportunity11 Opportunity to fire my service weapon Proper Stance5Practicing stances Use of force instruction4 Reviewing use of force – policy statement Policy and procedures review4 Where you can carry your handgun off duty Quality training4Great teachers List two things that you benefited from the most:

13. Level 1 – Open-ended CategoryNumber of CommentsExample of Comments Yes10Yes, to be retrained Yes, more often6Yes, probably semi-annually Should this training be provided on an annual basis?

14. Level 1 – Open Ended Summary 47% of responses indicate that participants would prefer more hands on time at the firing range Participants value the opportunity to practice firing their weapon during training Policy and procedure instruction also significantly impacted participants’ knowledge regarding accuracy and use of their firearm Trainees felt the course fosters better efficiency, proficiency, and consistency when it comes to application as needed in the field. All participants agree that the training should be provided on an annual basis, if not more often – 100% stated the training should be provided on an annual basis – Approximately 31% voted semi-annually

15. Analysis Level 1 – Overall positive response – Respondent 1 – Outlier? Misunderstanding? – Minimal variance – Strongest area: “Use of Force” v. “Use of Deadly Force” IAW NC State Law – Weakest areas (tie): Overall impact on marksmanship and Identification of individual handgun parts

16. Level 2

17. Results and Data Level 2 – Observation Checklist Meets Standards or Does Not Meet Standards Individual participant data and comments provided by instructor Summary of data and comments created by Team 4 – Exam Average score for class provided by instructor No further analysis conducted – Tool not created by Team 4 and individual data not available

18. Level 2 - Observation

19. Level 2 – Observation Eleven separate measures – Safety – Stance – Accuracy – Loading/unloading revolver – Loading/unloading semi-automatic – Drawing and holstering weapon – Malfunctions – Night procedures without emergency equipment – Night procedures with emergency equipment – Day qualification – Obstacle course performance

20. Analysis Level 2 – Observation Every participant met standards in all categories – Strongest area: Safety – Weakest area: Stance Attempted to avoid subjective judgments – Exam All students achieved passing grade Average score: 92% Individual scores not available

21. Level 3

22. Results and Data Level 3 – Obstacle Course Controlled simulation dictated and governed by NCJA Limited data provided and no analysis conducted

23. Analysis Level 3 – Obstacle Course Graded as pass or fail

24. Conclusions Positive response from students to program All measures of success met by all participants Actual impact of class on performance? Changes to the qualification time period?

25. Areas for Improvement Larger and more diverse sample size Development of more in-depth level 3 observation tool to accompany simulation Further observation and analysis of on the job performance

26. Q & A Questions?