Error Rates and Random Match Probabilities (RMP) Based on the RUGER 10-Barrel Test and the GLOCK Cartridge Case Tests James E. Hamby, Ph.D., David J. Brundage, M.S., Steven A. Norris, B.S., Nicholas D.K. Petraco, Ph.D., and James W. Thorpe, Ph.D.

Conclusions on Research Conducted over nearly 25 years involving the RUGER 10-Barrel Test, GLOCK Cartridge Case Tests Base error rates were established for both studies using advanced Bayesian techniques

The 10-Barrel Test The Identification of Bullets Fired From 10 Consecutively Rifled RUGER 9mm Pistol Barrels A Research Project Involving 626 Scientists From Labs In 24 Countries

James E. Hamby, Ph.D., International Forensic Science Laboratory & Training Centre, Indianapolis, Indiana David J. Brundage, M.S., Forensic Science Consultant, Nashville, Tennessee James W. Thorpe, Ph.D., University of Strathclyde, Glasgow, Scotland (Retired)

A total of 240 test sets were produced by Dave Brundage, Mickey French and Jim Hamby – all then employed at the Indianapolis-Marion County Forensic Laboratory, Indianapolis, IN Recently 20 additional test sets were produced by Hamby and 10 sets sent to Europe and other countries for examination. The various test sets have used 4 different types of 9mm ammunition. Of the 626 participants in this research project, nine have used some type of ‘ballistics’ imaging equipment. Brundage and Hamby have since retired from the laboratory. Mickey French is now a Special Agent with the Bureau of Alcohol, Tobacco, Firearms & Explosives (BATFE)

Of the 626 participants in this worldwide project,9,382 of the possible 9,390 unknown bullets were identified. (15 unknowns x 626 participants = 9,390) Three bullets were mutilated during test firing (tank rash) and five ‘unknown’ bullets could not be associated to the known bullets by two young examiners-in-training - listed as inconclusive. ‘Tank’ rash occurs when the bullet strikes the bottom of the tank obliterating the individual striae required for identification -

This worldwide research project and numerous other research projects reported in the scientific literature – the AFTE Journal, the Journal of Forensic Sciences, and others - since the early 1930’s should provide data that will obviate some of the concerns within the legal community -especially in the United States.

Request for Assistance We request that those members who haven’t participated ‘take’ the test in their laboratory & submit their answers! We also request that you return the set if you are no longer using it as there are labs that have requested test sets. Some labs may have two sets in their possession.

GLOCK Cartridge Case Test The Examination, Evaluation and Identification of 9mm Caliber Cartridge Cases Fired from 1,632 GLOCK Semiautomatic Pistols Manufactured Over a 23 Year Period - Using Optical Microscopy, Confocal Microscopy and Computational Pattern Recognition

This paper was presented at the 2013 AAFS Meeting by Petraco & Hamby with Norris as a co-author. The 1,632 fired cases were examined and the original 617 fired cases evaluated using the heritage IBIS.

1,632 GLOCK Cartridge Cases

This GLOCK research involved both IBIS and examiner based results This GLOCK research involved both IBIS and examiner based results. (over 1,300,000 exams) Nick Petraco will discuss the Error Rates/RMP for both research projects in a few minutes.

Modified SWGGUN Power Point Using the excellent SWGGUN power point, I have made a few changes that work well when testifying especially in Frye or Daubert Hearings. The following are a few of the changes. Our beloved AV man–Mike–has the power point if you are interested.

Definition: Subjective Examination The results of a subjective examination are based on an individual’s opinion based on his/her training and experience. This does not mean that this type of examination is unreliable or unscientific. There is subjectivity in every science and in every test, whether it is a doctor diagnosing a head cold, a physician examining an x-ray or a cytotechnologist examining slides for cancer cells (pap smears).

National Institute of Justice, Washington, D.C. Grant or Accepted Government Teaching Programs for Firearms Identification National Institute of Justice, Washington, D.C. NIJ - in conjunction with the National Forensic Science Technology Center (NFSTC) – funded a distance learning program titled ‘Firearms Examiner Training Program’ to augment training for new examiners, Federal Bureau of Investigation (FBI) Since 1986, the FBI has offered a one week course titled ‘Specialized Techniques in Firearms Identification’. The course has been attended by several hundred firearms examiners. Bureau of Alcohol, Tobacco and Firearms (ATF) Since 1999, the ATF has offered a one-year long course titled the ‘National Firearms Examiner Academy. To date, 140+ firearms examiners have attended the NFEA. (Funded by ATF and NIJ) California Criminalistics Institute (CCI), CA DOJ For several years, CCI has offered a variety of specialized courses in firearms and toolmark related fields.

United Nations Office on Drugs and Crime (UNODC) Grant or Accepted Government Teaching Programs for Firearms Identification United Nations Office on Drugs and Crime (UNODC) In 2013, the UNODC – with funding from the Canadian Government – is developing a limited service forensic laboratory for the Palestinian Civil Police (PCP) Temporary Forensic Science Laboratory (TFSL) in Ramallah, Palestine.. The TFSL will have the following forensic specialties: Forensic Chemistry – Drug and Arson Analysis Questioned Documents Analysis Firearms and Tool Mark Identification United States Department of State In 2014, the US Department of State – through the Bureau of International Narcotics and Law Enforcement Affairs, US Embassy, Belize, funded a program to strengthen the Belize National Forensic Science Laboratory by providing technical assistance related to the handling, processing and analysis of ballistics (firearms / tool marks) related evidence. Other programs such as ICITAP teach firearms and tool mark Identification around the world.

Summary Firearm & Toolmark Identification meets the reliability standard put forth by both Daubert or Frye requirements. The science has been accepted in every state and Federal court that uses Daubert or Frye. Examples: A recent (9-11) Indiana Supreme Court Decision (Frye) upheld the reliability of Firearms and Tool Mark Identification; and a recent (12-11) Kansas District Court Decision (Frye) upheld ‘general acceptance’ in the field. A recent (7-13) Florida Court Issued a Daubert decision upholding the science. Cases are available for review on the SWGGUN website (under ARK) at www.swggun.org.

Data Analysis Average Examiner Error Rate and Random Match Probability The Good News: 10-Barrel Test: # examiner errors committed = 0 GLOCK Cartridge Case Test: # of falsely matched cartridge cases = 0 THIS SHOULD BE GREAT! We WANT error rates of 0% for good classification systems!!

Data Analysis Average Examiner Error Rate and Random Match Probability You have to deal with the U.S. court system 0% error rates present an opportunity to muddy waters with the “unrealistic study” criticisms Little court interest in understanding the technicalities of estimating small error rates. The Bad News:

Data Analysis Average Examiner Error Rate and Random Match Probability So what can we do? 0% error rate is the “frequentist” estimate “Bayesian” statistics provide complementary methods Can work much better in estimating small probabilities We looked to sports statistics for low scoring games

Data Analysis Average Examiner Error Rate and Random Match Probability For 10-Barrel we need to estimate a small error rate For GLOCK we need to estimate a small random match probability (RMP) Use Bayesian “Beta-binomial” method when no “failures” are observed (Schuckers)

Data Analysis Bayesian Methodology Basic idea of the reverend Bayes: Prior Knowledge × Data = Updated Knowledge a + b a Error Rate/RMP = U(a,b) × Beta-Binomial(data | a,b) Posterior(a,b | data) Get updated estimates of Error rate/RMP

Data Analysis Bayesian Methodology So given the observed data and assuming “prior ignorance” Posterior error rate/RMP distributions: Posterior Dist. 10-Barrel Posterior Dist. GLOCK RMP 0.000086% [0.0000020%, 0.00031%] Average Examiner Error Rate 0.011% [0.00023%, 0.040%]

Data Analysis Extrapolations 10-Barrel GLOCK Actual: 626 participants Actual: 1632 cartridge cases RMP (%) Examiner Error Rate (%) # of Participants # Cartridge Cases Examined With No Errors in I.D.

Done