Presentation on theme: "Error Rates and Random Match Probabilities (RMP) Based on the RUGER 10-Barrel Test and the GLOCK Cartridge Case Tests James E. Hamby, Ph.D., David J."— Presentation transcript:
1 Error Rates and Random Match Probabilities (RMP) Based on the RUGER 10-Barrel Test and the GLOCK Cartridge Case Tests James E. Hamby, Ph.D., David J. Brundage, M.S., Steven A. Norris, B.S., Nicholas D.K. Petraco, Ph.D., and James W. Thorpe, Ph.D.
2 Conclusions on Research Conducted over nearly 25 years involving the RUGER 10-Barrel Test, GLOCK Cartridge Case Tests Base error rates were established for both studies using advanced Bayesian techniques
3 The 10-Barrel TestThe Identification of Bullets Fired From 10 Consecutively Rifled RUGER 9mm Pistol BarrelsA Research Project Involving 626 Scientists From Labs In 24 Countries
4 James E. Hamby, Ph.D.,International Forensic Science Laboratory & Training Centre, Indianapolis, IndianaDavid J. Brundage, M.S.,Forensic Science Consultant, Nashville,TennesseeJames W. Thorpe, Ph.D.,University of Strathclyde, Glasgow, Scotland(Retired)
5 A total of 240 test sets were produced by Dave Brundage, Mickey French and Jim Hamby – all then employed at the Indianapolis-Marion County Forensic Laboratory, Indianapolis, INRecently 20 additional test sets were produced by Hamby and 10 sets sent to Europe and other countries for examination. The various test sets have used 4 different types of 9mm ammunition.Of the 626 participants in this research project, nine have used some type of ‘ballistics’ imaging equipment.Brundage and Hamby have since retired from the laboratory. Mickey French is now a Special Agent with the Bureau of Alcohol, Tobacco, Firearms & Explosives (BATFE)
6 Of the 626 participants in this worldwide project,9,382 of the possible 9,390 unknown bullets were identified. (15 unknowns x 626 participants = 9,390)Three bullets were mutilated during test firing (tank rash) and five ‘unknown’ bullets could not be associated to the known bullets by two young examiners-in-training - listed as inconclusive.‘Tank’ rash occurs when the bullet strikes the bottom of the tank obliterating the individual striae required for identification -
8 This worldwide research project and numerous other research projects reported in the scientific literature – the AFTE Journal, the Journal of Forensic Sciences, and others - since the early 1930’s should provide data that will obviate some of the concerns within the legal community -especially in the United States.
9 Request for Assistance We request that those members who haven’t participated ‘take’ the test in their laboratory & submit their answers!We also request that you return the set if you are no longer using it as there are labs that have requested test sets. Some labs may have two sets in their possession.
10 GLOCK Cartridge Case Test The Examination, Evaluation and Identification of 9mm Caliber Cartridge Cases Fired from 1,632 GLOCK Semiautomatic Pistols Manufactured Over a 23 Year Period - Using Optical Microscopy, Confocal Microscopy and Computational Pattern Recognition
11 This paper was presented at the 2013 AAFS Meeting by Petraco & Hamby with Norris as a co-author. The 1,632 fired cases were examined and the original 617 fired cases evaluated using the heritage IBIS.
14 This GLOCK research involved both IBIS and examiner based results This GLOCK research involved both IBIS and examiner based results. (over 1,300,000 exams) Nick Petraco will discuss the Error Rates/RMP for both research projects in a few minutes.
15 Modified SWGGUN Power Point Using the excellent SWGGUN power point, I have made a few changes that work well when testifying especially in Frye or Daubert Hearings. The following are a few of the changes. Our beloved AV man–Mike–has the power point if you are interested.
16 Definition: Subjective Examination The results of a subjective examination are based on an individual’s opinion based on his/her training and experience. This does not mean that this type of examination is unreliable or unscientific.There is subjectivity in every science and in every test, whether it is a doctor diagnosing a head cold, a physician examining an x-ray or a cytotechnologist examining slides for cancer cells (pap smears).
17 National Institute of Justice, Washington, D.C. Grant or Accepted Government Teaching Programs for Firearms IdentificationNational Institute of Justice, Washington, D.C.NIJ - in conjunction with the National Forensic Science Technology Center (NFSTC) – funded a distance learning program titled ‘Firearms Examiner Training Program’ to augment training for new examiners,Federal Bureau of Investigation (FBI)Since 1986, the FBI has offered a one week course titled ‘SpecializedTechniques in Firearms Identification’. The course has been attended by several hundred firearms examiners.Bureau of Alcohol, Tobacco and Firearms (ATF)Since 1999, the ATF has offered a one-year long course titled the ‘National Firearms Examiner Academy. To date, 140+ firearms examiners have attended the NFEA. (Funded by ATF and NIJ)California Criminalistics Institute (CCI), CA DOJFor several years, CCI has offered a variety of specialized courses in firearms and toolmark related fields.
18 United Nations Office on Drugs and Crime (UNODC) Grant or Accepted Government Teaching Programs for Firearms IdentificationUnited Nations Office on Drugs and Crime (UNODC)In 2013, the UNODC – with funding from the Canadian Government – is developing a limited service forensic laboratory for the Palestinian Civil Police (PCP) Temporary Forensic Science Laboratory (TFSL) in Ramallah, Palestine.. The TFSL will have the following forensic specialties:Forensic Chemistry – Drug and Arson AnalysisQuestioned Documents AnalysisFirearms and Tool Mark IdentificationUnited States Department of StateIn 2014, the US Department of State – through the Bureau of International Narcotics and Law Enforcement Affairs, US Embassy, Belize, funded a program to strengthen the Belize National Forensic Science Laboratory by providing technical assistance related to the handling, processing and analysis of ballistics (firearms / tool marks) related evidence.Other programs such as ICITAP teach firearms and tool mark Identification around the world.
19 SummaryFirearm & Toolmark Identification meets the reliability standard put forth by both Daubert or Frye requirements. The science has been accepted in every state and Federal court that uses Daubert or Frye.Examples: A recent (9-11) Indiana Supreme Court Decision (Frye) upheld the reliability of Firearms and Tool Mark Identification; and a recent (12-11) Kansas District Court Decision (Frye) upheld ‘general acceptance’ in the field. A recent (7-13) Florida CourtIssued a Daubert decision upholding the science. Cases are available for review on the SWGGUN website (under ARK) at
20 Data Analysis Average Examiner Error Rate and Random Match Probability The Good News:10-Barrel Test: # examiner errors committed = 0GLOCK Cartridge Case Test: # of falsely matched cartridge cases = 0THIS SHOULD BE GREAT!We WANT error rates of 0% for good classification systems!!
21 Data Analysis Average Examiner Error Rate and Random Match Probability You have to deal with the U.S. court system0% error rates present an opportunity to muddy waters with the “unrealistic study” criticismsLittle court interest in understanding the technicalities of estimating small error rates.The Bad News:
22 Data Analysis Average Examiner Error Rate and Random Match Probability So what can we do?0% error rate is the “frequentist” estimate“Bayesian” statistics provide complementary methodsCan work much better in estimating small probabilitiesWe looked to sports statistics for low scoring games
23 Data Analysis Average Examiner Error Rate and Random Match Probability For 10-Barrel we need to estimate a small error rateFor GLOCK we need to estimate a small random match probability (RMP)Use Bayesian “Beta-binomial” method when no “failures” are observed (Schuckers)
24 Data Analysis Bayesian Methodology Basic idea of the reverend Bayes:Prior Knowledge × Data =Updated Knowledgea + baError Rate/RMP =U(a,b) × Beta-Binomial(data | a,b)Posterior(a,b | data)Get updated estimates of Error rate/RMP
25 Data Analysis Bayesian Methodology So given the observed data and assuming “prior ignorance”Posterior error rate/RMP distributions:Posterior Dist. 10-BarrelPosterior Dist. GLOCKRMP%[ %, %]Average Examiner Error Rate0.011%[ %, 0.040%]
26 Data Analysis Extrapolations 10-BarrelGLOCKActual:626 participantsActual:1632 cartridge casesRMP (%)Examiner Error Rate (%)# of Participants# Cartridge Cases ExaminedWith No Errors in I.D.