1. Improving the Quality of Investigative Decision Making Craig Bennell, Ph.D. Carleton University Ottawa, Ontario, Canada cbennell@connect.carleton.ca

2. Two-Alternative Decisions Is this suspect lying? Will this negotiation end in failure? Are these crimes linked? Was this bite mark made by an adult? Is this suicide note fake? Is this allegation of rape false? To improve decision-making we need a valid measure of performance...

3. How Should We Measure Performance? SET A DECISION THRESHOLD COUNT THE FREQUENCIES OF DECISION OUTCOMES CALCULATE PERCENTAGE CORRECT EXAMINE THE EVIDENCE

4. Relatively Inaccurate Evidence

5. Relatively Accurate Evidence

6. ROC ANALYSIS (Bennell, 2005)

7. ROC ANALYSIS (Bennell, 2005) An AUC of.60 indicates that there is a 60% chance that a randomly selected positive case will be higher on evidence X than a randomly selected negative case

8. The Accuracy of Diagnostic Systems: The Polygraph CQT and GKT The Control Question Test –Relevant (R) and Control (C) questions –For each R-C pair a score of -3 to +3 is given –Greater than +5: truthful, Less than -5: deceitful The Guilty Knowledge Test –Multiple choice questions –One alternative relates to guilty knowledge (GK) –Elevated responses to GK questions: deceitful How accurate are these procedures?

9. Accuracy scores for 52 lab studies examining the validity of the polygraph (National Academy of Science, 2003) Median AUC for GKT =.88 Median AUC for CQT =.85

10. Identifying Accurate Predictors: The Fake Suicide Note What discriminates between real and fake notes? –Structure variables –Content variables –Combinations of the two How can we tell if some are better than others? How can we determine how much better?

11. Probability of false alarms 1 - Structure model (AUC =.69) 2 - Content model (AUC =.74) 3 - Optimal model (AUC =.82) 4 - Inclusive model (AUC =.85) Optimal: Average sentence length (shorter in genuine) Positive affect (more in genuine) 4 3 2 1 (Jones & Bennell, 2005)

12. Setting Appropriate Thresholds: Linking Serial Burglaries Many behaviours can be used Inter-crime distance is very useful (AUC=.80) –The closer two crimes are to one another the more likely it is that they are linked When should we decide that crimes are linked? What is an appropriate threshold?

13. (Bennell & Canter, 2002) 2.50km (72% H, 25% FA) 1.50km (58% H, 11% FA) 3.50km (85% H, 45% FA) 1.10km (49% H, 10% FA)

14. Comparing Decision Makers: Bite Mark Identification It is often unclear as to the source of a bite mark –Non-accidental (adult) –Accidental (child) How well can people perform on this task? Is expertise required?

15. (Whittaker et al., 1998) Senior forensic dentists (AUC=.69) Student dentists (AUC=.69) Junior forensic dentists (AUC=.68) General dentists (AUC=.62) Social workers (AUC=.63) Police officers (AUC=.62)

16. Conclusions ROC analysis provides a way of calculating accuracy scores that are not dependent on arbitrary thresholds ROC analysis also provides a way of identifying thresholds that meet the demands of specific situations The result for police investigations will likely be improved decision-making accuracy and utility