1. Steve Lund and Hari Iyer steve.lund@nist.gov hari@nist.gov
Considering the Reliability of and Alternatives to Likelihood Ratios Used to Explain Forensic Evidence
Steve Lund and Hari Iyer

2. Disclaimer
Viewpoints expressed are our own and are not intended to reflect the viewpoints of anyone else at NIST
We are only discussing what makes sense to us and are not trying to claim that these are new perspectives

3. Illustration: Comparing Toolmark StriaeQ

4. Comparing Toolmark Striae – IllustrationQ
After digitally capturing and computer aligning of the striae from R and Q we can determine the maximum number of consecutively matching striae.

5. Comparing Toolmark Striae – Specific CaseQ
The maximum number of consecutive matching striae is 6

6. Reference Collection
Ground truth known 4

7. Reference Collection
Ground truth known

8. 4
Reference Collection
Ground truth known
Chance of getting a 4 when comparing marks from the same tool:
0.0159
0.0262
0.0471
2.52e-07
0.0006
0.0060
Chance of getting a 4 when comparing marks from different tools:
Min = 2.65

9. 4
Reference Collection
Ground truth known
Chance of getting a 4 when comparing marks from the same tool:
0.0159
0.0262
0.0471
2.52e-07
0.0006
0.0060
Chance of getting a 4 when comparing marks from different tools:
Min = 2.65; Max=187000

10. Some Comments on Probability
Probability is not a description of what’s been observed
Some might say it’s a personal belief, based on what information is currently available
Others might view it as a best guess for what would be observed, if we could collect unlimited data
Under either case, there is no single right answer.

11. Some Comments on LRs
Under the perspective of personal probabilities, the question is what is my LR, not what is the LR of any one particular expert
Expert can’t create an exact answer for anyone else’s LR, but may be able to characterize a range under different assumptions
Asking multiple experts may illustrate spread (at least partially)
Under the perspective of predicting what unlimited data would illuminate, use statistical inference tools to assess whether each considered model is consistent or inconsistent with currently available data
LR value is plausible if it corresponds to at least one plausible model
LR value only implausible if there’s no corresponding plausible model – this can be hard to demonstrate

12. Some Comments on LRs What’s the best value?
Under either perspective of probability
a best guess is expected to be wrong
an uncertainty is intended to characterize how far off you might be
What’s the best value?
Is the offered value reasonable?
What is the range of “reasonable” interpretations?

13. Reliability
Our working notion: To be reliable, a report/testimony must have a low risk of being wrong (in a way that affects the ultimate decision)

14. Reliability: Expert’s intended meaning

15. Reliability: Recipient’s interpretation
Rule 403. Excluding Relevant Evidence for Prejudice, Confusion, Waste of Time, or Other Reasons
The court may exclude relevant evidence if its probative value is substantially outweighed by a danger of one or more of the following: unfair prejudice, confusing the issues, misleading the jury, undue delay, wasting time, or needlessly presenting cumulative evidence.

16. LR Reliability Concern
Don’t want experts or recipients to falsely believe that any reasonable LR assessment would differ only negligibly from offered value
Guard against extreme values resulting from choice of model

18. LR Reliability Concern
Don’t want recipients to falsely believe that any reasonable LR assessment would differ only negligibly from offered value
Guard against extreme values resulting from choice of model
If providing an LR, try to investigate and express the range of reasonable values

19. Among different forms for expressing expert opinion (e. g
Among different forms for expressing expert opinion (e.g., categorical conclusions or probabilities of source), LRs have some clear advantages
Federal Rules of Evidence permit but do not advocate for experts communicating in the form of an opinion
Experts could focus on presenting an overview of the information available to them that would serve as the basis of an expert interpretation (cards are on the table)

20. Reference Collection
Ground truth known 11

23. Thank you!