Causal networks in evidential reasoning David Lagnado UCL

Causal networks in evidential reasoning David Lagnado UCL

Leonard Vole accused of murdering a rich elderly lady Miss French
Romaine, Vole’s wife, was to testify that he was with her at time of murder Vole befriended French and visited her regularly including night of murder But instead Romaine appears as witness for prosecution Testifies that Vole was not with her, returned later with blood on his jacket, and said “I’ve killed her” Vole needed money French changed her will to include him; shortly after he enquired about luxury cruises Maid testified Vole was with French at time of death Letters written by Romaine to lover – reveals her plan to lie and incriminate Vole Blood on Vole’s jacket same type as French Vole is acquitted!

‘The jury's tasks are to weigh up the evidence, decide what has been proved and what has not and return a verdict based on their view of the facts’

‘Jurors evaluate evidence and reach a conclusion not by means of a formula, mathematical or otherwise, but by the joint application of their individual common sense and knowledge of the world to the evidence before them’

Forensic science approach
Report likelihood ratio, but leave prior and integration to the fact-finder

A difficult task! Lay people left with a huge task
Must integrate complex bodies of interrelated evidence Forensic evidence, witness testimonies, alibis, confessions expert evidence … Assess credibility of witnesses Conflicting arguments and stories Uncertainty, ambiguity, incompleteness … Need to reach a singular verdict

How do people do it? How should they do it? Jurors, judges, lawyers, investigators, …

A puzzle People are sophisticated reasoners in some respects
Constructing stories and explanations; causal reasoning; naive physics; counterfactual thinking; social reasoning; argumentation; problem solving … But can stumble on basic probability and logic problems

Cognitive science approach
How do people reason about evidence? What representations do they use? What inference processes? How does this compare with normative theories? Bayesian approaches (eg RSS guidelines) BNs for complex legal evidence (Dawid et al, 2011; Taroni et al, 2006; Fenton et al., 2013) How can we improve it? 9

Causal networks Story model (Pennington & Hastie, 1986, 1991, 1992, 1993) People construct stories to make sense of evidence Use scripts and schemas based on causal knowledge Focus on intentional actions, plans, beliefs of agents Broad empirical support But need to supplement with: How people represent evidence/reliability Probabilistic inference Normative benchmark

Representing evidence and reliability
Story model focuses on building a story/mental reconstruction of what happened But need to include ‘story of the trial’ Reliability & credibility of witnesses Evidence about reliability Cross-examination, contradictions, inconsistencies, etc Complex interactions between all these variables Bayes net approach based on legal idioms? (Fenton et al, 2103; Lagnado et al, 2013)

Legal idioms Evidential reasoning using simple building blocks
Capture generic causal schemas Reusable and combinable Qualitative structure Based on Bayesian networks Support various forms of reasoning, including predictive, diagnostic, hypothetical, counterfactual Fenton & Neil, 2012; Fenton, Neil & Lagnado, 2013; Lagnado, Fenton & Neil, 2013

Key idioms Cf reliabilty
Schum, Friedman, Dawid & Hepler; Bovens & Hartmann Hypothesis Evidence Evidence idiom Evidence-reliability reliability Crime Eyewitness Evidence Motive and opportunity Crime Evidence Alternative account Explaining away Motive Crime Opportunity Evidence

Represent a ‘plausible’ story of what happened
Vole in Will Vole poor Vole Motive Vole visits French Maid hears Vole Vole strikes French French dies Blood on Vole Vole returns to wife

Represent the evidence for the story
Vole visits French Maid hears Vole Vole strikes French French dies from blow Blood on Vole Vole in Will Vole poor Vole Motive Vole returns to wife Maid testimony Blood report Coroner report Documents Wife testimony

Represent the reliability of the evidence
Vole visits French Maid hears Vole Vole strikes French French dies from blow Blood on Vole Vole in Will Vole poor Vole Motive Vole returns to wife Maid testimony Wife testimony Blood report Coroner report Documents Reliability

And evidence for/against reliability
Vole visits French Maid hears Vole Vole strikes French French dies from blow Blood on Vole Vole in Will Vole poor Vole Motive Vole returns to wife Maid testimony Wife testimony Blood report Coroner report Documents Reliability Letters to lover

Status of framework Normative Prescriptive Descriptive
Formal model for probabilistic inference Prescriptive Guide to interpreting complex evidence and improving inference Not just to show if inference violates norms, but also what’s needed for good inference Descriptive Do people’s inferences conform to BN models? Qualitatively? Quantitatively?

Empirical Studies Witness testimonies (Connor Desai & Lagnado, 2016)
Alibi evidence (Lagnado, 2011; Lagnado et al, 2013) Discredited evidence (Lagnado & Harvey, 2009) Explaining away Lots of work on argumentation by Hahn, Oaksford, Harris, Corner … All show capability for qualitative inference that fits with BNs

Alibi evidence The defendant says that he was not at the scene of the crime when it was committed Often crucial evidence Treated with suspicion especially when provided by relative or friend (Culhane & Hosch, 2004; Olson & Wells, 2004) Often hard to provide strong alibi ‘where else would law-abiding citizens be at 3am but in bed with their spouse?’ Very little formal or empirical work 20

Witness testimony Woman is assaulted whilst walking home
X arrested on suspicion of assault Witness claims to have seen X leaving crime scene Simple model (reliability idiom) Priors & CPTs for illustration X ~X scene 1 0.1 scene reliable ~scene reliable scene ~reliable ~scene W 1 0.5

Witness testimony suppose we independently know X at crime scene
Testimony raises probability X did it suppose we independently know X at crime scene This ‘screens off’ witness report from guilt Tells us nothing extra about guilt

Alibi testimony suppose we know X was at crime scene
X claims he was not at crime scene SIMILAR MODEL? Alibi lowers guilt suppose we know X was at crime scene Intuitively this DOES NOT ‘screen off’ witness report from guilt Now we know X is lying! And natural to assume that he’s more likely to lie if he’s guilty

Alibi model New variable ‘motive to lie’
Add link from X committed crime CPTs X ~X Motive to lie 0.9 0.3 scene motive ~scene motive scene ~motive ~scene A 0.9 1 0.1

Alibi model This raises guilt
Alibi lowers guilt (not much) Prior to alibi report suppose you know X was at crime scene This raises guilt But adding the alibi raises it even further The false alibi is diagnostic of guilt

Alibi model Same model applies if someone else (friend, relative) is giving alibi for X But only if their motivation to lie depends on whether or not X is guilty eg If they know whether or not he committed the crime Otherwise no link from guilt to motive to lie and screening off applies as in witness model

Alibi model For example if mother gives alibi for son
And is motivated to lie But this does not depend on whether or not he is guilty Then her false alibi is not diagnostic of his guilt

Experiments Background story
Man arrested on suspicion of assault… fits victim’s description … no prior convictions … ALIBI 10 miles from crime scene at time of assault suspect mother Knows vs does not know if son is guilty driver CCTV Suspect clearly shown at crime scene close to time of assault CONTROL – CCTV evidence only No alibi given

Results Fits predictions of BN models
- Adverse inference drawn only if alibi giver knows whether suspect is guilty (ie suspect himself and ‘mother knows’)

Is lying dependent on guilt?
Suppose that suspect is guilty (not guilty), how likely is it that X would lie about his whereabouts at the time of the crime? Fit with alibi model assumptions – For suspect and ‘mother knows’, lying is diagnostic of guilt

Summary People draw inferences in line with proposed alibi model
Caution about assuming that they do so in a fully Bayesian way But use of qualitative networks seems key BN analysis clarifies what inferences to draw from a false alibi

Couple accused of intentionally harming their baby
They took the baby to the hospital when they discovered bleeding in his mouth But the doctors found bruises on the baby’s body And an X-ray revealed fractures But child was suffering from a rare blood disorder Causes bruising and bleeding Family courts ruled child abuse, and baby placed in adoption independent radiologist argued there were no fractures Court exonerates couple -- now fighting to get child back

Simplified Bayes net of case
Abuse Disorder Fractures X-Ray report2 X-Ray report1 Test Bruises Simplified Bayes net of case

Abuse and disorder increase (diagnostic inference)
Bruises Fractures Test Bruises= true X-Ray report1 X-Ray report2 Abuse and disorder increase (diagnostic inference) Fractures increase (predictive inference)

X-ray report increases prob of fractures and abuse
Disorder Fractures X-Ray report2 X-Ray report1 Test Bruises X-ray report1= true X-ray report increases prob of fractures and abuse Disorder is reduced ‘explaining away’ Main evidence in family court

Evidence for criminal court Positive test raises prob of disorder
Abuse Disorder Fractures X-Ray report2 X-Ray report1 Test Bruises Test= true Evidence for criminal court Positive test raises prob of disorder Lowers prob of abuse ‘explaining away’ Lowers prob of fractures

Complex patterns of inference – what do lay people do?
Abuse Disorder Fractures X-Ray report2 X-Ray report1 Test Bruises New X-ray report undermines previous report Reduces prob abuse And further boosts disorder Complex patterns of inference – what do lay people do?

Experiments Participants given abuse case
(150 in 3 very similar studies) Evidence presented sequentially Make probability judgments after each new piece of evidence Conditional probabilities elicited after main experiment

Probabilities P(abuse) = .40 P(disorder) = .28
F .70 .14 Fractures X-Ray report2 X-Ray report1 Test Bruises D ~D T .94 .08 F ~F R1 .91 .10 A & D A & ~D ~A & D ~A&~D B .90 .75 .18 F ~F R2 .90 .16 Use participants’ conditional probabilities to parameterize individual BNs

Compare actual judgments with posteriors generated by BNs using each individual’s parameters
Bruises Disorder Abuse Test Fractures X-Ray report2 X-Ray report1

Bruises Disorder Abuse Test Fractures X-Ray report2 X-Ray report1

Good qualitative fit with BN model +high individual correlations
Bruises Disorder Abuse Test Fractures X-Ray report2 X-Ray report1 Good qualitative fit with BN model +high individual correlations but a few discrepancies

Disorder drops with report2?
Bruises Disorder Abuse Test Fractures X-Ray report2 X-Ray report1 Disorder drops with report2? General doubt in reliability of hospital test?

Bruises have less impact than predicted by model
Disorder Abuse Test Fractures X-Ray report2 X-Ray report1 Bruises have less impact than predicted by model Excessive ‘explaining away’

Tendency to assume exclusive causes?
In experiment1 subjects told- “Doctors suggested two possible causes for the bleeding: abuse or a blood disorder” Bruises did not change probability of abuse or disorder Did subjects assume exclusive and exhaustive causes? bruises Abuse Disorder

Clarify in exp2 “Note that it is possible that both causes are true: e.g. that a child has been abused and has the disorder; it is also possible that neither are true, and that the symptoms arise due to other causes.” Now probability of abuse does increase (but still slightly less than predicted) bruises Abuse Disorder

Integrity of coat corrupted
Explaining away Barry George case FDR BG shot JD Integrity of coat corrupted P(FDR | cause1) = 1/100 P(FDR | cause2) = 1/100 Forensic experts argued that FDR irrelevant because equally (low) probability on either cause Likelihood ratio = 1

Integrity of coat corrupted
BUT Causes neither exclusive nor exhaustive FDR BG shot JD Integrity of coat corrupted P(FDR | cause1) = 1/100 P(FDR | cause2) = 1/100 So even if FDR does not discriminate between the two causes FDR still probative of guilt (vs not guilt) Both causes are raised

Conclusions Studies suggest people conform to qualitative prescripts of BNs Complex pattern of updating well fit by individual BN models But slight deviations from predicted model Tendency to treat alternative causes as exclusive?

Consistency between multiple witnesses
Idioms approach applies when there are multiple witnesses, whose testimonies might cohere or conflict

Experiments Based on transcript of actual trial (R v Capel)
Uses court dialogue which includes cross-examination of witnesses Should draw more attention to issues of credibility (trial not pre-selected for this, but typical of most jury trials)

Case Summary Defendant, Samuel Capel (SC), charged with assaulting Jay Dineen (JD), occasioning actual bodily harm (ABH) SC pleads not guilty

Prosecution argue that SC punched JD in face … unprovoked attack
Defence argue that SC punched JD in self-defence, and that JD was drunk and aggressive and pushed SC first Witnesses for Prosecution JD – claimed attack unprovoked, and he was not drunk JD’s friend – JD only drank 4 pints Policeman - saw SC punch JD, and SC run away JD did not seem drunk Witnesses for defence SC – claimed JD pushed him and he hit back in self-defence SC’s friend – supported SC’s claims Barman – did not mention any push, saw SC punch JD JD DRANK 8 PINTS JD WAS DRUNK SAID HE SAW JD PUSH SC CONSISTENT CONDITION INCONSISTENT CONDITION

Simplified Bayes net of case
SC guilty JD drunk Police testimony Force of punch JD provoked SC SC runs away DG testimony Barman testimony JD credible Stitches/hospital Police testimony JD testimony

Focus on consistency of three key witnesses
SC guilty JD drunk Drunk vs not drunk Police testimony JD provoked SC Friend testimony Barman testimony JD credible 8 pints vs 4 pints Push vs not push JD testimony JD “I only drank a few pints”

Empirical results When victim testimony inconsistent -
Victim judged less credible Defendant judged less guilty Victim credibility positively correlated with judged probability of guilt judgments of credibility through trial (similar for probability of guilt) friend police barman

Modeling with Bayes net
SC guilty JD drunk Police testimony JD provoked SC Friend testimony Barman testimony JD credible JD testimony

Use participants’ conditional probabilities to parameterize the graph
P(provoke|drunk) = .65 P(provoke|~drunk) = .50 Police says ‘drunk’ P(police| drunk) = .83 P(police| ~drunk) = .58 SC guilty JD drunk Police testimony JD provoked SC Friend testimony Friend says ‘8 pints’ Barman testimony JD credible P(friend| drunk) = .66 P(friend| ~drunk) = .31 P(barman| provoke) = .77 P(barman| ~provoke) = .35 JD testimony C&~D C&D ~C&D ~C&~D JD says few pints .80 .50 .65 .76 P(credible| provoke) = .37 P(credible| ~provoke) = .76

Use Bayesian updating to compute final probability of guilt
Based on each individual’s model

For consistent condition –
Victim credible Unlikely to be drunk Unlikely to have provoked defendant

For inconsistent condition –
Victim less credible Likely to be drunk Likely to have provoked defendant

Good fits with actual judgments both overall and on an individual basis
(NB but consistency of testimonies clearly not only deciding factor in case – need more complete model)

Summary Key role of credibility Modeled using idiom approach
Not claiming people use fully Bayesian reasoning (which is too complex!) But they seem to follow qualitative prescripts of Bayes net

Broader framework People’s causal mental models are critical to reasoning (Sloman & Lagnado, 2015) Both to construct stories about what happened and to evaluate how well these are supported by the evidence Idioms give building blocks for evidence evaluation

Understanding the good & bad in human reasoning
BNs both over- and under-estimate human reasoning Complex cases require complex reasoning People construct and use causal models but these are necessarily simplified and inference is heuristic* Hence possible biases - but these heuristics are also what allow us to be smart *Heuristics as ‘resource rational’ – cf Bramley et al, 2016; Goodman, Griffiths etc

Thank You! Norman Fenton Martin Neil Tamara Shengalia
Saoirse Connor Desai Phil Dawid Amanda Hepler William Twining

Understanding bias Reasoning errors due to shortcomings in people’s mental models, parameters and inference Over-simplified models & inference Flawed assumptions Misinterpreting/misusing probabilistic information Covers miscarriages of justice and biases in lab studies Suggests way to improve -

Improving probabilistic reasoning
How can we improve statistical reasoning? Thompson, Koehler, Gigerenzer, et al Cf. recent research by Hsu, Dewitt, Fenton, Lagnado But more needed – with better understanding of how people reason (and how they should) Use what people are naturally good at (eg causal reasoning; diagrams; frequencies; mental simulation) to improve what they are less competent at (eg statistical reasoning)

Causal networks in evidential reasoning David Lagnado UCL

Similar presentations

Presentation on theme: "Causal networks in evidential reasoning David Lagnado UCL"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Causal networks in evidential reasoning David Lagnado UCL

Similar presentations

Presentation on theme: "Causal networks in evidential reasoning David Lagnado UCL"— Presentation transcript:

Similar presentations

About project

Feedback