Decision making as a model 2. Statistics and decision making

Bayesian statistics: p(H|D)  p(H) ∙p(D|H) If H refers to possible values of θ: pdf(θ|D)  pdf(θ) ∙L (θ|D)NB: L: Likelihood function! From about 1925 Bayesian approach in inductive statistics was marginalised (now a come back)

In “classial ” statistics frequentist interpretation of probability is preferred Hypotheses are TRUE or FALSE (we don’t know which for certain - not a matter of probability) and are accepted of rejected based on data D and likelihood p(D|H) e.g. test of significance

Statistic S compute pdf(S|H 0 ) (for sample of n) If p is small, reject H 0, you could accept some alternative SxSx p Fisher Null hypothesis about some population parameter do experiment (  S x, p) probability density

Neyman & Pearson pdf(S|H 0 )pdf(S|H 1 ) do experiment, compute S x and choose between H 0 and H 1 Statistic S Specify H 0,H 1 and their pdf’s. Decide on a criterion based on β p(type II error) and α p(type I error)

Neyman & Pearson more suitable for decision making than for science! For completeness: Likelihood approach without priors: Fisher, Royall p(H|D)  p(H)∙p(D|H) Irrespective of p(H): how strong is D’s support for H ? Example: model selection: AkaikeAIC = -log(L) + k BIC = -log(L) + k log(n)/2

Military technology (WW2): Signal Detection Theory Application of Application of Neyman-Pearson to processing sonar or radar signals on noisy background

Hypothesis 0: there is no signal, only noise Hypothesis 1: there is a signal and noise NB.1 On the basis of some “evidence” I have to act, although I do not know which H is true! NB.2This is typically a “classic” approach, but at the end Bayes will creep in by the back door!

“Evidence”, e.g.…..???? 1.Effect (= a value of “Evidence”) of signal is variable (according to a probability distribution). 2. Effect of Noise is also variable. Probability density Problem: is this “Evidence” (= a point on x-axis) the effect of a signal (+ noise) or of noise only? fundamental assumptions of signal detection theory

3. If signal is weak, distributions overlap and errors are unavoidable, whichever criterion is adopted “No” “Yes”

Signal (+noise) (only) noise miss hit correct rejection false alarm Terminology:

The stronger the signal (or the better the detector) … the further the distributions lie apart

“No” “Yes” Given some sensitivity (= a distribution for noise and one for signal) several response criteria can be adopted Dependent on van personal preference or “pay off” in this situation: -How bad is a miss, how important is a hit? -How bad is a false alarm, how important is a correct rejection? -Hoe often do signals occur? (think of Bayes!)

Two types of applications: 1.Normative: distributions are known, try to find optimal criterion (for optimal behavior) -Is that a hostile plane? -Does this mammogram indicate a malignancy? -Is there a weapon in this suitcase? -Can we admit this student to this school? -What is the best cut-off score for this test?

Two types of application: 2.Descriptive: Behavior is known, try to reconstruct distributions and criterion as a rational model How good is this person in detecting a v among u’s? Is this person inclined to say “yes” in a recognition test? How well judges or juries are able to distinguish between the guilty and the innocent? Do judges and lay juries differ in their bias for convicting or acquitting? How good is this test?.

Hit rate = Proportion hits (of signal trials) False Alarm Rate = Proportion false alarms (of noise trials) “No” “Yes” An experiment with noise (blank) and and signal (target) trials: A strict (“high”) criterion results in few hits and few false alarms

false alarms hits “No” “Yes” A lax “low” criterion results in more hits and more false alarms -given the same sensitivity

connects points in a Hit/FA- plot, resulting from adopting several criteria given the same sensitivity (= same distributions) ROC-curve characterises detector sensitivity (or signal strength) independent of criterion important: sensitivity and criterion theoretically independent The ROC-(response operating characteristic) curve

ROC-curve Receiver Operating Characteristic Relative Operating Characteristic Isosensitivity Curve false alarms hits Same sensitivity (for this signal), several criteria

Greater sensitivity: ROC-curve further from diagonal false alarms hits (Perfection would be: all hits and no false alarms)

Suggests two types of measure for sensitivity (independent of criterion:) 2.Area under ROC-Curve: A 1.distance between signal and noise distributions (e.g. d ' )

No distinction between signal and noise: A =.50 (ROC-curve reflects only bias for saying “yes” or “no”)

Perfect distinction between signal and noise: A  1.

Types of measures for criterion: 2. Likelihood ratio p(x c |S)/p(x c |N) = h/f (e.g. β) h f 1. Position on op x-axis (e.g. c) 3. Position in ROC-plot (left down. vs right up) 4. Slope of tangent on ROC c

Signal Detection Theory is applied in many contexts! Breast cancer?

PSA-indices for screening prostate cancer FA rate Hit rate

Psychodiagnosis: 1.How good is this test distinguishing relevant categories? 2.What is good cut-off score (at which score should I hire the candidate/admit the student / send the cliënt to a psychiatrist or an asylum? Control group patients Test score

Comer & Kendall 2005: Children’s Depression Inventory detects depression in a sample of anxious and anxious + depressive children Several cut-off scores

What are the costs missing a weapon/explosive at an airport? What are the costs a false alarm? What are the costs of screening (apparatus, personnel, delay)?