True and Error Models of Response Variation in Judgment and Decision Tasks Michael H. Birnbaum

Overview I review three papers that are available at my Website that involve application and evaluation of models of variability in response to choice problems. Two papers are co-authored with Jeff Bahra on with tests of transitivity, stochastic dominance, and restricted branch independence. Our findings consistently rule out assumptions of iid that are required in certain models, such as the approach of Regenwetter, Dana, and Davis-Stober (2011) Psych Review. Transitivity is often satisfied, but a few show evidence of intransitive preferences. No individual satisfied CPT or the priority heuristic.

Testing Algebraic Models with Error-Filled Data Algebraic models assume or imply formal properties such as stochastic dominance, coalescing, transitivity, gain-loss separability, etc. But these properties will not hold if data contain “error.”

Some Proposed Solutions Neo-Bayesian approach (Myung, Karabatsos, & Iverson. Cognitive process approach (Busemeyer) “Error” Theory (“Error Story”) approach (Thurstone, Luce) combined with algebraic models. Random preference model: choices independent; no errors: variability due to iid sampling from mixture. Loomes & Sugden.

Variations of Error Models Thurstone, Luce: errors related to separation between subjective values. Case V: SST (scalability). Harless & Camerer: errors assumed to be equal for certain choices. Sopher & Gigliotti: Allow each choice to have a different rate of error, assumed transitivity. Birnbaum proposed using repetitions within block as estimates of error rates. Birnbaum & Gutierrez, 2007; Birnbaum & Schmidt, 2008.

Basic Assumptions of TE model (2 errors model) Each choice in an experiment has a true choice probability, p, and an error rate, e. The error rate is estimated from (and is the “reason” given for) inconsistency of response to the same choice by same person over repetitions

One Choice, Two Repetitions AB A B

Choices are not Independent In this model, choices are not independent, in general. If there is a mixture of true preferences, there will be violations of independence. This contrasts with the assumption of iid used by Regenwetter and colleagues.

Solution for e The proportion of preference reversals between repetitions allows an estimate of e. Both off-diagonal entries should be equal, and are equal to:

Estimating e

Estimating p

Testing if p = 0

True and Error Model to Individuals When applied to individuals, it is assumed that each person has a “true” set of preferences within a trial block. True preferences might differ between blocks, if the person has a mixture. If so, violates independence. A mixture could arise if a person’s parameters change in response to experience.

By Testing Individuals… we can tailor the experiment (or devise a fish net) to “catch” violations that otherwise slip through a static study of a property. we can see if people “learn” from internal feedback and change their behavior. we can test if a model that holds for one property can predict the results of other tests.

Recent Studies with Jeffrey Bahra Tested participants with many replications. Tests of transitivity Basic tests of critical properties, including stochastic dominance, coalescing, LCI, UCI, and others. Today: RBI and SD (+ transitivity)

Stochastic Dominance A: 10 tickets to win $10 5 tickets to win $90 85 tickets to win $98 B: 5 tickets to win $10 5 tickets to win $12 90 tickets to win $99 Each person received two such problems in each repetition block of 107 choice problems. Blocks were separated by at least 50 unrelated choice problems.

SD is a critical property This test of stochastic dominance lies outside the probability simplex on three branch gambles. CPT with ANY strictly monotonic utility function and decumulative weighting function must satisfy stochastic dominance in this choice. We don’t need to estimate any parameters or assume any particular functions to refute CPT.

Non-nested Models

Testing CPT Coalescing Stochastic Dominance Lower Cum. Independence Upper Cumulative Independence Upper Tail Independence Gain-Loss Separability TAX:Violations of:

Testing TAX Model 4-Distribution Independence (RS’) 3-Lower Distribution Independence 3-2 Lower Distribution Independence 3-Upper Distribution Independence (RS’) Res. Branch Indep (RS’) CPT: Violations of:

Critical Tests: LS Models CPT and TAX satisfy transitivity LS Models violate transitivity (includes PH) LS Models satisfy priority dominance, integrative independence, and interactive independence. This properties systematically violated. See my JMP 2010 article. PH satisfies SD in these tests and it also violates RBI in the same way as CPT.

Restricted Branch Independence Weaker version of Savage’s “sure thing” ax. 3 equally likely events: slips in urn. (x, y, z) := prizes x, y, or z, x < y < z RBI: (x, y, z)  (x', y', z)  (x, y, z')  (x', y', z')

TAX, CPT, PH Violate RBI 0 < z < x' < x < y < y' < z' (x, y) is “Safe”, S (x', y') is “Risky”, R (z, x, y)  (z, x', y')  w L u(z) + w M u(x) + w H u(y) > w L u(z) + w M u(x') + w H u(y')

TAX: SR ' Violations

S = (z, x, y) vs R = (z, 5, 95)

RBI distinguishes models of RDM EU and SWU (Edwards, 54) imply RBI Original prospect theory implies RBI Cumulative Prospect theory violates RBI CPT with inverse-S weighting function implies RS’ pattern of violations TAX model violates RBI with SR’ pattern

Testing iid Assumptions Each person’s data: rows represent trial blocks and the columns represent choice problems. Smith & Batchelder (2008) technique: random permutations within columns for each person. We then calculate two statistics on original data and on 10,000 simulations of the data. Variance of preference reversals and correlation between mean preference reversals and difference between repetitions.

A Test of Independence & Stationarity Within each subject, calculate the number of reversals between each pair of repetitions. 107 choices. 20 reps, so 20*19/2 = 190 pairs of reps. Correlate the average no. reversals with the difference in reps. For 59 participants, mean r = Only 6 were negative. Similar results for Regenwetter et al data. Second test: Variance of no. reversals.

Tests of Independence

Transitivity Findings Results were surprisingly transitive. True and Error Model Fit data fairly well. Data violate independence of choices. People differ in true preferences and people differ in “noise” levels. No one satisfied the priority heuristic One person satisfied linked viols of transitivity consistent with LS models.

Very Few Intransitive Cases No one showed pattern predicted by PH in all three designs. Of the 59*3 = 177 Matrices, perhaps 4 show credible evidence of intransitivity. This change of procedure did not produce the higher rates of intransitivity conjectured.

Allais CC Paradox (JMP’04) Choose Between: A = ($40, 0.2; $2, 0.8) B = ($98, 0.1; $2, 0.9) Choose Between: C = ($98, 0.8; $40, 0.2) D = ($98, 0.9; $2, 0.1) Many Choose B > A and C > D

Analysis of the Paradox: ($40,.2; $2,.8)  ($98,.1; $2,.9)  (Coalescing) ($40,.1;$40,.1;$2,.8)  ($98,.1;$2,.1;$2,.8)  (RBI) ($40,.1;$40,.1;$98,.8)  ($98,.1;$2,.1;$98,.8)  (Coalescing) ($98,.8; $40, 0.2)  ($98,.9; $2,.1)

Four Theories Compared RBI holds (**cancellation) RBI fails Coalescing holds (*combination) EU, CPT**, OPT* CPT Inverse-S => RS’ Coalescing fails OPTTAX, RAM SR’

Error Models & EU One error model: P(SR’) = P(RS’) Two error model allows P(RS’)>P(SR’) Two error model: P(R’) > 1/2 implies P(R) > 1/2 Four error model allows above All models imply P(SR) = P(RS); P(SS,SR’)=P(SR’,SS), etc.

Results (refute 4 error EU) SS’SR’RS’RR’ SS’ SR’5413 RS’ RR’121214

Results (split form RBI) SS’SR’RS’RR’ SS’ SR’ RS’74147 RR’16639

Summary Most people conform to transitivity A very small number do not, but not consistently in all three designs as predicted by LS models. Data appear to violate the assumptions of random utility model; in particular, they show evidence against the assumptions of i.i.d.

Summary (cont’d) No one satisfied CPT, except those consistent with EU No one satisfied PH People appear to change between blocks. We suspect that parameters change systematically during the study. Suspect TE model oversimplified and incomplete.

Available at my Website Tests of LS models (JMP 2010) Psych Review Comment RP Model (2011) Reanalysis of Regenwetter et al data testing iid, in JDM. Birnbaum & Bahra: Testing transitivity in Linked Designs (submitted) Birnbaum & Bahra: Testing SD and RBI (submitted) Soon: Birnbaum & Schmidt Allais paradoxes with 4 errors model