1. Linear Judgment Models: What Do They Suggest About Human Judgment?
Psychology 466: Judgment & Decision Making Instructor: John Miyamoto 10/07/2015: Lecture 02-2
Note: This Powerpoint presentation may contain macros that I wrote to help me create the slides. The macros aren’t needed to view the slides. You can disable or delete the macros without any change to the presentation.

2. Outline Four linear judgment models Finish: Model of the judge
SMART model
Unit weighting model
What do studies of linear judgment models show about human judgment?
Psych 466, Miyamoto, Aut '15

3. Four Linear Judgment Models
Using multiple regression on objective data for which the true state is known.
Using multiple regression on judgment data where the true state is not known.
SMART (Simple Multi-Attribute Rating Technique)
Unit Weighting Model
Except for Model 4 (unit weighting), the models do NOT describe everyday judgment.
Some natural models are similar to these models.
Using any of these models can improve human performance.
Same Slide with Expanded Comments about Each Model
Psych 466, Miyamoto, Aut '15

4. Judgment Data Used in Baron's Chapter 20
COL = college GPA. This is the criterion. This is what the judge wants to predict.
SAT = SAT score; REC = judge's rating of the recommendation; ESS = judge's rating of the student's essay; GPA = high school GPA. These are the cues.
Comment re Qualitative Variables
Psych 466, Miyamoto, Aut '15

5. Method 2: Model of the Judge
Researcher has available the scores for SAT, REC, ESS and GPA.
The values of the criterion (COL) are NOT available.
Researcher asks the judge to make intuitive, global predictions for these cases. This produces the column labeled "JUD” (next slide).
Not Available
Not Available
Same Slide Except JUD Column Added to Table
Psych 466, Miyamoto, Aut '15

6. Method 2: Model of the Judge
Researcher has available the scores for SAT, REC, ESS and GPA.
The values of the criterion (COL) are NOT available.
Researcher asks the judge to make intuitive, global predictions for these cases. This produces the column labeled "JUD."
Not Available
Not Available
Same Slide Except Column Labeled “MUD” Added to Table
Psych 466, Miyamoto, Aut '15

7. Method 2: Model of the Judge
It is an accident that in this example, JUD and MUD are identical
Compute a regression model that predicts JUD (Model of the jUDge or MUD).
“Policy Capturing”.
Not Available
Not Available
See ‘e:\p466\nts\baron.quant.jdmt.r-code.docm’ for R-code that computes the prediction from the model of the judge.
Example:
MUD = (–5x10–18)·SAT ·REC ·ESS ·GPA  4.8
Use the MUD model to predict college GPA (COL) for these cases or future cases.
Discussion of Model of the Judge
Psych 466, Miyamoto, Aut '15

8. Discussion of Case 2: Model of the Judge (MUD)
Empirical findings for Case 2 are the same as for Case 1
MUD more accurate than the intuitive judgments of the judge.
Judges think that they make their judgments by means of a complex, interactive, nonlinear evaluation of the cues.
Whether or not this is really true, studies show that a much simpler combination of the cues produces better predictions.
We don't need to know the value of the criterion (COL) in order to find a statistical formula (prediction equation) that can outperform the judge. .
Why Does MUD Outperform the Judge?
Psych 466, Miyamoto, Aut '15

9. Why Does MUD Outperform the Judge?
The model of the judge (MUD) is a model of the judge’s decisions, not of the criterion (reality).
The MUD outperforms the judge because the statistical formula is consistent – it treats each case by the same formula.
A human judge has all sorts of random variations in his or her judgment. These random variations simply increase the inaccuracy (error) of the judge's predictions.
A human judge sees various “special circumstances” (anecdotal memories) that suggest that a specific case should be judged differently from the standard procedure.
Four Methods – SMART Method is Next
Psych 466, Miyamoto, Aut '15

10. Four Ways to Compute a Statistical Prediction Model
Method 1: Multiple regression applied to existing data. Called a “proper linear model”
Method 2: Multiple regression applied to a judge’s predictions Called a “model of the judge”
Method 3: SMART Method with "importance" weights Called the SMART method or importance weighting method
Method 4: Unit weighting model Called the “unit weighting model or unit weighting method”
Next
SMART Method
Psych 466, Miyamoto, Aut '15

11. Method 3: The SMART Method
Simple Multi-Attribute Rating Technique (SMART)
A.k.a. Importance Weighting Model
This technique lets us rank order the individual cases (high school students).
A rank order tells us who is predicted to do better or worse. This is enough to guide our selection of students.
The SMART method can be modified to actually predict college GPA (COL), but we won’t go into this complication because it is not needed in a choice situation.
Step by Step Explanation of SMART Method
Psych 466, Miyamoto, Aut '15

12. Method 3: The SMART Method
Step 1: Convert the predictors, SAT, REC, ESS and GPA to z-scores.
Step 2: The judge decides what is the relative importance of these variables, e.g., SAT is twice as important as ESS, GPA is 50% more important than SAT, etc.
Step 3: Create a prediction equation that reflects the judge’s judgment as to the relative importance of the predictor variables.
Example of Prediction of MAUT Model = Predicted Score = 2·Z.sat + 3·Z.rec + 1·z.ess + 2·Z.gpa
Tables Showing Original Data, Z-Scores, and Prediction
Psych 466, Miyamoto, Aut '15

13. Method 3: The SMART Method
Z-Scores
R-code for the table to the right is in ‘e:\p466\nts\baron.quant.jdmt.r-code.doc’
Table on the left shows the initial data.
Table on the right shows the z-scores for the predictor variables, and the predicted rating for each student.
Predicted Score = 2·Z.sat + 3·Z.rec + 1·z.ess + 2·Z.gpa
The “Predicted” column tells you who is predicted to do better or worse. It does not tell you the predicted GPA.
Example: Computing the Predicted Score for One Student
Psych 466, Miyamoto, Aut '15

14. Method 3: The SMART Method
Z-Scores
R-code for the table to the right is in ‘e:\p466\nts\baron.quant.jdmt.r-code.doc’
Example of Prediction of MAUT Model = Student Score = 2·Z.sat + 3·Z.rec + 1·z.ess + 2·Z.gpa
Example for 1st student: 2(2.39) + 3(.32) + 1(.41) + 2(1.58) = 9.31
Comment on How to Convert Prediction to a Predicted College GPA
Psych 466, Miyamoto, Aut '15

15. Method 3: The SMART Method
Z-Scores
R-code for the table to the right is in ‘e:\p466\nts\baron.quant.jdmt.r-code.doc’
To convert the “Predicted” column to an actual predicted college GPA, you need to furnish a guess as to the mean and variance of college GPA’s (unnecessary for your assignment).
Producing a predicted college GPA by this method requires a few math tricks that are not worth discussing in Psych 466.
Findings for the SMART Method
Psych 466, Miyamoto, Aut '15

16. Findings for Method 3: SMART Method
Findings for Method 3 are the same as Method 2 (Model of the Judge)
SMART method is more accurate than the intuitive judgments.
The predicted ratings that are computed by the SMART method correlate better with actual results than do the judges intuitive predictions.
More Discussion of the SMART Method
Psych 466, Miyamoto, Aut '15

17. Discussion of Method 3: SMART Method
The SMART method does not require that we know the value of the criterion (COL) for a set of known cases.
We don't need a complicated calculation of the optimal regression weights (Methods 1 & 2 require this calculation).
The SMART Method is better than intuitive judgment because ….
It approximately captures the relative importance of the predictor variables, and ….
It is consistent – it is not bothered by shifts of attention, fatigue, distractions, anecdotal memories, etc.
Four Methods – Next We Discuss Unit Weighting Model
Psych 466, Miyamoto, Aut '15

18. Four Ways to Compute a Statistical Prediction Model
Method 1: Multiple regression applied to existing data. Called a “proper linear model”
Method 2: Multiple regression applied to a judge’s predictions Called a “model of the judge”
Method 3: SMART Method with "importance" weights Called the SMART method or importance weighting method
Method 4: Unit weighting model Called the “unit weighting model or unit weighting method”
Next
Unit Weighting Method
Psych 466, Miyamoto, Aut '15

19. Reminder: What Are the Weights?
Method 1: Proper Linear Model PRE = ·SAT ·REC ·ESS ·GPA  5.161
Method 2: Prediction Equation for the Judge’s Predictions MUD = (–5x10–18)·SAT ·REC ·ESS ·GPA  4.8
Method 3: Predicted Scores Based on Importance Weights: Student Score = 2·Z.sat + 3·Z.rec + 1·z.ess + 2·Z.gpa
The weights are the numbers that are assigned to the predictor variables.
Unit Weighting Model: Assign +1 to all positive attributes; Assign –1 to all negative attributes.
Student Score = 1·Z.sat + 1·Z.rec + 1·z.ess + 1·Z.gpa
Further Explanation of the Unit Weighting Model
Psych 466, Miyamoto, Aut '15

20. Method 4: Unit Weighting Model
Z-Scores
R-code for the table to the right is in ‘e:\p466\nts\baron.quant.jdmt.r-code.doc’
Table on the right shows the z-scores for the predictor variables, and the predicted rating for each student.
Example for Case 1: 1(2.39) + 1(.32) + 1(.41) + 1(1.58) = 4.70
Findings for the Unit Weighting Model
Psych 466, Miyamoto, Aut '15

21. Findings for Method 4: Unit Weighting Model
Unit weighting model more accurate than the intuitive judgments of the judge.
Judges think that they make their judgments by means of a complex, interactive, nonlinear evaluation of the cues. Whether or not this is really true, studies show that a much simpler combination of the cues produces better predictions.
We don't even need to calculate optimal regression weights.
All we need to know is which cues are positively related to the criterion and which are negatively related to the criterion.
Discussion of the Unit Weighting Model (cont.)
Psych 466, Miyamoto, Aut '15

22. Discussion of Method 4: Unit Weighting Model (cont.)
Unit weighting model outperforms human judge because it is consistent - it treats each case by the same formula.
A human judge has all sorts of random variations in his or her judgment. These random variations simply increase the inaccuracy (error) of the judge's predictions.
A human judge may be influenced by anecdotal memories. The unit weighting model is not influenced by these memories.
Comment: Results for unit weighting model show that the critical weakness of the human judge is inconsistency, not the inability to produce optimal weights.
Table: Pros and Cons of Prediction Methods
Psych 466, Miyamoto, Aut '15

23. Only if you want to make a quantitative prediction
This table is Table 3 in ‘e:\p466\hnd.02-2a.p466.a13.docm’. The table was copied to an image and pasted into these slides.
Repeat this Table Without the Red Rectangles
Psych 466, Miyamoto, Aut '15

24. This table is Table 3 in ‘e:\p466\hnd. 02-2a. p466. a13. docm’
This table is Table 3 in ‘e:\p466\hnd.02-2a.p466.a13.docm’. The table was copied to an image and pasted into these slides.
Conclusions
Psych 466, Miyamoto, Aut '15

25. What We Have Learned from the Study of Linear Judgment Models
Human judges typically believe that they use complex judgment processes to make judgments from complex cues. (This may be true.)
The part of the human judgment process that validly predicts the criterion is well modeled by a linear model.
We don't need to use optimal regression models to outperform human judges.
We don't need to know the value of the criterion in order to create a model that outperforms human judges.
Evidence: Model of the judge, unit weighting and importance weighting models outperform the human judge.
Note: These methods work only because the human judge has some valid knowledge of the relationship between the cues and the criterion.
Sample Size Affects the Predictive Accuracy of Multiple Regression & MUD
Psych 466, Miyamoto, Aut '15

26. Sample Size Affects the Accuracy of Multiple Regression Model and Model of the Judge
This point is emphasized by Gigerenzer: If you are using a multiple regression model or the model of the judge, the predicted accuracy of the model is bad if the sample size is too small.
If the sample size is small, then the regression weights will tend to be inaccurate. (Technically, the variance of the regression weights increases as the sample size gets smaller.)
If the sample size is small, the unit weighting model can be more accurate than multiple regression model or the model of the judge. (This point is emphasized by Gigerenzer.)
Why Do Statistical Model Outperform Humans?
Psych 466, Miyamoto, Aut '15

27. Why Do Statistical Models Outperform Human Judges?
Human judgment is affected by internal random variation; statistical model is not.
Human judgment is affected by vivid individual cases (anecdotes); statistical model is not.
Speculation: Human judge tries to fit information into a story; statistical model ignores story; it just adds up the evidence But is the human preference for stories bad?
General Discussion of Linear Judgment Models
Psych 466, Miyamoto, Aut '15

28. General Discussion of Linear Models
Why aren’t linear judgment models used more widely in practical decision making?
College or graduate admissions
NIH or NSF grant review committees
Political decisions like where to locate a prison; where to locate a homeless shelter;
The Denver bullet study
Are linear judgment methods dehumanizing, e.g., when choosing who will be admitted to a college?
END
Psych 466, Miyamoto, Aut '15