1. CHAPTER 2 Personality Assessment, Measurement, and Research Design © 2015 M. Guthrie Yarwood

2. Outline I. 4 Sources of personality data II. Reliability / Validity in Personality III. A little more on observer reports © 2015 M. Guthrie Yarwood

3. I. Sources of Personality Data Self-Report Data (S-Data) Observer-Report Data (O-Data) Test-Data (T-Data) Life-Outcome Data (L-Data) © 2015 M. Guthrie Yarwood

4. Self-Report Data (S-Data) Information provided by a person, such as through a survey or interview Limitations of S-data? © 2015 M. Guthrie Yarwood

5. O-Data Information provided by someone else about another person  Professional personality assessors  People who actually know the target person Naturalistic vs. Artificial Observation Limitations? © 2015 M. Guthrie Yarwood

6. Test-Data (T-Data) Information provided by standardized tests or testing situations Situation designed to elicit behaviors that serve as indicators of personality © 2015 M. Guthrie Yarwood

7. Test-Data Creativity Example What are unusual uses for common objects – bricks, knives, newspapers? Answers to hypothetical events  What would happen if people went blind?  What would happen is people shrank to 12 inches tall? © 2015 M. Guthrie Yarwood (Paul Silvia’s work)

8. Test-Data: Other Examples Mechanical recording devices, e.g., “Actometer” used to assess children’s activity Physiological data Projective Tests Ex: Fairy Tale TestFairy Tale Test Limitations? © 2015 M. Guthrie Yarwood

9. Life-Outcome Data (L-Data) Information that can be gleaned from events, activities, and outcomes in a person’s life that is available for public scrutiny—e.g., marriage, speeding tickets Can serve as important source of “real life” information about personality Ex: implicit egotism: people gravitate toward people, places, things that resemble the self © 2015 M. Guthrie Yarwood (Pelham and colleagues’ work)

10. Issues in Personality Assessment Links among different data sources Fallibility of personality measurement  All sources of data have limitations  Results that replicate through “triangulation” are most powerful © 2015 M. Guthrie Yarwood

11. You are a personality psychologist and would like to measure the personality trait risk-taking (i.e., sensation seeking). How could you measure risk-taking using each of the four data sources? S-Data O-Data T-Data L-Data © 2015 M. Guthrie Yarwood

12. Evaluation of Personality Measures How do we know whether our scale is a “good scale?” Types of Errors Reliability Validity Threats to Reliability and Validity © 2015 M. Guthrie Yarwood

13. Extraneous vs. Confounding Variables © 2015 M. Guthrie Yarwood Extraneous variables are variables that may compete with the independent variable in explaining the outcome of a study. Confounding variable: an extraneous variable that does indeed influence the dependent variable. A confounding variable systematically varies or influences the independent variable and also influences the dependent variable.

14. Random vs. Constant Errors Random Errors (unsystematic): extraneous variables whose average influence on the outcome is the same in both (or all) conditions (Aronson et al., 1990)  Affects reliability AND validity Constant (systematic) Errors: influences all the scores in one condition in the same direction and has no effect or a different effect on the other condition.  Affects ONLY validity  Confounding variables © 2015 M. Guthrie Yarwood

15. Clarification Check! IV: Type of show  Condition 1: Extraverts watch comedy show  Condition 2: Extraverts watch neutral show DV: Self-reported emotions  Extraverts report more positive emotions in Comedy Condition than in Neutral Condition © 2015 M. Guthrie Yarwood

16. Clarification Check! – Random Errors Extraverts in conditions 1 and 2 are placed in very hot rooms. The hot temperature lowers their self-reported levels of positive emotions. But, we still find that extraverts report more positive emotions when watching the comedy show than the neutral show. Accurate: We find differences in self-reported positive emotions (on 5-point scale):  Comedy Cond. = 4.8, Neutral Cond. = 2.5 Random Errors: In presence of hot temperature, we find differences in self-reported positive emotions  Comedy Cond. = 3.0, Neutral Cond. = 1.5 © 2015 M. Guthrie Yarwood

17. Clarification Check! – Random Errors Extraverts in conditions 1 and 2 are placed in very hot rooms. The hot temperature lowers their self-reported levels of positive emotions. But, we still find that extraverts report more positive emotions when watching the comedy show than the neutral show. Accurate: We find differences in self-reported positive emotions (on 5-point scale):  Comedy Cond. 1 = 4.8, Neutral Cond. 2 = 2.5 Random Errors: In presence of hot temperature, we find differences in self-reported positive emotions  Comedy Cond = 3.0, Neutral Cond.2 = 1.5 Hot temperature lowers the scores in both conditions!! © 2015 M. Guthrie Yarwood

18. Clarification Check! – Constant Errors Extraverts in Comedy Cond. are in a 70◦ room. Extraverts in Neutral Cond. are placed in a 78◦ room. In Neutral Cond. only, the hot temperature lowers self- reported levels of positive emotions. We find that the comedy show results in more positive emotions than the neutral show. Accurate: We do not find differences in self-reported positive emotions  Comedy Cond. = 4.8, Neutral Cond. = 4.6 Constant Error: We find differences in positive emotions due to the hot temperature, not due to the manipulation!  Comedy Cond. = 4.8, Neutral Cond. = 2.5 © 2015 M. Guthrie Yarwood

19. Clarification Check! – Constant Errors Extraverts in Comedy Cond. are in a 70◦ room. Extraverts in Neutral Cond. are placed in a 78◦ room. In Neutral Cond. only, the hot temperature lowers self- reported levels of positive emotions. We find that the comedy show results in more positive emotions than the neutral show. Accurate: We do not find differences in self-reported positive emotions  Comedy Cond. = 4.8, Neutral Cond. = 4.6 Constant Error: We find differences in positive emotions due to the hot temperature, not due to the manipulation!  Comedy Cond. = 4.8, Neutral Cond. = 2.5 Hot temperature lowers the scores in Condition 2, but not Condition 1!! © 2015 M. Guthrie Yarwood

20. Reliability The ability of a test to measure an attribute consistently  Does this extraversion scale measure the true level of each participant’s extraversion (over time)?  Are coders following the same method? To achieve reliability we want to reduce random error. © 2015 M. Guthrie Yarwood

21. Threats to Reliability (random error) Participant Characteristics:  fatigue, motivation, boredom Testing situation:  time of day, room temperature Testing Instrument:  instructions, rating scale, items, reading level. Experimenter Characteristics and Errors:  interactions with participants; incorrect observations of participants; incorrect coding of behavior © 2015 M. Guthrie Yarwood

22. Estimating Reliability Test-Retest Coefficient Parallel-Forms Coefficient Internal Consistency Coefficient Interrater (interobserver) Reliability © 2015 M. Guthrie Yarwood

23. Validity Validity: The degree to which a test or measurement accurately measures or reflects what it claims to measure. Internal Validity: Did the experimental treatments make a difference in this specific experimental instance? External Validity: Generalizability; To what populations, settings, treatment variables, and measurement variables can the effect be generalized?  **Never completely answerable © 2015 M. Guthrie Yarwood

24. Internal Validity Degree to which test measures what it claims to measure 5 types of internal validity  Face validity  Predictive or criterion validity  Convergent validity  Discriminant validity  Construct validity © 2015 M. Guthrie Yarwood

25. Threats to Internal Validity (errors) Affected by random and constant errors  Random (unsystematic) Errors: same errors that affect reliability  Constant Errors (systematic): errors that affect measurement of variable; does not affect reliability © 2015 M. Guthrie Yarwood

26. Knowledge Check! You are conducting a study on the personality traits associated with the frequency of exercising. For your study, which of the following poses a threat to validity, but not reliability? A. All the participants are bored. B. The construction outside the laboratory window is very loud. C. You recruited participants from the Rec Hall. D. The experimenter who greets all participants is very rude. © 2015 M. Guthrie Yarwood

27. Knowledge Check! You are conducting a study on the personality traits associated with the frequency of exercising. For your study, which of the following poses a threat to validity, but not reliability? A. All the participants are bored. B. The construction outside the laboratory window is very loud. C. You recruited participants from the Rec Hall. D. The experimenter who greets all participants is very rude. © 2015 M. Guthrie Yarwood

28. Threats to Internal Validity – MRS SMITH Maturation Regression to the Mean Selection of Subjects Selection by Maturation Interaction Mortality Instrumentation Testing History © 2015 M. Guthrie Yarwood

29. Regression to the Mean Example © 2015 M. Guthrie Yarwood In an experiment involving reading instruction, subjects grouped because of poor pre-test reading scores show considerably greater gain than do the groups who scored average and high on the pre-test.

30. Regression to the Mean Example © 2015 M. Guthrie Yarwood In an experiment involving reading instruction, subjects grouped because of poor pre-test reading scores show considerably greater gain than do the groups who scored average and high on the pre-test. Poor Average / Mean High Pre-test

31. Regression to the Mean Example © 2015 M. Guthrie Yarwood In an experiment involving reading instruction, subjects grouped because of poor pre-test reading scores show considerably greater gain than do the groups who scored average and high on the pre-test. Poor Average / Mean High Pre-test After reading Instruction

32. Selection by Maturation Interaction © 2015 M. Guthrie Yarwood GroupPre-testPost-Test (after head start) Head Start Intervention Middle-class children65/100 Disadvantaged children65/100 Control Group – No Intervention Middle-class children65/100 Disadvantaged children65/100

33. Selection by Maturation Interaction © 2015 M. Guthrie Yarwood GroupPre- test Post-Test: 6 months Post-Test: 12 months Post-Test: 18 months Head Start Intervention 65/10070/10075/10080/100 Control Group – No Intervention 65/100

34. Selection by Maturation Interaction © 2015 M. Guthrie Yarwood GroupPre- test Post-Test: 6 months Post-Test: 12 months Post-Test: 18 months Head Start Intervention 65/10070/10075/10080/100 Control Group – No Intervention 65/100 Ss in Intervention are middle class, while Ss in control group are disadvantaged. Over time, Intervention Ss show improvement of post- test due to better health care, greater parental support, greater access to resources, etc.

35. Identify the Threat to Validity! In a short experiment designed to investigate the effect of computer-based instruction, Ss missed some instruction because of a power failure at school. A. History B. Mortality C. Testing D. Instrumentation © 2015 M. Guthrie Yarwood

36. Identify the Threat to Validity! In a short experiment designed to investigate the effect of computer-based instruction, Ss missed some instruction because of a power failure at school. A. History B. Mortality C. Testing D. Instrumentation © 2015 M. Guthrie Yarwood

37. Identify the Threat to Validity! In a health experiment designed to determine the effect of various exercises, those Ss who find the exercise most difficult stop participating. A. Selection of Subjects B. Mortality C. Testing D. Maturation © 2015 M. Guthrie Yarwood

38. Identify the Threat to Validity! In a health experiment designed to determine the effect of various exercises, those Ss who find the exercise most difficult stop participating. A. Selection of Subjects B. Mortality C. Testing D. Maturation © 2015 M. Guthrie Yarwood

39. Estimating Internal Validity Content Validity Criterion-Related Validity  Concurrent Validity  Predictive Validity Construct Validity  Convergent Validity  Discriminant Validity © 2015 M. Guthrie Yarwood

40. Content Validity Definition: whether the content of a test elicits a range of responses that are representative of the entire domain or universe of skills, understandings, and other behaviors a test is designed to measure. To assess: compare tests’ content with an outline of specifications concerning subject matter to be covered in test. © 2015 M. Guthrie Yarwood

41. Openness to Experience ????? © 2015 M. Guthrie Yarwood

42. Openness to Experience Behaviors? Perceptions? Thoughts/Cognitions? Feelings/Emotions? © 2015 M. Guthrie Yarwood

43. Criterion-Related Validity Definition: procedures in which the test scores of a group of people are compared with ratings, classifications, or other measures of performance. © 2015 M. Guthrie Yarwood

44. Concurrent Validity – A type of criterion-related validity Concurrent Validity: when a test is administered to people in various categories, to determine whether test scores of people in 1 category are significantly different from people in other categories.  Clinical vs. non-clinical group  Different socioeconomic levels © 2015 M. Guthrie Yarwood

45. Group A shows more openness to experience Group B shows less openness to experience Concurrent Validity © 2015 M. Guthrie Yarwood

46. High IQ group shows more openness to experience Low IQ group shows less openness to experience Concurrent Validity © 2015 M. Guthrie Yarwood

47. Predictive Validity – A type of criterion-related validity Predictive Validity: how accurately test scores predict criterion scores. Indicated by correlation between test score (the predictor) and a criterion of future performance (what the test predicts) © 2015 M. Guthrie Yarwood

48. Predictive Validity Openness to Experience (Predictor) ????? (Criterion) © 2015 M. Guthrie Yarwood

49. Predictive Validity Openness to Experience College Major Well-being, Psychological Adjustment Openness to Experience © 2015 M. Guthrie Yarwood

50. Construct-Related Validity Definition: extent to which scale measures a particular construct or psychological concept To assess: need to determine whether an assessment instrument that presumably measures a certain personality variable is actually doing so © 2015 M. Guthrie Yarwood

51. Convergent Validity – A Type of Construct-Related Validity Convergent Validity: the measure has high correlations with other measures or methods of measuring the same construct © 2015 M. Guthrie Yarwood

52. Convergent Validity Our Openness to Experience Self-Report Scale Same Construct, Same Measurement Our Openness to Experience Self-Report Scale Same Construct, Different Measurement © 2015 M. Guthrie Yarwood

53. Convergent Validity Our Openness to Experience Self- Report Scale Costa & McCrae’s Self-Report OE Dimension Observer-Report OE Dimension Our Openness to Experience Self- Report Scale © 2015 M. Guthrie Yarwood

54. Discriminant Validity – A Type of Construct-Related Validity Discriminant Validity: the measure has low correlations with measures of different constructs © 2015 M. Guthrie Yarwood

55. Discriminant Validity Our Openness to Experience Self-Report Scale Different Construct, Same Measurement Different Construct Different Measurement Our Openness to Experience Self-Report Scale OR © 2015 M. Guthrie Yarwood

56. Discriminant Validity Our Openness to Experience Self-Report Scale Self-Report Sensation Seeking Scale Clinical Diagnosis of Schizotypal PersonalityDisorder Our Openness to Experience Self-Report Scale © 2015 M. Guthrie Yarwood

57. Discriminant Validity Our Openness to Experience Self-Report Scale Self-Report Sensation Seeking Scale Clinical Diagnosis of Schizotypal PersonalityDisorder Our Openness to Experience Self-Report Scale These correlations should be significant, but lower than correlations for convergent validity. © 2015 M. Guthrie Yarwood

58. To assess convergent/discriminant validity The same construct using the same method (c) The same construct using different methods (c) Different constructs using the same method (d) Different constructs using different methods (d) Note: c = convergent; d=discriminant © 2015 M. Guthrie Yarwood

59. Reliability & Validity A test can be reliable, but not valid In other words, to be valid a test must first be reliable. NO VALDITY WITHOUT RELIABILITY! © 2015 M. Guthrie Yarwood

60. Reliability Internal Validity Test-Retest Coefficient Parallel-Forms Coefficient Internal Consistency Coefficient Interrater (interobserver) Reliability Content Validity Criterion-Related Validity  Concurrent Validity  Predictive Validity Construct Validity  Convergent Validity  Discriminant Validity Reliability and Validity: Summary © 2015 M. Guthrie Yarwood

61. A LITTLE MORE ON OBSERVER REPORTS (O-DATA) © 2015 M. Guthrie Yarwood

62. Self-report Observer report 1 Correlation 1 Observer report 1 Observer report 2 Correlation 2 Self-report Several different observers Correlation 3, 4, 5… © 2015 M. Guthrie Yarwood

63. Aggregating Scores Averaging the self-report and the observer report/s provides a clearer picture of personality than the self or observer report alone (Kolar et al., 1996) © 2015 M. Guthrie Yarwood

64. Let’s discuss the purpose of observer reports What is the purpose of an observer report? Compared to a self-report measure, what results would you expect from an observer report? What would you think if you did not obtain these results? Can you think of any threats to validity that observer reports may pose? © 2015 M. Guthrie Yarwood

65. Aggregating Scores A positive correlation between an observer report and a self-report, would provide evidence for which type of validity? A. Convergent B. Predictive C. Concurrent D. Discriminant © 2015 M. Guthrie Yarwood

66. Recall Reflection Post and TIPI! Disagree Strongly (1) Disagree Moderately (2) Disagree a little (3) Neither Agree nor Disagree (4) Agree a little (5) Agree Moderately (6) Agree Strongly (7) 1 Extraverted, enthusiastic 2Reserved, quiet r 3 Critical, quarrelsome r 4Sympathetic, warm 5 Dependable, self- disciplined 6 Disorganized, careless r 7 Anxious, easily upset r 8 Calm, emotionally stable 9 Open to new experiences, complex 10Conventional, uncreative r © 2015 M. Guthrie Yarwood

67. Self Scores! © 2015 M. Guthrie Yarwood

68. Observer Scores! © 2015 M. Guthrie Yarwood

69. Based on the following graphs, which dimension do you think shows the weakest correlation between self and observer scores? A. Extraversion B. Agreeableness C. Conscientiousness D. Emotional Stability E. Openness to Experience

70. © 2015 M. Guthrie Yarwood

71. Based on the following graphs, which dimension do you think shows the weakest correlation between self and observer scores? A. Extraversion B. Agreeableness C. Conscientiousness D. Emotional Stability E. Openness to Experience

72. LET’S CORRELATE THE SCORES! © 2015 M. Guthrie Yarwood TIPI DIMENSION TIPI TRAIT r 1 EXTRAVERSION Extraverted, enthusiastic.74 2Reserved, quiet r 3 AGREEABLENESS Critical, quarrelsome r.48 4Sympathetic, warm 5 CONSCIENTIOUSNESS Dependable, self-disciplined.52 6Disorganized, careless r 7 EMOTIONAL STABILITY (LOW NEUROTICISM) Anxious, easily upset r.07 8Calm, emotionally stable 9 OPENNESS TO EXPERIENCE Open to new experiences, complex.39 10Conventional, uncreative r

73. Note. Red = Self; Yellow = Observer © 2015 M. Guthrie Yarwood

74. Note. Red = Self; Yellow = Observer © 2015 M. Guthrie Yarwood

75. Summary and Evaluation Decisions about data source and research design depend on  (1) the purpose of study and  (2) threats to validity/reliability There is no perfect data source There is no perfect research design Assessing threats to reliability and validity will assist in selecting a data source and research design. Observer reports improve validity and reduce social desirability concerns © 2015 M. Guthrie Yarwood

76. Reminder – Paper Topic!! Find a group member (can do on ANGEL!) Select a topic! Submit Paper topic to drop-box  Due: Friday, January 30th at 9 AM ET Paper Topic Outline available on ANGEL. Guidance: Meet with Michelle or Celina Questions??? © 2015 M. Guthrie Yarwood

77. Chapter 3 (Dispositional Domain) Survey HEXACO  Follow link to access survey  Score your survey according to the instructions on ANGEL Big Five Model  Access through Chapter 3 Survey on course website © 2015 M. Guthrie Yarwood