Study of the day Misattribution of arousal (Dutton & Aron, 1974)

Can situations influence how much we like someone? Answer: Yes! Dutton & Aron’s bridge study Misattribution of arousal

Writing good survey questions

Learning objectives You will learn: What makes a quality instrument How to write questions What validity means How to assess validity What reliability means How to assess reliability Scale development

Quality instrument* Questions are determined by objectives Resist the temptation to ask questions that are interesting but not relevant to your hypothesis Concrete and clearly phrased Pilot with potential respondents Contain reverse-scored items

Learning objectives What makes a quality instrument How to write questions What validity means How to assess validity What reliability means How to assess reliability Scale development

Good Questions* 1.Are not double-barreled (asks only about one item per question) 2.Do not have double negatives 3.Are not phrased in a leading manner 4. Are not biased in their language or make participants feel uncomfortable 5. Are concrete and specific

Open-ended Advantages Gives you the respondents view without interference Good when unsure what type of answers to expect Difficulties Time-consuming for respondent Answers must be cataloged and interpreted Two types of questions Mussweiler, T., & Damisch, L. (2008). Going back to Donald: How comparisons shape judgmental priming effects. Journal of Personality and Social Psychology, 95(6), 1295.

Two types of questions Closed-ended Advantages Easier to interpret More conducive to statistical analyses Good for large samples Answers are usually more reliable and consistent Difficulties Researcher must be familiar enough with phenomenon to write questions May not accurately capture all of respondents opinions Less illustrative

Response Choices 1.Nominal: categorical, discrete, mutually exclusive, no inherent order 2.Ordinal: inherent order but not associated with numbers – Strongly agree, agree, neither agree nor disagree, disagree, strongly disagree – Excellent, very good, good, fair, poor 3.Continuous: numbers meaningful

Ordinal Measures* Include a “do not know” if appropriate Include a neutral response if appropriate Balance all responses Use a 5- or 7-point numbered scale

Don’t forget demographics Important for: 1. Describing sample (especially gender and race) 2. Exploring your findings Typically asked at end Can include gender, race/ethnicity, education, job, age, marital status, geographic place of residence, etc.

Learning objectives What makes a quality instrument How to write questions What validity means How to assess validity What reliability means How to assess reliability Scale development

Validity Assesses the construct of interest Measuring what it is supposed to Cannot be directly measured – must be inferred from evidence

Face Validity Appears to measure what it is supposed to Let’s say you want to assess how social someone is Which ones would be best? – How often do you socialize with friends? – How many parties did you attend last month? – Do you get angry easily? – How much do you enjoy meeting new people? – Do you enjoy playing sports?

Construct Validity Does it relate to other constructs as would be predicted? Assess by correlating it with other measures E.g. How social someone is should relate to how much they have conversations with other people -& maybe how healthy they are

Criterion Validity How well does the measure predict or estimate real- world outcomes? E.g., If you asked students how interested they were in majoring in computer science, the measure can be validated by looking at how it relates to who actually majors in computer science

Learning objectives What makes a quality instrument How to write questions What validity means How to assess validity What reliability means How to assess reliability Scale development

Reliability Consistency in the type of result a test yields E.g., a reliable person Not perfectly overlapping, but questions “hang together”

Measuring Reliability* Cronbach Alpha: Inter-item consistency Test – retest : Same measure twice to the same group Equivalent forms: Administer two forms of the test to the same group Split-half : Administer half the measures at a time (odd items vs. even items) Inter-rater Reliability (coding qualitative data): How much coders’ answers overlap (percent agree and kappa)

How to improve reliability? Use clear and concise measures Include more items – less distorted by chance factors Developing a scoring plan for qualitative coding

Learning objectives What makes a quality instrument How to write questions What validity means How to assess validity What reliability means How to assess reliability Scale development

Scale development* 1.Conceptualize the target construct What are you trying to measure? Lit review Open-ended questions 2.Write the items Start with a larger pool Choose an appropriate response format 3.Pilot the scale Analyze reliability (kappa) and validity (correlations with other measures) Factor analysis to see if there are subscales Decide which need to be eliminated 4.Pilot the new measures

Available scales* The easiest thing (and the thing looked upon favorably by reviewers) is to use other people’s scales You can find these in: Scale databases e.g. Google Journal articles Books of scales

Other Suggestions For Qualtrics Include a question that asks participants NOT to respond – Helps rule out random clicking In general: Put DVs as close to manipulation as possible

Congratulations! You have learned: Characteristics of good scales How to write good questions What validity means and how to assess it What reliability means and how to assess it The steps involved in developing a scale