Measurement Construction Psych 818

Focus Majority of measurement in social sciences relies upon responses to a probe item – Constructed response – Checklists – Multiple Choice – Likert response Many other options – Eye movements, reaction time (IAT), Observation, physiological measures, fMRI, etc...

Measurement Goals Sensitive – Measure changes in response to changes in the construct Accurate – High levels of the construct are indicated by high levels on the measure Specific – Sensitive to changes in ONLY the focal construct Replicable – Same level of the construct results in same measurement

Item Types Multiple Choice – Can't be beat for knowledge and ability testing Likert Response (surveys/questionnaires) – Attitudes, beliefs, opinions Checklists – Risk factors True/False – Knowledge, ability, attitudes – Good for special populations (e.g., kids)

Item Types Ipsative – Rank ordering...or – Present a scenario, then choose between two options ( I would A) Stay or B) Go ) – Induces artifactual negative correlations among the responses – Don't use without very good reason Forced Choice – Often results in nearly ipsative measures – Resulting distribution often does not reflect reality

Survey Item Principles Know your population – Reading level (Check in Word) Generally at or below 6 th grade level – Mental framework Use a small sample verbal protocol – You will always get an answer but it may not make any sense Get others to review your measure – Testing experts – Population experts

Survey Item Principles Clear instructions – Unless you absolutely must deceive, tell them what you're trying to measure. – Get all respondents in a similar mindset Cluster items measuring the same construct – Don't ask the respondent to flip mental frameworks back and forth Incorporate random response items if the survey is long

Survey Item Principles Put easy, uncontroversial items first in the survey Put demographics and controversial items late in the survey – Don't prime gender,age, or race early in the survey Keep it short...far shorter than you think you can get away with Now make it even shorter!

Survey Items - Stems/Probes Stem should assess a single, clear aspect of the construct – No double barrel items Stem should be concise Use action verbs and active voice Avoid “not”, “except”, and double-negatives If you must use a clause, make it right branching – Early-completing sentence with more words after the main verb than before

Survey Items - Stems/Probes Use correct grammar – Most common is subject verb agreement when trying to avoid gendered language Ask for exactly what you want. – Don't make the respondent guess – Ex: What's your income? Weekly, monthly, yearly? Pre or post tax? Total compensation or wages? Include interest and dividend income?

Survey Items - Stems/Probes Avoid leading questions – You don't smoke, do you? – Should taxes be raised even higher? Avoid false premises – What should we do to keep the economy from deteriorating even further? Avoid extreme language – forbid, never, always Don't ask questions that rely heavily on veridical memories – Humans don't store information that way

Survey Items – Response format Response format matters – Tree growth example 5 plus or minus 2 response options – Keep it an odd number (5,7,9) Don't be afraid to change the item or the response format – You can often do a better job than the measure developer – Be conservative when doing this with well- established scales

Survey Items – Response format Try to use verbal anchors that are as clear and as precise as possible – Goal: eliminate idiosyncratic interpretations – Behaviorally anchored rating scales (BARS) – Behavioral observation scales (BOS) “No opinion” or “don't know” response options? – Massive scoring problems – Trend is toward using “neutral” to capture this idea

Survey Items – Response format Use same response format across scales on the same survey...whenever possible.

Let's look at some scales.... Dogmatism Personality Locomotion/Assessment