Quality instrument* Questions are determined by objectives Resist the temptation to ask questions that are interesting but not relevant to your hypothesis Concrete and clearly phrased Pilot with potential respondents Contain reverse-scored items

Good Questions* 1.Are not double-barreled (asks only about one item per question) 2.Do not have double negatives 3.Are not phrased in a leading manner 4. Are not biased in their language or make participants feel uncomfortable 5. Are concrete and specific

Ordinal Measures* Include a “do not know” if appropriate Include a neutral response if appropriate Balance all responses Use a 5- or 7-point numbered scale

Learning objectives What makes a quality instrument How to write questions What validity means How to assess validity What reliability means How to assess reliability Scale development

Measuring Reliability* Cronbach Alpha: Inter-item consistency Test – retest : Same measure twice to the same group Equivalent forms: Administer two forms of the test to the same group Split-half : Administer half the measures at a time (odd items vs. even items) Inter-rater Reliability (coding qualitative data): How much coders’ answers overlap (percent agree and kappa)

Scale development* 1.Conceptualize the target construct What are you trying to measure? Lit review Open-ended questions 2.Write the items Start with a larger pool Choose an appropriate response format 3.Pilot the scale Analyze reliability (kappa) and validity (correlations with other measures) Factor analysis to see if there are subscales Decide which need to be eliminated 4.Pilot the new measures

Available scales* The easiest thing (and the thing looked upon favorably by reviewers) is to use other people’s scales You can find these in: Scale databases e.g. Google Journal articles Books of scales