Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect measures

Measurement Issues General steps –Consider limitations of measures selected –Collect or secure info/data –Summarize findings in writing

What is the relation between concepts, variables, instruments & measures?

Concepts Program is based on conceptual basis of why people behave the way they do

Why do you think people behave the way they do? Think of food and nutrition issues

Variables A theory has variables Variables define concepts Theory states how the variables interact or are related

Variables Variables of the theory are what you measure Variables are the verbal or written abstractions of the ideas that exist in the mind

Why should an intervention be based on a theory?

Why use theory? Know what you are to address in the intervention Makes evaluation easier Know what to measure to evaluate

Figure 6.1 A simple social learning theory model for reducing salt in the diet Comes next

Fig. 6.1 Social learning theory

Need measurements and instruments to assess changes in the variables of interest

Instruments Something that produces a measure of an object Series of questions to measure the variable, concept Includes instructions

Measures The numbers that come from the person answering questions on the instrument

Figure 6.2 Relation among models, variables, measures, and an instrument Comes next

Fig. 6.2

Based on why you think people behave the way the do, list possible variables to consider to measure this variable. What might be variables of the social learning theory?

What about variables that would verify if a change has or has not taken place?

Figure 6.1 A simple social learning theory model for reducing salt in the diet Comes next See how the program links with the theory & what measure

Fig. 6.1 Social learning theory

Reliability The extent to which an instrument will produce the same result (measure or score) if applied two different or more times.

Reliability X = T + E X is measure T is true value E is random error

Reliability Measurement error reduces the ability to have reliable and valid results.

Reliability Random error is all chance factors that confound the measurement. Always present Effects reliability but doesn’t bias results

Reliability Figure 6.5 Distribution of scores of multiple applications of a test with random error A is true score a is measure

Fig. 6.5 Distribution of scores of multiple applications of a test and random error

Distribution Can have the same mean with two different distributions Figure 6.6 next

Fig. 6.6 Two distributions of scores around the true mean

Which distribution has less variability? Which distribution has less random error?

Sources of Random Error Day-to-day variability Confusing instructions Unclear instrument Sloppy data collector

Sources of Random Error Distracting environment Respondents Data-management error

What can you do to reduce random error and increase reliability?

Variability & the Subject What you want to measure will vary from day to day and within the person

Variability & the Subject Intraindividual variability –variability among the true scores within a person over time

Figure 6.7 True activity scores (A, B, C) for 3 days with three measures (a, b, c) per day Comes next

Fig. 6.7 True activity (A, B, C) for 3 days with three measures (a, b, c) per day

Variability & the Subject Interindividual variability –variability between each person in the sample

Figure 6.8 Interindividual (A, X) and intraindividual (A1, A2, A3) variability for two people (A, X) in level of physical activity Comes next

Fig. 6.8 Interindividual (A, X) and intraindividual (A1, A2, A3) variability for two people (A, X) in level of physical activity

Assessing Reliability Need to know the reliability of your instruments Reliability coefficient of 1 is highest, no error Reliability coefficient of 0 is lowest, all error

Factors of Reliability Type of instrument –observer –self-report Times instrument applied –same time –different time

Figure 6.9 Types of reliability Comes next

Fig. 6.9 Types of reliability

Assessing Reliability Interobserver reliability –have 2 different observers rate same action at same time –reproducibility

Assessing Reliability Intraobserver reliability –1 observer assesses same person at two different times –video tape the action & practice

Assessing Reliability Repeat method –self-report or survey –repeat the same item/question at 2 points in survey

Assessing Reliability Internal consistency –average inter-item correlation among items in an instrument that are cognitively related

Assessing Reliability Internal consistency –Cronbach’s alpha –0.70 & above a good score

Assessing Reliability Test-retest reliability (internal consistency method) –same survey/test at 2 different times to same person

Validity Degree to which an instrument measures what the evaluator wants it to measure

Bias Systematic error that produces a systematic difference between an obtained score and the true score Bias threatens validity

Bias Figure 6.10 Distribution of scores of multiple applications of a test with systematic error Comes next

Fig Distribution of scores of multiple applications of a test with systematic error

What will basis do to your ability to make conclusions about your subjects?

Figure 6.11 Effect of bias on conclusions Comes next

Fig Effect of bias on conclusions

Types of Validity Face Content Criterion

Face Validity Describes the extent to which an instrument appears to measure what it is suppose to measure How many veg did you eat yesterday?

Content Validity Extent to which an instrument is expected to cover several domains of the content Consult a group of experts

Criterion Validity How accurate is a less costly way to measure the variable compared to the valid and more expensive instrument

What can lower validity? Guinea pig effect –awareness of being tested Role selection –awareness of being measured may make people feel they have to play a role

What can lower validity? Measurement as a change agent –act of measurement could change future behavior

What can lower validity? Response sets –respond in a predictable way that has nothing to do with the questions

What can lower validity? Interviewer effects –characteristics of the interviewer affects the receptivity and answers of the respondent

What can lower validity? Population restrictions –if people can’t use the method of data collection, can’t generalize to others

End of reliability and validity Questions Look at CNEP Survey