1. ScWk 240 Week 6 Measurement Error Introduction to Survey Development “England and America are two countries divided by a common language.” George Bernard Shaw 1ScWk 240 Week 6

2. Review--Variables: When we use the word “outcome” are we talking about the independent variable or the dependent variable?___________ In an evaluative study, the primary independent variable is the __________ “This study examines the impact of acculturation on the use of health services by immigrants, controlling for income, age, and health status. – What is the dependent variable? ___________ – What is (are) the independent variable(s)? _______________________________________________________________ 2ScWk 240 Week 6

3. Review: Conceptualization We’re talking about complex ideas – Depression – Self esteem – Acculturation How would you conceptualize “acculturation”? – Give a one sentence conceptual definition – What dimensions would you include? ScWk 240 Week 63

4. Operationalization – the next step “Operationalization” – the process of defining a specific variable, and identifying the relevant attributes of the variable Example – Depression “loss of energy” dimension – Variable: “loss of energy” – Measured by: (CIDI instrument) During the period when your sadness was most severe and frequent, did you feel tired or low in energy even when you had not been working very hard? – Attributes: – Yes – No – Don’t know ScWk 240 Week 64

5. Now operationalize “acculturation” Pick one dimension For that dimension, describe the variable in a simple statement – First, give the variable a label (name) in 1-3 words – Include in the simple statement “…as measured by…” For that variable, identify one set of attributes ScWk 240 Week 65

6. Small group exercise: Pick a concept as a group Identify three dimensions of that concept Choose one dimension For that dimension, identify and describe one variable in a simple statement – First, give the variable a label (name) in 1-3 words – Include in the simple statement “…as measured by…” For that variable, identify one set of attributes ScWk 240 Week 66

7. Student Report What is “culturally competent measurement” in research? Relate this to one or more (of the 5) areas of the Transcultural Perspective. Elisse ScWk 240 Week 67

8. “Error” in Measurement What is “measurement error”? Measurement error occurs when we obtain data that do not accurately reflect the concept we want to measure. There are two main sources of measurement error: – Random error: inconsistent error. There’s no pattern or predictability – Systematic error: consistent error. There is pattern and predictability ScWk 240 Week 68

9. Random Error – inconsistent response – Environmental factors: i.e. surveys administered in a hot room will vary from surveys administered in a more comfortable setting – Questions with jargon or technical terms will be interpreted differently by different respondents (or other factors related to a poorly constructed instrument) – Written survey response will differ by educational level or reading ability of respondents ScWk 240 Week 69

10. Systematic error—consistently and predictably “wrong” response Social desirability bias’ – respondents portray themselves in the best light ‘Acquiescent response set’ – Agree or disagree with everything Cultural ‘bias’ – cultural definitions of concepts different than those intended by researcher (i.e. you can always expect physical somatization as a stated primary symptom of depression in many non- Western cultures) Bias from wording of questions – i.e. “Are you resistant to treatment?” ScWk 240 Week 610

11. Review: Potential for systematic or random error? 1.People are more alert in the morning than after lunch ____________ 2.The meaning of “living independently” varies cross- culturally __________ 3.Male respondents tend to minimize their aggressive tendencies _________ 4.A 16 y.o. taking a test: “I just circle ‘C’ each time and hope for the best” (hint-this is not Acquiescent Response) _________ 5.The person entering data is not careful enough and mis-codes some of the responses ________ 11ScWk 240 Week 6

12. Student reports 1.What’s the difference between “reliability” and “validity” of a measure? Glorialee 2.How is validity related to cultural issues? Dawei ScWk 240 Week 612

13. Measuring error—Validity and reliability The presence of error is not “all or nothing” – there’s always going to be some error We have to measure it to understand its impact Reliability – measures random error— consistency of meaning Validity – measures systematic error— accuracy of meaning ScWk 240 Week 613

14. Reliability Concerned with whether or not a particular technique done repeatedly would yield the same result each time – Inter-rater reliability—Two researchers rating the same behavior: do they rate them the same? – Test-retest reliability—Is the response from the first test the same as the response from a re-test? – Alternate forms reliability—Is there the same response from slightly different versions of same measure? – Internal consistency reliability & use of coefficient alpha— Are similar questions (about the same concept) in survey are answered in the same way? ScWk 240 Week 614

15. You‘ll see this in your lit review… Methods section—measures used: the author will report on reliability testing (if done) using the coefficient alpha – A number that tells you, overall, how well the questions in an instrument measuring the same concepts correlate with each other (internal consistency reliability) – A coefficient alpha is considered good if it is.80 or above – The author will then report “The instrument showed good internal consistency reliability” ScWk 240 Week 615

16. Validity Concerned with the accuracy of meaning—are you measuring what you originally intended? Types of validity – Face validity – experts’ understanding of concept – Content validity – Does measure cover all relevant dimensions? – Criterion-related validity – Does measure of concept agree with (or relate with) another related concept (external criterion)? – Construct validity – Does measure agree with or correlate with measures of other theoretically related concepts? ScWk 240 Week 616

17. Validity examples 1.Validity studies show that a set of questions about alcohol dependence correlates well with the observed daily amount of alcohol consumption. Criterion-related validity 2.Most experts would agree that children with Attention Deficit Disorder with Hyperactivity have difficulty staying on tasks (even those the child likes). Face validity 3.Couples who test as having low levels of marital satisfaction communicate less effectively, leading to high levels of conflict and, eventually, divorce. Construct validity 4.The Beck Depression Scale covers those dimensions of depression that are most supported by evidence. Content validity 17ScWk 240 Week 6

18. Student reports for next week 1. Dawei What are some strategies for ensuring reliability when developing survey questions?___Amanda What are some strategies for ensuring validity when developing survey questions?____Sarah What is the main difference between “open- ende” and “closed-ended” questions? Selene ScWk 240 Week 618

19. Concepts to know Measurement error Random error Systematic error Reliability (Inter-rater, Test-retest, Alternate forms, Internal consistency) Coefficient alpha Validity (Face, Content, Criterion-related, Construct) ScWk 240 Week 619