Lecture 17

Survey validity Thanks to Dr. Neupert (my wife) for slides

2. Survey: Questionnaire & Interview – Positive: People can be the best experts on their own behavior Unobservable data (i.e. opinions & beliefs) – Negative: Social desirability, memory

Validity A. Construct: does the operational definition “ fit ” ? B. Internal: causal conclusions C. External: generalizability

Reliability A. Consistency or stability of a measure – 1. True score – 2. Measurement error – Observed score = True score + Measurement error B. Assessment – 1. Not directly observed/tested – 2. Pearson correlation

2. Pearson correlation (cont) a. test-retest reliability: 2 points in time b. internal consistency: 1 point in time – i. split-half – ii. Cronbach ’ s alpha c. interrater reliability

Construct validity A. How well does the operational definition “ fit ” ? B. Face validity C. Criterion-oriented validity – 1. Predictive validity – 2. Concurrent validity – 3. Convergent validity – 4. Discriminant validity

Measurement scales A. Nominal: no numeric properties B. Ordinal: rank order C. Interval: difference is meaningful D. Ratio: absolute zero

Surveys A. Match questions to research objectives B. Types of questions: – 1. attitudes and beliefs – 2. facts and demographics – 3. behaviors

C. Question wording 1. Avoid: – a. technical terms – b. vagueness – c. grammatical errors – d. memory overload – e. misleading information – f. double-barreled questions – g. loaded questions – h. negative wording – i. yea-saying and nay-saying

CRITIQUE (from Quaid) 1. Unfamiliar technical term : sibling. The following term may be unfamiliar to some respondents: "sibling" in the first sentence. 2. Complex syntax Sentence 1: The question is either ungrammatical or difficult to parse syntactically. (4 modifiers for a noun are counted.) 3. Unfamiliar technical term : environmental. The following term may be unfamiliar to some respondents: "environmental" in the first sentence. 4. Unfamiliar technical term : you ’ re. The following term may be unfamiliar to some respondents: "you ’ re" in the first sentence. Example (bad): Did or did not you ’ re monozygotic, genetically identical sibling have the same environmental resources as you? Example (better): Were you raised in the same household as your twin?

2. Do: – a. keep it simple – b. reverse the direction of some items – c. make items emotionally neutral

D. Responses to questions 1. Closed-ended 2. Open-ended 3. Number of response alternatives 4. Rating scales: labeled extremes – a. graphic – b. semantic differential – c. non-verbal

5. Labeling response alternatives (cont) a. vague quantifiers b. odd vs. even numbers of responses c. quantitative differences between intervals

E. Formatting 1. PROOFREAD!!! 2. consider the sequence 3. clearly demarcate questions/items and responses 4. keep response formats consistent within a scale

Examples FAMILISM SCALE (Reliability.84; Mean 47.77; S.D. 8.89) 1. Generally we don't all have meals at the same time in our family. R 2. Our family discuss a lot of things together. 3. We usually go out together at least once a week in our family. 4. I care about my family above everything else. 5. Our family has always been a close-knit family. 6. I am very proud of my family. 7. Members of our family have tended to grow away from one another. R 8. We're not as close as we used to be in our family. R 9. Our family does a lot of things together. 10. We always co-operate with one another in our family. 11. Family life can be terribly dull and boring. R 12. I sometimes feel like a stranger in my family. R 13. Family life these days is not what it used to be.

Familism scale 1. If someone has the chance to help a person get a job, it is always better to choose a relative rather than a friend. 2. When someone has a serious problem, only relatives can help. 3. When looking for a job a person should find a job near his/her parents even if it means losing a better job somewhere else.

ECONOMIC CONSERVATISM SCALE (Perhaps better referred to as a "Socialism" scale) (Reliability.82; Mean 41.14; S.D. 9.84) "Am satisfied with" 1. Existing levels of foreign ownership of Australian industry. 2. The housing and land situation. "Am in favour of": 3. Moves to control poverty in Australia. R 4. Increased government spending on pensions and other social service payments. R 5. Moves towards government control of the health insurance schemes. R 6. Greater control of the unions by the government. 7. Power for the Federal government to control prices. R 8. Power for the Federal government to control incomes. R 9. A free dental service provided by the government. R 10. Government control of the big industries such as steel. R 11. Higher rate of taxation on the large income earner. R 12. Government power to control big businesses and monopolies. R

Final Project Design your own survey! – Find an interesting question and population – Design your sampling plan – Collect Data – Analyze using R Write 5 page paper on your results Due November 25 (just before Thanksgiving)

Final presentation During the last class (December 3) all students will be required to give a short presentation – Select one of the three projects – Make a powerpoint presentation (no more than 3- 5 slides) – Present your results to the class