1. ScWk 240 Week 8 Sampling (continued) Intro to Causality and Group Designs “The tendency of the casual mind is to pick out or stumble upon a sample which supports or defies its prejudices, and then to make it the representative of a whole class.” Walter Lippman (1889-1974), journalist

2. Review: Name the implied type of reliability or validity testing: 1.Two clinical observers of parent-child interaction mostly agree on whether the parent’s response to the child’s behavior is flexible or rigid _______________ 2.A researcher wants to make sure that the instrument she is developing covers all the various dimensions of the concept “empowerment.” __________________ 3.A new instrument called the Self Perceived Empowerment Scale has a coefficient alpha of.85 __________________ 4.A person scoring higher on a depression scale (more depressed) will also often score higher on an anxiety measurement __________________ 5.A person scoring higher on a depression scale (more depressed) will also often have a diagnosis of depression after a clinical interview _______________ Inter-rater reliability Test-retest reliability Internal consistency reliability Criterion-related validity Content validity Face validity Construct validity 2

3. News flash—much much better definition of construct validity: Extent to which an instrument accurately measures one concept. For example, an instrument designed to measure alcoholism accurately measures alcoholism (not depression or other diagnosis). In addition, construct validity can also refer to the extent to which a measure of a construct (like alcoholism) is correlated with another theoretically-related measure (such as “addictive personality”) Source: Faulkner, C. A. & Faulkner, S. S. (2009). Research methods for social workers: A practice-based approach. Chicago, IL: Lyceum 3

4. Student report 1.What is the main difference between “open- ended” and “closed-ended” questions? Selene 2.Discussion question for class: what are the implications for reliability and validity for both types of questions? 4

5. Two main types of sampling: Probability sampling -- everyone in the study population has an equal chance of selection into your study. Non-probability sampling -- everyone in the study population does not have an equal chance of selection ***Discussion: what are the pros and cons of both? 5

6. The Logic of Probability Sampling Overall purpose of probability sampling: to allow the researcher to estimate characteristics (parameters) of the study population from a sample’s characteristics – e.g. “How much does the sample represent the population from which it was drawn?” Why do we want to “estimate population characteristics”? – Because in most cases we don’t really know the real population characteristics (but we’d like to know!) So, we estimate a population parameter (e.g. average depression score of the population), from a sample statistic (average depression score of the sample) Since research means “never having to say you’re certain”, we won’t be 100% perfect in our estimates, although we can quantify how confidently we feel our sample matches the population. How? Stay tuned for ScWk 242. 6

7. Probability Sampling Example: “How do stress levels of teen parents compare with stress levels of other youth?” Probability sample of teen parents Population: All youth From the sample statistic we can “infer” stress levels in the larger population of youth Statistic – comparison of stress levels Probability sample of non-teen parent youth Select random samples of teen parents and non-parent youth 7

8. How to do probability sampling Simple random sample—assigning numbers to potential participants and selecting numbers (people) randomly. Similar to picking them out of a hat Systematic random sample—choosing every kth person--selecting persons at a predetermined interval Stratified random sampling--Simple or systematic random sampling with sub-groups Cluster sampling—a multi-stage procedure of randomly choosing levels of analysis units (e.g. cities, neighborhoods, schools, classrooms) 8

9. Probability sampling “ripped from the headlines…” http://www.pbs.org/newshour/bb/politics/july -dec12/makingsense_10-12.html 9

10. Non-probability sampling Quota sampling—sample chosen based on predefined characteristics of study so that sample will have same proportion of those characteristics as in the study population  Think of a journalist standing on a street corner interviewing people who walk by, but who also wants the opinions of both men and women ***What’s the difference between this and stratified random sampling? Snowball sampling—for difficult-to-locate populations: finding more sample participants based on recommendations from others in the study  You’re interviewing a victim of domestic violence and you ask “Can you think of anyone else at this shelter who might be able to talk with me about this?” Purposive sampling —selecting sample that is thought to yield the most comprehensive understanding of the study’s topic  You put up a flyer inviting students to participate whose parents immigrated to the US Convenience sampling (a.k.a. “availability sampling”)—selects participants simply based on their immediate availability. (Note—participants may also coincidentally comprise a purposive sample, but not necessarily)  You decide to interview staff of a county mental health agency about their training in cultural competence 10

11. Two families of sampling Probability sampling Simple random sampling Systematic sampling Stratified random sampling Cluster sampling Non-probability sampling Quota sampling Snowball sampling Purposive sampling Convenience sampling 11

12. Student report Name some strategies to recruit difficult-to- reach populations, and retain them for follow up -- Annalisse 12

13. What type of sampling is this? (hint: first decide if it’s prob or non-prob) 1.For an experimental study of cognitive behavioral therapy, every 10 th adult with anxiety disorders is selected randomly from a sampling frame 2.An agency manager walks down the hall and asks staff opinions about the new caseload policy 3.The same agency manager makes sure to ask both women & men 4.For a study on standardized testing in schools, researchers randomly select cities in California, then randomly select elementary, middle and high schools in those cities 5.A social work researcher seeks out Latino women who were victims of domestic violence Simple random sampling Systematic sampling Stratified random sampling Cluster sampling Quota sampling Snowball sampling Purposive sampling Convenience sampling 13

14. Student report What is required in order to prove causality? Jenna 14

15. Establishing Causality—What are the Requirements*? 1.Time sequence: the cause (independent variable, or treatment) precedes the effect (dependent variable, or outcome) 2.The IV and the DV are associated. (They must be related or correlated statistically.) 3.The relationship between the IV and DV cannot be explained by a third variable (or rival hypothesis—an alternate explanation) *Original source: John Stuart Mill (1859). A system of logic 15

16. Causality and an Evaluative Study— ”Does group therapy reduce anxiety?” 1.Time sequence 2. Correlation (or association) Those getting group therapy will also have reduced level of anxiety. (Correlation alone doesn’t ensure causality!) 3.Outcome not caused by 3 rd variable (rival hypothesis) X Group therapy (indep variable) Y Anxiety (dep variable) Group tx (X 1 ) Medication (X 2 ) Anxiety 16

17. Set up experiment to show causality* 1.Independent variable should precede DV in time* 2.Treatment is “manipulated” as an IV 3.Establish statistical association between IV and DV* 4.Eliminate rival hypotheses* (How? See 5.) 5.Use control group with Random assignment Indep var (X)Dep var (Y) Indep var (X) Treatment No treatment Attributes: Treatment is associated with some change in the DV X2X2 X1X1 Y Assign to treatment? Assign to control (no tx)? ? Sample 17

18. Random Assignment Example: “The impact of a parenting class on teen parents’ stress levels” Those in teen parenting class All teen parents From the sample statistic we can “infer” how the population of teen parents will benefit from the class Statistic –how much better teens in parenting class did Those not in teen parenting class Randomly assign a sample of teens to parenting class, and those not in class 18

19. Alternative Ways to Manipulate Independent Variable (not complete list) Experimental GroupControl Group Experimental TreatmentNo treatment Large amount of experimental treatment Small amount of experimental treatment Experimental TreatmentOther type of tx (or treatment-as-usual) Experimental treatment2 control groups: 1)Other type of tx 2)Experimental treatment plus other type of tx 19

20. Random sampling, random assignment (they’re different & easy to confuse) Study Population N =10,000 Probability sampling Sample n=100 Experimental Group n=50 Control Group n=50 Random* assignment (or Non-probability procedure) *Note: random assignment can also be stratified within the experimental and control groups 20

21. Student reports next week: What is the most important difference between experimental group designs and all others? (Hint—has to do with group assignment) -- Ashley Give an example of a one group, pretest posttest design -- Kristen 21

22. Concepts to know Three requirements for establishing causality Random assignment Rival hypothesis Experimental group Control group “infer” from a sample statistic “Manipulating the treatment variable” X is independent variable, Y is dependent variable 22