1. SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014

2. Research Design in SoTL

3. Commitment to Outcomes Research

4. Learning Objectives Identify the basic elements of four common quasi- experimental designs used in outcomes research Identify the basic elements of four common quasi- experimental designs used in outcomes research Discuss strategies to “control” factors that might impact validity of conclusions drawn from designs Discuss strategies to “control” factors that might impact validity of conclusions drawn from designs Be able to identify ethical issues in educational outcomes research and resources for IRB review Be able to identify ethical issues in educational outcomes research and resources for IRB review

5. Outcome Research is … Important Important Rewarding Rewarding Necessary Necessary

6. And often… Difficult Difficult Time-consuming Time-consuming Small effect sizes Small effect sizes

7. Type of Research Descriptive/Qualitative Question: What Is? descriptive, generative naturalistic, observational, constructivist Samples: small, targeted Assessment: interviews, observations inductive formative, “thick description”, expansive Inferential/Quantitative Question: What Works? hypothesis testing, confirmatory empirical, statistical, comparative Samples: large, representative Assessment: scores, rates deductive summative, precise, reliable

8. Formal Design Focus of SoTL Focus of SoTL Actions as educators Actions as educators Student learning and performance Student learning and performance Question guides choice of design Question guides choice of design Source Material: Source Material: Cook & Campbell (1979) – Quasi-Experimentation: Design & Analysis Issues for Field Settings Cook & Campbell (1979) – Quasi-Experimentation: Design & Analysis Issues for Field Settings Trochim (2006) – Research Methods Knowledge Base Trochim (2006) – Research Methods Knowledge Base Online: http://www.socialresearchmethods.net/kb/ Online: http://www.socialresearchmethods.net/kb/http://www.socialresearchmethods.net/kb/ NSF – User-friendly Handbook for Mixed Methods Designs NSF – User-friendly Handbook for Mixed Methods Designs Online: http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf97153 Online: http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf97153 http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf97153

9. Optimal Design -- Descriptive No single preferred design No single preferred design Common features Common features Focus on individual and collective experiences of students Focus on individual and collective experiences of students Context is described Context is described Search for patterns Search for patterns Developmental Developmental Multiple assessment methods and assessment opportunities Multiple assessment methods and assessment opportunities Subgroup analysis Subgroup analysis

10. Optimal Design -- Inferential

11. Randomized “Clinical” Trial

12. Real Life Quasi-experimental designs Quasi-experimental designs Non-random assignment to groups Non-random assignment to groups Frequently used (and effective designs Frequently used (and effective designs Single group repeated measures design Single group repeated measures design Non-equivalent groups pre-test/post-test design Non-equivalent groups pre-test/post-test design Cohort design Cohort design Single group crossover (ABAB) design Single group crossover (ABAB) design

13. Uninterpretable Designs One-shot case study One-shot case study Single-group pre-test/post-test design Single-group pre-test/post-test design Static group comparison Static group comparison X O O 1 X O 2 X O O

14. Single Group Repeated Measures/Time Series Add multiple observations to track learning/change Add multiple observations to track learning/change Goal is to study the trajectory of learning Goal is to study the trajectory of learning Depth of knowledge Depth of knowledge Understanding of concepts Understanding of concepts Application Application Integration/synthesis Integration/synthesis Testing of ideas Testing of ideas O 1 O 2 X O 3 O 4 O 5 O 6

15. Non-equivalent Groups Pre-test/Post-test Design Compare two classes Compare two classes New vs. standard New vs. standard Willing/interested colleagues Willing/interested colleagues Commitment to method Commitment to method Pre-test/Post-test essential Pre-test/Post-test essential Targeted measurement Targeted measurement O 1 X O 2 O 1 O 2

16. Cohort Design What if you are the only one who teaches this? What if you are the only one who teaches this? Or colleagues are not “willing” Or colleagues are not “willing” Compare outcomes to yourself Compare outcomes to yourself But be very, very careful But be very, very careful Risk – nothing else can change Risk – nothing else can change Assumes stable cohort Assumes stable cohort O 1 X O 2

17. Single Group Cross-over Design Multiple units/cases Multiple units/cases Replicate (ABAB) Replicate (ABAB) Standard assessment strategy Standard assessment strategy X A1 O 1 O 2 O 3 X B1 O 4 O 5 O 6 X A2 O 7 O 8 O 9 X B2 O 10 O 11 O 12

18. Less “Control” Rule out competing explanations of findings Rule out competing explanations of findings Argument/Logic Argument/Logic Measurement Measurement Student demographics Student demographics Possible pre-test differences Possible pre-test differences Characteristics of “treatment” Characteristics of “treatment” Design Design Statistical controls Statistical controls Preventive action Preventive action

19. Sound Measurement Existing standards Existing standards Independence of observations, where possible Independence of observations, where possible Conceptual comparison Conceptual comparison Multiple dimensions/skills Multiple dimensions/skills

20. Your Turn… Choose design that best “fits” your case Choose design that best “fits” your case Type of class Type of class Possibility of comparison Possibility of comparison Assessment strategy Assessment strategy

21. IRB Review Educational research Educational research Human participants Human participants Type of research Type of research Non-research/exempt/expedited Non-research/exempt/expedited Primary Issues Primary Issues Coercion Coercion Informed consent Informed consent

22. Everything IRB www.irb.emory.edu www.irb.emory.edu www.irb.emory.edu Social-behavioral studies Social-behavioral studies Multi-disciplinary team Multi-disciplinary team Rebecca Rousselle Rebecca Rousselle Carol Corkran Carol Corkran

23. Non-Research New category New category Systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge Systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge PI can make this determination PI can make this determination If it is just for your own evaluation, it is not research If it is just for your own evaluation, it is not research

24. Exempt vs. Expedited Most surveys, interviews, or observations of public behavior are presumed exempt if the participant cannot be identified (no links from the person to the data) or the responses/data can not harm the participant. Most surveys, interviews, or observations of public behavior are presumed exempt if the participant cannot be identified (no links from the person to the data) or the responses/data can not harm the participant. IRB determines this IRB determines this

25. Coercion Major issue Major issue Students must be free not to participate Students must be free not to participate

26. Informed Consent Description of the purpose of the study Description of the purpose of the study Who is being studied Who is being studied What will be required What will be required Risks/benefits Risks/benefits Compensation Compensation Legal review Legal review Contact information Contact information Summary of findings Summary of findings