Presentation on theme: "How to Implement a Randomization-Based Introductory Statistics Course: The CATALST Curriculum Bob delMas, Laura Le, Nicola Parker and Laura Ziegler Funded."— Presentation transcript:
How to Implement a Randomization-Based Introductory Statistics Course: The CATALST Curriculum Bob delMas, Laura Le, Nicola Parker and Laura Ziegler Funded by NSF DUE-0814433
Overview of Workshop Overview of CATALST Course TinkerPlots Introduction UNIT 1: Chance Models and Simulation UNIT 2: Models for Comparing Groups – Randomization Methods UNIT 3: Estimating Models Using Data – Bootstrap Methods Assessment Results
University of Minnesota Project Team FACULTY Joan Garfield Andy Zieffler Bob delMas GRADUATE STUDENTS Rebekah Isaak Laura Le Laura Ziegler
Our Collaborators Allan Rossman, Cal Poly State U Beth Chance, Cal Poly State U John Holcomb, Cleveland State U George Cobb, Mt. Holyoke Coll. Herle McGowan, NCSU Instructors at different institutions who have implemented CATALST
CATALST Course: Teaching Students to Really Cook Metaphor from Alan Schoenfeld (1998) Many intro stats classes teach how to follow “recipes” but not how to really “cook.” Able to perform routine procedures and tests Don't have the big picture that allows them to solve unfamiliar problems and to articulate and apply their understanding. Someone who knows how to cook knows the essential things to look for and focus on, and how to make adjustments on the fly.
CATALST: The Inference Model The core logic of inference as the foundation (Cobb, 2007) Model: Specify a model to reasonably approximate the variation in outcomes attributable to the random process Randomize & Repeat: Use the model to generate simulated data and collect a summary measure Evaluate: Examine the distribution of the resulting summary measures
Radical Content No t-tests; Use of probability for simulation and modeling (TinkerPlots) Coherent curriculum that builds ideas of models, chance, simulated data, inference from first day Immersion in statistical thinking Activities based on real problems, real data Course materials contain all classroom activities and homework assignments Lesson plans posted at CATALST website
Radical Pedagogy Student-centered approach based on research in cognition & learning, instructional design principles Minimal lectures, just-in-time as needed Cooperative groups to solve problems “Invention to learn” and "test and conjecture" activities [develop reasoning & promote transfer] Writing & whole class discussion (wrap-up)
Why TinkerPlots? Fathom ® is a viable option for building models and simulation, but also challenging to students (Maxara & Biehler, 2006, 2007; Biehler & Prommel, 2010) TinkerPlots™ was chosen because of unique visual (graphical interface) capabilities Allows students to see the devices they select (e.g., sampler, spinner) Easily use these models to simulate and collect data Allows students to visually examine and evaluate distributions of statistics
3 CATALST Units (14 Week Semester) Chance Models and Simulation Learning to use the core logic of inference (George Cobb) Specify a chance model (sampling or random assignment) Generate a trial, Collect measure, Repeat many times Evaluate fit of chance model to the observed data Models for Comparing Groups Randomization Tests Random Assignment Studies Observed Data Studies Design: Random Assignment and Random Sampling Drawing valid conclusions using logic of inference Estimating Models Using Data Bootstrap Method Standard Error of a Sample Statistic Confidence Intervals
Workshop Overview Summary You will work through shortened versions of three activities from the CATALST course. You will do one activity from: Unit 1 - Chance Models and Simulation Unit 2 - Models for Comparing Groups Unit 3 - Estimating Models Using Data Preliminary Assessment Results
Agenda for Session 1 An introduction to TinkerPlots One activity from Unit 1 on Chance Models and Simulation A modified version of the Matching Dogs to Owners
An introduction to TinkerPlots: Modeling Dice
Matching Dogs to Owners: Learning Goals Develop the reasoning of statistical significance Understand how observed result can be judged unlikely under a particular model, Begin the process of statistical thinking
At this point students have…. Modeled random behavior in TinkerPlots Coins, dice Colors of M&M candies Effects of “One Son” or “One of Each” child policies (using a stopping criterion) …and Introductory readings on hypothesis testing have been assigned Matching Dogs to Owners: Student Preparation
Matching Dogs to Owners: Research Question http://media-cache- lt0.pinterest.com/192x/97/b8/56/97b856be7d0a265348a6f2ea71531ce0.jpg
Matching Dogs to Owners: Research Question Are humans able to match dogs to owners better than blind luck?
Matching Dogs to Owners: Simulation For each numbered person in the following photos, write down the letter for the dog that you think is owned by that person.
1. _____ A B 2. _____ 3. _____ 4. _____ 5. _____ E D C F 6. _____
A B 2. _C__ 3. _F__ 4. __E__ 5. __B__ E D C F 6. __D__ 1. _A__
Matching Dogs to Owners: Research Question Are humans able to match dogs to owners better than blind luck?
Matching Dogs to Owners: Directions Let’s do the modified activity Again, put on your “student hat”! Afterward, you will get to put your “teacher hat” back on
Matching Dogs to Owners: Wrap-up What does the “blind guessing” model mean?
Matching Dogs to Owners: Wrap-up Why do we use the “blind guessing” model to simulate our data?
Matching Dogs to Owners: Wrap-up What does it mean to have evidence that supports or does not support the “blind guessing” model?
Matching Dogs to Owners: Wrap-up Tinkerplots steps
Matching Dogs to Owners: Reflection What about this activity (content, format…) do you think might help maximize student learning in your classroom? What are your hesitations / what do you think might hinder student learning in your classroom? What questions do you still have about the implementation of such a course? Presuming that you wanted to implement these activities in your courses, how comfortable would you feel doing so? Why or why not?
Matching Dogs to Owners: Reflection What about this activity (content, format…) do you think might help maximize student learning in your classroom?
Matching Dogs to Owners: Reflection What are your hesitations about this activity / what do you think might hinder student learning if you were to use it in your classroom?
Matching Dogs to Owners: Reflection What questions do you still have about the implementation of this type of activity?
Matching Dogs to Owners: Reflection Presuming that you wanted to implement these activities in your courses, how comfortable would you feel doing so? Why or why not?
Matching Dogs to Owners: Building the Model & Simulation
Agenda for Session 2 Two activities from Unit 2 on Comparing Groups A modified version of the Sleep Deprivation Activity A modified version of the Contagious Yawns Study Homework
Develop the need for a summary measure to compare two groups with quantitative data (e.g., different summary measures provide different information regarding characteristics of the data; some more relevant than others) Sleep Deprivation: Learning Goals
Learn how to determine if a single observed difference is real and important or just due to chance (the need to consider the observed result in the distribution of results that are possible under the null model) Find the approximate p-value from simulated data and draw a conclusion Sleep Deprivation: Learning Goals
Sleep Deprivation: Prior Knowledge Informal idea of p-value, but the term is introduced in this activity Basic idea of comparing groups Randomization test (by hand)
Students read the abstract from Stickgold, R., James, L., & Hobson, J. A. (2000). Visual discrimination learning requires sleep after training. Nature Neuroscience, 3(12), 1237- 1238. The reading is a scientific article abstract that introduces the context of and provides motivation for the Sleep Deprivation activity. Sleep Deprivation: Student Preparation
Begin with a discussion on how to measure whether or not the amount of sleep effects test performance. Have students come up with different methods. Describe how that relates to the activity. Sleep Deprivation: Preliminary Discussion
Does the effect of sleep deprivation last, or can a person “make up” for sleep deprivation by getting a full night’s sleep in subsequent nights? Stickgold, R., James, L., & Hobson, J. A. (2000). Visual discrimination learning requires sleep after training. Nature Neuroscience, 3(12), 1237-1238. Sleep Deprivation: Research Question
21 Human Subjects 11 Sleep Deprived 10 Unrestricted Sleep Sleep Deprivation: Study Design
Sleep Deprivation: Directions Let’s do the modified activity Again, put on your “student hat”! Afterward, you will get to put your “teacher hat” back on
Sleep Deprivation: Sample Wrap-Up Questions What was our null model? Why do we need to conduct a test, why can't we just look at the observed difference? What is the purpose of random assignment? Where was the plot centered? Why does that make sense? What is a p-value? What conclusion did you come to for the sleep study?
Sleep Deprivation: Building the Model & Simulation
Contagious Yawns Study: Homework Assignment Research Question: Are yawns contagious?
MythBusters [Photograph]. Retrieved April 11, 2013, from: http://store.discovery.com/?v=discovery_shows_mythbusters Contagious Yawns Study: Study Design 50 Human Subjects 34 Yawn Seed Planted 16 No Yawn Seed Planted
Subject Yawned Subject Did Not YawnTotal Yawn Seed Planted 102434 Yawn Seed Not Planted 41216 Total 143650 Contagious Yawns Study: Data
Describe how you would create a model to answer the research question. Describe the simulation process you would use to answer the research question. Contagious Yawns Study: Questions
Contagious Yawns Study: Building the Model & Simulation
Quantify the strength of evidence/p-value for the observed result. Note that the difference in the proportion that yawned was 0.044. In light of your previous answer, would you say that the results that the researchers obtained provide strong evidence that yawning is contagious? Explain your reasoning based on your simulation results. Contagious Yawns Study: Homework Questions
What was the purpose of this homework assignment? How did it differ from the Sleep Deprivation Activity? Contagious Yawns Study: Reflection
Agenda for Session 3 Overview of Unit 3 on Estimating Models Using Data A demonstration of the Kissing the Right Way Activity
Unit 3 Overview: Estimating Models Using Data Sampling Bias, precision, and size of population Comparing Hand Spans Formalizes summary measure of variation (standard deviation) Kissing The Right Way Intro to confidence intervals Memorizing Letters Part II Confidence interval for effect size Why 2 Standard Errors?
Kissing the Right Way
Kissing the Right Way: Learning Goals Understand what standard error is Understand the difference between standard deviation and standard error Understand what margin of error is Understand what an interval estimate is Understand the purpose of an interval estimate
Kissing Study Activity Collect data! What percentage of couples lean their heads to the right when kissing? How can we find out?
Kissing Study Activity 124 Couples 80 Couples Lean Right 44 Couples Lean Left
Kissing Activity How much variation is there in the estimate from sample-to-sample? Sixty-five percent of the couples observed leaned to the right. What percentage of all couples lean to the right? Use sample as a substitute for the population
Kissing Study: Building the Model & Simulation
Kissing Study: Building the Model & Simulation
Kissing Study: Building the Model & Simulation
Kissing the Right Way: Sample Wrap-Up Questions What is standard error? What is margin of error? When would we need to use margin of error? How would you interpret the margin of error? What is an interval estimate? What is the purpose of an interval estimate? How would you interpret your interval estimate? What do you think the purpose was of this activity?
Agenda for Session 4 Summary of Assessment Results Overview of Assessment Instruments Comparison of CATALST and Non- CATALST students
CATALST: Assessment GOALS: 27 forced-choice items Study design Reasoning about variability Sampling and sampling variability Interpreting confidence intervals and p-values Statistical inference Modeling and simulation MOST: Measure of statistical thinking 4 real-world contexts open-ended and forced-choice items AFFECT: attitudes and perceptions
Performance of the CATALST (n = 289) & non-CATALST (n = 440) groups on the GOALS test
Bootstrapped Confidence Interval Limits for Each Item (CATALST: n = 289; non-CATALST: n = 440)
GOALS Item 1 Design & Conclusions A recent research study randomly assigned participants into groups that were given different levels of Vitamin E to take daily. One group received only a placebo pill. The research study followed the participants for eight years to see which ones developed a particular type of cancer during that time period. What is the primary purpose of the use of random assignment for making inferences based on this study? CATALSTNON-CATALSTResponse Options 17.0%38.2% a.To ensure that a person doesn’t know whether or not they are getting the placebo. 66.7%29.1% b.To ensure that the groups are similar in all respects except for the level of VitamE. 16.3%32.7% c.To ensure that the study participants are representative of the larger population.
A local television station for a city with a population of 500,000 recently conducted a poll where they invited viewers to call in and voice their support or opposition to a controversial referendum that was to be voted on in an upcoming election. Over 5,000 people responded, with 67% opposed to the referendum. The TV station announced that the referendum would most likely be defeated in the election. Select the best answer below for why you think the TV station's announcement is valid or invalid. CATALSTNON-CATALSTResponse Options 5.2%22.3% a.Valid, because the sample size is large enough to represent the population. 4.8%17.5% b.Valid, because 67% is far enough above 50% to predict a majority vote 7.2%16.8% c. Invalid, because the sample is too small given the size of the city 82.7%43.4% d. Invalid, because the sample is not likely to be representative of the population GOALS Item 3 Design & Conclusions
GOALS Items 23, 24 & 25 Understanding p-Values For questions 23-25, indicate whether the interpretation of the p-value provided is valid or invalid. STATEMENT CORRECT RESPONSECATALST Non- CATALST The p-value is the probability that the $5 incentive group would have the same or lower success rate than the “do your best” rate. INVALID82.3%58.0% The p-value is the probability that the $5 incentive group would have a higher success rate than the “do your best” group. INVALID58.4%48.9% The p-value is the probability of obtaining a result as extreme as was actually found, if the $5 incentive is really not helpful. VALID70.0%51.4% ANSWER ALL THREE ITEMS CORRECTLY 39.5%19.3%
MOST Test: Statistical Thinking Example of an Open-Ended Item: Context Consider an experiment where a researcher wants to study the effects of two different exam preparation strategies on exam scores. Twenty students volunteered to be in the study, and were randomly assigned to one of two different exam preparation strategies, 10 students per strategy. After the preparation, all students were given the same exam (which is scored from 0 to 100). The researcher calculated the mean exam score for each group of students. The mean exam score for the students assigned to preparation strategy A was 5 points higher than the mean exam score for the students assigned to preparation strategy B.
Open-ended Question 7.Explain how the researcher could determine whether the difference in means of 5 points is large enough to claim that one preparation strategy is better than the other. (Be sure to give enough detail that someone else could easily follow your explanation in order to implement your proposed analysis and draw an appropriate inference (conclusion).) MOST Test: Statistical Thinking
MOST Test: Sample Scoring Rubric ComponentRandomization-BasedParametric-Procedures Modeling: Hypothesis Describes the null model of no difference Describes the Null and Alternative hypotheses Context: Hypothesis Identifies the context in definition of null model Identifies the context in definition of null and alternative hypotheses Modeling: TestDescribes the simulation: statistic to collect & repetition States a z-test & describes checking assumptions ConclusionsDescription identifies how conclusion can be drawn from the test results Context: Conclusion Conclusion presented in the context of the problem: Refers to the percentage of breakups for Monday or percentage of breakups for each day of the week
MOST Test: Preliminary Findings CATALST students include a higher percentage of procedural components For all students, responses are weakest on making conclusions and including context CATALST students more likely to describe the use of technology in their responses CATALST students tend to describe steps in more detail CATALST students more likely to map a problem to a previously solved problem Non-CATALST students more likely to present a non- statistical explanation for outcomes
Affect Survey: Percent Agree or Strongly Agree Non-CATALST (n = 453) CATALST (n = 283) Learning to use (software/TinkerPlots TM ) was an important part of learning statistics. 68.7%81.5% I would be comfortable using (software/TinkerPlots TM ) to test for a difference between groups after completing this class. 66.0%91.0% I would be comfortable using (software/TinkerPlots TM ) to compute an interval estimate for a population parameter after completing this class. 61.7%83.7% Learning to (use software/create models in TinkerPlots TM ) helped me learn to think statistically. 61.6%84.5%
CATALST: Interview Study Examined statistical reasoning of 5 tertiary students after Chance Models and Simulation unit Each of two novel problems asked students to reason about the likelihood of a surprising result After five weeks, students’ demonstrated consideration of sampling variability when drawing inferences Students’ solutions typically: Considered the value most likely to occur under the chance model (expected value) Demonstrated an awareness of sampling variability Quantified the degree of “unusualness”
CATALST publications Garfield, J., delMas, R. & Zieffler, A. (2012). Developing statistical modelers and thinkers in an introductory, tertiary-level statistics course. ZDM: The International Journal on Mathematics Education. Ziegler, L. and Garfield, J. (2013) Exploring students' intuitive ideas of randomness using an iPod shuffle activity. Teaching Statistics, 35(1), 2- 7. Isaak, R., Garfield, J. and Zieffler, A. (in press). The Course as Textbook. Technology Innovations in Statistics Education. Garfield, J., Zieffler, A., delMas, R. & Ziegler, L. (under review). A New Role for Probability in the Introductory College Statistics Course. Journal of Statistics Education. delMas, R., Zieffler, A. & Garfield, J. (under review). Tertiary Students' Reasoning about Samples and Sampling Variation in the Context of a Modeling and Simulation Approach to Inference. Educational Studies in Mathematics.
Thank You for your Participation Catalysts for Change (2012). Statistical thinking: A simulation approach to modeling uncertainty. Minneapolis, MN: CATALST Press. (Purchase at amazon.com) If you have any questions about the CATALST course, please contact anyone of the Pis: Joan Garfield: firstname.lastname@example.org Robert delMas: email@example.com Andy Zieffler: firstname.lastname@example.org We appreciate your feedback – please fill out the workshop evaluation