1. Authentic Discovery Projects in Statistics GAMTE Annual Conference October 14, 2009 Dianna Spence Robb Sinn NGCSU Math/CS Dept, Dahlonega, GA

2. Agenda Overview of Project Scope and Tasks – Dianna Teaching Model & Discovery Materials Developed – Robb Instructor Observations During Pilot – Dianna Findings (so far) – Robb

3. NSF Grant Project Overview Grant Title: “Authentic, Career-Specific Discovery Learning Projects in Introductory Statistics” Project Goals: Increase students’...  knowledge & comprehension of statistics  perceived usefulness of statistics  self-beliefs about ability to use and understand statistics Tasks:  Develop Instruments  Develop Research Constructs and Projects  Develop Materials and Train Instructors  Measure Effectiveness

4. Interdisciplinary Team Disciplines Represented  Biology/Ecology  Criminal Justice  Psychology  Sociology Tasks of Team Members  Identify authentic research constructs  Define instrument/measurement of construct  Suggest simple statistical research projects  Nursing  Physical Therapy  Education  Business

5. Exploratory Study Fall 2007 Instrument Validation and Concept “Trial Run” Based on 10 sections of Introductory Stats 4 experimental sections  Used authentic discovery projects  n=113 participants out of 128 students 88% participation rate 6 control sections  Did not use authentic discovery projects  n = 164 participants out of 192 students 85% participation rate

6. Exploratory Results: Content Knowledge Instrument  21 multiple choice items  KR-20 analysis: score = 0.63 Results  control mean: 8.87; experimental mean = 10.82  experimental mean 9 percentage points higher  experimental group significantly higher (p <.0001)  effect size = 0.59 Instrument shortened to 18 items for full study

7. Exploratory Results: Perceived Usefulness of Statistics Instrument  12-item Likert style survey; 6-point scale  5 items reverse scored  score is average (1 – 6) of all items  Cronbach alpha = 0.93 Results  control mean: 4.24; experimental mean = 4.51  experimental group significantly higher (p <.01)  effect size = 0.295 Instrument unchanged for full study

8. Exploratory Results: Statistics Self-Beliefs Beliefs in ability to use and understand statistics Instrument  15-item Likert style survey; 6-point scale  score is average (1 – 6) of all items  Cronbach alpha = 0.95 Results  control mean: 4.70; experimental mean = 4.82  difference not significant (1-tailed p =.1045)  effect size = 0.15 Instrument unchanged for full study

9. Full Study: Pilot of Developed Materials 3 institutions  1 university (6 undergraduate sections)  1 2-year college (2 sections)  1 high school (3 sections) 7 instructors Quasi-Experimental Design  Spring 2008: Begin instructor “control” groups  Fall 08 - Fall 09: “Experimental” groups

10. Teacher Training – Pilot Instructors Took place before pilot of materials Half a day training Follow-up meetings Work sessions Individual Mentoring

11. Teacher Training Workshop For Secondary Teachers 1 day workshop Follow-up online assignments PLU credit available Units covered 1.Designing Quality Variables and Constructs 2.Hands-on Survey Design Session 3.Project Organization, Phases, Assessment, and Rubrics 4.Best Practices and Avoiding Pitfalls (Panel Discussion) 5.Technology Tools and Hands-On Data Analysis 6.Team Presentations (Participants share their work product) 7.Instructor Observations from First Implementations

12. What we discovered… …about discovery learning

13. Instructional Model: Discovery Learning in Statistics Authentic Research Projects  Experiencing the Scientific Method  Discovering Statistical Methods in Context Design of Research Question Definition of Variables Demographic Data Representative Sampling Issues Data Collection Appropriate Analyses of Data Interpretation of Analyses

14. Project Format Linear regression  Variables student selects often survey based constructs  Survey design  Sampling  Regression analysis t-tests  Variables may use data previously collected  Designs Independent samples Dependent samples  Hypotheses

15. Materials Developed (Web-Based) Instructor Guide  Project overview Timelines Implementation tips Best practices  Handouts for different project phases  Evaluation rubrics  Links to student resources

16. Materials Developed (Web-Based) Student Guide  Overall Project Guide Help for each project phase  Technology Guide  Variables and Constructs

17. Critical Issue: Defining Variables We advise that our students:  Try to measure interests, obsessions, or priorities  Narrow their focus Some great variable ideas we’ve seen include:  Number of text messages sent / received during class  Pairs of shoes owned  Minutes spent getting ready this morning  Allowance money per week / month  The car you drive regularly is ______ years old Better than “rich parents” variable for correlations  Age you were when you had your first real kiss  Interest in an MRS degree (Scale of 1 to 10)

18. Critical Issue: Defining Variables We advise that our students:  Try to measure interests, obsessions, or priorities  Narrow their focus Some great variable ideas we’ve seen include:  Number of text messages sent / received during class  Pairs of shoes owned  Minutes spent getting ready this morning  Allowance money per week / month  The car you drive regularly is ______ years old Better than “rich parents” variable for correlations  Age you were when you had your first real kiss  Interest in an MRS degree (Scale of 1 to 10)

19. Critical Challenge Good mathematics teachers without experience doing quantitative research struggled with “numericizing” variables. Work Around  Give teachers “hands on” experience  Workshop: “Make it Real” Teachers developed their own survey questions Math majors worked during workshop –Copied/distributed survey to classes near Math office –Entered data sets into an Excel spreadsheet Teachers analyzed data and presented findings about their research question by end of day

20. Instructor Experiences and Observations

21. Importance of Project Structure Intermediate goals Defined deliverables and project phases Student accountability at each phase Class time needed for project guidance Requirements for final report  outline  template  prior work samples

22. Instructor Experiences and Observations Setting Student Expectations Students underestimate time/effort required Students often unclear on exactly what to do once they have collected the data Students should be prepared for results that may be weak, non-significant, etc.  realistic view of statistics  avoid too much disappointment

23. Instructor Experiences and Observations Student Expectations – Quotes Shared by Instructors “The main thing that we have learned is that statistics take time. They cannot be conjured up by a few formulas in a few minutes. The time and effort that is put into a small research project such as this is significant. On a large scale, one can quickly understand the kind of commitment of money and time that is required just to obtain reasonable data.” “While our results did not meet our initial expectations, this is not an utter disappointment. Before this project, statistics looked simple enough for anyone to sit down and do, but now it is evident that it requires more creativity and critical thinking than initially expected. Overall, it was an edifying experience.”

24. Instructor Experiences and Observations Resolving Team Issues Set guidelines for team communication Assign roles for different team members Individual accountability for group result Independent project options for some projects Assigned teams vs. student-selected

25. Our Results

26. Instrument Perceived Usefulness  Pretest: 50.42  Posttest: 51.40  Significance: p = 0.208 Self-Efficacy for Statistics  Pretest: 59.64  Posttest: 62.57  Significance: p = 0.032** Content Knowledge  Pretest: 6.78  Posttest: 7.21  Significance: p = 0.088*

27. Self-Beliefs Statistics Self-Efficacy  Self-efficacy improved significantly overall Strong Gains –SE for Regression Techniques ( p = 0.035 ) –SE for General Statistical Tasks ( p = 0.018 ) Little or No Improvement –SE for t-test Techniques ( p = 0.308 ) Perceived Utility for Statistics  Students perceptions of the usefulness of statistics improved slightly but not significantly Perceived Utility ( p = 0.208 )

28. Performance Gains Concept Knowledge: 3 Components  Regression Techniques Moderately Significant ( p = 0.086 )  T-test Usage Moderately Significant ( p = 0.097 )  T-test Inference No gain Instrumentation timeline  Difficult to “squeeze in” time for the instrumentation and all t-test topics for first time instructors

29. Summing Up Impact on Students Students experienced strong gains in their confidence for regression and data analysis tasks Students experienced moderate gains in their content knowledge

30. Impact on Teachers of Mathematics Training Teachers  Experiential Learning Instructors needed authentic experiences before they felt comfortable guiding student efforts  Mentoring Providing quality feedback to students during their study design phase was the most critical feature in helping teams produce high quality projects Future Challenges  Discovery Learning Student freedom in choosing topics and variables requires a great deal of instructor creativity.

31. For more information Project Website  http://radar.ngcsu.edu/~djspence/nsf/ http://radar.ngcsu.edu/~djspence/nsf/ Instructional Materials Home  http://radar.ngcsu.edu/~rsinn/nsf/ http://radar.ngcsu.edu/~rsinn/nsf/ Contact Us  Robb: rsinn@ngcsu.edursinn@ngcsu.edu  Dianna: djspence@ngcsu.edudjspence@ngcsu.edu