Presentation on theme: "How Data Mining Contributes to Efficacy Studies and Course Redesign Answering Your Questions: Why? What? How? Claire Masson Doug Paetzell Yun Jin Rho."— Presentation transcript:
How Data Mining Contributes to Efficacy Studies and Course Redesign Answering Your Questions: Why? What? How? Claire Masson Doug Paetzell Yun Jin Rho Rasil Warnakulasooriya 24 Sept 2011
Tracking and Analyzing Student Data3 Your Comments from the pre-workshop survey… 1. Why are statistics useful in commenting on success – Show me some data on redesigned programs – How do I determine student success 2. What should we do after the redesign – How do I use the data more effectively – Show me ways to select the best data from pilot efforts 3. How should data be gathered to evaluate the redesign – What is the best way to collect data – What kind of data can be collected
Tracking and Analyzing Student Data4 How to Collect DataDoug & Claire 1. Setting Expectations – Determine goals for course redesign – Effects of increased rigor 2. Pilot the Program – Small program to full implementation – Pace of redesign – Gradual improvement 3. Review some Basic Statistics – Interchangeable learning aids – Why class size matters 4. Getting Started – Worksheets, Checklists, Templates
Tracking and Analyzing Student Data5 Setting Expectations What are the specific goals of the course redesign? 1
Tracking and Analyzing Student Data6 First Step: Setting Expectations Determine a Primary Goal Set Quantifiable Expectations set a specific goal to frame the redesign. What is the problem were trying to solve? Guiding questions – What percent do we want to increase student grades? – How many students do we want increase class size without raising costs? – What qualitative effects do we want to see in our classroom as a result of increased rigor?
Tracking and Analyzing Student Data7 The Flow of Redesign Set Goal Evaluate Resources Design Course Select Measurement Tools Implement Course Prepare Data Interpret Data Adjust Course Analyze Data Increase Learning Effect Size by 0.5 Computer Lab Available? IRB? Emporium Model Compare Final Exam Scores Compare Historical / Current Exam Scores Run Thru Semester / Give Last Years Final Exam Results: Statistically Valid? Results: Educationally Valid? Apply Lessons Learned
Tracking and Analyzing Student Data8 Was this Redesign Successful? TEACHER SAYS YES: For probably the first time, all students are engaged in working on homework on a regular basis. The rigor of the homework assignment has increased, and even as weve implemented grading with no partial credit, success rates have increased in the course. -Rebecca Muller, Mathematics Instructor -Southeastern Louisiana University
Tracking and Analyzing Student Data9 Effects of Increased Rigor The quality of the class experience has changed. Students come with constructive questions…Class time is more productive. -Kathleen Almy, Associate Professor Rock Valley College MyStatLab saves me from using class time to explain and re-explain how to solve problems. Because students are more prepared to learn and more proactive in their learning, I can convey more complicated, robust concepts to them. It makes teaching the course more fun to teach. -Gwen Terwilliger, Ph.D., Professor Emeritus -University of Toledo
Tracking and Analyzing Student Data10 Pilot the Program The Gradual Process 2
Tracking and Analyzing Student Data11 The Road to Success Pilot vs. Full Rollout 1.Importance of deadlines and benchmarks with assignments Teachers who monitor student participation have higher retention rates 2. Grade inflation may skew the effects Beginning phases of redesign may require remediation for students who were passed along previously 3. A good pilot often requires three phases of implementation to achieve success Source:
Tracking and Analyzing Student Data12 The Three Phases Year 1: Develop and aggregate course material Year 2: 1 st Semester: Course development 2 nd Semester: Campus pilots 2 nd Semester: Course revision Year 3: 1 st Semester: Campus full implementation 2 nd Semester: Convert course material for full campus 2 nd Semester: Develop customized best practices plan and rework curriculum
Tracking and Analyzing Student Data13 Examples of Gradual Improvement Effect of Course Redesign on Reducing Copying Over Time Decrease in copy rate over the four courses Decrease between Year 0 and Year 1 due to studio format redesign Decrease from Year 2 to Year 3 due to assigning to ABC grades instead of pass-no pass record Traditional Year 1 Pilot Year 2 Full Course Year 3 Full Course Palazzo/Lee/Warnakulasooriya/Pritchard Patterns, Correlates, and Reduction of Homework Copying Physics Education Research 6, (2010)
Tracking and Analyzing Student Data14 Examples of Gradual Improvement Effect of Course Redesign on Improving Pass Rates Over Time Jackson State Community College conducted three pilots before full implementation Largest increased occurs after initial transition Continued improvement results from numerous adjustment
Tracking and Analyzing Student Data15 Review some Basic Statistics Null Effect / Class Size / p-value 3
Tracking and Analyzing Student Data16 Similar Learning Aids Study Design Yields Null Effect Null Effects are NOT Negative. Comparing one learning model to another with the same intervention goal, remediation, often yields same results: null / no effect.
Tracking and Analyzing Student Data17 Same Content / Different Platform Study Design Yields Null Effect Teacher assigned the same content, so there should be no expectation of improvement. Mean exam scores with standard error bars for A&P (7 exams) using CC in 2010 vs. MAP in 2010 vs. MAP in CourseCompass v Mastering for A&P
Tracking and Analyzing Student Data18 The Meaning Behind Class Size MyITLab p-value <0.05 statistically significant But educationally significant? Effect size: 0.44 p-value >0.05 not statistically significant MyMathLab 67%64% error bars overlap error bars dont overlap # of students inside bars
Tracking and Analyzing Student Data19 Combining data and confounding factors Combining data must be thoughtfully considered. 1. Is it okay to combine your own sections of one class if the same material is covered. (Consider student population v night / day classes)? 2. Is it okay to combine your student data with a colleague at a different institution (CC and 4-year research schools), administering different exams, etc.? 3. Is it okay to combine data with other instructors at your school teaching the same course?
Tracking and Analyzing Student Data20 Getting Started Worksheets, Checklists, Templates, Examples 4
Learn more… Case Studies: White Papers: (Math): Making the Grade (English): Vision in Action (Sci/Eng): Make Learning Part of the Grade Peer-Reviewed Journal Articles: (MATH): Brewer/Becker Online Homework Effectiveness for Underprepared and Repeating College Algebra Students, Journal of Computers in Mathematics and Science Teaching 29(4), (2010) (BIO): Rayner: Evaluation and Student Perception of MasteringBiology as a Learning and Formative Assessment Tool in a First Year Biology Subject ATN Assessment Conference (2008) (PHYS): Palazzo/Lee/Warnakulasooriya/Pritchard Patterns, Correlates, and Reduction of Homework Copying Physics Education Research 6, (2010)