Scientifically Informed Web- Based Instruction Financial and Intellectual Support: The William and Flora Hewlett Foundation Carnegie Mellon University through the Office of Technology for Education and the Eberly Center for Teaching Excellence Joel M. Smith, Vice Provost, CIO Candace Thille, Director of OLI

Improvement in post-secondary education will require converting teaching from a solo sport to a community-based research activity Herb Simon, Last Lecture Series, Carnegie Mellon, 1998

What Is a Community Based Research Activity in Teaching? Theory-based Feedback Loops Diversity of Perspective, Roles & Contexts

Open Learning Initiative

OLI Project Goals Develop exemplars of scientifically based online courses and course materials that enact instruction and support instructors Provide open access to these courses and materials Develop a community of use that contributes to the evaluation, iterative improvement, and ongoing growth of the courses and materials.

OLI is Currently a Small Scale Community Based Research Activity in Teaching Theory Based: Course designs based on current theories in the learning sciences Feedback Loops: Courses record student activity for robust feedback mechanisms Diversity of Perspectives, Roles and Contexts: Courses developed and deployed by teams that include faculty content experts, learning scientists, software engineers

Theory Based: Build on Prior/ Informal Knowledge

What is a Cognitive Tutor? A computerized learning environment whose design is based on cognitive principles and whose interaction with students is based on that of a (human) tutori.e., making comments when the student errs, answering questions about what to do next, and maintaining a low profile when the student is performing well.

Theory Based: Provide Immediate Feedback in the Problem Solving Context

Theory Based: Promote Coherence

Theory Based: Multiple Representations With Explicit Connections

Theory Based: Promote Authenticity, Flexibility & Applicability Learning environments with ambiguous problems that require flexible application of procedural knowledge

Feedback Loops in Learning

Evaluation Chemistry: Post-test scores by treatment group show significant positive correlation for the OLI treatment. Most significant indicator was time spent in Virtual Lab Activities – made all other variables drop out. Biology: End of the 3rd week showed an advantage for the OLI section. There was a positive and significant association between students time spent working on particular activities and performance on quiz questions testing the corresponding topics even after total time with OLI has been regressed out

Evaluation Statistics 1 st Study:

n Average % correct Pre Post Increase: 7.9% [t(487) = 13.8, p <.001] n Average % correct Pre Post Increase: 11.7% [t(23) = 4.7, p <.001] CAOS Sample: CAOS Sample: CMU OLI Course Sample: CMU OLI Course Sample: Evaluation

Measured learning Outcome % correct CAOS% correct CMU PrePostPrePost Box plots provide accurate estimates of % data above & below only for quartiles Correctly estimate and compare SDs for different histograms Correlation does not imply causation Calculating appropriate conditional probabilities given table of data Evaluation

Student Satisfaction –End of course survey for online section: All students reported at an increase in their interest in statistics. 75% Definitely Recommend 25% Probably Recommend 0% Probably not Recommend 0% Definitely not Recommend

Student Quotes I have found it to be one of the fastest/most efficient ways to learn new material. Compared to reading the text, the interactivity makes it harder to skim/gloss over the material so I retain more at the end of the session.

Student Quotes I really like the way you can learn individually and at your own pace. If I understand something, I can move through it quickly and take more time on challenging things. "This is so much better than reading a textbook or listening to a lecture! My mind didnt wander, and I was not bored while doing the lessons. I actually learned something."

Feedback Loop – Current Research Instructors can use such data to adjust their teaching to students needs.

The Vision Instructor assigns students to work through online instruction System collects data as students work System automatically analyzes and organizes the data to present instructor with the students current learning state Instructor reviews this data summary and adapts instruction accordingly

The Anticipated Benefits Instructors get a window onto students progress They can adapt their teaching accordingly Students get better feedback to monitor and adjust their learning Strengthens the student-instructor connection

Core OLI Community Faculty Content Experts Learning Scientists Human Computer Interaction Software Engineers Evaluation/Assessment Specialists Learners A community of scholars from diverse disciplines who are committed to improving quality and access to instruction. The collaborative nature of the OLI course design process inspired participating faculty to rethink their approach to classroom teaching.

Extended Community: Ecology of Use and Reuse OLI Workshops in USA, Qatar, Mexico OLI Tools and support for adapting, localizing content Formal and informal evaluation Carnegie Foundation KEEP Toolkit - document scholarship of teaching

Feedback from Evaluation The success of the implementation of the OLI course materials, in terms of learning gains and a renewed student enthusiasm for statistics, led the Department of Psychology at Universidad de los Andes to completely change the instructional sequence in research methodology.

Extended Community: Ecology of Use and Reuse OLI Appliance in locations with poor internet access Collaborative Development with institutions in US, Chile, India, Qatar Developers workshop in US, Taiwan

The Challenge of Scale Theory Based : Learning Theories are like toothbrushes: Everyone has one and no one wants to use anyone else's. Feedback Loops: The nature of valuable feedback is contextual Ecology of Use and Reuse: Mix and mash-up of content in isolation.

Improvement in post-secondary education will require converting teaching from a solo sport to a community-based research activity Herbert Simon