1. Scientifically Informed Web- Based Instruction Financial and Intellectual Support: The William and Flora Hewlett Foundation Carnegie Mellon University through the Office of Technology for Education and the Eberly Center for Teaching Excellence Joel M. Smith, Vice Provost, CIO Candace Thille, Director of OLI

2. Improvement in post-secondary education will require converting teaching from a solo sport to a community-based research activity Herb Simon, Last Lecture Series, Carnegie Mellon, 1998

3. What Is a Community Based Research Activity in Teaching? Theory-based Feedback Loops Diversity of Perspective, Roles & Contexts

4. Open Learning Initiative

5. OLI Project Goals Develop exemplars of scientifically based online courses and course materials that enact instruction and support instructors Provide open access to these courses and materials Develop a community of use that contributes to the evaluation, iterative improvement, and ongoing growth of the courses and materials.

6. OLI is Currently a Small Scale Community Based Research Activity in Teaching Theory Based: Course designs based on current theories in the learning sciences Feedback Loops: Courses record student activity for robust feedback mechanisms Diversity of Perspectives, Roles and Contexts: Courses developed and deployed by teams that include faculty content experts, learning scientists, software engineers

7. Theory Based: Build on Prior/ Informal Knowledge

8. What is a Cognitive Tutor? A computerized learning environment whose design is based on cognitive principles and whose interaction with students is based on that of a (human) tutori.e., making comments when the student errs, answering questions about what to do next, and maintaining a low profile when the student is performing well.

9. Theory Based: Provide Immediate Feedback in the Problem Solving Context

10. Theory Based: Promote Coherence

11. Theory Based: Multiple Representations With Explicit Connections

12. Theory Based: Promote Authenticity, Flexibility & Applicability Learning environments with ambiguous problems that require flexible application of procedural knowledge

13. Feedback Loops in Learning

14. Evaluation Chemistry: Post-test scores by treatment group show significant positive correlation for the OLI treatment. Most significant indicator was time spent in Virtual Lab Activities – made all other variables drop out. Biology: End of the 3rd week showed an advantage for the OLI section. There was a positive and significant association between students time spent working on particular activities and performance on quiz questions testing the corresponding topics even after total time with OLI has been regressed out

15. Evaluation Statistics 1 st Study:

16. n Average % correct Pre48843.3 Post48851.2 Increase: 7.9% [t(487) = 13.8, p <.001] n Average % correct Pre2455.8 Post2466.5 Increase: 11.7% [t(23) = 4.7, p <.001] CAOS Sample: CAOS Sample: CMU OLI Course Sample: CMU OLI Course Sample: Evaluation

17. Measured learning Outcome % correct CAOS% correct CMU PrePostPrePost Box plots provide accurate estimates of % data above & below only for quartiles 22.2 50.0 Correctly estimate and compare SDs for different histograms. 31.5 46.4 66.7 83.3 41.8 46.4 59.3 75.0 Correlation does not imply causation 51.9 49.4 48.1 70.8 Calculating appropriate conditional probabilities given table of data 49.6 47.4 70.4 70.8 Evaluation

18. Student Satisfaction –End of course survey for online section: All students reported at an increase in their interest in statistics. 75% Definitely Recommend 25% Probably Recommend 0% Probably not Recommend 0% Definitely not Recommend

19. Student Quotes I have found it to be one of the fastest/most efficient ways to learn new material. Compared to reading the text, the interactivity makes it harder to skim/gloss over the material so I retain more at the end of the session.

20. Student Quotes I really like the way you can learn individually and at your own pace. If I understand something, I can move through it quickly and take more time on challenging things. "This is so much better than reading a textbook or listening to a lecture! My mind didnt wander, and I was not bored while doing the lessons. I actually learned something."

21. Feedback Loop – Current Research Instructors can use such data to adjust their teaching to students needs.

22. The Vision Instructor assigns students to work through online instruction System collects data as students work System automatically analyzes and organizes the data to present instructor with the students current learning state Instructor reviews this data summary and adapts instruction accordingly

23. The Anticipated Benefits Instructors get a window onto students progress They can adapt their teaching accordingly Students get better feedback to monitor and adjust their learning Strengthens the student-instructor connection

24. Core OLI Community Faculty Content Experts Learning Scientists Human Computer Interaction Software Engineers Evaluation/Assessment Specialists Learners A community of scholars from diverse disciplines who are committed to improving quality and access to instruction. The collaborative nature of the OLI course design process inspired participating faculty to rethink their approach to classroom teaching.

25. Extended Community: Ecology of Use and Reuse OLI Workshops in USA, Qatar, Mexico OLI Tools and support for adapting, localizing content Formal and informal evaluation Carnegie Foundation KEEP Toolkit - document scholarship of teaching

26. Feedback from Evaluation The success of the implementation of the OLI course materials, in terms of learning gains and a renewed student enthusiasm for statistics, led the Department of Psychology at Universidad de los Andes to completely change the instructional sequence in research methodology.

27. Extended Community: Ecology of Use and Reuse OLI Appliance in locations with poor internet access Collaborative Development with institutions in US, Chile, India, Qatar Developers workshop in US, Taiwan

28. The Challenge of Scale Theory Based : Learning Theories are like toothbrushes: Everyone has one and no one wants to use anyone else's. Feedback Loops: The nature of valuable feedback is contextual Ecology of Use and Reuse: Mix and mash-up of content in isolation.

29. Improvement in post-secondary education will require converting teaching from a solo sport to a community-based research activity Herbert Simon www.cmu.edu/oli cthille@cmu.edu joelms@cmu.edu