1. Lessons Learned Findings from Ten Formative Assessments of Educational Initiatives at MIT (2000-2003) Lori Breslow, Ph.D. and The Staff of the Teaching and Learning Laboratory

2. The Context From 1999 to the present, MIT has undertaken approximately 40 experiments in educational innovation. TLL has assessed: Subjects 1.00,* 6.001*, 6.002x,* 8.02 (TEAL),* 8.224, HST582J, 18.03, Mission 200X* Programs The Undergraduate Exchange with University of Cambridge,* Residence-Based Advising* Technologies PIVoT, Cross Media Annotation System (XMAS),* MetaMedia,* PRS Space The d’Arbeloff/TEAL classroom (26-152) *Denotes multi-year or multi-semester assessment

3. Agenda Four lessons about the use of educational technology Observations on best practices in design, implementation, and assessment Where do we go from here?

4. Four Lessons about Educational Technology Educational technologies have contributed to gains in learning Educational technologies that have met a specific educational need that has been unmet or poorly met by traditional media have been more successful Too much new technology, too many technologies, technologies that are poorly integrated into the curriculum have been less successful Studying the relationship between educational technologies and the contexts in which they are embedded has yielded important understandings

5. Technology-Enabled Active Learning Increased Learning Gains Source: Dori, Y. & Belcher, J. (2004) “How Does Technology-Enabled Active Learning Affect Undergraduate Students’ Understanding of Electromagnetic Concepts?” under review at The Journal of the Learning Sciences Lesson #1 Learning gains = %Correct post-test - % Correct pre-test 100% - %Correct pre-test

6. A Problem-Based Module Increased Learning Gains Source: Greenberg, J. Smith, N. & Newman, J. (2003) “Instructional Module in Fourier Spectral Analysis, Based on Principles of How People Learn,” Journal of Engineering Education Lesson #1

7. What Has Worked Online lectures to teach students basic concepts Visualizations to help students see what cannot be seen Archives of visual images to help students strengthen different kinds of literacies Wireless laptops to learn programming Remote online laboratories Technologies that provide instantaneous, more individualized feedback Lesson #2

8. Online Lectures Motivate Students Lesson #2

9. Online Lectures Were More Effective in Intro Computer Science Lesson #2 Source: Newman, J. (2002) “6.001 Report on the Outcomes of Online Learning,” unpublished assessment report

10. Visualizations Help Students See What Cannot Be Seen An animation used in Physics II, a course in electromagnetism” (8.02T). Click to see how field lines move. For more information on the Technology-Enabled Active Learning project, go to http://web.mit.edu/8.02t/www. http://web.mit.edu/8.02t/www

11. MetaMedia Creates Visual Archives that Expand Literacy MetaMedia homepage. For more information on MetaMedia, go to http://metaphor.mit.edu/. http://metaphor.mit.edu/

12. iLab Makes Laboratory Facilities Available Remotely The Flagpole Project allows students to take readings remotely. For more information on iLab, go to http://icampus.mit.edu/projects/iLab.shtml. http://icampus.mit.edu/projects/iLab.shtml

13. Two Other Effective Applications of Educational Technology Using wireless laptops to learn programming in “Introduction to Computers and Engineering Problem Solving” (1.00) Improving feedback –To the students through hint and check buttons used to answer problems embedded in online lectures (6.001) –To the faculty through a personal response system that records student answers to conceptual questions asked during class (8.02T) Lesson #2

14. What Hasn’t Worked Technologies designed for interaction (e.g., discussion boards) have been less successful at MIT When students need to learn the technology before they can learn the material, they have two things to master Too much technology can be detrimental When technology is poorly integrated into the curriculum, it is either not used at all or not used effectively by the students Lesson #3

15. Applications That Needed Some Work Functionality of –MetaMedia –XMAS –Simulation in “Biomedical Signal Image and Processing” Discussion boards in “Exploring Black Holes” Unchat in Mission 200X Technology in 8.02T –PRS –PowerPoint Laptops when used in lectures in 1.00 Lesson #3

16. Edtech and the Learning Environment Edtech has been most successful when there are strong connections between it and –Learning objectives –Pedagogies –Assessment The same technology will have different effects in different environments Edtech exerts its impact by changing the nature of information in the system Lesson #4

17. There Are Strong Connections in TEAL Objective: to increase students’ conceptual and analytical understanding about the nature of electromagnetic fields Technology’s contribution –Visualizations –Desktop experiments –PRS Lesson #4

18. But the Same Technology May Have Different Effects... In different learning environments –PIVoT as used at MIT, RPI, and Wellesley In different situations –Laptops in 1.00 during active learning and during lecture For different students –For example, less prepared MIT students were significantly more likely than better prepared MIT students to think PIVoT helped their conceptual understanding –But at RPI, students with better high school preparation derived greater benefit than those with weaker preparation Lesson #4

19. Edtech Changes the Nature of Information in the System Information Has Five Properties FormMagnitudeVelocityDirectionAccess Source: Nystrom, C. (1973) “Towards a Science of Media Ecology: The Formulation of Integrated Conceptual Paradigms for the Study of Human Communication Systems,” unpublished doctoral dissertation Lesson #4

20. How Edtech Changes Information in the System and the Impact If form changes student literacies expand If magnitude changes students can be overwhelmed by information If accessibility and direction changes relationship between students and instructor shifts If velocity changes feedback can be more instantaneous Lesson #4

21. Best Practices in Design, Implementation, and... Design –Begin with learning objectives –Research what is already known or has been done related to the innovation Implementation –Estimate the amount of resources (time, funding, space, etc.) needed; then increase it –Assume mid-course corrections will be needed

22. ... Assessment Assessment at MIT works best when: It is formative It is collaborative, which means –Demands on faculty time must be honored –Differences in research in the “hard” and “soft” sciences must be made explicit Quantitative and qualitative methodologies are used

23. Next Steps The longer term impact of the earliest projects are being studied –TEAL –Mission 200X –6.002x The “lessons learned” will be disseminated through a variety of venues at the Institute and in the wider academic community The next set of research questions of interest to the faculty and the TLL staff will be prioritized