1. Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics Teaching Conference October 27, 2011 www.economistsdoitwithmodels.com

2. Motivation Personal bias Increasing fiscal constraints at both the university and K-12 levels Increased technology availability and cost effectiveness Adoption rate of online learning is very high 2.6 million students took at least one online course in fall 2005 80 percent of doctoral/research institutions in the US offer online courses Virtually every institution with over 15,000 students offers online courses Online courses also increasing prevalent at the K-12 level

3. Motivation Personal bias Increasing fiscal constraints at both the university and K-12 levels Increased technology availability and cost effectiveness Adoption rate of online learning is very high 2.6 million students took at least one online course in fall 2005 80 percent of doctoral/research institutions in the US offer online courses Virtually every institution with over 15,000 students offers online courses Online courses also increasing prevalent at the K-12 level Online instruction represents a potential opportunity for both cost savings and educational enhancement, but the effect on student understanding needs to be understood before new modes of instruction are widely adopted

4. Brown and Liedholm 2002 Data from Principles of Microeconomics at Michigan State University 3 modes of instruction: live, hybrid, virtual Same textbook, multiple-choice examinations, course web sites and email Live: 3 class hours per week Hybrid: 2 class hours per week plus online materials Required “Problems in Microeconomics” Virtual: Videos of live lectures plus hybrid materials

5. The Data (Means) LiveHybridVirtual Total Score (%)65.4964.5161.19 Subscore 172.2373.0671.84 Subscore 262.4858.7755.57 Subscore 358.0257.1350.34 Female0.520.460.42 Black0.170.050.03 Math Courses1.531.171.62 ACT22.8423.5224.46 GPA2.862.842.80 Credits51.4839.9556.42 N36325889

6. Score Regressions By Course Type LiveHybridVirtual Female-5.70-1.08-2.03 Black-4.70-5.94-4.93 Math Courses 1.650.69-3.74 ACT0.630.741.15 GPA15.9312.7215.06 Credits0.0300.044-0.052 Adjusted R 2 0.500.370.36

7. Score Predictions By Instruction Type Hypothetical Type Actual TypeLiveHybridVirtual Live65.4964.2059.72 Hybrid65.7664.5163.80 Virtual66.9866.0261.19

8. Discussion Differences in performance across modes of instruction increase as questions get higher level Differences could be at least partially attributable to differences in student effort Statistics on time spent on course work indicate that students in live sections put in more effort Gender differences largely go away in hybrid and virtual environments Could reflect differences in effort across genders Race differences also go away to some degree Selection bias is present since students are selecting into different forms of the course

9. Discussion Differences in performance across modes of instruction increase as questions get higher level Differences could be at least partially attributable to differences in student effort Statistics on time spent on course work indicate that students in live sections put in more effort Gender differences largely go away in hybrid and virtual environments Could reflect differences in effort across genders Race differences also go away to some degree Selection bias is present since students are selecting into different forms of the course Selection bias can only be eliminated via a randomized experiment…

10. Figlio, Rush and Yin 2010 Large introductory course at a major research university Randomly assigned to live or internet lectures Students were given an incentive to participate in the experiment Volunteers not markedly different from non- volunteers All other aspects of the course held constant Multiple-choice exams Live lecture condition most similar to the hybrid condition introduced earlier

11. Scores By Method of Instruction Exam 1Exam 2Final Exam Average Score Live84.53676.69275.93979.940 Online83.30176.90474.30278.502 Difference1.235-0.2121.6371.440

12. Scores By Method of Instruction Exam 1Exam 2Final Exam Average Score Live84.53676.69275.93979.940 Online83.30176.90474.30278.502 Difference1.235-0.2121.6371.440 None of the differences are statistically significant!

13. Differences Matter for Some Demographics GroupLive-Online Differential White students1.117 Black students2.828 Hispanic students11.276 Asian students4.319 Male students3.480 Female students1.780 Low-achievers4.054 High-achievers1.169

14. Differences Matter for Some Demographics GroupLive-Online Differential White students1.117 Black students2.828 Hispanic students11.276 Asian students4.319 Male students3.480 Female students1.780 Low-achievers4.054 High-achievers1.169 Could again reflect differences in effort

15. Caveats Live/online contamination Attendance data suggests that some live students were watching lectures online However, live participants had better attendance than live non-participants Potential peer effects Representativeness of study volunteers Familiarity of students with online platforms may affect the live/online differential Results may not generalize to non-economics material

16. Caveats Live/online contamination Attendance data suggests that some live students were watching lectures online However, live participants had better attendance than live non-participants Potential peer effects Representativeness of study volunteers Familiarity of students with online platforms may affect the live/online differential Results may not generalize to non-economics material Do other studies support these findings?

17. Dept. of Education Meta Analysis Literature search from 1996 through July 2008 Studies screened for several attributes Contrasted an online to a face-to-face condition Measured student learning outcomes Used a rigorous research design Provided adequate information to calculate an effect size 51 independent effects identified 7 at the K-12 level, the rest with older students Various subjects examined, the most popular being medicine or health care

18. Findings On average, students in online learning conditions performed better than those receiving face-to-face instruction 11 individual effects significantly favored an online or hybrid condition 2 individual effects significantly favored face-to-face condition Differences were largest when pure face-to-face instruction contrasted with a hybrid of online and face to face Elements such as video or online quizzes do not appear to influence the amount that students learn in online courses

19. Discussion Because online conditions usually included additional learning time and instructional elements, it’s not clear that the online medium is driving results This is supported by the observation that studies in which online learners spent more time on the course than their face-to-face counterparts saw greater benefits from online instruction

20. Key Success Factors for Technology Integration Leverage top instructors Winner-take-all market Use technology to supplement and economize, not purely to replace Enable efficient division of labor Provide incentives/commitment devices/etc. to dissuade procrastination and lack of engagement

21. For Further Reading… David N. Figlio, Mark Rush, and Lu Yin, “Is It Live or Is It Internet? Experimental Estimates of the Effects of Online Instruction on Student Learning.” National Bureau of Economic Research Working Paper 16089. Byron W. Brown and Carl E. Liedholm, “Can Web Courses Replace the Classroom in Principles of Microeconomics?” American Economic Review, May 2002 (Papers and Proceedings), 92(2), pp. 444-448. U.S. Department of Education, Office of Planning, Evaluation, an Policy Development, Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies, Washington, D.C., 2009.

22. For more information, go to http://www.economistsdoitwithmodels.com/Carlson