1. The Role of Automation in Undergraduate Computer Science Chris Wilcox Colorado State University 3/5/2015

2. Problem: Expanding Enrollments  “Bachelor’s degree production increased by a double digit percentage for the third straight year. The number of new undergraduate computing majors among U.S. computer science departments rose an astonishing 29.2 percent, 22.8 percent among those departments reporting both this year and last year. This is the fifth straight year of increased enrollment in computing majors by new students.” *1 The Role of Automation in Undergraduate Computer Science 2 1) Taulbee Survey 2013 Computer Research Association 2) CSU Enrollment Numbers for Introductory Course 2010-2014 3/5/15

3. Problem: Decreasing Funding  “Colorado has reduced its support for higher education by nearly 69.4 percent, from $10.52 in fiscal 1980 (and a peak of $13.85 in fiscal 1971) to $3.22 by fiscal 2011. At this rate of decline Colorado appropriations will reach zero in 2022, 11 years from now.” *1 The Role of Automation in Undergraduate Computer Science 3 1) American Council on Education “State Funding: A Race to the Bottom” 2) Center on Budget and Policy Priorities “A Formula for Decline…” 3/5/15

4. Solution: Increased Automation  Automated Grading Proprietary web-based framework for the automated submission and assessment of student programs.  Peer Instruction Partial inverted classroom with peer instruction quizzes automated using iClicker technology.  Online Tutorials – MyProgrammingLab, Zyante  Bulletin Boards – Blackboard, Piazza  Grading Frameworks – Blackboard, Canvas  Web Portal – Proprietary The Role of Automation in Undergraduate Computer Science 43/5/15

5. Key Questions  Question: How do automated tools negatively or positively affect student learning?  Question: Do the benefits of automation, such as resource savings, outweigh the disadvantages? The Role of Automation in Undergraduate Computer Science 5 Answer: Based on my research outcome, we can conserve resources (grade less, teach more) while maintaining or improving academic performance. 3/5/15

6. Automated Grading  250 students * (12 assignments + 12 labs + 6 quizzes) = around 7500 programs to grade during one semester!  Let’s make it very easy, maybe 6 minutes per program, so 6 * 7500 = 45000 minutes = 750 hours.  Who has the time for that? That’s more than two teaching assistants working 20 hours per week, which is all I have.  From an economic viewpoint, that’s $10-15K based on average graduate student compensation.  From a pedagogic viewpoint, shouldn’t teaching assistants and instructors maximize time spent in front of students? The Role of Automation in Undergraduate Computer Science 63/5/15

7. Software: Automated Grading The Role of Automation in Undergraduate Computer Science 73/5/15

8. Software: Test Results The Role of Automation in Undergraduate Computer Science 83/5/15

9. Student Feedback Cycle The Role of Automation in Undergraduate Computer Science 9 045678321 Workflow with Manual Grading Final Feedback Preliminary Feedback Post Date Due Date Workflow with Automated Grading 3/5/15

10. Software: Course Calendar The Role of Automation in Undergraduate Computer Science 103/5/15

11. Evaluation – Exam Scores The Role of Automation in Undergraduate Computer Science 113/5/15

12. Evaluation – Withdrawal Rates The Role of Automation in Undergraduate Computer Science 123/5/15

13. Evaluation – Program Submissions The Role of Automation in Undergraduate Computer Science 133/5/15

14. Evaluation – Online Tutorial The Role of Automation in Undergraduate Computer Science 14 SemesterCategory Sample Size Class Percentage Average Total Statistical Significance Fall 2013Everyone252 100% 78.02% Fall 2013Control121 48% 76.60% base population Fall 2013Treatment110 44% 80.09% t = 1.73 p = 0.0421 Fall 2013 Self Unassigned 8 3% 68.20% t = 2.34 p = 0.0212 3/5/15

15. Resource Savings  Each time we grade a student program with automation we save ~16 hours, and we give many assignments and programming quizzes.  Each time we grade a peer instruction quiz with automation, it saves us around ~6 hours of teaching resource, and we give one per week.  Equivalent to at least one TA each semester, or 300 hours at approximately $20 per hour, which is around $6000, not counting tuition, and we use automated grading in 5-6 courses, twice a year. The Role of Automation in Undergraduate Computer Science 153/5/15

16. Automation Caveats However, there are some negative aspects of automated program grading that we have observed:  Automated testing can be perceived as unforgiving and overly strict by students.  Clearly not as beneficial as one on one code reviews for design and coding style issues.  Can create an unhealthy dependence on preliminary testing, which limits the acquisition of testing and debugging skills by the student. The Role of Automation in Undergraduate Computer Science 163/5/15

17. Threats to Validity  Inherent in empirical studies performed on students in a university setting.  External Validity: Automation does not degrade academic performance, and can actually improve it.  Internal Validity: Are there extraneous factors that could account for observed differences? Semester to semester variation in students. Staff turnover among teaching assistants. Changes to course format and content.  However, we were able to tightly control the critical factors: curriculum, assignments, and exams. The Role of Automation in Undergraduate Computer Science 173/5/15

18. Conclusions  The automation of key processes can save a significant amount of scarce resources without negatively impacting academic performance.  The benefits of automation are both tangible, such as higher exam scores, and intangible, such as increased student engagement and interest.  We view this as a fortunate outcome, since expanding enrollments will inevitably provide incentive for even more automation. The Role of Automation in Undergraduate Computer Science 183/5/15