Boris Milašinović Faculty of Electrical Engineering and Computing University of Zagreb, Croatia

Motivation Courses Programming (and software engineering) Data Structures and Algorithms >= 700 students on each course Old examination system 2 mid term exams Classic written exam as an option => no continuous assessment! How to introduce continuous assessments? 2

Continuous assessment Mid term exams and final exam Periodical small tests Homeworks In class assessment Problems: Manual reviewing process is usually more precise but time consuming Lack of teaching staff Solution: Using software for automatic evaluation of assignments Complete automatic reviewing would be to strict  Mid term and final exams reviewed manually  Homeworks and quizzes (multiple choice tests) with automatic evaluation 3

Grading scheme (1 point = 1% of final points) Programming and software engineering 1st mid-term exam: 15 (points) 2nd mid-term exam: 25 Final exam : 30 Homeworks: 3 x 2 = 6 Quizzes: 6 x 3 = 18 Class activity: 6 Algorithms and Data Structures 1st mid-term exam: 15 2nd mid-term exam: 20 Final exam : 30 Homeworks: 3 x 3 = 9 Quizzes: 6 x 3 = 18 Class activity: 8 To pass a student must collect at least 50 points and at least 8 (of 30) points on the final exam Grades are awarded by Gaussian distribution 4

Multiple-choice tests (Quizzes) 30 minutes, 12 questions, 5 possible answers (only 1 is correct) Programming and software engineering 6 quizzes (3 points per quiz) Correct answer: 0.25 points Incorrect: Algorithms and Data Structures: 3 quizzes (6 points per quiz) Correct answer: 0.5 points Incorrect: point is equal to 1% of final score In order to retain the initial quality new questions had to be added every year to enlarge the questions database. Time span between the assessments should be reduced (all students should do the test in the same day) 5

Multiple-choice tests (Quizzes) – Histograms of results in 2007/08 Programming and software engineering Algorithms and Data Structures 6

Suitability of Multiple-choice tests (Quizzes) Advantages: Simple to create and easy to run Quick (quiz lasts 30 minutes) Has good results distribution Helps achieving continuous assessment Disadvantages: Backwash effect: Students tend to study the matter needed to pass the test, not the matter representing the core knowledge being thought Does not represent real knowledge of programming! Students learn how to recognize answers. 7

Homeworks Students are given either whole programs or individual functions as programming assignments On upload, students' code is joined with the code previously defined by the teachers and compiled. Upon the successful compilation, program is run against the predefined tests and its output is compared with the expected results. Tests with fixed set data Randomly generated inputs Time to collect the assignment: 7-10 days Time to solve: 2 days from the time of assignment collection 8

Avoiding ambiguities in homeworks test definitions Switching to automatic evaluation can bring problems. The lack of experience (on behalf of students and teachers) Avoiding ambiguities in test definitions Exams for automatic evaluation must be precisely defined First test with automatic evaluation: 300 received s with complaints on automatic evaluation process (more than 40%) Now: less than 40 complaints per test (5%) => mostly with request for explanation of student’s solution errors 9

Homeworks - Histograms 10

Suitability of automatically evaluated homeworks Advantages Improve students programming skills More time to create assignments, but no manual review evaluation later Disadvantages Minor error can lead to zero points for the assignment More tests refines the grading scale Running pre-tests examples before the final submission Poor distribution (although not unusual for homeworks) Easy to cheat Suspicious situations: similar solutions too short time between assignment collection and submission 11

Correlation between results on each assignment and student’s final grade – Algorithms and Data Structures (2007/08) 1st mid- term exam 2nd mid- term exam Final exam Quiz 1Quiz 2Quiz 3Homework 1 Homework 2 Homework 3 Classroom activity TotalGrade 1st mid- term exam 0,670,640,530,510,370,140,240,310,450,800,75 2nd mid- term exam 0,67 0,680,470,550,370,120,220,310,480,840,80 Final exam 0,640,68 0,520,580,520,180,250,340,530,900,86 Quiz 10,530,470,52 0,530,410,200,270,390,420,660,56 Quiz 20,510,550,580,53 0,490,190,280,400,470,720,62 Quiz 30,37 0,520,410,49 0,210,290,440,410,610,51 Homework 1 0,140,120,180,200,190,21 0,230,260,110,260,21 Homework 2 0,240,220,250,270,280,290,23 0,400,260,370,27 Homework 3 0,31 0,340,390,400,440,260,40 0,280,480,31 Classroom activity 0,450,480,530,420,470,410,110,260,28 0,660,62 Total0,800,840,900,660,720,610,260,370,480,66 0,92 Grade0,750,800,860,560,620,510,210,270,310,620,92 12

E1E1 E2E2 FEFE Q1Q1 Q2Q2 Q3Q3 Q4Q4 Q5Q5 Q6Q6 H1H1 H2H2 H3H3 CACA TGRGR 1st mid-term exam 1,000,650,590,430,400,550,440,460,400,170,120,200,280,760,69 2nd mid-term exam 0,651,000,690,48 0,610,490,560,500,160,120,200,400,870,85 Final exam 0,590,691,000,520,550,640,590,680,640,180,170,330,480,910,87 Quiz 1 0,430,480,521,000,550,560,500,480,510,240,200,290,410,620,55 Quiz 2 0,400,480,55 1,000,590,570,55 0,210,220,340,500,640,57 Quiz 3 0,550,610,640,560,591,000,570,620,590,220,210,320,420,740,68 Quiz 4 0,440,490,590,500,57 1,000,700,650,29 0,430,510,680,56 Quiz 5 0,460,560,680,480,550,620,701,000,700,24 0,410,530,740,66 Quiz 6 0,400,500,640,510,550,590,650,701,000,230,250,420,510,700,59 Homework 1 0,170,160,180,240,210,220,290,240,231,000,240,25 0,270,20 Homework 2 0,12 0,170,200,220,210,290,240,250,241,000,220,24 0,16 Homework 3 0,20 0,330,290,340,320,430,410,420,250,221,000,320,380,26 Class activity 0,280,400,480,410,500,420,510,530,510,250,240,321,000,570,50 Total number of points 0,760,870,910,620,640,740,680,740,700,270,240,380,571,000,93 Grade 0,690,850,870,550,570,680,560,660,590,200,160,260,500,931,00 Programming and software engineering (2007/08) 13

Prediction of grades after 1/3 of the semester Classification Matrix (Algorithms and Data structures-2007/08) Rows: Grades Columns: Predicted grades Percent ,73% % ,55% ,78% ,80% Total 52,67% homework 1 quiz 1 mid-term exam 1 homework 2 quizzes 1 mid-term exam Classification Matrix (Programming and software enginereeing-2007/08) Rows: Grades Columns: Predicted grades Percent ,76% % % ,13% ,49% Total 55,53%

Prediction of grades after 2/3 of the semester Classification Matrix (Algorithms and Data structures-2007/08) Rows: Grades Columns: Predicted grades Percent ,32% ,39% ,53% ,31% ,24% Total 65,02% homeworks 2 quizzes 2 mid-term exam 2 homeworks 4 quizzes 2 mid-term exam Classification Matrix (Programming and software enginereeing-2007/08) Rows: Grades Columns: Predicted grades Percent ,27% ,95% ,5% ,62% ,97% Total 71,59%

Conclusion Why is prediction not more accurate? Extremely poor prediction for grade 2! Reason: Significant number of students learn just to pass Gaussian distribution of grades 15%-35%-35%-15% Nevertheless: Very good prediction about the number of students that will pass the exam What about various assessments methods Requires more work but covers more aspects of assessment Eliminates backwash effect Continuous assessment 16