1. ICALT 2001 How students learn to program: Observations of practical tasks completed Dr. Pete Thomas & Carina Paine Centre for Informatics Education Research Computing Department

2. ICALT 2001 Presentation Details Student Environment AESOP Software Tasks Completed Tool Research Questions Results Conclusions Current Work

3. ICALT 2001 Student Environment The Open University –A distance education institution –Adults learning at home M206 –Computing: An Object-oriented Approach –5000 students per year –LearningWorks Smalltalk –Computer based practical work using LearningBooks

4. ICALT 2001 LearningBook Architecture LearningBook Sessions ExercisesDiscussions Tasks

5. ICALT 2001 LearningBook - Workspace

6. ICALT 2001 AESOP Software Recorder –Records a student’s use of a LearningBook (LB) –Automatically & invisibly launched –‘Significant’ student events recorded with a time & date stamp Replayer –Recreates students’ actions on screen Analyser –A search engine for examining recordings

7. ICALT 2001 Tasks Completed Tool Students are set a number of ‘tasks’ in a LearningBook (LB) This tool compares the ‘tasks’ students complete with tasks set in a LB Student recordings are compared to an ‘ideal recording’

8. ICALT 2001 Some Research Questions Earlier ad-hoc investigations showed students did not complete all tasks set. Does the number of tasks set in a LearningBook affect the amount of tasks a student completes? Is the time a student spends studying a LearningBook related to the number of tasks set?

9. ICALT 2001 0 10 20 30 40 50 60 70 80 0306090 No. of tasks set No. of tasks completed Results: tasks completed in LBs Students complete about 80% of tasks set r s = 0.9074, significant correlation (95% confidence)

10. ICALT 2001 50 60 70 80 90 100 050100150200 Time taken (mins) % of tasks completed Results: time spent in LB The more time spent, the smaller the proportion of tasks set are completed, r s = -0.9048, significant correlation (95% confidence)

11. ICALT 2001 Results - Task Difficulty Values of 3 metrics for task difficulty A: average time taken per task set B: average time taken per task completed C: average recommended time per task set

12. ICALT 2001 Results - Task Difficulty 0 1 2 3 4 5 6 910121314151620 LearningBook Task difficulty 0 10 20 30 40 % Tasks not completed Task difficulty % Tasks not completed The more difficult the task, the higher the % of tasks not completed r s = 0.762, significant correlation (95% confidence)

13. ICALT 2001 Conclusions Students seldom complete all practical tasks The more time students spend on practical work the more likely they are to stop before completing all the work set Students perception of task difficulty is different to that of the course designers A sitting lasts for 30 - 40 mins

14. ICALT 2001 Current Work Perform similar analyses on a more substantial & representative collection of recordings –2000 data involves 200 students Investigate how students tackle their assignments Investigate any relationship between –amount of practical work completed & –resulting assignments and exam scores Investigate tasks attempted by students Improve the tool’s accuracy