Presentation is loading. Please wait.

Presentation is loading. Please wait.

Johnny can’t Program (and neither can Johann) Lynda A. Thomas Department of Computer Science, University of Wales, Aberystwyth, Wales, UK.

Similar presentations


Presentation on theme: "Johnny can’t Program (and neither can Johann) Lynda A. Thomas Department of Computer Science, University of Wales, Aberystwyth, Wales, UK."— Presentation transcript:

1

2 Johnny can’t Program (and neither can Johann) Lynda A. Thomas Department of Computer Science, University of Wales, Aberystwyth, Wales, UK

3 Slide 2 of about 50 Two Multi-institutional and Multi-national Studies of Beginning Programmers (and some attempts at solutions) A multi-national, multi-institutional study of assessment of programming skills of first-year CS students Report by the ITiCSE 2001 Working Group Mike McCracken (et al.) SIGCSE Bulletin (Dec 2001) A multi-national study of reading and tracing skills in novice programmers Report by the ITiCSE 2004 Working Group Raymond Lister (et al.) SIGCSE Bulletin (to appear) Various papers by Lynda Thomas and Mark Ratcliffe

4 Slide 3 of about 50 Outline Outline the problem Report on McCracken Study Report on Lister Study Some possible remedies Discussion

5 Slide 4 of about 50 Programming is hard! du Boulay (1989) describes the sources of difficulty, and the need to deal with all of them at once: General orientation, notional machine, notation, structures, pragmatics Soloway and Spohrer (1989) note deficits in understanding of concepts, shortcomings in planning and testing, and much more Winslow (1996) notes that novices lack detailed mental models, are limited to surface knowledge and approach the problem line-by-line as opposed to in chunks

6 Slide 5 of about 50 Does OO make it worse? Rist (1996) suggests that is not so much that OO is different, it is more Wiedenbeck (1999) compared OO and Procedural novices and discovered that although for short programs there was little difference in comprehension, programs with multiple classes were not understood as well. However, in a later study it appeared that almost all difference was coming from weaker students.

7 Slide 6 of about 50 On a Personal Note 20 years of experience, US and UK Old assignments - would I give them now? (note how more comes into this (e.g.. solitaire)) There is a lot more to teach The population is different Many of my students cannot program (30%?) Is it my fault?

8 Slide 7 of about 50 McCracken Study Participants from 8 universities, 5 countries were given a choice of 3 problems that students ‘should’ be able to solve (the problems were calculators). Then gave the problem to an entire class of students at the appropriate level. Solutions were marked by several researchers.

9 Slide 8 of about 50 McCracken, et al. (2001) Most were end of first year students Remember, 8 universities, 5 countries … … it says something about our discipline. The “Picassos” “Finger painters”

10 Slide 9 of about 50 Why??? According to McCracken et al, solving a programming problem involves the following … –(1) Abstract the problem from informal description, –(2) Generate sub-problems, –(3) Produce sub-solutions, –(4) Re-compose, and –(5) Evaluate and iterate

11 Slide 10 of about 50 Why??? According to McCracken et al, solving a programming problem involves the following … –(1) Abstract the problem from informal description, –(2) Generate sub-problems, –(3) Produce sub-solutions, –(4) Re-compose, and –(5) Evaluate and iterate “Oh, my students know how to code, but they can’t design.”(or “problem solve”)

12 Slide 11 of about 50 Lister Study The nature of the McCracken study does not isolate the exact problem This study attempted to separate the steps so that we could see whether the problem was at the problem solving/design step or at the programming level Students attempted 12 multiple choice questions. –Generic, 3GL, iterative processes on arrays –Could be translated into many languages, as it happens …Java (11 universities) and C++ (1 university) –5 from US, 3 from Scandinavia, 3 from Australasia, 2 from UK

13 After the while loop finishes, “count” contains what value? a) 3 b) 2 c) 1 d) 0 (Formatting compressed to fit question into this space.) int[] x1 = {1, 2, 4, 7}; int[] x2 = {1, 2, 5, 7}; int i1 = x1.length-1; int i2 = x2.length-1; int count = 0; while ((i1 > 0 ) && (i2 > 0 )) { if ( x1[i1] == x2[i2] ) { ++count; --i1; --i2; } else if (x1[i1] < x2[i2]) --i2; else --i1; } ie. 3 Multiple Choice Question 2, of 12 (a “fixed code” question)

14 Correct, 65% Wrong, 23% After the while loop finishes, “count” contains what value? a) 3 b) 2 c) 1 d) 0 (Formatting compressed to fit question into this space.) int[] x1 = {1, 2, 4, 7}; int[] x2 = {1, 2, 5, 7}; int i1 = x1.length-1; int i2 = x2.length-1; int count = 0; while ((i1 > 0 ) && (i2 > 0 )) { if ( x1[i1] == x2[i2] ) { ++count; --i1; --i2; } else if (x1[i1] < x2[i2]) --i2; else --i1; } ie. 3 Multiple Choice Question 2, of 12 (a “fixed code” question)

15 But first, the preamble to the question … If any two numbers in an array of integers, not necessarily consecutive numbers in the array, are out of order (i.e. the number that occurs first in the array is larger than the number that occurs second), then that is called an inversion. For example, consider an array “x” that contains the following six numbers: There are 10 inversions in that array, as: Multiple Choice Question 8, of 12 (a “skeleton code” question) x[0]=4 > x[3]=2 x[0]=4 > x[4]=1 x[0]=4 > x[5]=3 x[1]=5 > x[3]=2 x[1]=5 > x[4]=1 x[1]=5 > x[5]=3 x[2]=6 > x[3]=2 x[2]=6 > x[4]=1 x[2]=6 > x[5]=3 x[3]=2 > x[4]=1

16 int inversionCount = 0; for ( int i=0 ; i x[j] ) ++inversionCount; } … the “xxxxxx” should be replaced by: a) ( int j=0 ; j

17 int inversionCount = 0; for ( int i=0 ; i x[j] ) ++inversionCount; } … the “xxxxxx” should be replaced by: a) ( int j=0 ; j

18 int inversionCount = 0; for ( int i=0 ; i x[j] ) ++inversionCount; } … the “xxxxxx” should be replaced by: a) ( int j=0 ; j

19 int inversionCount = 0; for ( int i=0 ; i x[j] ) ++inversionCount; } … the “xxxxxx” should be replaced by: a) ( int j=0 ; j

20 int inversionCount = 0; for ( int i=0 ; i x[j] ) ++inversionCount; } … the “xxxxxx” should be replaced by: a) ( int j=0 ; j

21 Slide 20 of about 50 Students across all institutions (N=556), percentage correct for each of the 12 MCQs. Q8, 3 rd hardest Q2, 6 th hardest

22 Slide 21 of about 50 Moral: a difficult question at one place is difficult elsewhere

23 Slide 22 of about 50 QuartileScore Top10-12 Second 8-9 Third 5-7 Bottom universities!!! 7 countries!!! >500 students (If you failed more than 25% of class) Quartiles on the 12 Multiple Choice Questions The third quartile determines the “bottom passing” student.

24 Slide 23 of about 50 QuartileScores Top10-12 Second8-9 Third5-7 Bottom0-4 McCracken et al., 2001 Lister et al., 2004

25 Slide 24 of about 50 The Story So Far Performance Data … and now …

26 Slide 25 of about 50 The Story So Far Performance Data … and now … Doodles –Did students draw pictures and what sort “Doodles”

27 Slide 26 of about 50 “Doodles”

28 Slide 27 of about 50 Doodle Performance Percentage of correct answers when students (N=56) use a particular doodle type on any question (12). Doodle Category%CorrectData Points … …… Synchronized Trace (S) 77%73 Trace (T) 75%215 … …… Blank Page (B) 50%256

29 Slide 28 of about 50 Blank Doodles 6 th hardest, 65% correct 3 rd hardest, ~50% correct

30 Slide 29 of about 50 Doodle Discussion Students appear to lack good doodling strategies Should we teach doodling (see later) –Assess doodling?

31 Slide 30 of about 50 The Story So Far Performance Data Doodles … and now …

32 Slide 31 of about 50 The Story So Far Performance Data Doodles … and now … Interviews, “Think Out Loud” –Transcripts from 38 students

33 Slide 32 of about 50 A transcript from Question 2 This time I'm going to look to see what the question wants first. So trying to find the value of count. So now I'm looking to see what each variable is set to and what's happening inside the program. I'm going to go ahead and write down that x1's length is 4 and x2's length is 4. And then e loop, 4 is greater than 0 and 4 is greater than 0. So inside the loop, the, checking to see if the 5 th, checking, noticing that I didn't subtract 1 from each of i1 and i2. So changing i1 to 3 and i2 to 3. So when the 3 rd spot, last spot of each array, they're equal, so it enters the first if loop so count is now equal to 1; i1 equals 2 and i2 equals 2. Now we're going through the loop again. 2 is greater than 0 and 2 is greater than 0 so we're inside the loop. Checking the first "if" are the second indexes equal to each other? no. So going to the else if and ‘is 4 less than 5?’, which is true so i2 is subtracted 1. I2 now equals 1 and we're going back to the while loop again. 2 is greater than 0 and 1 is greater than 0. So checking the first if ‘is 4 equal to 2?’, which is false so checking the second if. “Is 4 less than 2?’, which is false so doing the else statement which is subtract 1 from i1. so i1 now equals 1. So going through the while loop again. 1 is greater than 0 and 1 is greater than 0. So checking the first if "is 2 equal to 2?" which is true so increment count by 1. Count is now 2. Subtract 1 from i1 which becomes 0 and i2 which becomes 0. So now the while loop fails because 0 is not greater than 0. So it asks for the value of count which is 2.

34 Slide 33 of about 50 Possible Remedies? Other languages? Force students to produce doodles? Support for collaborative work? Technology? –Integrated Development Environments (IDEs) For example, BlueJ (Kolling etc.), TogetherJ (Borland), …..

35 Slide 34 of about 50 In particular we wanted: support for scaffolding of key concepts support for collaborative learning, and a way of collecting information on how our students actually approach the design and coding process So, we decided to build our own

36 Slide 35 of about 50 Four experiments at Aberystwyth and how they have affected the design of our IDE – Vortex 1.Learning Styles 2.Scaffolding with Visual Materials 3.Pair Programming 4.Collaborative Design

37 Slide 36 of about 50

38 Slide 37 of about Learning Styles Various approaches: left/right brain, Meyers Briggs, Kolb Learning styles, … Felder-Silverman model of preferred learning style. This identifies what is easy for the student. Felder believes that students need to improve other styles of learning too.

39 Slide 38 of about 50 Active: try things out, work with others Reflective: think things through, work alone Sensing: concrete, practical, facts, procedures Intuitive: conceptual, innovative, theories and meanings Visual: pictures, diagrams, flow-charts Verbal: written or spoken explanations Sequential: incremental, orderly steps Global: holistic, learn in large leaps Felder believes that all undergrad education should be inductive (specific to general) not deductive

40 Slide 39 of about 50 Active: try things out, work with others Reflective: think things through, work alone Sensing: concrete, practical, facts, procedures Intuitive: conceptual, innovative, theories and meanings Visual: pictures, diagrams, flow-charts Verbal: written or spoken explanations Sequential: incremental, orderly steps Global: holistic, learn in large leaps

41 Slide 40 of about 50 Experimented and concluded… Some (small) evidence that Active and Visual (biggest groups) learners needed more support Coursework mark Exam mark n Overall63%56 %107 Active61 %52 %58 Visual61 %54 %82

42 Slide 41 of about 50 So, what did that mean for Vortex? First, all first year students are encouraged to reflect on their learning style and pointed to types of materials In the design of Vortex, we wanted to: –Add visual material, and –Encourage active learning

43 Slide 42 of about Scaffolding with Visual Materials person1 Fred Aberystwyth person2 Bill Borth Person person1 = new Person(“Fred”, “Aberystwyth”); Person person2 = new Person(“Bill”, “Borth”); person2 = person1;

44 Slide 43 of about 50 person1 Fred Aberystwyth person2 Bill Borth Person person1 = new Person(“Fred”, “Aberystwyth”); Person person2 = new Person(“Bill”, “Borth”); person2 = person1; person1.setAddress(“Llan”); What is person2’s address?

45 Slide 44 of about 50 What we wanted To encourage our students to draw Object Diagrams How? By providing them with partially completed diagrams as scaffolding. Ultimately add this facility to Vortex?

46 Slide 45 of about 50 Research Questions Is drawing some kind of Object diagram correlated with success in solving multiple-choice tracing questions? Does providing students with scaffolding in the form of partially completed Object diagrams help them correctly answer multiple-choice tracing questions? Do students who have been provided with this scaffolding continue to use it in such multiple- choice questions?

47 Slide 46 of about 50 Is drawing some kind of Object diagram correlated with success in solving multiple- choice tracing questions? Group nFollow-up Test Avg - Just on tracing questions Beginners who do not use diagrams 2347% Beginners who do use diagrams 4568% The answer is a (very) qualified yes (compare Lister)

48 Slide 47 of about 50 Does providing students with scaffolding in the form of partially completed Object diagrams help them correctly answer multiple- choice tracing questions? We thought that the answer to this would be an obvious ‘yes’ but it appears to be ‘not really’ 36% for people given diagrams 28% for those not given diagrams

49 Slide 48 of about 50 Do students who have been provided with this kind of scaffolding continue to use it? Not very surprisingly in light of last question, ‘no’

50 Slide 49 of about 50 Why? In Hegarty and Narayanan model, the viewer: decomposes the system into simpler components, constructs a static model by making representational connections to prior knowledge andother components, integrates information between different representations (e.g. text and diagrams), hypothesizes lines of action, and finally constructs a dynamic mental model by mental animation.

51 Slide 50 of about 50 Why? In Hegarty and Narayanan model, the viewer: decomposes the system into simpler components, constructs a static model by making representational connections to prior knowledge andother components, integrates information between different representations (e.g. text and diagrams), hypothesizes lines of action, and finally constructs a dynamic mental model by mental animation. We did They did?

52 Slide 51 of about 50 We short circuited the process? Or, When you are really lost the diagram looks like just one more thing to understand!

53 Slide 52 of about 50 So, what did that mean for Vortex? We are trying harder to give students more practice in drawing object diagrams themselves. But in the design of Vortex, we are holding off on adding this facility.

54 Slide 53 of about Pair Programming “a programming technique where two people program with one keyboard, one mouse and one monitor” [Beck] Part of the XP set of techniques Good results! (Williams and Kessler) –Programs passed 15% more of test cases –95% of students agree “I was more confident in our assignments because we pair programmed.” –84% of the students report enjoying the experience of programming more when working in a pair

55 Slide 54 of about 50 Would it work with our students? 18 year old males straight out of school. We do a lot of team-building exercises in the first year and feel that many of our students need them! It often seems to be the students who have better social skills who are the ones who lack confidence in their programming ability.

56 Slide 55 of about 50 The Two Extremes Code Warriors I have had no trouble at all completing programming tasks to date, in fact they weren't challenging enough. I love to program and anticipate no difficulty with this course. Code-a-phobes I don't like programming and I don't think I am any good at it. I can write simple programs but have trouble writing new programs for solving new problems.

57 Slide 56 of about 50

58 Slide 57 of about 50 Results Ability vs. Confidence Performance in the course was not that different Average for whole class: 59.7% Average for Warriors:62.8% Average for Phobes55.7% None statistically significant

59 Slide 58 of about 50 Strategy for Experiment Idea introduced in lecture Students then given brief paper to read Then experiment in regular 2 hour closed lab situation Done twice –With ‘opposite’ partner (at extremes) –Then again with ‘similar’ partner

60 Slide 59 of about 50 Results of ‘opposite’ experience Overall, students enjoyed the experience (66%) and thought that it helped them produce a better solution (66%). But only 53% of the warriors reported enjoyment of the experience and only 47% of them thought that pair programming led to a better solution (statistically significant).

61 Slide 60 of about 50 Achievement totally opposite pairs – somewhat opposite pairs – similar pairs –

62 Slide 61 of about 50 Results of ‘similar’ experience Overall: 64% enjoyed experience and 65% thought it led to a better solution Warriors: 58% of them enjoyed the experience and 67% thought it led to a better product Overall: 44% of the students reported liking the second experience more than the first (and 38% just as much). Phobes: 60% enjoyed this experience more, and none less, than last time.

63 Slide 62 of about 50 Relevant Conclusions Overall, first-year students like pair programming and believe that it helps them achieve good solutions. The students whom we have identified as ‘warriors’ like pair programming the least. Students with less self-confidence seem to enjoy pair-programming the most. Students produce their best work when paired with students of somewhat, but not very, different levels of confidence. (some evidence)

64 Slide 63 of about 50 So, what did that mean for Vortex? We confirmed that many of our students liked and seemed to learn more when they could work in groups. The theory that collaborative work is best done in mixed ability groups seems to have some validity for our students. Design goal of basing VorteX around support for collaborative working was valid. Vortex should encourage the pair programming idea that pairs ‘talk’ about what they are doing and why. (This was the most frequently mentioned advantage.)

65 Slide 64 of about 50 VorteX tool has indeed been constructed To provide a fully interactive, collaborative development environment that enables the capture of novice design knowledge and gives structured feedback to educators. Student explanation for decisions give three benefits: they generate in-line documentation; they force the users to think about each action they are performing (see PP experiment); and they provide the instructor with valuable data about student misconceptions.

66 Slide 65 of about 50 Collaborative Design Considerable evidence that collaborative work is helpful, but not without its difficulties. Slavin suggested three qualities for successful collaborative working projects: 1.team reward, 2.individual accountability, and 3.equal opportunities for success

67 Slide 66 of about 50

68 Slide 67 of about Experimenting with Vortex This experiment is different because we now have the tool. Groups in lab used Vortex for a design session Randomly assigned groups

69 Slide 68 of about 50 Feedback from using Vortex “I found the experience both enjoyable and insightful.” “It made it easy, I could make stupid mistakes anonymously.” “If a decision is reached about two different ways of doing something and the group is split on the matter, it is more difficult to resolve, but being anonymous makes it easier to say what you need to.”

70 Slide 69 of about 50 Observations Every group (except the one that met face to face!) rated the exercise as 'very positive'. Many groups never met face to face even at the end of the lab. Students with disabilities were not disadvantaged (as often happens). Much more peer support and less need for tutors.

71 Slide 70 of about 50 So, what did that mean for Vortex? We have confirmed that working collaboratively through Vortex is useful. We have addressed Slavin’s points. We have collected a huge amount of data to analyse. The enthusiasm for anonymity was unexpected and interesting and needs to be investigated further.

72 Slide 71 of about 50 Three original design goals support for scaffolding of key concepts support for collaborative learning, and a way of collecting information on how our students actually approach the design and coding process We have accomplished last two and provided an active learning environment with some visual support

73 Slide 72 of about 50 Reflection and Conclusions Process has perhaps been more important than what we have done. It is easy to decide what is right for students based on what is right for you. A different mindset has been established in staff: Understanding students is the key to success!


Download ppt "Johnny can’t Program (and neither can Johann) Lynda A. Thomas Department of Computer Science, University of Wales, Aberystwyth, Wales, UK."

Similar presentations


Ads by Google