1. Computing Education Research Anders Berglund Informationsteknologi Uppsala universitet Uppsala anders.berglund@it.uu.se

2. Informationsteknologi Anders Berglund, Department of Information Technology Computing education research at Uppsala University How do our students understand computer science concepts? How to teach computer science?

3. Informationsteknologi Anders Berglund, Department of Information Technology Computing Education Research This talk What can we say about how grading in an internationally distributed project-based course? and Why and how to do research in computing education? Two parallel stories My secret agenda: Inspire to Computing Education Research

4. Informationsteknologi Anders Berglund, Department of Information Technology 1. Background A research approach? Phenomenography 2. Results concerning grading The setting The teachers’ grades The students’ peer evaluation The experience of being graded Results concerning grading 3. Summary Computing Education Research Some research results Choosing a research approach Computing Education Research

5. Informationsteknologi Anders Berglund, Department of Information Technology What does it mean to learn something? Unfortunately (?)…. General case: A “meaningless” question It all depends on “what you mean by learning” or “how you see things”

6. Informationsteknologi Anders Berglund, Department of Information Technology Quantitative/Qualitative research Quantitative research is grounded on “… the assumption that features of the social environment constitute an objective reality … collecting numerical data on observable variables” Qualitative research is grounded on “… the assumption that individuals construct a social reality in the form of meanings and interpretations. … studying … intensively in natural settings” Implications for the role of the researcher, the concept of evidence, interpretation etc. (Gall, Borg & Gall, 1996)

7. Informationsteknologi Anders Berglund, Department of Information Technology A research approach/methodology/ framework Offers a way to perform research in learning Organizes “ways to see things” A lens with a certain focus With a specific research approach: Some issues get clearer, others blurred Offers theoretical stand on learning, ways to see possibilities and limitations, opens to communicate with other researchers etc.

8. Informationsteknologi Anders Berglund, Department of Information Technology Research approach/methodology/framework Phenomenography (Marton & Booth, 1997) Takes the learners’ perspective. Aims at analysing and describing the variation in students’ experience (understanding, learning). A empirical, qualitative research approach Data is often collected through interviews Outcome: A few qualitatively different ways, in which something is understood within a student cohort Examples: TCP, experience of being graded

9. Informationsteknologi Anders Berglund, Department of Information Technology Phenomenography Researcher The researcher studies the different ways in which the students understand TCP The students study TCP

10. Informationsteknologi Anders Berglund, Department of Information Technology Why the learners’ perspective? A teacher’s understanding of how her students understand and learn about something (CS concepts, for example TCP) is a good tool for improving teaching. A change that is not perceived as “good” by the students does not improve learning.  Example: Grades are not the driving force for most students who take (a certain) project course.

11. Informationsteknologi Anders Berglund, Department of Information Technology 1. Background A research approach? Phenomenography 2. Results concerning grading The setting The teachers’ grades The students’ peer evaluation The experience of being graded Results concerning grading 3. Summary Computing Education Research Some research results Choosing a research approach Computing Education Research

12. Informationsteknologi Anders Berglund, Department of Information Technology Background: The Runestone initiative A project course in computer systems A project-based course in distributed systems, real-time programming and computer communication. Third/fourth year students majoring in CS. International collaboration for students who do not go on exchanges. Experience of collaboration over ICT tools.

13. Informationsteknologi Anders Berglund, Department of Information Technology Project course in computer systems The Runestone project USASweden 3 + 3 advanced CS students per team 16 teams in total No lectures Tutoring by e-mail and chat Communication by e-mail and chat

14. Informationsteknologi Anders Berglund, Department of Information Technology Student project Student project: Produce a software system to control a (modified) Brio labyrinth from any Web-browser. The task demands computer communication solutions. The task requires collaboration within the team of 6.

15. Informationsteknologi Anders Berglund, Department of Information Technology Both the process and the project are graded, in relation to the teams’ own plan Process grade is based on weekly meetings Components of grade: Team performance, in relation to the team’s own plan Individual contribution Peer evaluation The instructor’s decision. Team members are graded by “their” instructor Different grading schemes in Sweden and US Sweden: pass/fail US: A to E Grading in Runestone Problematic???

16. Informationsteknologi Anders Berglund, Department of Information Technology Analysing the grading in Runestone Teachers’ distribution of grades ( quantitative) Peer evaluation The students’ evaluation of each others’ contributions ( quantitative) Students’ experienced purpose of being graded ( qualitative)

17. Informationsteknologi Anders Berglund, Department of Information Technology Grades Grades awarded by the instructors, according to the Runestone scheme (Max = 100, Pass ≈ 60) To all students83,61 To Americans81,55 To Swedes85,05

18. Informationsteknologi Anders Berglund, Department of Information Technology Peer evaluation From Swede to Swede22,25 From Swede to American18,79 From American to American20,07 From American to Swede20,07 Each student awarded USD 120.- to his team-mates Then, what is the driving force?

19. Informationsteknologi Anders Berglund, Department of Information Technology The experienced purpose of being graded Cat Getting a good grade … 1. 2. 3. … has a value on its own … is a tool to reach other aims … is sub-ordinated to other aims Focus is on The grade per se The benefits of a good grade Me and the team My team and other teams

20. Informationsteknologi Anders Berglund, Department of Information Technology Results on grading Getting a good grade is not the driving force for most students in this project. ”Me in the team” or ”My team in front of other teams” is often important. How generalizable are these results? Research in computing education research are normally situationally bound How can we use this in our teaching? Both ”hard” results and insights gained by doing the research are useful.

21. Informationsteknologi Anders Berglund, Department of Information Technology 1. Background A research approach? Phenomenography 2. Results concerning grading The setting The teachers’ grades The students’ peer evaluation The experience of being graded Results concerning grading 3. Summary Computing Education Research Some research results Choosing a research approach Computing Education Research

22. Informationsteknologi Anders Berglund, Department of Information Technology On Computing Education Research Theoretically sound research in students’ learning in Computing can serve to improve teaching. Different research approaches offer various contributions to our understanding of students’ learning. The perspective on ”reality”, what can be studied, what can be known, what the researcher’s role is, how research is performed etc. varies. This talk: Some examples of qualitative research

23. Informationsteknologi Anders Berglund, Department of Information Technology Example, research approach Constructivism A family of traditions Jean Piaget, 1896 - 1980 Reality is rejected or irrelevant Knowledge is constructed by each individual No firm methodology Passive learning will fail Extremely influential in school teaching

24. Informationsteknologi Anders Berglund, Department of Information Technology Empirical results from constructivism in CS Education Students construct rules for parameters. They are only sometimes successful. (Fleury, 1991) Students construct their own understanding of variables. (Paz, 1996; and others) Software visualization in itself does not help students understanding (Mulholland, 1997)

25. Informationsteknologi Anders Berglund, Department of Information Technology Applications of constructivism in Computer Science Education Think twice when using visualizations Explicitly teach the model of the computer Don’t start with abstractions Teach planning, teach to avoid “bricolage” Don’t run to the computer Organize “closed labs” (from Ben-Ari, 2001)

26. Informationsteknologi Anders Berglund, Department of Information Technology Example, research approach: A socio-cultural perspective Thinking/Learning is not influenced by the environment ⇔ Thinking/Learning is an interaction between the individual and the environment A family of traditions in research into learning Common “source of inspiration”: Lev Vygotsky (1896 – 1934)

27. Informationsteknologi Anders Berglund, Department of Information Technology Applications of the soci-cultural tradition Why do teams of students interpret a programming task so differently? (Holland & Reeves, 1996) Why do our students hand in “incorrect” programs? (Ben-David Kolikant, 2005) The example: Open source community - Linux

28. Informationsteknologi Anders Berglund, Department of Information Technology Example, research approach: Critical enquiry Critical enquiry (research with “a mission”, often to address power imbalances) Feminist research Cannot be defined in ontological stand or research methodology Instead, it questions fundamental principals and values and aims to implement changes.

29. Informationsteknologi Anders Berglund, Department of Information Technology Feminist research, an example of critical research Are there factors within computer science itself, that preserves the currently dominating gender structure? (Björkman & Trojer, 2002) “We consider it of vital and decisive importance that gender research is done from within computer science.” “Such research on the core of computer science and its knowledge production would serve to enrich computer science as well as education within computer science.”

30. Informationsteknologi Anders Berglund, Department of Information Technology Why phenomenography? A personal view Appropriate for the research questions “Complex” “answers” desirable Statistical “answers” hard or impossible to get “Close to” learners Students’ perspective A way for students to “talk” to teachers/organisers Data stems from individuals Computer science is in focus Results “talk to” computer scientists Supports deployment of results in teaching Relevant for recognition of CSE research within CS Competence at hand/tradition Shirley Booth I like it Selecting a research approach Read! Create a network!

31. Informationsteknologi Anders Berglund, Department of Information Technology Literature Clancy, M., Stasko, J., Guzdial, M., Fincher, S., & Dale, N. (2001). Models and Areas for CS Education Research. Computer Science Education, 11(4), 323-341 Fincher, S., & Petre, M. (2004) Computer Science Education Research, London, UK: Taylor & Francis Berglund, A., Daniels, M. and Pears, A. (in press). Qualitative Research Projects in Computing Education Research: An Overview. To appear in the Proceedings of the 8th Australasian Computer Science Education Conference, Hobart, Australia. Berglund, A. 2005. Learning computer systems in a distributed project course: The what, why, how and where. Acta Upsaliensis Universitatis. Uppsala Dissertations from the Faculty of Science and Technology 62 Read!

32. Informationsteknologi Anders Berglund, Department of Information Technology CeTUSS CeTUSS ska verka för anpassning av teknikutbildning till samhällets och studenternas behov. Genom att utveckla och sprida information om lärandemiljöer som är Personligt meningsfulla Socialt relevanta Gränsöverskridande Baserad på samarbete (lokalt och internationellt) Workshops, kurser etc. Create a network!

33. Informationsteknologi Anders Berglund, Department of Information Technology