1. Validating a theoretical framework for analysing computer programming learning processes
Presentation to the ACARA Digital Technologies National Working Group
6 December 2017
Matt Bower, Jens Siemon, Garry Falloon, Jennifer Lai

2. How do we know…
Playing with robots and online programming environments can be fun, but how do we know if it is developing students’ computational thinking capabilities?
What pedagogical strategies actually help students to become better computer programmers and problem solvers using technology?

3. Research problem
How do we describe and analyse the process by which people learn to write computer programs? 
Important in order to understand how teachers might better support the development of computer programming capabilities?
Activity Theory has been used as framework for analysing how activities are undertaken within social constructivist contexts
Subject pairs of students working together, Object is the computer program, the tools (instruments) that can be used by subjects to develop the program are computers, integrated development environments, etc.
Activity theory does not encompass any of the specific computer programming processes that people undertake
To better understand teachers might change things to improve the decision process, and give us the way to impact what teachers will do to affect what students will do
Limited studies on how people write computer programs/investigate the computer programming process.
Figure 1. The elements of an activity system and their interrelationships (Engeström, 1987)

4. Computational thinking & notional machine
Computational Thinking: Solving problems, designing systems, and understanding human behaviour, by drawing on the concepts fundamental to computer science (Wing, 2006)
Notional Machine: The notional machine is an abstract version of the computer, “an idealised, conceptual computer whose properties are implied by the constructs in the programming language employed” (du Boulay, et al., 1989)
These concepts can be used to theoretically ground how people go about performing computer programming process
Theory should represent reality - model requires empirical validation
Notional machine
OFFICE | FACULTY | DEPARTMENT

5. Research model
Thematic analysis using an informed grounded theory approach, starting with an initial framework.
Task specification
Computational thinking
Requirements checking
Reconstruction
Deconstruction
Help seeking
Technology
Organising
Off topic
Unclassifiable
Reflections on learning
Design
17 categories and we had the definitions of the categories and we made changes via the entire process
Evaluating
Decoding
Encoding
Notional machine
Implementation
Testing Debugging

6. Methodology
Data was collected from 10 pairs of students completing a scratch programming activity:
5 pairs of pre-service teachers with little experience
5 pairs of third year computing students
Program a story game where a hero has to overcome a challenge in order to defeat the villain(s).
Each pair spent approximately 40 minutes to undertake the task.
Both finished the IQ tests; Did the Computational thinking tests (similar CT scores) ; they did not have any prior programming knowledge; decent scratch products

7. Data analysis – Thematic analysis
DUAL CODING BY TWO RESEARCHERS
Nvivo 11 to conduct the thematic analysis
Two raters coded the first two transcripts independently to identify areas of coding discrepancy. After each of the two videos was coded the pair met to analyse the reasons for the differences and agree upon a category. The inter-rater reliability improved from the first to the second video, leaving the team with a sense of conceptual agreement about the meaning and boundaries of the categories. Following this, the final X videos were coded by one team member, who flagged potential utterances where coding could potentially have been ambiguous. The second coder was then consulted on each of these, and the pair formed a consensus about the most appropriate category.

8. Refined coding scheme and final results
Three categories were merged into others because they were not observed in majority pairs (Reconstruction Requirements checking; Decoding  Evaluation; Technology  Help seeking)
Percentages of each category observed shown below.
Task specification (0.2%)
Computational thinking
Requirements checking/Reconstruction (0.3%)
Deconstruction (1.4%)
Help seeking/Technology (0.8%)
Organising (1.0%)
Off topic (1.2%)
Unclassifiable (3.3%)
Reflections on learning (1.4%)
Design (11.1%)
Top 5 categories highlighted in red
Evaluating/Decoding (3.5%)
Encoding
(15.3%)
Notional machine
Implementation (47.0%)
Testing (9.2%)
Debugging (4.5%)

9. Some significant differences between computing and education students
Chi-square test: = , df = 13, p-value < 2.2e-16
The categorisations are dependent on different disciplines (e.g. education students or computing students)
Significant difference between the expected and observed frequencies

10. Framework was able to show that
Computing students had relatively greater focus on deconstructing the problem
Computing students made relatively greater focus on evaluating the program
Education students made relatively greater focus on help seeking
Computing students made relatively greater focus on implementation
Education students made relatively greater focus on reflections on learning
Organising was not significant (p=0.0067) but very low value may indicate that with larger sample education pairs tended to dedicate more attention to groupwork
1. Think more about the relationship between tasks and design. 2. Computing students are more skilful to evaluate the program. 3. Education students are less knowledgeable on how to use Scratch hence they have to seek help, like watching the tutorials or asking others. 4. Education students did more self-reflections exercises in their discipline.
OFFICE | FACULTY | DEPARTMENT

11. Summary and future work
Study has empirically validated a theoretical framework for describing computer programming processes and shows, using a single contrast, how it can be used to perform educational analysis
Further research could investigate how programming process differs for:
Different tasks (e.g. more complex specifications)
Different languages (e.g. C++, Python, Blockly)
Different programming environment (e.g. visual vs. text interfaces)
Different Cohorts (e.g. children, experts, gender)
Different teacher interventions (e.g. forms of scaffolding and modelling)

12. Thank you! Q&A Computational thinking Notional machine
Task specification
Computational thinking
Requirements checking /
Reconstruction
Deconstruction
Help seeking / Technology
Organising
Off topic
Unclassifiable
Reflections on learning
Design
Evaluating /
Decoding
Encoding
Notional machine
Implementation
Testing Debugging