Pedagogy and Performance: The challenge of measurement Conference on Quantitative Applications in Education Research Stellenbosch August 2015 Ursula Hoadley & Jaamia Galant, UCT

SPADE project research question How can we account for poor schools performing above expectations given their socio-economic contexts relative to low performing schools in similar socio-economic contexts?

Sample School 1 School A Set 1 School 2 School 3 School B Set 2 School 4 School 5 School C Set 3 School 7 School 6 School D Set 4 School 8 School 9 School E Set 5

Research question: pedagogy Is there a relationship between better school achievement outcomes and pedagogic practice in mathematics and home language at the Grade 3 level?

Classroom Observations 43 Grade 3 teachers across the 14 schools were selected. Each of these teachers was observed teaching a Home Language, Mathematics and First Additional Language lesson on the same day. Data was collected both through a closed observation schedule and videoing of the three lessons.

Measuring pedagogy background: Van der Berg et al (2010) 136 Grade 1 and Grade 3 classrooms in 45 schools in the Western Cape. Describe classroom features found in weakly performing schools: – Slow pacing; low content complexity and cognitive demand; weak assessment and little explicit verbal feedback; limited teacher knowledge; dominance of everyday discourse. Conclude: in these largely functional, but weakly performing schools, there is an attention to form rather than substance.

Constituting an ‘ideal pedagogy’ Strong and clear evaluation (explanation and explicit verbal feedback) Individualising Coherence in lesson topics / components Variable learner / topic directed pacing Text-based pedagogy (vs oral) Reading and writing extended text in language Conceptual focus and elaboration in Mathematics

Developing a pedagogic score Used closed classroom observation schedule and detailed lesson narratives to code pedagogic practices. Coded along 10 dimensions of pedagogy (see handout 1) Score for each dimension was derived from ‘ideal pedagogy’. For example: – text-based classroom discourse type was weighted more than oral- based discourse type; – extended engagement with text was weighted more than restricted engagement with text; – extended elaboration of evaluative criteria was weighted more than restricted feedback or minimal elaboration of evaluative criteria; – individualized reading practices was weighted more than communalized reading practices; – individualizing students was weighted more than uniform treatment of learners.

Categorising teachers based on pedagogic scores Teachers’ scores on all dimensions were summed to get a cumulative overall pedagogic score for Language and Mathematics separately. Each teacher was then given a pedagogic percentage score relative to the maximum score for the ideal pedagogy privileged in our coding for Language and Mathematics. Using the maximum possible score for the ideal pedagogy in Mathematics and Language, we categorized teachers as follows: – ‘Good’, if their cumulative overall pedagogic score was above 70% of the maximum total score possible; – ‘Moderate’, if their cumulative overall pedagogic score was between 50-69% of the maximum total score possible; – ‘Weak’, if their cumulative overall pedagogic score was below 50% of the maximum total score possible.

Developing a school pedagogic score For most schools, teachers at each school were either all ‘good’, all ‘weak’, or a mixture of ‘good’ and ‘moderate’ or ‘moderate’ and ‘weak’. We summed the individual teacher percentage scores to get an average percentage pedagogic score for Mathematics and Language for each school. We excluded 3 of the schools for Language and 2 for maths given too great a variation between teachers in these schools (ie. They included weak, moderate and good teachers).

Sample School 1 School A Set 1 School 2 School 3 School B Set 2 School 4 School 5 School C Set 3 School 7 School 6 School D Set 4 School 8 School 9 School E Set 5

Schools ranked by performance

Comparing school Mathematics pedagogic scores with school performance

Comparing school Language pedagogic scores with school performance

No relationship between school pedagogic scores and school performance. Why? Within school range - Particular teachers may be elevating the overall score of the school, or conversely weak teachers lowering the score in others. The data shows that across the sample there is variation, but no exemplary practice Within this narrow band of performance, ‘good’ still does not represent stand out practice

Going back to individual teacher pedagogic scores Changes in the form of pedagogy: Higher levels of individualising Working at grade level Time on task More texts and text-based activity

Form vs Substance: Example from a reading lesson The teacher calls ten learners to mat. Each reads individually. They sit in a row, so most of the children don’t hear the learner who is reading. The teacher spends most of the time not listening to the learners read but regulating the behaviour of the rest of the class. The majority of her utterances are regulative of the whole class rather than focused on the reading group. When she does attend to the reading of the individual there is no work on fluency, expression or punctuation. When a learner struggles with a word no strategies are provided for the child to derive the word on their own. There is questioning around the meaning of words and two questions related to simple factual recall, but no questions relating to textual interpretation, inference or connecting the text to children’s worlds. No effort at encouraging mutual listening or pleasure in reading.

Teachers with higher pedagogic scores are realising the form rather than the substance of an improved pedagogy. We describe this substance as that which connects the particular pedagogic instance to a more general principle or subject trajectory. We distinguish between evaluative criteria – which teachers with high pedagogic scores engage in – and evaluative rules, the ordering principal or knowledge object that connects fragments of the pedagogy. Focusing in on evaluation

Graded group reading Teacher P, who in a small reading group, reads with learners in a graded group a story about a boy who was too proud. The reading of the text is fragmented by constant interjection [insertion of evaluative criteria] in the pedagogy including aspects related to vocabulary, punctuation, the mechanics of reading (how to turn a page); the value of reading (it makes you ‘clever’). By the end of the reading session it is very difficult for the researcher, let alone struggling readers, to retrieve the narrative. Reading as a sustained activity where text communicates meaning does not emerge. The activity ends abruptly and learners are given questions related to a completely separate and unrelated set of pictures to the text just read. Here we have the form of guided group reading, which theoretically aims for enhanced focus on reading for meaning and comprehension and more individualized evaluation by the teacher. We also find strong framing over the evaluative criteria with constant intervention by the teacher. But the focus of the evaluative practice in relation to student productions is decoding (although without specific strategies to decode unfamiliar words) and not retrieving meaning from the text.

Maths example In a mathematics lesson of Teacher Z, with learners on the mat, the teacher explains that they must count out 10 beans in front of them, then make equal groups with 10 beans. As they make equal groups with the beans, the teacher writes each representation on the board as a number sentence, first as repeated addition, and then explains shorter way to write as multiplication (e.g =10 or 5x2=10). She ends up with multiple representations of the same number (10) on the board and makes sure that learners are aware they are making groups with the same number all the time (indicator 1). Teacher uses 12 as a second example, which learners also have to break up into equal groups. Teacher again writes multiple representations of 12 on the board, showing different equal groups as repeated addition or multiplication (e.g = 12; 2 x 6 = 12; = 12; = 12). Learners are then instructed to use their beans to represent their own number greater than 12 by breaking the number up into equal groups, similar to the examples on the board. The teacher makes some points either to the whole class or to individual learners so as to clarify what is expected of them in the task. In particular, the teacher checks on learners as they are making groups with beans and makes the learners check their totals, then writes the number sentence representing their groups on the board (indicator 2). Learners are seldom asked to give reasons for their answers. When learners have to verify their totals, they don’t actually count in their groups, they count in ones to check. The teacher pauses with errors, but rather than engaging with the error, she often makes learners change the number they are working with to make it easier for them to make equal groups (indicator 3).

Conclusion Is the study of pedagogy amenable to variabalisation (measurement through isolated variables)? Form has shifted, but doesn’t get to the organising / ordering principles – this is the substance of pedagogy. The substance or ordering principle of pedagogy concerns the strength of the evaluative rules. The challenge is the measurement of the evaluative rule across a large sample.