Robert Coe Neil Appleby Academic mentoring in schools: a small RCT to evaluate a large policy Randomised Controlled trials in the Social Sciences: Challenges and Prospects University of York, Sept 2006

2 Mentoring In England, part of the KS3 Strategy ‘Learning Mentors’ part of Excellence in Cities Now part of ‘Every Child Matters’ ‘Mentoring’ means a lot of different things –Group / individual; location; duration; goals (Sipe and Roder, 1999) Academic mentoring in schools –Takes place in school –Aims to improve academic achievement

3 Evidence of effectiveness Meta-analysis by DuBois et al (2002) –Modest positive overall effect (SMD=0.14) –“evidence of an overall favorable effect of mentoring is notably lacking under circumstances in which participating youth have been identified as being at risk solely on the basis of individual- level characteristics (e.g. academic failure).” UK review by Hall (2003) –“There is a very poor evidence base in the UK. Claims are made for the impact of mentoring but there is as yet little evidence to substantiate them.” (p15) –“Mentoring is in danger of being unsuccessful if any of the following conditions apply: social distance and mismatch between the values of mentor and mentee inexpert or untrained mentors mismatch between the aims of the mentoring scheme and the needs of the person being mentored conflict of roles such that it is not clear whether the mentor is to act on behalf of the person being mentored or of ‘authority’.” (p24)

4 Justification for the current policy of Learning Mentors “Since their introduction in 1999, learning mentors have made a significant contribution to re-engaging pupils in learning and helping to raise standards in primary and secondary education. For example, research has shown that pupils receiving support from learning mentors were one and a half times more likely to achieve five or more GCSEs at grades A* to C than young people with similar prior attainment who had not been mentored. 1 In many cases learning mentors succeeded in raising pupil performance above the levels indicated by these pupils’ Key Stage 3 outcomes.” Supporting the new agenda for children’s services and schools: the role of learning mentors and co-ordinators. DfES, 2005,

5 To be worth considering, a study would have to Be conducted in the UK Include a comparison group Provide evidence of initial equivalence and the reasons for membership of mentoring/comparison groups Describe the ‘mentoring’ intervention adequately Include an outcome measure of academic achievement

6 Neil Appleby’s Experiment A randomised controlled trial involving 20 underachieving Y8 (12-13 year-old) students Matched in pairs on ability and gender Randomly allocated: one of each pair mentored, the other not Mentored group had 20 mins individually every two weeks (11 sessions) –‘It nearly killed me’ –Cost estimated at between £170 and £410 per mentored pupil –Represents between 8-19% of the school’s annual per pupil funding for the whole of their education

7 What the teachers said about the mentored students … “**** is a changed person this year she has progressed greatly and is a superb helpful student.” “Better now, has achieved more, more confident.” “Generally a great improvement recently.” “****’s attitude and effort have improved over the year. He is a lot pleasanter and more willing to participate in lessons particularly oral work, he responds well to praise.”

8 What they said about the control group … “Has improved overall this term.” “****’s attitude and effort have improved over the last few months, she is now trying very hard to achieve her target. Great effort.” “Commended for attitude and progress.” “**** has settled since the beginning of the year.” “**** has undergone quite a transformation since September. Her attitude towards the teacher and her learning have improved drastically and she should be congratulated.”

9 Change in Teachers’ Ratings of progress, effort and attitude (English, maths and science combined)

10 What this proves If you identify a group of underachieving pupils at a particular time and then come back to them after a few months, many of them will have improved, whatever you did. Others (the ‘hard cases’) will not have improved, whether mentored or left alone. The interpretation of this would have been very different without a ‘control’ group

11 Academic achievement Effect size = 0.03

12 Effect size = -0.16

13 Effect size = 0.23

14 Ethical issues University Ethics Advisory Committee questioned the size of the study: ‘It seems a rather small number to exclude a type 2 error? ‘ Withholding treatment from control group Informed consent Schools are not very ethical –Pupils never give consent for anything –Basic human rights (eg privacy) are violated all the time

15 Challenges and prospects Limitations of ‘evaluation by participants’ Importance of control group Why does education suffer from these fashions? You can do RCTs in education

16 Dr Robert Coe Curriculum, Evaluation and Management (CEM) Centre University of Durham Mountjoy Research Centre 4 Stockton Road Durham DH1 3UZ Tel: (+44/0) Fax: (+44/0)