Robert Coe Neil Appleby Academic mentoring in schools: a small RCT to evaluate a large policy Randomised Controlled trials in the Social Sciences: Challenges.

Slides:



Advertisements
Similar presentations
TRANSFORMING EDUCATION THROUGH EVIDENCE. The Centre for Effective Education SCHOOL OF Education Conducting Educational Randomised Control Trials in Disadvantaged.
Advertisements

What you don’t see can’t matter? The effects of unobserved differences on causal attributions Robert Coe CEM, Durham University Randomised Controlled Trials.
School-based interventions for increasing physical activity and well-being: The MOVE project’s design and conceptual framework Katie Thomson, Sarah Curtis.
Peter Finlayson Quality improvement Officer February 2013.
Deprivation and the Pupil Premium - what you need to know. After prior attainment, poverty is the strongest predictor of a child’s future life-chances.
Sue Rogers Director of Education KGA Presentation GCSE and Post 16 plus Closing the Gap.
∂ What works…and who listens? Encouraging the experimental evidence base in education and the social sciences RCTs in the Social Sciences 9 th Annual Conference.
Closing the progress gap. Key issues addressed by the study This study explored: – approaches to closing the gap for disadvantaged pupils –effective leadership.
T HE DURHAM SHARED MATHS PROJECT. Peer Tutoring and EEF CEM involvement in Peer Tutoring Fife and Vic’s research Educational Endowment Foundation.
Children’s Trust Network 19 October 2011 Education Attainment and Progression Anthony May Corporate Director for Children, Families and Cultural Services.
Dr Ian Abrahams Combining randomised control trials with qualitative research approaches: The best of both worlds York
STEM: Progression from GCSE to A Level Andrew Powell (Schools Analysis and Research Division) and Aileen Clement (Curriculum Unit) Presentation at DCSF.
Session 3: Trial management Sarah Miller (Queens, Belfast) Laura Dunne (Queens, Belfast)
Youth mentoring and the well-being of young people: Evidence from an Irish mixed- methods evaluation Dr Bernadine Brady Child & Family Research Centre.
Neil Harrison Senior Research Fellow Bristol Centre for Research in Lifelong Learning and Education (BRILLE) University of the West of England.
Trials and Tribulations Peter Tymms York Outline Introduction Background issues –Impact of educational policies –Level of intervention –Monitoring.
Gorefield Primary School
Assessment January ‘Attainment’ & ‘Progress’ Attainment: This is the score, grade, mark or level that is achieved from a particular task or activity.
Assessment policy Fundamental principles Assessment for learning Assessment of learning Recording Reporting Reflect what is known to be effective teaching.
Evaluation of Math-Science Partnership Projects (or how to find out if you’re really getting your money’s worth)
Education Policies and Structures Their impact on the delivery of Gaelic Games.
Thinking Skills 1 of 23. Why teach thinking skills? Is it really that important? Creative and critical thinking abilities are not inborn as was once believed.
Essex Virtual School March / April 2011 Designated Teacher for Children Looked After Training.
Raising Academic Standards for all School Development Planning Initiative.
Welcome & Introduction.  The largest educational research unit in a UK university (75 staff)  1.1 million assessments are taken each year  More than.
Year 7 Settling – in Evening. Assessment Process and Ability Grouping.
Raise Your Chances! Get your attendance right! What does “Good Attendance” mean?
ARROW Trial Design Professor Greg Brooks, Sheffield University, Ed Studies Dr Jeremy Miles York University, Trials Unit Carole Torgerson, York University,
#unseenchildren Unseen Children: Access and Achievement in Education Sean Harford HMI Regional Director Ofsted.
Level 7 Research Project Laura Bridge Robert Owen EBITT.
Raising standards, improving lives. Tackling disadvantage – lessons from Ofsted inspections and research John Kennedy Interim Regional Director, London.
University of Durham D Dr Robert Coe University of Durham School of Education Tel: (+44 / 0) Fax: (+44 / 0)
Every Child Matters Improving outcomes for children in the UK Dr Gillian Pugh DBE Oslo, April 2006 Early interventions for infants and small children in.
Programme Information Incredible Years (IY)Triple P (TP) – Level 4 GroupPromoting Alternative Thinking Strategies (PATHS) IY consists of 12 weekly (2-hour)
1 Data Linkage for Educational Research Royal Statistical Society March 19th 2007 Andrew Jenkins and Rosalind Levačić Institute of Education, University.
Goal and Target Setting - What’s my role? Module 3.
Developing teachers’ mathematics subject knowledge in early years settings and primary schools to improve the attainment of all pupils Liz Woodham, NRICH.
CLOSING THE GAPS – REDUCING INEQUALITIES IN OUTCOMES FOR CHILDREN AND YOUNG PEOPLE BIRMINGHAM ACHIEVEMENT GROUP SEMINAR DECEMBER 2008 JOHN HILL RESEARCH.
Getting Strategic Provision Management in Schools.
Quality First Teaching for All SENJIT 21 st May 2013.
ECAS Meeting Friday 17 th June  The Achievement for All (AfA) project aimed to improve the outcomes of all children and young people with special.
Gosforth Park First School Literature Works in the Primary Classroom: Talk for thinking.
Primary.  There was a greater level of improvement in Literacy than Numeracy for both FSME and Non-FSME pupils.  Boys showed a greater level of.
Year 10 GCSE Information Evening WELCOME. Year 10 GCSE Information Evening Update on national changes to GCSEs – Jennifer Howe, Assistant Headteacher.
Monday 16 th November What is AMA?  An Academically More Able pupil is one who excels far beyond the ordinary or the expected.  Pupils who display.
Impact of two teacher training programmes on pupils’ development of literacy and numeracy ability: a randomised trial Jack Worth National Foundation for.
National Improvement Framework Aims of this presentation: Share information on the draft National Improvement Framework To discuss and share views on.
Belfast VPs’ cluster group Value Added May Why Measure Value Added? As a means of self-evaluation Raising academic standards As a measure of effectiveness.
Pinchmill Lower School Assessment in the New Curriculum November 2015.
ABMP Student Success Curriculum Topic 1: Transitioning.
Quality First Teaching for All. Quality First Teaching for ALL The most effective way to narrow the gaps! A Top Priority for Schools! Context and Background.
ItemEnglishMaths National A*-C6661 National boys A*-C6061 National girls A*-C7362 National FSM A*-CNA National boys FSM A*-CNA National girls.
Welcome!. To explain what Assertive Mentoring is To see how it will work in our school To see how it will benefit you and your child Today’s aims.
Assessment Information Evening 17 th September 2015.
SEF Describing good or better achievement and standards What is laid down, ordered, factual is never enough to embrace the whole truth: life spills over.
New Assessment Arrangements at Old Cleeve. No more Levels? This academic year the Government abolished the use of levels (eg 3C, 4A etc) in school as.
Empowering Informed Decisions Using RAISEonline data to improve governor effectiveness Dave Thomson Head of Data Analysis, RM Education.
Accelerating the Achievement of Vulnerable Groups May 2016 Jo Pallett Head of Virtual School
Planning and assessing Session 3.  Aims and Principles; what are those in settings like? How do they compare with personal aims/ principles?  Journal.
Blackshaw Primary School.  DfE – statutory assessments:  Reception – Baseline, EYFS profile  Year 1 (and 2) - Phonics Check  Year 2 and 6 - end of.
“University is a waste of time” Myths and methods in engaging parents in widening access work Dr. Graeme Atherton, Director, National Education Opportunities.
Common Core Presentation Sharelius Holmes Week 2: Discussion 2 Dr. Sherry Leialoha-Waipa.
Raising Achievement Project YOU CAN DO THIS!!!. The ‘rationale’ for this group... The Year 10 report analysis shows that you, as a group, are significantly.
St Patrick’s Academy Achieving excellence together The Academy aims to promote the maximum attendance from all our students so they may achieve their full.
Hertfordshire County Council The Role of the Secondary Assessment Co-ordinator Day One 5 th July 2005.
Moving to Life Without Levels
Statutory Assessment at SPRINGFIELD PRIMARY SCHOOL
WPS Assessment Information Evening
Charlton Kings Junior School
The English RCT of ‘Families and Schools Together’
Presentation transcript:

Robert Coe Neil Appleby Academic mentoring in schools: a small RCT to evaluate a large policy Randomised Controlled trials in the Social Sciences: Challenges and Prospects University of York, Sept 2006

2 Mentoring In England, part of the KS3 Strategy ‘Learning Mentors’ part of Excellence in Cities Now part of ‘Every Child Matters’ ‘Mentoring’ means a lot of different things –Group / individual; location; duration; goals (Sipe and Roder, 1999) Academic mentoring in schools –Takes place in school –Aims to improve academic achievement

3 Evidence of effectiveness Meta-analysis by DuBois et al (2002) –Modest positive overall effect (SMD=0.14) –“evidence of an overall favorable effect of mentoring is notably lacking under circumstances in which participating youth have been identified as being at risk solely on the basis of individual- level characteristics (e.g. academic failure).” UK review by Hall (2003) –“There is a very poor evidence base in the UK. Claims are made for the impact of mentoring but there is as yet little evidence to substantiate them.” (p15) –“Mentoring is in danger of being unsuccessful if any of the following conditions apply: social distance and mismatch between the values of mentor and mentee inexpert or untrained mentors mismatch between the aims of the mentoring scheme and the needs of the person being mentored conflict of roles such that it is not clear whether the mentor is to act on behalf of the person being mentored or of ‘authority’.” (p24)

4 Justification for the current policy of Learning Mentors “Since their introduction in 1999, learning mentors have made a significant contribution to re-engaging pupils in learning and helping to raise standards in primary and secondary education. For example, research has shown that pupils receiving support from learning mentors were one and a half times more likely to achieve five or more GCSEs at grades A* to C than young people with similar prior attainment who had not been mentored. 1 In many cases learning mentors succeeded in raising pupil performance above the levels indicated by these pupils’ Key Stage 3 outcomes.” Supporting the new agenda for children’s services and schools: the role of learning mentors and co-ordinators. DfES, 2005,

5 To be worth considering, a study would have to Be conducted in the UK Include a comparison group Provide evidence of initial equivalence and the reasons for membership of mentoring/comparison groups Describe the ‘mentoring’ intervention adequately Include an outcome measure of academic achievement

6 Neil Appleby’s Experiment A randomised controlled trial involving 20 underachieving Y8 (12-13 year-old) students Matched in pairs on ability and gender Randomly allocated: one of each pair mentored, the other not Mentored group had 20 mins individually every two weeks (11 sessions) –‘It nearly killed me’ –Cost estimated at between £170 and £410 per mentored pupil –Represents between 8-19% of the school’s annual per pupil funding for the whole of their education

7 What the teachers said about the mentored students … “**** is a changed person this year she has progressed greatly and is a superb helpful student.” “Better now, has achieved more, more confident.” “Generally a great improvement recently.” “****’s attitude and effort have improved over the year. He is a lot pleasanter and more willing to participate in lessons particularly oral work, he responds well to praise.”

8 What they said about the control group … “Has improved overall this term.” “****’s attitude and effort have improved over the last few months, she is now trying very hard to achieve her target. Great effort.” “Commended for attitude and progress.” “**** has settled since the beginning of the year.” “**** has undergone quite a transformation since September. Her attitude towards the teacher and her learning have improved drastically and she should be congratulated.”

9 Change in Teachers’ Ratings of progress, effort and attitude (English, maths and science combined)

10 What this proves If you identify a group of underachieving pupils at a particular time and then come back to them after a few months, many of them will have improved, whatever you did. Others (the ‘hard cases’) will not have improved, whether mentored or left alone. The interpretation of this would have been very different without a ‘control’ group

11 Academic achievement Effect size = 0.03

12 Effect size = -0.16

13 Effect size = 0.23

14 Ethical issues University Ethics Advisory Committee questioned the size of the study: ‘It seems a rather small number to exclude a type 2 error? ‘ Withholding treatment from control group Informed consent Schools are not very ethical –Pupils never give consent for anything –Basic human rights (eg privacy) are violated all the time

15 Challenges and prospects Limitations of ‘evaluation by participants’ Importance of control group Why does education suffer from these fashions? You can do RCTs in education

16 Dr Robert Coe Curriculum, Evaluation and Management (CEM) Centre University of Durham Mountjoy Research Centre 4 Stockton Road Durham DH1 3UZ Tel: (+44/0) Fax: (+44/0)