Campbell Collaboration Colloquium 2012 Copenhagen, Denmark The effectiveness of volunteer tutoring programmes Dr Sarah Miller Centre for Effective Education Queen’s University Belfast

The review team Dr Sarah Miller Deputy Director, Centre for Effective Education, QUB Professor Gary Ritter College of Education and Health Professions, University of Arkansas Pauline Connolly Centre for Effective Education, QUB Nicole Craig Centre for Effective Education, QUB Naoimh Fox Centre for Effective Education, QUB

Overview Purpose of the current review Summary of the methods and findings from the original Ritter et al 2006 review Rationale for updating and extending Ritter et al 2006 review Value of systematic reviews Potential limitations of reviews

The current review The current review is an update and extension of Gary Ritter’s 2006 review of the effectiveness of volunteer tutoring programmes Context of the Ritter et al 2006 review: limited and conflicting nature of existing evidence Objective of the Ritter et al 2006 review: to summarise the cumulative empirical evidence on the effectiveness of volunteer tutoring programmes in improving academic skills of elementary school children

Summary of methods (Ritter et al 2006) Types of studies Randomised field trials, standard care control, published after 1985, English language studies in the United States. Specialised programmes were excluded. Types of participants Tutors: adult, non-professional tutors Tutees: Kindergarten to Grade 8 (age 5 to 13/14 years old) Types of interventions Regular tutoring sessions with academic focus at least 1 month in duration

Summary of methods (Ritter et al 2006) Types of outcome measures Standardised assessment of math and reading skills: Reading global Reading letters and words Reading comprehension Reading oral fluency Writing Mathematics global

Summary of methods (Ritter et al 2006) Search results 969 unique study reports identified Application of the inclusion and exclusion criteria resulted in 28 study cohorts to be included in the review Methodological quality Transparency of allocation procedures, integrity of random assignment design, existence of high level of attrition, existence of problems re programme fidelity No studies eliminated from the analysis on this basis

Summary of findings (Ritter et al 2006) Data synthesis Total n=1,676 study participants (873=intervention, 803=control) Reading: n=1,077 students, 24 studies (excluding 1 outlier study) Maths: n=643, 5 studies Types of tutors Primarily parents: n=338, 5 cohorts College age tutors: n=899, 12 cohorts Community volunteers: n=439, 11 cohorts

Summary of findings (Ritter et al 2006) Types of tutees Grade 1 (age 6-7 years): n=770, 14 cohorts Grade 2 and above (age 7+ years): n=906, 14 cohorts Programme structure Highly structured programmes: n=919, 15 cohorts Not highly structured: n=757, 13 cohorts Source of publication Refereed journals: n=772, 15 cohorts Other sources: n=904, 13 cohorts

Summary of findings (Ritter et al 2006) OutcomeN of studies N of tutored students in analysis Effect size95% CI Reading - overall *(.18,.42) Reading - global *(.05,.48) Letters & words *(.27,.56) Comprehension (-.6,.42) Oral fluency *(.14,.46) Writing *(.19,.71) Mathematics (-.18,.72)

Summary of findings (Ritter et al 2006) No significant difference in effect size by tutor group (i.e. parent, college student, community volunteer) No significant difference in effect size by Grade (i.e. programmes aimed at children in Grade 1 or Grade 2+) Highly structured programmes were significantly more effective (ES=.59) than programmes that were not highly structured (ES=.14) No evidence of publication bias

Rationale for updating and extending the review To include studies conducted outside the United States To include programmes that have employed both volunteer and paid tutors to improve literacy and numeracy outcomes To include studies that aim to improve literacy for children who are learning English as an additional language To include relevant studies that have been conducted since 2006

Context for updating and extending the review PhD students designing evidence based programmes in the area of literacy, numeracy and language. These are being evaluated using an RCT design. We have recently conducted two large trials of a volunteer tutoring programme (with children aged 8-9 years) and found small effect sizes related to reading:

Context for updating and extending the review RCT 1 (n=734) Reading comprehension (ES , p=0.98) RCT 2 (n=512) Decoding (ES +0.15, p=0.01) Reading rate (ES +0.22, p=0.01) Reading accuracy (ES =.07, p=0.28) Reading Fluency (ES +0.14, p=0.05) Comprehension (ES +0.05, p=0.55)

Value of systematic reviews Systematic reviews produce an unbiased account of the cumulative state of evidence in relation to a research question/hypothesis They provide information relating to the strengths and weaknesses of the existing evidence and the rigour of the methods used Determine the magnitude of the relationship under investigation Locate and include all relevant research

Potential limitations Included studies in reviews often contain small sample sizes Only four trials in four major reviews in the area of tutoring had a sample size greater than 250 Trials with small sample sizes and large effect sizes are potentially over represented in the literature (Slavin & Smith, 2009) –Use of less robust methodologies –Super-realisation –Treatment inherent measures

Potential limitations Linking outcomes to theory - theory development should not be neglected. In the case of volunteer tutoring: –Chall’s stages of reading development –Provides a useful framework for understanding children’s transition through the developmental stages –Successful transition at age 8-9 (to Stage 3 in Chall’s theory) requires practice reading familiar materials –Precisely the aim of many reading focused tutoring programmes

Challenges associated with conducting a systematic review Narrowing down the research question and defining variables The time required to undertake a review and volume of data to process and extract Studies not reporting sufficient data to use in meta analysis and non response of authors