The issue of retention in distance learning: a strategic approach to using data to inform retention interventions Alison Gilmour and Avinash Boroowa The Open University

Struggling students want to be noticed.

Overview: our retention project Proactive student support Predictive modelling Studying more than 120 credits Focus of the session: retention project which draws together and is informed by our previous experiences of proactive student support, in particular the pre-course calls, as well as the use of predictive modelling to try to target resource most effectively. Keen to share our approach and interim findings, as well as to get your views on the nature of this type of student support and how it compares with work that has been done previously in OUiS. We also want to open up discussion on some of the difficult topics that we’ve encountered as part of this project to seek your views. Will return to predictive modelling and the nature of student support later in the presentation. Key is that our project objective is to … Objective To improve the retention rate of ‘at risk’ students as defined by the Strategy and Information Office Predictive Model

Multiple project drivers Overview Multiple project drivers

Project Team Various staff from The Open University in Scotland including ALs, Educational Advisors and Learning Development Strategy & Information Office Learning & Teaching Innovation A key aspect of this project, and one we feel is quite distinct from previous retention projects at the OUiS is the collaborative approach which we feel has been a key strength in shaping all aspects from the design and methodology, through to collection and analysis of data and next steps. It’s the input of different types of expertise from whether that be statisticians, data analysts, student advisors, associate lecturers, learning development etc. Another key element of the project is in the collation of evidence (not just related to the outcomes for students, but also operational data associated with the interventions). Although this is being run as a pilot project in Scotland, we are feeding into the body of evidence being generated as part of the Early Alert Indicators / Data Analytics projects.

What are the student probabilities? Phase 1: SIOPM What are the student probabilities? Overarching – three phases to the project, currently in phase 2 Phase 1 took place between February 2016 – June 2016, where the Strategy and Information office compared the actual performance of the entire Scottish cohort in 2014J to the predictive model run against the same cohort. The project team felt confident that the predictive model was fit for use with a Scottish cohort. Correctly identifying more non-completers Within IET, statistical models have been trialled to identify the expected pass rate for cohorts of students taking particular modules. These models use information relating to module design, and both demographic characteristics and study history of module students to yield a predicted probability that each student will pass the module. The Open University is interested in ensuring that all students receive appropriate and timely support. Models of this kind could, in principle, be used to identify students who are at risk of dropping out or failing their modules so that they could be targeted using appropriate interventions by the relevant Student Support Team. Pilot work is being undertaken to determine the usefulness of this approach… Probabilities are generated for: - retention to the above milestones - module completion - module pass - return in the following academic year

How are the student probabilities generated? Phase 1: The SIOPM How are the student probabilities generated? Different resilience factors influence the model at different points in the module.

Testing the application of predictive modelling with a Scottish Cohort Phase 1: February – June 2016 Testing the application of predictive modelling with a Scottish Cohort Our aim is to reduce the number of non-completions How accurate was the SIOPM in predicting non-completions for students in Scotland for 2014J? IOPM October 2014 completion predictions were compared with actual completion status at the end of the module How does the using the SIOPM to predict non-completion, compare with using a single variable? Correctly identifying more non-completers Within IET, statistical models have been trialled to identify the expected pass rate for cohorts of students taking particular modules. These models use information relating to module design, and both demographic characteristics and study history of module students to yield a predicted probability that each student will pass the module. Predictive modelling is not currently used to enhance the attainment or the experience of individual students. However, the Open University is interested in ensuring that all students receive appropriate and timely support. Models of this kind could, in principle, be used to identify students who are at risk of dropping out or failing their modules so that they could be targeted using appropriate interventions by the relevant Student Support Team. Pilot work is being undertaken to determine the usefulness of this approach…

Identifying ‘at risk’ students Step 1: Comparing Selection Methods

Identifying ‘at risk’ students Step 2: Further Refinement of the Selection

Identification of ‘at risk’ students Step 2: Further refinement of the selection The lower the range selected, the higher the percentage of non-completers within the selection. Student Probabilities allow users to refine the selection to identify a number of students that suits the capacity and resources available, by narrowing the selection range. The example shows the number of students in each probability band, and this can be used to focus on one or more bands to suit both the target number of students, and the target range.

Phase 2: Retention Intervention Nature of Intervention Rationale: To ensure students who were identified as ‘at risk’ were contacted to offer additional support if required.

Interim Data The first interim analysis highlights a positive impact on retention on day 14 of students contacted via the project, the Intervention Group, compared to the Control Group, and that this is statistically significant Retention to 50% Fee Liability Point (31st December) for Intervention and Control Groups. Again, this shows a statistically significant positive difference in retention for those students in the Intervention Group compared to those in the Control Group. Caution: these are only active withdrawals; this contains all students receiving the intervention including those who only received SMS and Email. Not only is the actual retention to 50% is better for the intervention group, this also converts into an increased projected number of completion and pass (but we need to be cautious as these are still predictions). In terms of the student number gain, the sizes of the two start groups are aligned. (So if we assume that the control and intervention group were of the same size, this is what the gain would be. )

Some caution … but encouraging signs in the interim data Next Steps and Phase 3 Some caution … but encouraging signs in the interim data Data on completion (July 2017) Disaggregation of students within Intervention Group (a difference between those who were contacted by SMS/ Telephone compared to SMS/ Email) Consideration of students who moved out of the band Currently exploring the potential of broader contextual evidence that would allow us to better understand student behaviour: Mining of student records for both Control and Intervention Groups (to consider specific module intervention and the broader intervention landscape) Analytics such as VLE behaviour Planning for Phase 3 – Changing the intervention? Running additional interventions with the same band? Running the same intervention with another band?

Contact Alison Gilmour, The Open University in Scotland alison.gilmour@open.ac.uk Broader Project Team includes: Avinash Boroowa, Lucy Macleod, Hannah Jones, Galina Naydenova, Rebecca Ward and Christothea Herodotou.