Evaluating WP in in specialist contexts Why Evaluate? 27th November 2017 Evaluating WP in in specialist contexts Kate Wicklow Senior Policy Adviser, GuildHE Kate.Wicklow@guildhe.ac.uk

National Policy Drivers Why Evaluate? 27th November 2017 National Policy Drivers Look at specific interventions small and specialist providers undertake Major barriers to evaluation Introduction to Developmental Evaluation Kate.Wicklow@guildhe.ac.uk

National Policy Drivers Why Evaluate? 27th November 2017 National Policy Drivers Access Agreements – measures access and participation TEF – measures final grades, learning gain, employment outcomes OfS Registration Data – will measure both Access and Success Value for Money – NCOP, Access funding etc. All require different types of data to show your impact Kate.Wicklow@guildhe.ac.uk

All small/specialist institutions have a problem with small datasets. Why Evaluate? 27th November 2017 What is a Small/Specialist institution Less than 10,000 students Multidisciplinary A specialist (with all courses falling in 1-2 course codes) A discipline focused HEI (Finance, agriculture, STEM) Often rural, costal or suburban All small/specialist institutions have a problem with small datasets. Kate.Wicklow@guildhe.ac.uk

Work within rural and coastal communities to raise aspirations Why Evaluate? 27th November 2017 Specific events for students with Physical or Specific Learning Difficulties or Disabilities Work within rural and coastal communities to raise aspirations  What are My Options Theatre in Education Working with Employers to ensure skills are useful to local region Workshops for creative teachers Residential summer school for Year 12s (Get Set) After school creative classes/Saturday Schools Demystify audition/interview process Primary age arts/farming classes Kate.Wicklow@guildhe.ac.uk

Why Evaluate? 27th November 2017 Top down approach “As part of your evaluation strategy we will be looking for increased demonstration of the effectiveness of your access agreement spend, and we will be looking to see that robust evaluation of the outcomes of your activities and interventions, particularly in key strategic areas such as school attainment, has informed your decision on where to invest your access agreement expenditure.” Kate.Wicklow@guildhe.ac.uk

What should we evaluate? Why Evaluate? 27th November 2017 What should we evaluate? All your Access Agreement commitments Demographic access and outcomes Employment trajectory School/Community engagement Crawford et al. 2017 Kate.Wicklow@guildhe.ac.uk

Small/Specialist Challenges Why Evaluate? 27th November 2017 Small/Specialist Challenges There is no widespread planning of evaluation activities during the Access and Participation Agreement planning process. Data is very hard to get hold of. It is prohibitively expensive to engage in national data banks such as HEAT, and buying data from organisations such as UCAS was also deemed to be too expensive for the value they could derive. Outreach managers are ’novice evaluators’. Need more CPD to undertake evaluation more effectively. Much of the outreach activity was delivered outside of WP team – need better levers to ensure everyone understands the importance of evaluation. Far more comfortable in describing a self-evaluation approach, however, very few practitioners can describe a time where they had collected robust data to back up their self-reflections Resources issues means that there is no ‘specialist’ to analyse the data, or academic to take on a formal robust evaluative approach. The work never ends! Kate.Wicklow@guildhe.ac.uk

Theory of Change Kate.Wicklow@guildhe.ac.uk Why Evaluate? 27th November 2017 Theory of Change http://learningforsustainability.net/theory-of-change/ Kate.Wicklow@guildhe.ac.uk

Kate.Wicklow@guildhe.ac.uk Why Evaluate? 27th November 2017 https://oxfamblogs.org/fp2p/what-happens-when-20-middle-east-decision-makers-discuss-theories-of-change/ Kate.Wicklow@guildhe.ac.uk

Developmental Evaluation Why Evaluate? 27th November 2017 Developmental Evaluation Michael Patton developed DE after being asked to undertake an external evaluation of a complex project. “Developmental evaluation is a way of being useful in innovative settings where goals are emergent and changing rather than predetermined.” (Patton, 1994 p318) The model calls for project teams to “collaborate to conceptualize, design and test new approaches in a long term, ongoing process of development, adaptation and intentional change.” (Smith, 1994 p220) “developmental evaluation is first and foremost about doing what makes sense.” grounded in pragmatism and situational adaptation (Patton 2011, p55) Kate.Wicklow@guildhe.ac.uk

Why Evaluate? 27th November 2017 Lessons learned Need more events like this to discuss different approaches Need formal CPD to help develop planning, and evaluation skills No one size fits all model Need better national datasets to draw from Need to share findings and be more collaborative Kate.Wicklow@guildhe.ac.uk

Thank You Any Questions? Kate.Wicklow@guildhe.ac.uk Why Evaluate? 27th November 2017 Thank You Any Questions? Kate.Wicklow@guildhe.ac.uk