1. Evaluating WP in in specialist contexts
Why Evaluate? 27th November 2017
Evaluating WP in in specialist contexts
Kate Wicklow
Senior Policy Adviser, GuildHE

2. National Policy Drivers
Why Evaluate? 27th November 2017
National Policy Drivers
Look at specific interventions small and specialist providers undertake
Major barriers to evaluation
Introduction to Developmental Evaluation

3. National Policy Drivers
Why Evaluate? 27th November 2017
National Policy Drivers
Access Agreements – measures access and participation
TEF – measures final grades, learning gain, employment outcomes
OfS Registration Data – will measure both Access and Success
Value for Money – NCOP, Access funding etc.
All require different types of data to show your impact

4. All small/specialist institutions have a problem with small datasets.
Why Evaluate? 27th November 2017
What is a Small/Specialist institution
Less than 10,000 students
Multidisciplinary
A specialist (with all courses falling in 1-2 course codes)
A discipline focused HEI (Finance, agriculture, STEM)
Often rural, costal or suburban
All small/specialist institutions have a problem with small datasets.

5. Work within rural and coastal communities to raise aspirations
Why Evaluate? 27th November 2017
Specific events for students with Physical or Specific Learning Difficulties or Disabilities
Work within rural and coastal communities to raise aspirations 
What are My Options Theatre in Education
Working with Employers to ensure skills are useful to local region
Workshops for creative teachers
Residential summer school for Year 12s (Get Set)
After school creative classes/Saturday Schools
Demystify audition/interview process
Primary age arts/farming classes

6. Why Evaluate? 27th November 2017
Top down approach
“As part of your evaluation strategy we will be looking for increased demonstration of the effectiveness of your access agreement spend, and we will be looking to see that robust evaluation of the outcomes of your activities and interventions, particularly in key strategic areas such as school attainment, has informed your decision on where to invest your access agreement expenditure.”

7. What should we evaluate?
Why Evaluate? 27th November 2017
What should we evaluate?
All your Access Agreement commitments
Demographic access and outcomes
Employment trajectory
School/Community engagement
Crawford et al. 2017

8. Small/Specialist Challenges
Why Evaluate? 27th November 2017
Small/Specialist Challenges
There is no widespread planning of evaluation activities during the Access and Participation Agreement planning process.
Data is very hard to get hold of. It is prohibitively expensive to engage in national data banks such as HEAT, and buying data from organisations such as UCAS was also deemed to be too expensive for the value they could derive.
Outreach managers are ’novice evaluators’. Need more CPD to undertake evaluation more effectively.
Much of the outreach activity was delivered outside of WP team – need better levers to ensure everyone understands the importance of evaluation.
Far more comfortable in describing a self-evaluation approach, however, very few practitioners can describe a time where they had collected robust data to back up their self-reflections
Resources issues means that there is no ‘specialist’ to analyse the data, or academic to take on a formal robust evaluative approach.
The work never ends!

9. Theory of Change Kate.Wicklow@guildhe.ac.uk
Why Evaluate? 27th November 2017
Theory of Change

10. Kate.Wicklow@guildhe.ac.uk Why Evaluate? 27th November 2017

11. Developmental Evaluation
Why Evaluate? 27th November 2017
Developmental Evaluation
Michael Patton developed DE after being asked to undertake an external evaluation of a complex project.
“Developmental evaluation is a way of being useful in innovative settings where goals are emergent and changing rather than predetermined.” (Patton, 1994 p318)
The model calls for project teams to “collaborate to conceptualize, design and test new approaches in a long term, ongoing process of development, adaptation and intentional change.” (Smith, 1994 p220)
“developmental evaluation is first and foremost about doing what makes sense.” grounded in pragmatism and situational adaptation (Patton 2011, p55)

12. Why Evaluate? 27th November 2017
Lessons learned
Need more events like this to discuss different approaches
Need formal CPD to help develop planning, and evaluation skills
No one size fits all model
Need better national datasets to draw from
Need to share findings and be more collaborative

13. Thank You Any Questions? Kate.Wicklow@guildhe.ac.uk
Why Evaluate? 27th November 2017
Thank You
Any Questions?