1. Curriculum.leeds.ac.uk SEC/DigiFest 2016 Evidencing Excellence Evaluating the Leeds Curriculum the story so far… Karen Llewellyn Mitch Waterman Martin Purvis 1 http://curriculum.leeds.ac.uk/

2. Evaluating the Leeds Curriculum Workshop plan 1.Brief overview of the Leeds Curriculum 2.Introduction to Programme Theory Evaluation 3.Evaluation undertaken so far 4.Group reflections and discussion on findings 5.Summary and next steps 2

3. Rationale for change in 2010  HE landscape changing: fees increase, increased competition from UK and overseas  Shift in student motivation: value for money, greater focus on employability  Development of more rounded graduates to compete more effectively in the workplace who can meet employer demands  Provision of greater clarity around the information and guidance which informs student choice, greater scrutiny with KIS  Improved articulation of what a Leeds degree can offer in an increasingly competitive market: what makes us and our graduates distinctive 3

4. Enhancing the Leeds Curriculum Original 8 objectives 1.Ensure that research is integrated with learning and teaching in all programmes and that staff and students understand what we mean by this 2.Provide cutting-edge curricula which maintain the University’s position in attracting high quality applicants in an increasingly competitive market 3.Provide the opportunity in all programmes for students to broaden their study within or outside the main discipline area 4.Provide clarity on what a prospective and current student can expect from a University of Leeds education 5.Enhance the employability of our graduates through the development of programmes which challenge and prepare our students to meet the demands of the outside world 6.Make efficient use of learning and teaching space 7.Provide a structure for the academic year which best facilitates research and learning and teaching of the University 8.Ensure the sustainability of learning and teaching going forward 4 Were these (are these?) the right objectives?

5. Embedding the Leeds Curriculum 2015 Learning in a research environment 1.Research-based Learning (including the FYP) 2.Core Programme Threads (CPT) Ethics and Responsibility Global and Cultural Insight Employability 3. Broadening 5

6. Leeds Curriculum: Broadening Broadening - within and/or outside a degree programme: Through 10 Discovery Themes 1. Creating Sustainable Futures 2. Enterprise and Innovation 3. Ethics, Religion and Law 4. Exploring the Sciences 5. Language and Intercultural Understanding 6. Media, Culture and Creativity 7. Mind and Body 8. Personal and Professional Development 9. Power and Conflict 10. Technology and its Impacts 6

7. Embedding the Leeds Curriculum 2015 Plus Assessment – mapping assessments to programme level learning outcomes Communications - to both internal and external audiences Moving from project to business-as usual 1.Measure success – short and long term evaluations 2.Disseminate findings - to sustain continuing enhancement Q: Is there anything about the Leeds Curriculum that is new to you? 7

8. Evaluating the Leeds Curriculum How to evaluate?  Before vs after comparison?  Data/evidence issues  Which stakeholders?  Complex emergent issues  The context changing Complex, dynamic, extended in time, multiple stakeholders: Programme Theory Evaluation (e.g. Pawson, 2006, Rogers, 2008, Westhorp, 2012) 8

9. Evaluating the Leeds Curriculum Programme Theory Evaluation: An example of a logic model (Taylor, et al. 1996) 9

10. Evaluating the Leeds Curriculum Programme Theory Evaluation 1.The ‘Programme’ is the change, or intervention that we are seeking to achieve. 2.The ‘Theory’ is how we think this will work, what it will do. 3.The evaluation is the assessment of the degree to which the ‘theory’ holds true. 4.Critical elements; describing the programme detail, and the theory. 5.Differentiating between assessment of the success of actually implementing programme elements (e.g. Broadening, RBL, CPT), and assessment against objectives. 6.The evaluation is a framework that surrounds this, and in common with most evaluations of large scale transformative change, develops as the programme unfolds. 10

11. Evaluation: the story so far … Short Term Evaluation: did we get CPT/RBL in place?  CPTs (Ethics & Responsibility, Global & Cultural Insight, Employability), agreed, mapped, identified in most schools.  RBL mapped (somewhat vaguely) in all schools.  CPT/RBL assessment performed by former RBL/CPT Working Group, and followed up in Programme Reviews and Annual School Reviews.  Detailed guidelines and checklist for Final Year Project confirmed and circulated; all schools now have mandatory FYP (of many different forms) in place for students starting Sept. 2014.  FYP supervision training in place for schools where compulsory FYP not historically a requirement. 11

12. Evaluation: the story so far … Short Term Evaluation: did we get Broadening in place?  Discovery Themes agreed  Theme Leaders appointed  Agreed flexibility in broadening models – most programmes via student choice in accessing discovery modules – or by demonstration of exposure to Discovery Themes in specified programme content  Addressed information deficit – Broadening website (within LfL) and Fairs 12

13. Evaluation: the story so far … Short Term Evaluation: did we get Shape of the Academic Year and Assessment in place?  No! Several rounds of consultation led the Assessment Working Group to recommend suspending the AY element.  WG recommended auditing of assessment to Programme, Level and Module learning outcomes; underway (to be reported in forthcoming round of ASRs). Did we get Communications in place?  In part, but given diversity and scale, a challenge.  Some tensions between LC comms objectives and those of Marketing/Comms more generally  In reality, had to rely on those involved. 13

14. Evaluation: the story so far … Short Term Evaluation: did we get Systems and Process in place?  Effectively, yes (though see issues below).  New drafting forms.  Amendments to online enrolment.  QA checks through ASRs and Programme Reviews.  New ‘controls’ written into the planned Integrated Programme Administration System (IPAS). 14

15. Evaluation: Issues so far … Given LC’s scale and development over early years, always likely to be issues ….  Discovered that not all programmes had been mapped for CPT/RBL.  Initial recommendation for assessment auditing softened, leading to delays  Assessment WG devoted too much time to shape of the AY, only latterly focusing on Assessment  Some programme publicity made no mention of a new compulsory FYP so had to delay  Compulsory nature of FYP meant that some programmes require amendment for Methods training 15

16. Evaluation: Issues so far … Given LC’s scale and development over early years, always likely to be issues ….  Challenges associated with JH FYP (credit weighting, supervision, interdisciplinary, funding, allocation)  LC Project funding meant ‘early’ transition to business as usual  Revised deliberative structures make governance more complex 16

17. Evaluation: OUTCOMES so far … There has not as yet been a complete cohort to complete the LC, however, some initial outcomes can be considered 17 Most progs identify CPT/RBL FYPs in place for all UG students Assurances about relationship between LOs and assessment still to be determined Discovery Themes in place, students choosing Test through focus groups and other stakeholders (e.g. employers) Test through profiles, focus groups and stakeholders Assess duplication, gaps, amend programmes, IPAS confirmation

18. Broadening Outcomes: Overall  Positive student feedback on Broadening  Little evidence of damaging student frustration about lack of capacity/timetable clashes  Increased L2 discovery module enrolment  Increased (but still uneven) moves to accommodate discovery modules in UG programmes  BUT: uneven internal awareness amongst staff/students  Limited profile in marketing for student recruitment 18

19. Curriculum.leeds.ac.uk Discovery Module Enrolments 2015-16: by Theme and Faculty * School of Computing only; ** School of Psychology only ArtsFBSLUBSENG*ENVESSLMAPSM&H**PVAC Creating Sustainable Futures 1761824541498936319 Enterprise & Innovation 240195821463106611559 Ethics, Religion & Law 12703625731864407211173 Exploring the Sciences 1889110913135413653524 Language & Inter- cultural Understanding 9728121457226801397 Mind & Body 36611816749120021324067 Media, Culture & Creativity 704984228115346193 Personal & Professional Development 48237608188616420190131 Power & Conflict 1099817119941020346 Technology & its Impacts 2211510224349124237

20. Broadening Outcomes: Information  Broadening website/fairs – well-used and appreciated  BUT: not equally well-publicised by all Schools to incoming students  Website not integrated with timetabling information and on- line enrolment 20

21. Broadening Outcomes: Themes  Enable students to appreciate and navigate the breadth of available opportunities  Successful as means to review UoL’s existing offer and encourage new modules/flexible delivery  Confirms importance of Theme Leaders in defining/executing academic vision 21

22.  BUT: cannot yet fully document impact on student behaviour/attitudes  Patchy communication between Schools and Theme Leaders  Only a minority of modules delivered flexibly  Limited competition for DTL posts  Awaiting RAM revisions to facilitate inter-disciplinary teaching  Question visibility of alternative models of Broadening ‘within’ degree programmes 22 Broadening Outcomes: Themes

23.  Large group, all stakeholders including students  Internal and external experts on PTE  Reports to new Education Steering Group  Likely to continue working till two cohorts have completed  Project Manager and part-time Project Officer  Honest, detailed, confident evaluation: what went well, what did not, did the ‘Programme’ work, was the ‘theory’ sound?  Lots of interviews and focus groups 23 Evaluation: The Evaluation Group

24. 1. Given the fact that only some ‘outcomes’ are as yet identifiable, what do you make of the story so far? 2. What are your thoughts on the evaluation approach? 3. Who should the Evaluation Group be speaking with and about what? 4. Is there anything missing? 5. Would you like to be involved further? 24 Evaluation: Reflections

25. 1. The Evaluation Group now has a description of the Programme and the Theory (though heavily influenced by what’s happened, and arguably parochial) 2. Focus group and interview activity with multiple stakeholders 3. Recommendations to Education Steering Group 4. Heavily influenced design of IPAS, to try to assure some future proofing 5. A series of reports and publications 6. What more should there be? 25 Evaluation: Next Steps

26. Leeds Curriculum Evaluation References Pawson, R. (2006) Evidence Based Policy: A Realist Perspective London Sage Publications Pawson, R. (2010) Middle range theory and programme theory evaluation: from provenance to practice, in J. Vaessen and F.L. Leeuw (eds) Mind the Gap: Perspectives on Policy Evaluation and the Social Sciences. New Jersey Transaction Publishers Rogers, P. (2008) Using Programme Theory to Evaluate Complicated and Complex Aspects of Interventions. Evaluation 14(1): 29-48 Taylor-Powell, E., Steele, S., & Douglah, M. (1996). Planning a program evaluation. Retrieved December 2015, from University of Wisconsin- Extension-Cooperative Extension, Program Development and Evaluation Unit Web site: http://www.uwex.edu/ces/pdande/evaluation/evaldocs.html http://www.uwex.edu/ces/pdande/evaluation/evaldocs.html Westhorp, G. (2012) Using complexity-consistent theory for evaluating complex systems. Evaluation 18(4): 405-420 26

27. Curriculum.leeds.ac.uk SEC/DigiFest 2016 Evidencing Excellence Thank you Any comments/questions, please email: k.a.llewellyn@leeds.ac.uk 27 http://curriculum.leeds.ac.uk/