1. Embedding Institutional Curricular Priorities in the First Year Embedding Institutional Curricular Priorities in the First Year A Case Study from RGU Roger McDermott, Gordon Eccleston, Garry Brindley School of Computing The Robert Gordon University Aberdeen, UK rm@comp.rgu.ac.uk

2. HEA-ICS: 2008 2 Plan of Talk Introduction Institutional Priorities: Where do they come from? The RGU context The School of Computing The Evaluation Problem Adapting the Kirkpatrick model

3. HEA-ICS: 2008 3 Institutional Curricular Priorities External: Social/Political – QA/QE Scottish Funding Council Quality Assurance Agency for Higher Education Subject Benchmarks ELIR – Enhancement Led Institutional Review Higher Education Academy Scottish Enhancement Themes Environmental Demographic change Technological change

4. HEA-ICS: 2008 4 The Ideal FY Curriculum First Year Enhancement Theme 2008 report on Curriculum Design for the First Year. Kathy Bovill, Kate Morss and Catherine Bulley (QMU) tried to identify key features of an ideal first year. Work was based on investigation of perspectives from academic staff students Also consulted previously published literature, case studies.

5. HEA-ICS: 2008 5 Ideal Curriculum (Literature) Orientation of students to increase social and academic engagement, connectedness to university, sense of direction and future career Development of learning skills Student-centred, active learning, through problem- based, project-based and group learning Collaborative learning or learning communities to enhance transferable skills and lend a sense of belonging Formative assessment and feedback Progressive skills development Time and structures for reflecting on learning

6. HEA-ICS: 2008 6 Ideal Curriculum (Staff) Co-ordinated programme level approach Small group work Problem-based learning Student choice Early formative feedback Use most experienced staff to teach first year students Involving students in curriculum design Opportunities for personal contact between students/staff Clear communication between staff and students about all elements of the curriculum

7. HEA-ICS: 2008 7 Ideal Curriculum (Student) More attention on assessment and timely feedback More challenging work Students being involved in curriculum design in a role that is more than just feedback Student participation in designing timetabling and curriculum structure

8. HEA-ICS: 2008 8 The Old RGU First Year

9. HEA-ICS: 2008 9 What was wrong with the Old Structure? Disenchantment in sem 1 disengagement in sem 2. Little daily contact between personal tutors and tutees Little provision for students to learn study skills, group/ team-working. 12 week first semester teaching session Relentless teaching schedule Diagnostic assessment and remediation difficult to schedule due to time constraints.

10. HEA-ICS: 2008 10 An Integrated Curriculum The structure of the new Foundation Year should place strong emphasis on the idea that, as a discipline, Computing is seen as a unified and self-supporting whole. Introductory teaching should, wherever possible, reflect and convey this notion of integration to the student. Integrated Curriculum As a consequence, prominence should be placed on the concept of an Integrated Curriculum.

11. HEA-ICS: 2008 11 An Integrated Curriculum This integration should manifest itself at structural and pedagogical levels, as well as that of student-support: Integration of teaching across more fluid thematic boundaries Integrative assessment Partner-module assessment Synoptic assessment Integration of Student-Support

12. HEA-ICS: 2008 12 The New Foundation Year

13. HEA-ICS: 2008 13 30 credit themes which run over two semesters greater flexibility to manage a balanced and engaging teaching style can deal specifically with issues of transition. Refocus of subject content on three major themes: Software Design and Development, Problem-Solving and Modelling, and Information Systems. Main Points

14. HEA-ICS: 2008 14 2-semester 15 credit module to address soft-skills, collaborative and group-working, PDP, promote employability skills and foster study skills to encourage independent learning. Used (but monitored!) portfolio-based assessment to allow students to build up work over two semesters Promoted the introduction of joint assessment opportunities between the major themes and the collaborative working module. Built time for remediation into the timetable at the start. Used Social Software – Blogs, Wikis, Virtual Social Spaces – to provide a vehicle for collaboration. Other Points

15. HEA-ICS: 2008 15 Other Points Strict attendance monitoring and used participation in formative assessment as a Learning Objective. Set up a dedicated Foundation Year Teaching Team, composed of enthusiastic and approachable staff with proven teaching ability to oversee the academic delivery. Use the same staff to provide pastoral, academic and remediation support for students.

16. HEA-ICS: 2008 16 Assessment Assessment was accomplished by coursework. Group projects – extremely popular with students. introduction of structured peer and self-assessment for the collaborative work, greatly enhanced the degree of peer socialisation in the cohort and seemed to promote the connectedness. 2-semester modules Sem 1 used mainly for quasi-formative assessment (passed through participation in the process) Summative assessment was reserved for Sem 2.

17. HEA-ICS: 2008 17 Results Concern that long modules simply allow students to fail 30 credits at a time rather than 15 credits. Evidence from 2007-8 suggests that this fear may be allayed by rigorous module administration. Individual occasions of assessment were used to demonstrate and record competence and achievement in a variety of outcomes across the different themes. Subsequent assessment occasions also allowed students to exhibit competence in prior learning outcomes which they may not have acquired at their first attempt.

18. HEA-ICS: 2008 18 Results For First Year students who were not failed for non- submission, the 2007-8 pass rates for the three 30 credit modules, based on a similar size cohort (~75), were, on average, 5% higher than the corresponding 2006-7 figure for equivalent pairs of 15 credit modules. The headline figure for non-submissions for the 30 credit modules were somewhat higher than those for pairs of 15 credit modules, but the narrative changed from that of slow disengagement throughout the 2006-7 session, to one where students did not submit because they failed to engage from the outset.

19. HEA-ICS: 2008 19 Results The majority of these were non-progressing students referred from the previous first year cohort who simply did not attend after the second or third week (and who, for administrative reasons, could not be withdrawn). Few students progressively disengaged throughout the year and none did so solely in the second semester. This is in contrast to previous years where failure in the first semester modules was a major trigger for disengagement in the second semester.

20. HEA-ICS: 2008 20 The Evaluation Problem What criteria for success? Institutional Drivers Achievement rates Retention rates What about assessing student learning? What is the nature of the evidence? How do you provide such evidence? What about enhancement of career skills?

21. HEA-ICS: 2008 21 The Evaluation Problem Kirkpatricks Model In order to classify areas of evaluation in business training, Donald Kirkpatrick (1959) created what is still one of the most widely used models. His four levels of evaluation are: Level 1: Reaction - a measure of satisfaction Level 2: Learning - a measure of learning Level 3: Behavior - a measure of behavior change Level 4: Results - a measure of contribution to the organisation Some dispute over whether this is a model or an evaluation taxonomy.

22. HEA-ICS: 2008 22 The Evaluation Problem Kirkpatricks Four Level Evaluation Model The model seeks to measure: Reaction of student - what they thought and felt about the training Learning - the resulting increase in knowledge or capability Behaviour - extent of behaviour and capability improvement and implementation/application Results - the effects on the business or environment resulting from the student/staff performance.

23. HEA-ICS: 2008 23 Kirkpatricks Model Level 1: Reaction of Students to the first year experience Affective Reactions Did the student enjoy the first year? Did they engage with the material? Do they consider the material relevant? Did they like the operational set-up? Did they feel that level of effort required to make the most of the learning was sustainable? Do they feel confident about the practicability and potential for applying the learning?

24. HEA-ICS: 2008 24 Kirkpatricks Model Level 2: Learning Evaluation Increase in Knowledge How closely did the lecturers aims and objectives for students on the course match with the actual learning experience of the students? How far has the students level of competence or expertise in the subject improved over the period of the first year? How far is a positive result for one student reflected across the cohort.

25. HEA-ICS: 2008 25 Level 3: Application of Learning Change in Behaviour Did the students put their learning into effect in subsequent tasks? Were the relevant skills and knowledge used appropriately? Was there any measurable change in the activity and performance of the students and was any change in behaviour and new level of knowledge sustained? Was the student able to transfer their learning to another person? Is the student aware of their change in behaviour, knowledge, skill level? Kirkpatricks Model

26. HEA-ICS: 2008 26 Level 4: Results In the original context, this was the most important (and hardest to assess). Measures the effect on the organisation or environment resulting from the improved performance of the trainee. Typical metrics would be key performance indicators measuring quantifiable aspects of business performance, e.g. numbers of complaints, staff turnover, attrition, failures, wastage, non-compliance, quality ratings, achievement of standards and accreditations, growth, retention, etc. Vital to be able to calculate a Return on Investment! Kirkpatricks Model

27. HEA-ICS: 2008 27 Kirkpatricks Model Levels 1 – 3 can be seen as evaluation at the individual or cohort level. Evaluation in these areas can be related to the usual feedback mechanisms student questionnaires staff and student focus groups statistics relating to student performance What about Level 4? A new T&L initiatives contribution to Institutional success must be a factor in the overall evaluation.