Great excavations Unearthing institutional datasets to strengthen and inform university practice Clare Milsom, Martyn Stewart & Elena Zaitseva Liverpool John Moores University

National Student Survey Qualitative comments National Student Survey Satisfaction scores Module feedback First destination statistics Progression data Completion data Benchmarking Grade profiles League tables Enrolment data Demographic data External examiner comments Student feedback Student Union Surveys Attendance data

Problems with combining data sources Organisational separation Conceptual separation & validity Poor representation

A role for institutional researchers to transform information into knowledge, applying analytical approaches to uncover the knowledge residing in data, and improving the processes through which this knowledge is transferred throughout the university Terenzini 1993; Davenport & Prusak 1998; Teodorescu 2006, Yorke 2009 ‘The university is awash with data: but is it awash with institutional intelligence?’ Longden & Yorke (2009, p66) Unearthing meaning in data

CASE STUDY 1 : Library Sources of data: 1.Mirror NSS: first and second - February 2.NSS 2011: third year students – Jan to April 3.Student registration survey 2011: all students – end Sept mid Oct 4.Student enrolment 2011: all students

CASE STUDY 1 : Library NSS and ‘Mirror’ Surveys: two most semantically important themes words words words Year 1 (mirror)Year 2 (mirror)Year 3 (NSS) FavourableCourseFavourableCourseFavourableCourse Unfavourable Lectures Unfavourable Time Unfavourable Library Total 3533 respondents 1191 students left positive comments 1318 students left negative comments

CASE STUDY 1 : Library computers work students coursework time feedback Lack facilities

CASE STUDY 1 : Library library is great and has fantastic atmosphere (first year student) library resources are fantastic! (first year student) Library's & library staff are great-good range of electronic resources & staff always willing to take time to help-even if you feel silly asking obvious question they sort of make up for the lack of enthusiasm & investment in resources elsewhere would give 10/10 defo. Big up library!!! (first year student) The library and computer resources have always been reliable, even off campus and any problems have been dealt with quickly. Admin staff are always helpful and on hand. (second year student) The Library resources, including staff are supportive to my learning needs. (second year student) I'm grateful for the library and its resources, I make full use of them when I have an assignment to do. (second year student)

Lack of working computers at crucial times in the year, also the regular broken printers makes it a long and difficult process to work within the library. (second year student) Lack of available/working computers between the hours of 11am-5pm in the library has occasionally disrupted work, especially during group projects where meet up time is restricted. (second year student) Also, staff in the library are sometimes a bit rude and speak to you like you are in primary school, it’s quite patronising. (second year student) Yes within the library silent zones there is still too much talking, this impacts on my abilty to concentrate (first year student) just lack of space in the library to complete work and print work off nearer to deadline dates and around dinner times (first year student) The Electronic Library can be confusing especially when searching through journals for a particular topic (first year student) CASE STUDY 1 : Library

FavourableNeutralUnfavourable Provided, needs and coursework Closely linked to UniversityComputers, available, people, assignments Law, English, Business, Finance Health, Nursing, Pharmacy, environmental sciences, Forensic Education, ITT, Outdoor education, Dance, Tourism, CASE STUDY 1 : Library

Aldam Library associated with university Aldham Library being open 24 h is appreciated PC access remains a problem - lot of them (PCs) are out of order; waiting time to get access to computer – minutes (15 mins being mentioned) Lack of up to date books; too much old material in the library CASE STUDY 1 : Library

Avril Students’ life is placement – centred: Library and IT facilities not available over Christmas, [but] we had assignments to do. Think it is really hard having to do assignments and revise for exams while on placement! Avril Course associated with skills. University close to placement. CASE STUDY 1 : Library

Marsh Course associated with university Marsh The library isn't big enough – lack of general workspace and computers (40% of comments); waiting time to get access to computer – hour(s) Resources (mainly books) are limited or non- available - There are daily fights in the libraries for resources (30% of comments) Level of support/help should be geared towards needs of a particular student - we need some input from real people in the LRC who are switched on and not IT specialists whom expect you to grasp it first time; moody library staff CASE STUDY 1 : Library

Student registration survey 2011: opens comments relating to support StaffFavourable Unfavourable ModulesUnfavourable CASE STUDY 1 : Library

Favourable Friendly Fast Efficient Helpful Enrolment open text comments CASE STUDY 1 : Library

CASE STUDY 2: ‘Feedback on my work has been prompt’ National Student Survey (Q7): ‘ Feedback on my work has been prompt Low satisfaction score (sector-wide & institutionally). Many universities, including LJMU, introduce feedback turn-around policy Rise in satisfaction score of institution (5 percent points) & in 53% of programmes, but persists as area of most dissatisfaction

Predictors of ‘Overall satisfaction’ : Year 1 and 2 Student Experience Survey 2012

Relational frequency analysis of the content of free text comments from the 2011 National Student Survey, LJMU, generated by the semantic analysis software Leximancer Tm

‘Promptness’ & ‘late’ tied up in wider conceptions of utility of feedback Interventions wider than policy intervention: Better communication of function of feedback, Feedback as development: formative feedback, feed-forward, Make programme assessment ‘maps’ visible. CASE STUDY 2: ‘Feedback on my work has been prompt’ ‘But when I mean late feedback I mean it’s too late in the course..’ Well the work is handed in – so what’s the point of the feedback? ‘ ‘Can’t change the work’. It’s too late to do anything with it’

Data are representations of reality. Some better than others. - Combined data – unveiling ‘real-world’ complexities: interactions, perceptions, priorities - Looking beyond simplistic procedural interventions; cultural and behavioural change

‘a fraught juggling act between ‘economies of performance’ (results; league tables; KPIs) and ‘ecologies of practice’ (situated ethos and ideals; cultures & practices) Stronach et al (2002), plus Harris (2005), Clegg (2008), Cheng (2010), Gale (2011), Mathieson (2011) Supporting staff in use of data for enhancement: Studies on development of professional identity in a performance culture….

Cheng, M Audit cultures and quality assurance mechanisms in England: a study of their perceived impact on the work of academics, Teaching in Higher Education, 15, no.3: Clegg, S Academic identities under threat. British Educational Research Journal, 34, no.3, 329–345. Davenport, T.H. & Prusak, L. (1998). Working knowledge: how organizations manage what they know. Boston, MA: Harvard Business School Press Gale, H The reluctant academic: early-career academics in a teaching-oriented university, International Journal of Academic Development, 16, no. 3: Harris, S. (2005). Rethinking academic identities in neo-liberal times. Teaching in Higher Education, 10, no. 4: 421–433. Longden, B. & Yorke, M. (2009) Institutional research: what problems are we trying to solve? Perspectives: Policy & Practice in Higher Education, 13 (3), pp

Mathieson, M Developing academic agency through critical reflection: a sociocultural approach to academic induction programmes, International Journal for Academic Development, 16, no.3: Stronach, I. B. Corbin, O. McNamara, S. Stark and T. Warne Towards an uncertain politics of professionalism: teacher and nurse identities in flux. Journal of Education Policy. 17, no. 1: Teodorescu, D. (2006) Institutional researchers as knowledge managers in universities: envisioning new roles for the IR profession, Tertiary Education & Management, 12, pp Terenzini, P.T. (1993) On the nature of institutional research and the knowledge and skills it requires, Research in Higher Education, 34, 1-10.