Institutional Research What problems are we trying to solve? Mantz Yorke & Bernard Longden

Sorting out types of problems “Reports that say that something hasn't happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns the ones we don't know we don't know.” Donald H. Rumsfeld, Department of Defence - news briefing, February 12, 2002

The challenge facing any University is to Be smarter and better manage the known knowns; Inquire into university life to seek out the known unknowns and Reduce the risk of unknown unknowns as they relate to: Demographic changes National policy drivers International opportunities

Types of problem : Types of institutional research What’s going on type of research Why is it going on type of research ExplanatoryDescriptive Without theory, research is impossibly narrow. Without research, theory is mere armchair contemplation. (Silverman, 2001: 294) Two broad types of research

Institutional research - four faces Volkwein (1999) has provided a useful analysis of the aims of IR – The four Faces of IR. Research for improvement Information Educational research Audience Role Administrative, Institutional Formative & internal for improvement Summative & external Accountability Academic, Professional Presentation

Terenzini’s organisational intelligence Contextual intelligence Issues Intelligence Technical / Analytical Intelligence Hierarchy of complexity Factual knowledge analytical skills Major issues or decision areas facing the university Culture – knowledge of what a college is and where it has come from

What problems are we trying to solve? The view from inside - BL Volkwein (1999) has provided a useful analysis of the aims of IR – The four Faces of IR. Information Audience Role Administrative, Institutional Formative & internal for improvement Summative & external Accountability Presentation

Administrative/Formative-known knowns IR as Information Authority Administrative – formative and internal Emerging within institutions as: Information office Management information Planning and development Institutional Research Primary functions: Clarifying the known knowns Describing the university in data terms Educating the university community about itself

What problems are we trying to solve? Information authority ‘Base data’ about the institution – profile and fact book Statutory returns Validating data – is this really us? Size and shape of the institution to assist in setting targets and plans Terenzini’s organisational intelligence level – Technical/analytical intelligence In the UK these are well developed possibly over load the university with data The challenge is to reduce the plethora of tables and cross tabs – to identify Key Questions that are crucial to the organisation for success

What problems are we trying to solve? Presentation facility Marketing the best aspects – advocate role Audit and external accountability Performance measure League tables Terenzini’s organisational intelligence level – Technical analytical intelligence Spin and managing data Political pressure to maximise impact of data Denial about the message data gives Risks and concerns Primary functions: Conform data to agreed descriptors Performance Indicators Satisfy public funding legislative requirements

Student Data flow Student data Students Aggregate early return to HEFCE Data informs funding - audit Individual Summative activity data return to HESA Data passed to HEFCE who recreate December aggregate return But same data also used to inform public through: Publications, commodification of data Rankings

Data sources to assist with the questions UK Higher Education Statistics Agency Published data sets HEDI UCAS Unistat HEFCE (?) National Student Survey US National Centre for Education Statistics IPEDS – universe institutional census NPSAS – National Post Secondary Student aid Study BPS – Beginning Post Secondary - Longitudinal Study B&B – Baccalaureate & Beyond - Longitudinal Study National Science Foundation

Some propositions that we might need to address Should the IR questions be articulated more clearly and data collected in response, or should we continue to collect data and then use what we have for policy decision? Do universities make good use of the data within their own university? Does IR constitute value for money? Is data critical to teachers, administrators, decision makers? Is the emphasis between data collection and analysis appropriate? What lessons can be learned from the research community’s use of data sets in the US?’

Comparison – sometimes interesting UK HESA and US IPEDS – similar objectives and operation US interest in scientific advancement not reflected in the UK –Possibly RAE provides nearest comparison US positively encourages Institutional Research –Association for Institutional Research (US) in conjunction with National Centre for Education Statistics provides funded summer school to encourage analyses of public data UK implicitly discourages Institutional Research –HESA charge for the use of our data!

What problems are we trying to solve? The view from outside - MY Research for improvement Educational research Audience Role Formative for improvement Summative & external Accountability Academic, Professional

Academic/Formative: known knowns and unknowns IR as Research for Improvement Academic – formative and internal Analysing (afresh?) existing data, or investigating institutionally- focused research questions, with a view to influencing practice. Problem: such activity is often scattered within an institution, and unco-ordinated.

Research for improvement Examples: Student feedback, including NSS Outcomes of quality assurance Assessment results, cross-tabulated with demographics, subject disciplines, etc. Incidence of plagiarism Research assessment exercise What problems are we trying to solve?

Academic/Summative: known unknowns IR as Educational Research Academic – summative and external, though with some formative and internal relevance Investigation of issues which are of wider relevance than to a single institution, but which may be triggered by an internal institutional need. Evidence needs to be particularly robust. Potential for dissemination of findings beyond the individual institution, in which case criteria for peer review may well apply.

Educational Research Examples: Pedagogical innovations of various kinds Assessment of achievement in workplace settings Variation in honours degree classifications across subject disciplines and institutional types Relation of PT employment to achievement on FT programmes Students’ experiences of HE (FYE; PT; HE in FE, etc) Non-completion What problems are we trying to solve?

In the end what answer can we give to Gnothi Sauton – ‘Know thyself’?