1. The Research Excellence Framework Data and Audit May 2012

2. Presentation outline Citation data The submission system Contextual data REF 4 data Research doctoral degrees awarded Research income data Returning data Audit and data verification procedures

3. Citation data “Those panels using citation data will do so within the framework set out in ‘guidance on submissions’ (paragraphs 131-136). In particular, they will consider the number of times an output has been cited as additional information about the academic significance of submitted outputs. Panels will continue to rely on expert review as the primary means of assessing outputs, in order to reach rounded judgements about the full range of assessment criteria (‘originality, significance and rigour’).” (Panel Criteria and Working Methods, Part 1 paragraph 51)

4. Citation data Sub-panel 1: Clinical Medicine Sub-panel 2: Public Health, Health Services and Primary Care Sub-panel 3: Allied Health Professions, Dentistry, Nursing and Pharmacy Sub-panel 4: Psychology, Psychiatry and Neuroscience Sub-panel 5: Biological Sciences Sub-panel 6: Agriculture, Veterinary, and Food Science Sub panel 7: Earth Systems and Environmental Sciences Sub-panel 8: Chemistry Sub-panel 9: Physics Sub-panel 11: Computer Science and Informatics Sub-panel 18: Economics and Econometrics Will only be used by the following sub-panels:

5. Obtaining Citation data Items matched to Scopus via the submission system Via the web interface When entering individual outputs For whole submissions On bulk import Support from Elsevier available

6. Obtaining Citation data

11. Support from Scopus Elsevier will provide technical support on the matching process during the submissions phase Initial means of support will be by email We will publish further details of support arrangements before the submissions period

12. Contextual citation data Sub-panels using citation data will also receive contextual data about citation rates to help inform their interpretation of the citation counts.

13. Contextual citation data Scopus have assigned journals to ~350 subject categories Contextual data will be provided based on these subject categories, for each year The average number of times items were cited The number of citations required to be in the top 1%, 5%, 10% and 25% of papers in that category The data will also be provided at the level of UOA by combining subject categories, after the submission system closes. Data will be provided for the years 2008 to 2012

14. Contextual citation data (2) HEIs will be given the same data when the submissions system opens, except: Data will be provided for the years 2007 to 2011 UOA level data cannot be provided Unable to assign Scopus subject categories to UOAs until all REF data has been collected. Data will be available as an Excel workbook to download via the submission system. Data will be available shortly after the submission system opens HEIs will have access to the same data as the panels after the close of the submissions system

15. Questions?

16. Environment data All submissions to include data on: - Research doctoral degrees awarded (REF4a) - Research income (REF4b) - Research income in-kind (REF4c ) Definitions are aligned with HESA returns Data relates to whole unit of assessment - not just submitted staff Sub-panels 8, 9, 19, 25 and 26 request specific additional data, to be included within the environment template (REF5)

17. Environment data (2) HESA data will be presented to institutions at three stages: - Data for 2008-09, 2009-10 and 2010-11 in May 2012 - Excel workbook containing data - Guidance note explaining how the data were extracted - Data for 2011-12 in April 2013 - Doctoral degree awarded data for 2012-13 will be available from HESA when the 2012-13 HESA student record is submitted

19. Research doctoral degrees awarded Taken from the HESA student record All instances who successfully complete with: D00: “Doctorate degree obtained primarily through advanced supervised research written up as a thesis/dissertation” D01: “New Route PhD” An award is only counted the first year it is given to an instance An award is not counted if the instance is not closed.

20. Subject mapping Awards data are mapped to: RAE 2008 UOAs Cost Centres Data linked between years back to 2007-08 To map awards to cost centres and RAE UOAs To detect instances awarded multiple times

21. Research income Taken from the HESA Finance Statistics Return (FSR) FSR Table 5b: External research grants and contracts Academic departments (Head 1) Academic services (Head 2) Less any income passed on to other institutions or organisations as part of a collaborative project or subcontracted work (Head 5)

22. Research income-in-kind Data on research income in kind will be provided to institutions from: The UK Research Councils Health research funding bodies: The National Institute of Health Research (NIHR) The Chief Scientist Office, Scottish Government Health and Social Care Directorates The Welsh Government’s National Institute for Social Care and Health Research (NISCHR) The Health and Social Care Research & Development (HSC R&D), Northern Ireland.

23. Income-in-kind data template InstitutionDepartment Researcher title Researcher surname Researcher initials Facility Income-in-kind (£ by academic year) 2008-092009-102010-112011-122012-13

24. Returning data Data should be mapped to REF UOAs Research degrees awarded for all students in those UOAs should be returned, not just those with supervisors returned to the REF Income associated with all staff in those UOAs should be returned, not just those who are submitted REF4 data should align with corresponding HESA, RCUK, health research funding bodies data as closely as possible Validation checks will be run within the submission system software before this data can be submitted

25. Returning data (2) The total number of research degrees awarded submitted by the HEI as a whole should not exceed the total number of degrees awarded provided in the HESA data by: More than 20 and 5% over the period 2008-09 to 2011-12 More 10 and 10% in any AY2008-09 to 2011-12 The total amount of research income submitted by the HEI as a whole should not exceed the total amount of income provided in the HESA data by More than £200,000 and 5% over the period 2008-09 to 2011-12 More than £100,000 and 10% in any AY2008-09 to 2011-12. More than £100,000 and 10% in any category of income over the period 2008-09 to 2011 12 The total amount of research income in kind (REF4c) submitted by the HEI as a whole should not exceed the total amount of income in kind provided by the research councils and health agencies by: More than £100,000 and 5% over the period 2008-09 to 2011 ‑ 12 More than £50,000 and 10% in any AY2008-09 to 2011-12

26. Data adjustments Institutions should contact the REF team as soon as possible where they propose to return environment data that falls beyond the set tolerance limits. Requests should be directed to info@ref.ac.ukinfo@ref.ac.uk All requests must be submitted by 1 June 2013 at the very latest.

27. Questions?

28. Audit & data verification procedures Audit and data verification of REF submissions is necessary to: support principles of fairness and to adhere to good regulation provide sufficient confidence in the quality and reliability of data and assessment outcomes Any part of a REF submission can be audited and institutions should maintain accurate records in order to verify any information provided as part of their submission. Further guidance on audit and data verification will be published in June 2012 Audit procedures will involve the following processes: Routine checks Structured and targeted audits Systematic data comparisons Panel instigated audits

29. Audit – Staff eligibility A sample of randomly selected Category A and C staff will be audited in 1 - 4 units of assessment for every institution Institutions will be contacted promptly following the submissions deadline to request evidence of staff eligibility. Evidence may include employment contracts or other HR personnel records

30. Audit – Individual staff circumstances Submitted details for every individual with a clearly defined reduction in outputs will be examined to check that the reductions has been calculated correctly. Further information may be requested. All complex circumstances will be reviewed by EDAP and further information may be requested to inform judgements. A sample of staff with individual circumstances will be audited from each institution to verify the circumstances declared. Evidence will be largely based on: Staff contracts HR confirmation of leave Confirmation that appropriate evidence has been consulted

31. Audit – Research outputs Audit of research outputs will include a systematic data comparison of: Outputs that may have been returned to RAE2008 Outputs that are listed more than twice within a submission Panels may also raise audit queries on research outputs, these will be followed up, where necessary, by the REF audit team

32. Audit - Impact Routine audit will be carried out on a sample of impacts from each HEI. Queries will be informed by advice from the panels and are expected to peak in March – June 2014 Queries are expected to relate to: Claims made that can be corroborated through independent sources Eligibility of underpinning research

33. Audit – Environment Targeted audit will be undertaken where: The environment data (REF4a/b/c) is close to the system limits OR Where a comparison at subject level or cost centre suggests discrepancies in the allocation of data to REF UOAs If audited, HEIs will need to demonstrate how any additional data is eligible; and/or justify how this has been allocated to REF UOAs.

34. Audit – next steps Guidance on audit and data verification is due to be finalised and published on the REF web-site in June

35. Questions?