Law Sub-panel Generic Feedback - Impact UWE and BAM: maximising research impact 27 April 2016

Feedback Session for Law The sub-panel and assessors The submissions The process The outcomes

The law sub-panel and assessors 17 academic members 2 user members 8 academic assessors, including one who also assessed for SP22, SP23 (Criminology); some expert advisors for foreign language outputs 6 impact assessors

What was assessed Panels judged the overall quality of each submission, made by 67 units Quality of research outputs Impact of research on society The research environment 65% 20% 15% The review was based on data and information about the environment 5,525 research outputs by 1,553 Cat A staff were reviewed (3.36 per person) 225 impact case studies were reviewed

The Process – allocation Impact: Case studies – Assigned to one user member/assessor and two academic members as lead readers Templates – Assigned to one user member and two academic members as lead readers All judgments arrived at individually, then discussed, and then determined in plenary Strict procedures were in place to avoid conflicts of interest Consistency of judgement Calibration exercises were conducted at both main and sub-panel level for each element of the submission, with guidance from MPC feeding into sub-panel considerations

The outcomes: Development of sub-profiles Each impact case study, the impact template, was graded on a scale from 4* - U, using ‘half-marks’ for borderline judgments Each sub-profile for each submission was collectively agreed by the sub- panel and recommended to the main panel for approval. The impact sub-profile was aggregated with environment and research quality into the overall quality profile for each submission, according to the standard weightings NOW ONTO OUTCOMES FOR LAW Each impact case study, the impact template, and each scored section of the environment template was graded on a scale from 4* - U, using ‘half-marks’ for borderline judgments Criteria for impact were reach and significance – panel formed an overall view of that, taken as a whole, rather than assessing them separately. For the templates the panel considered the extent to which a unit’s approach described in the template was conducive to achieving impacts of ‘reach and significance;. 4 = outstanding; 3 = very considerable; 2= considerable; 1 = recognised but modest; U = little or no, impact ineligible or not underpinned by excellent research that produced the submitted impact Each sub-profile (impact, environment and research quality) for each submission was collectively agreed by the sub-panel and recommended to the main panel for approval. The overall quality profile for each submission aggregates the three sub-profiles, according to the standard weightings

Average overall quality profile for submissions in Law (FTE weighted) HIGH LEVEL OUTCOMES FOR LAW 27% of submissions were 4* 45% were 3* 24% were 2* 3% were 1* 1% was Unclassified

The average sub-profiles: Law compared with MPC Profile Type %4* %3* %2* %1* % Unclassified Law MPC Outputs 20.1 21.1 47.1 43.0 28.7 29.8 3.7 5.5 0.4 0.6 Impact 38.3 39.1 41.1 40.3 17.7 16.4 2.4 3.6 0.5 Environment 43.6 39.9 43.1 39.2 11.7 1.6 3.0 0.2 Overall 27 46 42 23 26 4 1 Overall, comparing averages, law did better than MPC (combining 4* and 3*) Impact scores broadly similar to the MPC average = 79.4% for both combining 4* and 3*.

Impact 225 impact case studies Subject-matter included (but was not limited to) criminal justice/human rights regulation of business, commerce, environment medicine and bio-ethics EU and international law Audience primarily policy and law-makers judiciary and the Bar Small number of examples related to public engagement and legal education

Impact case studies: strengths and weaknesses Gave clear indication of what was the underpinning research and explanation for its 2* (at least) quality Understood the distinction between dissemination and impact Gave clear explanation of how the research had been utilised to bring about change or effect Provided clear (and not overstated) account of the reach and significance of the impact When corroborating sources were followed up, they justified the claims made Failed to set out the thread of evidence linking the research to the impact and failed to establish the 2* quality of the research Did not understand the distinction between dissemination and impact Made unconvincing or overstated claims of the reach and significance of the impact When corroborating sources were followed up, they did not back up the claims made

Impact templates: strengths and weaknesses Provided evidence of thought in identifying pathways to impact and a strategic approach reflecting the unit’s own research context and priorities Understood the distinction between dissemination and impact Had (or planned) pro-active mechanisms for linking with end-users Assisted and supported staff to develop impact skills whilst recognising that not all research leads to non-academic impact Offered an unimaginative list of possible ways of engaging with end-users Did not demonstrate a clear understanding of impact Lacked a clear strategic approach to moving forward Were unambitious in their plans for achieving impact in future

Law’s quantitative data for research, compared with Politics Law (1,553 FTE) Politics and International Studies (1,275 FTE) Research income £77.4m £144,180m Doctoral awards 1,586 2,198

Further information www.ref.ac.uk includes: The results and submissions Summary data and analysis Panel overview reports Confidential feedback on submissions was provided to institutions in January Evaluations of the REF are currently being carried out by the funding bodies, with reports due in March