1. School of Business and Management Accounting for Research Quality: Research Audits and the Journal Rankings Debate Michael Rowlinson Professor of Organization Studies Co-editor The Association of Business Schools Academic Journal Quality Guide (version 4, 2010) Kent Business School Journal Rankings: Love ‘em or hate ‘em? Weds 30 2013

2. School of Business and Management Distribution of ratings for Business and Management in RAE2001 and RAE2008 RAE20084*3*2*1*Utotal no ftes5541,2221,098427343,337 % ftes17%37%33%13%1%100% RAE20015*543a3b21total no ftes280639664527276157122,555 % ftes11%25%26%21%11%6%0%100%

3. School of Business and Management Criticism of editors of the ABS Guide Biased (Hoepner & Unerman), self-appointed gerontocrats, who know they could never recognize academic excellence for themselves (Burrell) Auto-asphyxiation fetishists who can find time to endlessly revise their list but not for “the critical task of reading” (Willmott) If our academic leaders, such as Research Directors, “actually read lots of material” then they would be able to “recognize quality itself”, but the availability of lists distracts them from the necessary academic task of critical reading (Burrell)

4. School of Business and Management Feedback from the 2008 RAE panel “there was not a perfect correlation between the quality of a piece of work and its place of publication. Although much top-quality work was indeed published in what are generally regarded as leading journals, top quality work could also be found in journals occupying a lower position in conventional rankings. Similarly, some of the work considered that had been published in so-called leading journals was thought to be of less than top quality. The proportions in each of these categories also varied across sub-disciplines. There was also a considerable amount of work published in books or other formats, some of which was of world-leading quality. It would therefore be inappropriate in the future to use assessments of journal quality alone to assign quality ratings to individual items of work”

5. School of Business and Management Should the REF use The ABS List? (we woz robbed in the RAE 2008!)

6. School of Business and Management Four critiques of journal lists Expert peer review is sufficiently reliable so there is no need to consult journal quality lists –Therefore the best way to assess outputs for submission is to read them Variation in citations for articles in the same journal –Therefore the journal citation impact factor is not reliable as a guide to the quality of individual articles Subject Field Bias in Journal Ratings –The ABS Guide is allegedly biased against accounting, operations research, and inter-disciplinary research Journal rankings distort the REF as a research audit – if it wasn’t for journal rankings the REF would work well for resource allocation, which is it’s main function

7. School of Business and Management Was the 2008 RAE more inclusive than the ABS Guide? Proposition 1. –Outputs from more journals received the highest grades, 3 and 4, from the Business and Management RAE2008 panel than would be the case if the ABS Guide is used to rate outputs; Proposition 2. –More journal outputs received the highest grades, 3 and 4, from the Business and Management RAE2008 panel than would be the case if the ABS Guide is used to rate outputs; Proposition 3. –Outputs from more research active staff received the highest grades, 3 and 4, from the Business and Management RAE2008 panel than would be the case if the ABS Guide is used to rate outputs.

8. School of Business and Management

12. Count of ABS 2010 rated jnl outputs from individual staff RAE2008 countABS4ABS3ABS2ABS1nonABS 4118233431378 315841216449150 2320711451152345 1666940950478847 total1262229616086921420 cum total12622779325733683540

13. School of Business and Management

15. Alternative methods for assessing the quality of research outputs (RA2) Individualized citation metrics Informed by Web of Science, Scopus, or Publish or Perish Journal ratings (e.g. The ABS Guide) Informed by citation impact factors, peer review, or a hybrid combination Peer review (e.g. Bus and Mgt REF panel) Informed by expert judgment

16. School of Business and Management Conclusions Is it time to consider a more appropriate, inclusive, and economical form of ranking for research in business and management? A ranking system should recognize the importance of research in relation to teaching in business schools. Journal ratings would have role to play in an improved ranking system.

17. School of Business and Management RAE2008 Cum % bus & mgt outputs in journals