1. Estimating the Quality of Business and Management Journals from the UK Research Assessment Exercise John Mingers, Paola Scaparra, Kate Watson Kent Business School Centre for the Evaluation of Research Performance IMEC 2011, Hong Kong

2. John Mingers Many different journal rankings each with its own biases and prejudices They are based on often arbitrary criteria. They can be by peer review or behavioural (e.g, impact factors) The original Kent ranking was simply a statistical combination of other rankings “Objectivity results from a combination of subjectivities” (Ackoff) Journal quality Journal rankings 1. Journal rankings Why are they so contentious?

3. John Mingers Journal quality Journal rankings Paper quality Researche r quality Paper quality is unknown unless we peer review it – hence the RAE; so is researcher quality – no little Lion mark So we impute them from the journal ranking THEORY 1: The quality of a journal purely reflects the quality of its papers (Editors/publishers/common sense) THEORY 2: Low quality papers may be published in high quality journals and vice versa (RAE) It matters in terms of publication strategy and decision- making Is journal a good proxy for quality?

4. John Mingers 2. Reconstructing the 2008 RAE Grades Submission statistics for the last three RAEs Adapted from Geary et al (2004), Bence and Oppenheim (2004), RAE (2009a) a Totals differ slightly between different sources. Figures for 2008 are after data cleaning as described later 199620012008 No of submissions1009790 No. of staff submitted2300+3000+3300 Total no. of outputs8000+994212575 No. of journal papers (% of total) a 5494 (69%)7973 (80%) 11625 (92%) No. of journal titles127515821639 Mean outputs/journal4.35.07.1 Mean outputs/institution80.0102.5139.7

5. John Mingers Output Type Description199620012008 AAuthored book431285 BEdited book7760 CChapter in book863332 DJournal article797311374 EConference contribution29585 GSoftware31 HInternet publication24318 N Research report for external body 8098 T Other form of assessable output 18422 Total994212575 Number of publications by output type Adapted from Geary et al (2004), Bence and Oppenheim (2004), RAE (2009a). Categories with zero entries have been suppressed

6. John Mingers Pareto curve for the number of entries per journal in the 2008 RAE

7. John Mingers Numbers of journals in the RAE and the ABS list

8. John Mingers 2008 Rank 2001 Rank Journal Name2008 Total 2001 Total ABS Grade ABS subject area 12Journal of Management Studies2191164GEN MAN 28Human Relations171784ORG STUD 33Journal of the Operational Research Society1531133OR&MANSCI 45European Journal of Marketing146903MKT 510Organization Studies144754ORG STUD 614European Journal of Operational Research137613OR&MANSCI 76 International Journal of Operations and Production Management 134853OPS&TECH 89 International Journal of Human Resource Management 133763HRM&EMP 91Journal of Marketing Management1251273FINANCE 911Journal of Business Finance and Accounting125653MKT 114British Journal of Management1081074GEN MAN 1213Work, Employment and Society103644HRM&EMP 137British Journal of Industrial Relations99844HRM&EMP 14Regional Studies974SOC SCI 15Research Policy954SOC SCI 16 Service Industries Journal92592MKT 17Critical Perspectives on Accounting893ACCOUNT 1820Organization83553ORG STUD 19Accounting, Organisations and Society824ACCOUNT 20Journal of Business Ethics813ETH-GOV 20Journal of Business Research813GEN MAN Journals most frequently submitted to RAE

9. John Mingers Initial model (QP1) Let: j index the journals (j = 1.. no. of journals) g index the grades 0* - 4* (g = 0.. 4) i index the universities (i = 1.. no. of institutions) e ig be the estimated proportion of research at grade g for university i p jg be the estimated proportion of the outputs of journal j graded at grade g u ig be the actual proportion of research at grade g for university i n ij be the number of entries of journal j submitted by university i s.t. for each institution (i) and grade (g) for each journal (j) 2.1 The LP Model

10. John Mingers 2.2 Validity of the Results ABS 2009Kent 2007 Geary median ABS 1* or 4* RAE grade0.42 (574)0.37 (575)0.42 (416) ABS 2009.69 (574)0.48 (394) Kent 20070.49 (394) Geary median RAE grade 1* or 4*.61 (183) TypeModal gradeMean gradeNo of outputs No. of institutions Authored book22.3728568 Book chapter22.2433365 Edited book21.876128 External report11.3510235 Other11.3116947

11. John Mingers Journal Title% 4.0% 3.0RAE EntriesABS 2009 1* to 4* FT Top 40 y/n? Management International Review100.00.0273y Journal of the Academy of Marketing Science 100.00.0184 Accounting, Organisations and Society100.00.0824y Harvard Business Review100.00.0193y Journal of Applied Psychology100.00.0334y Abacus100.00.0252 Journal of Marketing Research100.00.0174y Journal of Financial Economics100.00.0394y Academy of Management Review100.00.0304y Fiscal Studies100.00.0132 Review of Financial Studies100.00.0274y International Journal of Industrial Organization 100.00.0144 American Economic Review100.00.0174y Information Systems Journal100.00.0183 Organization Science100.00.0284y MIT Sloan Management Review100.00.0233y Marketing Science98.20.0124y Communications of the ACM91.85.2133 Financial Accountability and Management87.54.3303 European Journal of Information Systems85.713.6493 TOP-20 JOURNALS BASED ON % 4* RECONSTRUCTED FROM RAE OUTPUTS

12. John Mingers ABS GradesRAE Estimated Grades All journal s Journal s not in RAE Journals in RAE and our list All our list Journals not in ABS Journals in ABS and our list 4*10%4%15%17%13%18% 3*24%12%31%29%28%31% 2*37%39%37%28%26%28% 1*27%45%17%23%19%22% 0*3%13%2% GPA2.171.742.432.342.092.41 PROPORTIONS OF JOURNALS IN PARTICULAR RANKS COMPARING ABS WITH RAE

13. John Mingers Conclusions from Table 9 Overall RAE grades were higher than overall ABS grades (cols 1, 4) but this was because of selectivity of submissions This can be seen by comparing the ABS submitted with the ABS not submitted (cols 2, 3) Comparing those journals that are in common the level of grading is very similar (cols 3,6) In the RAE, ABS journals were graded more highly than non-ABS journals (cols 5,6)

14. John Mingers Scattergram showing association between GPA and proportion of an institution’s submitted journals that are in ABS

15. John Mingers There are at least 3 possible explanations of this: Higher % ABS journals Better RAE grades Higher quality of department Higher % ABS journals Better RAE grades “RAE Bias” “Better depts. more mainstream” “Greater selectivity” Higher quality of department

16. John Mingers Journal TitleRAE Grade Entries% 4.0% 3.0% 2.0 Management Science42969.929.01.1 Group Decision and Negotiation4410000 Operations Research Letters4410000 Computational Optimization & Applications4310000 SIAM Journal on Optimization4310000 Journal of Heuristics4310000 Decision Sciences4310000 European Journal of Operational Research31370.098.21.8 Computers and Operations Research3200.067.932.1 Operations Research3200.098.91.1 Journal Royal Statistical Society A3926.244.828.8 Theory and Decision39080.019.9 IEEE Trans. on Systems, Man, & Cybernetics37084.415.5 Mathematical Programming37098.31.6 Advances in Applied Probability35081.018.9 Annals of Operations Research35096.23.7 Journal of Combinatorial Optimization34096.53.4 Computational Statistics & Data Analysis33082.017.9 TOP JOURNALS IN OPERATIONAL RESEARCH RANKED BY IMPUTED RAE GRADE

17. John Mingers 3. Technical conclusions Rankings are just a heuristic device and should not be taken as synonymous with quality We can use the RAE data to reconstruct the judgements they made

18. John Mingers 4. Strategic questions Current measurement regimes are hugely distorting to research: Narrow focus on types of outputs – ie “4*” English language journal articles Narrow focus on types of measurements Narrow focus on types of impact Should we stop now and develop a system that aims to evaluate quality in a variety of forms, a variety of media, through a variety of measures with the ultimate goal of answering significant questions? Adler, N. and Harzing, A-W. (2009) “When Knowledge Wins: Transcending the Sense and Nonsense of Academic Rankings”, Academy of Management Learning and Education 8, 1, 72-95