Presentation is loading. Please wait.

Presentation is loading. Please wait.

Estimating the Quality of Business and Management Journals from the UK Research Assessment Exercise John Mingers, Paola Scaparra, Kate Watson Kent Business.

Similar presentations

Presentation on theme: "Estimating the Quality of Business and Management Journals from the UK Research Assessment Exercise John Mingers, Paola Scaparra, Kate Watson Kent Business."— Presentation transcript:

1 Estimating the Quality of Business and Management Journals from the UK Research Assessment Exercise John Mingers, Paola Scaparra, Kate Watson Kent Business School Centre for the Evaluation of Research Performance IMEC 2011, Hong Kong

2 John Mingers Many different journal rankings each with its own biases and prejudices They are based on often arbitrary criteria. They can be by peer review or behavioural (e.g, impact factors) The original Kent ranking was simply a statistical combination of other rankings “Objectivity results from a combination of subjectivities” (Ackoff) Journal quality Journal rankings 1. Journal rankings Why are they so contentious?

3 John Mingers Journal quality Journal rankings Paper quality Researche r quality Paper quality is unknown unless we peer review it – hence the RAE; so is researcher quality – no little Lion mark So we impute them from the journal ranking THEORY 1: The quality of a journal purely reflects the quality of its papers (Editors/publishers/common sense) THEORY 2: Low quality papers may be published in high quality journals and vice versa (RAE) It matters in terms of publication strategy and decision- making Is journal a good proxy for quality?

4 John Mingers 2. Reconstructing the 2008 RAE Grades Submission statistics for the last three RAEs Adapted from Geary et al (2004), Bence and Oppenheim (2004), RAE (2009a) a Totals differ slightly between different sources. Figures for 2008 are after data cleaning as described later No of submissions No. of staff submitted Total no. of outputs No. of journal papers (% of total) a 5494 (69%)7973 (80%) (92%) No. of journal titles Mean outputs/journal Mean outputs/institution

5 John Mingers Output Type Description AAuthored book BEdited book7760 CChapter in book DJournal article EConference contribution29585 GSoftware31 HInternet publication24318 N Research report for external body 8098 T Other form of assessable output Total Number of publications by output type Adapted from Geary et al (2004), Bence and Oppenheim (2004), RAE (2009a). Categories with zero entries have been suppressed

6 John Mingers Pareto curve for the number of entries per journal in the 2008 RAE

7 John Mingers Numbers of journals in the RAE and the ABS list

8 John Mingers 2008 Rank 2001 Rank Journal Name2008 Total 2001 Total ABS Grade ABS subject area 12Journal of Management Studies GEN MAN 28Human Relations171784ORG STUD 33Journal of the Operational Research Society OR&MANSCI 45European Journal of Marketing146903MKT 510Organization Studies144754ORG STUD 614European Journal of Operational Research137613OR&MANSCI 76 International Journal of Operations and Production Management OPS&TECH 89 International Journal of Human Resource Management HRM&EMP 91Journal of Marketing Management FINANCE 911Journal of Business Finance and Accounting125653MKT 114British Journal of Management GEN MAN 1213Work, Employment and Society103644HRM&EMP 137British Journal of Industrial Relations99844HRM&EMP 14Regional Studies974SOC SCI 15Research Policy954SOC SCI 16 Service Industries Journal92592MKT 17Critical Perspectives on Accounting893ACCOUNT 1820Organization83553ORG STUD 19Accounting, Organisations and Society824ACCOUNT 20Journal of Business Ethics813ETH-GOV 20Journal of Business Research813GEN MAN Journals most frequently submitted to RAE

9 John Mingers Initial model (QP1) Let: j index the journals (j = 1.. no. of journals) g index the grades 0* - 4* (g = 0.. 4) i index the universities (i = 1.. no. of institutions) e ig be the estimated proportion of research at grade g for university i p jg be the estimated proportion of the outputs of journal j graded at grade g u ig be the actual proportion of research at grade g for university i n ij be the number of entries of journal j submitted by university i s.t. for each institution (i) and grade (g) for each journal (j) 2.1 The LP Model

10 John Mingers 2.2 Validity of the Results ABS 2009Kent 2007 Geary median ABS 1* or 4* RAE grade0.42 (574)0.37 (575)0.42 (416) ABS (574)0.48 (394) Kent (394) Geary median RAE grade 1* or 4*.61 (183) TypeModal gradeMean gradeNo of outputs No. of institutions Authored book Book chapter Edited book External report Other

11 John Mingers Journal Title% 4.0% 3.0RAE EntriesABS * to 4* FT Top 40 y/n? Management International Review y Journal of the Academy of Marketing Science Accounting, Organisations and Society y Harvard Business Review y Journal of Applied Psychology y Abacus Journal of Marketing Research y Journal of Financial Economics y Academy of Management Review y Fiscal Studies Review of Financial Studies y International Journal of Industrial Organization American Economic Review y Information Systems Journal Organization Science y MIT Sloan Management Review y Marketing Science y Communications of the ACM Financial Accountability and Management European Journal of Information Systems TOP-20 JOURNALS BASED ON % 4* RECONSTRUCTED FROM RAE OUTPUTS

12 John Mingers ABS GradesRAE Estimated Grades All journal s Journal s not in RAE Journals in RAE and our list All our list Journals not in ABS Journals in ABS and our list 4*10%4%15%17%13%18% 3*24%12%31%29%28%31% 2*37%39%37%28%26%28% 1*27%45%17%23%19%22% 0*3%13%2% GPA PROPORTIONS OF JOURNALS IN PARTICULAR RANKS COMPARING ABS WITH RAE

13 John Mingers Conclusions from Table 9 Overall RAE grades were higher than overall ABS grades (cols 1, 4) but this was because of selectivity of submissions This can be seen by comparing the ABS submitted with the ABS not submitted (cols 2, 3) Comparing those journals that are in common the level of grading is very similar (cols 3,6) In the RAE, ABS journals were graded more highly than non-ABS journals (cols 5,6)

14 John Mingers Scattergram showing association between GPA and proportion of an institution’s submitted journals that are in ABS

15 John Mingers There are at least 3 possible explanations of this: Higher % ABS journals Better RAE grades Higher quality of department Higher % ABS journals Better RAE grades “RAE Bias” “Better depts. more mainstream” “Greater selectivity” Higher quality of department

16 John Mingers Journal TitleRAE Grade Entries% 4.0% 3.0% 2.0 Management Science Group Decision and Negotiation Operations Research Letters Computational Optimization & Applications SIAM Journal on Optimization Journal of Heuristics Decision Sciences European Journal of Operational Research Computers and Operations Research Operations Research Journal Royal Statistical Society A Theory and Decision IEEE Trans. on Systems, Man, & Cybernetics Mathematical Programming Advances in Applied Probability Annals of Operations Research Journal of Combinatorial Optimization Computational Statistics & Data Analysis TOP JOURNALS IN OPERATIONAL RESEARCH RANKED BY IMPUTED RAE GRADE

17 John Mingers 3. Technical conclusions Rankings are just a heuristic device and should not be taken as synonymous with quality We can use the RAE data to reconstruct the judgements they made

18 John Mingers 4. Strategic questions Current measurement regimes are hugely distorting to research: Narrow focus on types of outputs – ie “4*” English language journal articles Narrow focus on types of measurements Narrow focus on types of impact Should we stop now and develop a system that aims to evaluate quality in a variety of forms, a variety of media, through a variety of measures with the ultimate goal of answering significant questions? Adler, N. and Harzing, A-W. (2009) “When Knowledge Wins: Transcending the Sense and Nonsense of Academic Rankings”, Academy of Management Learning and Education 8, 1, 72-95

Download ppt "Estimating the Quality of Business and Management Journals from the UK Research Assessment Exercise John Mingers, Paola Scaparra, Kate Watson Kent Business."

Similar presentations

Ads by Google