1. Funded through the ESRC’s Researcher Development Initiative Department of Education, University of Oxford Session 2.4: 3-level meta-analyses

4.  The odds-ratio is based on a 2 by 2 contingency table  The Odds-Ratio is the odds of success in the treatment group relative to the odds of success in the control group (in the present application males and females)

5. XLS

7. Abstract: Narrative reviews of peer review research have concluded that there is negligible evidence of gender bias in the awarding of grants based on peer review. Here, we report the findings of a meta-analysis of 21 studies providing, to the contrary, evidence of robust gender differences in grant award procedures. Even though the estimates of the gender effect vary substantially from study to study, the model estimation shows that all in all, among grant applicants men have statistically significant greater odds of receiving grants than women by about 7% Gender Differences in Peer Review ** Bornmann, L. (2007). Bias cut. Women, it seems, often get a raw deal in science—So how can discrimination be tackled? Nature, 445(7127), 566. ** Bornmann, L., Mutz, R. & Daniel, H. D. (2007). Gender differences in grant peer review: A meta- analysis. Journal of Informetrics, 1, 226–238.

8. Gender Differences in Peer Review Bornmann et al. conducted a multilevel to meta-analysis based peer reviews (grant applications & fellowship applications): 66 effect sizes from 21 studies and a total of 353,725 applications. They found a statistically significant but small effect in favour of men (an effective odds-ratio of 1.07). However, there was systematic variation in the effect sizes beyond random sampling error, suggesting that their results were not generalizable. Typically the next step would be to consider moderators: type of application (grants vs. pre- & post-doctoral fellowships), discipline, or country. However, they noted that : “Unfortunately, the inclusion of one or more of these characteristics into the calculation of the meta-analysis resulted in models that did not converge in the estimation process. This finding indicated that the model estimation became too complex by considering specific interaction effects or the included characteristics had no influence on the outcome, respectively.”

9. 9 Gender Differences in Peer Review (Meta-analysis data from Bornmann, Mutza, & Daniel,, 2007 Are women disadvantaged in Peer Reviews? Based on 66 outcomes from 21 studies, we evaluate whether there are systematic gender differences in success. Important moderator variables include type of peer review (grants, fellowships), discipline, country, and year.

10. 10 Gender Differences in Peer Review

11. Peer Review Mean: Unwted Study Level

12. Peer Review Mean: Wted by Ntot

13. Preliminary Box Plots: Total Sample Median 75 th %tile 25 th %tile 90 th %tile 10 th %tile Potential Outliers No Gender Difference Unweighted Weighted Median Effect Size slightly in favour of men Median Effect Size slightly in favour of women

14. No Gender Difference Preliminary Box Plots: Type

15. Preliminary Box Plots: Country No Gender Difference

16. Preliminary Box Plots: Discipline No Gender Difference

17. Preliminary Box Plots: Year No Gender Difference

19.  Is only a sample of studies from the entire population of studies to be considered. As a result, do want to generalise to other studies not included in the sample (e.g., future studies).  Variability between effect sizes is due to sampling error plus variability in the population of effects.  In contrast to fixed effects models, there are 2 sources of variance  Effect sizes are independent.

20. Where d j is the observed effect size in study j δ i s the mean ‘true’ population effect size u j is the deviation of the true study effect size from the mean true effect size and e j is the residual due to sampling variance in study j

21.  Like the fixed effects model, there are 2 general ways of conducting a random effects meta-analysis: ANOVA & multiple regression  The analogue to the ANOVA homogeneity analysis is appropriate for categorical variables  Looks for systematic differences between groups of responses within a variable  Multiple regression homogeneity analysis is more appropriate for continuous variables and/or when there are multiple variables to be analysed  Tests the ability of groups within each variable to predict the effect size  Can include categorical variables in multiple regression as dummy variables

23. Conclusions: Small effect size based on both Fixed & Random models. Slightly in favour of females for Fixed effects, slightly in favour of males for random effects Significant study-to-study variation so random effects and search for moderators appropriate.

24. Conclusions: Grants: NS effect in favour of women; Fellowships: significant effect in favour of men (but varies from study-to-study); Within variance NS but Var Comp significant; some study-to-study variation remains (particularly in fellowship applications);

25. Conclusions: Significant differences in favour of men for biomedical (2) and Social Sciences (3); other disciplines NS

26. Conclusions: BIG difference in favour of men in Sweden; smaller differences in favour of men in Germany and Europe; NS differences for other countries. Sweden

28. Website Address to get MLWIN Harvey Goldstein developed the MLWIN statistical package used here and has made many contributions to multilevel modeling, including meta-analysis.

29. 1. Click on the equation 2. make logOR the “y” variable 3. indicate a three level model with L3=study, L2=id, L3=LogOR 4. Click “done” button 1 2 3 4

30. 1.Click “Cons” in the equation 2.Tick “Fixed Parameter” “(study)” & “i(d)” but not “logOR” 3.Click the “done” button 1 2 3

31. 1.Now click “add term” button 2. This will bring up the “X-Variable” select SE (the standard error computed earlier) 3.Tick only the “logOR” box 4. Click “done” 1 2 3 4

32. Now we want to constrain the variance at level 1 to be fixed at 1.0. Under “model” select “constrain parameters”; will bring up “parameter constraint” window 1 1

33. In the parameter constraint window: 1. Click the “random” button; 2.Change “logOR: SE/SE” to 1; 3. Change “to equal” to 1”; 4. “store” the constraints in the first empty column (here “C27”); 5. Click the “attach random constraints” button; 6. Close “Parameter Constraint” Window 6 2&3 4 5 1

34. After Closing the “parameter constraint” window (last slide) Click on “start” button in “equation” window (may have to click estimates button to get values). Compute chi-square value in command interface window Conclusion: The mean effect size (-.101/.040) is significant. The chi-square (389.88) is signif; there is study-to- study variation. explore moderator variables ->pred c50->calc c51=(('logOR'-c50)/'se')**2->sum c51 to b1 = 389.88 ->cprob b1 65 = 5.6052e-045

35. Conclusion: The effect of type (-.196/.052) is highly significant The mean effect size (-.007/.034) NS for Type = grant (intercept) chi-sq (171.34) signif; remaining study-to-study variation.

36. Conclusion: The effect of DISC is highly significant (change in chi-sq = 389.88 -188.59 = 200.29 (df = 4). Men signif more successful than women in SocSci (relative to multidis, the reference category that is NS. ->pred c50->calc c51 = (('logor' - c50)/'se')**2->sum c51 to b1 = 188.59 ->cprob b1 61 =.1875e-014

37. Conclusion: The effect of CNTRY is highly significant (change in chi-sq = 389.88 -158.76 = 189.59 (df = 7). Men signif more successful is Swenden (but note large SE) and Germany relative to US (reference category which is NS). >pred c50->calc c51 = (('logor'-c50)/'se')**2->sum c51 to b1 = 158.76 ->cprob b1 58 = 1.7144e-011

38. Conclusion: The Linear effect of YEAR is NS. Notice that I changed the intercept to be 2000 (rather than “0” – which is completely out of the range. ->pred c50->calc c51 = (('logor' - c50)/'se')**2->sum c51 to b1 = 344.02 ->cprob b1 64 = 6.6568e-038

39. Conclusion: General pattern of results for each variable considered separately still evident. Reference category (Type = grants, Disc = Multi) still NS. Results should be interpreted cautiously because improper solution. Note that solution is technically improper (study level constrained to be non-negative) ->pred c50->calc c51 = (('logor' - c50)/'se')**2->sum c51 to b1 = 105.47 ->cprob b1 60 = 0.00026315

40. Conclusion: The change in chi-sq is NS, suggesting that there is no interaction. Results should be interpreted cautiously because improper solution. Note that solution is technically improper (study level constrained to be non-negative) ->pred c50->calc c51 = (('logor' - c50)/'se')**2->sum c51 to b1= 103.80 ->cprob b1 56 = 0.00010859

41. Conclusion: General pattern of results similar. Men signif more successful is Sweden (but note large SE) and Germany relative to reference category (US Grants).

42. Conclusion: The change in chi-sq is NS, suggesting that there is no interaction. Results should be interpreted cautiously because improper solution.

43. Conclusion: When all main effects are included, Type effect nearly unaffected. However, none of the disc effects are significant, although the Sweden and (marginally) Germany are still significant. Results should be interpreted cautiously because improper solution.

45. Caterpillar plot based on L1 residuals. Go to the “model” menu and select “residuals” option. This will bring up the “settings” window. Set “SD (comparative)” to 1.96; 3. Set “level” to “1logOR”; 4. click the “Calc” button; 5. click on the “plot” button to bring up the next window. In the “plot” window select “residual +/1 1.96SD x rank. This brings up the original graph. Clicking on the graph bring up a window to modify the graph (a bit)

47.  The mean effect size was very small, but significantly in favour of men. However, the results did not generalise across studies (there was study-to-study variation).  The effect size was significantly moderated by the type; it was almost exactly 0 for grants and in favour of men for fellowship applications. This difference was not moderated or mediated by other moderators.  There appeared to be some discipline effects (bias in favour of men in social sciences) and country effects (large bias in favour of men for Sweden). However, when all “main” effects included, discipline effects disappeared.  For Grant Proposals there was no evidence of any effect of gender on outcome.

48.  Purpose-built  Comprehensive Meta-analysis (commercial)  Schwarzer (free, http://userpage.fu- berlin.de/~health/meta_e.htm)  Extensions to standard statistics packages  SPSS, Stata and SAS macros, downloadable from http://mason.gmu.edu/~dwilsonb/ma.html  Stata add-ons, downloadable from http://www.stata.com/support/faqs/stat/meta.html  HLM – V-known routine  MLwiN  MPlus

49.  Bornmann, L. (2007). Bias cut. Women, it seems, often get a raw deal in science—So how can discrimination be tackled? Nature, 445 (7127), 566.  Bornmann, L., Mutz, R. & Daniel, H. D. (2007). Gender differences in grant peer review: A meta-analysis. Journal of Informetrics, 1, 226–238.  Cooper, H., & Hedges, L. V. (Eds.) (1994). The handbook of research synthesis (pp. 521–529). New York: Russell Sage Foundation.  Hox, J. (2003). Applied multilevel analysis. Amsterdam: TT Publishers.  Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park: Sage Publications.  Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage Publications.