This Article Full Text Full Text (PDF) All Versions of this Article: 0034654309334143v1 79/3/1290    most recent Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via Scopus Google Scholar Articles by Marsh, H. W. Articles by O’Mara, A. Social Bookmarking                         What's this?

Gender Effects in the Peer Reviews of Grant Proposals: A Comprehensive Meta-Analysis Comparing Traditional and Multilevel Approaches

University of Oxford

Lutz Bornmann and Rüdiger Mutz

ETH Zurich

Hans-Dieter Daniel

ETH Zurich, University of Zurich

Alison O’Mara

University of Oxford

HERBERT W. MARSH is a professor of education at Oxford University, 15 Norham Gardens Road, Oxford OX2 6PY, United Kingdom; e-mail: herb.marsh{at}education.ox.ac.uk. He spent much of his career in Sydney, Australia, after completing his PhD at the University of California, Los Angeles. He is widely published, with 350 articles in 70 journals, 60 chapters, 14 monographs, and 350 conference papers (with an h-index of 56). He founded the SELF Research Centre, which has 450 members and satellite centers at leading universities around the world. His academic interests include peer review, self-concept, students’ evaluations of teaching, and statistical methodology.

LUTZ BORNMANN is a postdoctoral researcher at the Professorship for Social Psychology and Research on Higher Education at ETH Zurich. Since 2004, he has published 30 articles in journals listed by Thomson Reuters (with a total of more than 200 citations, h index = 8), including three articles in the Essential Science Indicators as highly cited papers. His academic interests include peer review and bibliometric indicators, especially the h index.

RÜDIGER MUTZ holds degrees in psychology and economics. He is a senior researcher at the Professorship for Social Psychology and Research on Higher Education at ETH Zurich and has completed a PhD in biostatistics at the University of Freiburg, Germany. He is a recipient of a Young Investigator Prize of the German Society for Psychology. His academic interests include methodological questions of research on higher education (e.g., university rankings, h index, course evaluation, multilevel models, meta-analysis).

HANS-DIETER DANIEL is director of the Evaluation Office of the University of Zurich and a professor of social psychology and research on higher education at ETH Zurich. He is the author of Guardians of Science: Fairness and Reliability of Peer Review (1993/2004, Wiley-VCH). His academic interests include social science research methods, research evaluation, peer review, bibliometrics, evaluations of teaching by students, university dropouts, and university rankings.

ALISON O’MARA recently submitted her DPhil (PhD) in education at the University of Oxford, following the completion of a BPsych (Hons) at the University of Western Sydney, Australia. Her DPhil dissertation consisted of a series of meta-analyses examining methodological issues related to multivariate outcomes in the context of multidimensional self-concept. While completing her DPhil, she also worked as a research officer for the SELF Research Centre at the University of Oxford. She currently works as a research consultant in London and continues her collaboration with the SELF Research Centre.

Peer review is valued in higher education, but also widely criticized in terms of potential biases, particularly gender. We evaluate gender differences in peer reviews of grant applications, extending Bornmann, Mutz, and Daniel’s meta-analyses that reported small gender differences in favor of men (d = .04), but a substantial heterogeneity in effect sizes that compromised the robustness of their results. We contrast these findings with the most comprehensive single primary study (Marsh, Jayasinghe, and Bond) that found no gender differences for grant proposals. We juxtapose traditional (fixed- and random-effects) and multilevel models, demonstrating important advantages to the multilevel approach. Consistent with Marsh et al.’s primary study, there were no gender differences for the 40 (of 66) effect sizes from Bornmann et al. that were based on grant proposals. This lack of a gender effect for grant proposals was very robust, generalizing over country, discipline, and publication year

Key Words: meta-analysis • peer review • higher education • bias • gender • validity • generalizability • substantive–methodological synergy

This version was published on September 1, 2009

Review of Educational Research, Vol. 79, No. 3, 1290-1326 (2009)
DOI: 10.3102/0034654309334143


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?