Scholarly Communication

Owning the peer review process

If we have to do this work, we should own it

Charlotte Roh is instruction and reference librarian at California State University-San Marcos, email: croh@csusm.edu

As a new reference librarian, one of the most common questions I’ve encountered over the past year is some variation of “How do I find a peer reviewed scholarly article?” While the goal of professors who’ve assigned this task is to educate students on the nature of academic research, students are usually unclear on what “peer review” means and how to find this information.1 Often it is quite literally just a check box item amongst the facets.

On its face, peer review as a means of authentication seems valid. Why not have a qualified scholar or two review a work to see if it is accurate and relevant for publication? Unfortunately, as we know from countless studies, peer review is subject to systemic and individual bias across multiple genres and forms2 that undermines the legitimacy of the process. Within the open access movement, this has led to calls for more transparency. Scholars see a move to open peer review,3 especially as part of the movement for open science, as a solution to the inefficiencies and inequities of a closed process. Certainly, the technologies of publishing on the open web now make this easier than ever, with annotation tools, commenting features, and automation, and this would presumably solve for bad actors who take advantage of the peer review process to enact academic theft, fraud,4 or private retaliation.

However, some scholars are uncomfortable with open peer review as a solution. One study concluded, “Asking reviewers to consent to being identified to the author had no important effect on the quality of the review, the recommendation regarding publication, or the time taken to review, but it significantly increased the likelihood of reviewers declining to review.”5 Clues as to why scholars might be reluctant to be identified can be found in yet another study, where reviewers were asked whether they would agree to have their names revealed to the authors. Among those who agreed “reviews were of higher quality, were more courteous and took longer to complete.”6 This speaks to a fundamental problem in the culture of peer review: It is quite often critical and negative, sometimes even cruel. Transparency alone cannot solve the existing systemic issues and the existing culture around this work that is critical by nature.

The trope of “Reviewer #2” has spawned memes,7 unhappy tweets,8 and a Facebook group with more than 55,000 members called “Reviewer 2 Must Be Stopped!” In 2020, in response to this trope, an editor analyzed their own journal’s database and concluded that Reviewer #2 is no different from Reviewer #1 but that Reviewer #3 is much more likely to be negative.9 We can conclude that what is important is not the differences between reviewers, but the overall acceptance/rejection rates for journals that demonstrate how peer review is being used as a gatekeeping function, and not necessarily responsibly.

In response to the many known issues in peer review, there are those in the publishing ecosystem that would like to see it as a constructive and positive act rather than an unnecessarily negative one. Two examples of how this can be achieved are In the Library with the Lead Pipe and the new library publication up//root.10 In the Library with the Lead Pipe is transparent about the identities of authors and reviewers, communicative and flexible about publishing timelines, and their process includes a shared document on which reviewers and authors can all see the comments. The editors of up//root have been clear that they see peer review as relationship,11 and work to center the work of Black, Indigenous, and People of Color (BIPOC) in order to give reviewers and contributors agency and well-being through flexible publishing timelines, constructive feedback, and input into who will be working on an article. While up//root is able to pay both their authors and peer reviewers through a grant from SPARC, this is a rarity in academic publishing, nor is the publication strictly an academic one.

Implementing this kind of care and thoughtfulness to make the peer review process better requires time and education, and to address this every major publisher and quite a few associations have trainings, workshops, and certificates on how to properly undertake an academic review. The American Geophysical Union (AGU) has made a splash for their concerted efforts to increase diversity in their peer review demographics, mentor early career researchers who are learning to review, and expand the diversity of editorial boards.12 I am also particularly inspired by the participatory document “Anti-Racist Scholarly Reviewing Practices: A Heuristic for Editors, Reviewers, and Authors,”13 which serves as an excellent checklist for those who want to engage in anti-racist practices. The most recent Peer Review Week14 included events on diversity, identity, empathy, and the role of early career researchers. Peer Review Week is also a celebration of peer review and its role in scholarly communication and has in its brief history included themes on diversity, transparency, and recognition for reviewers.

While this is all good, I am discomfited by this insistence that peer review is to be celebrated and defended because 1) it is a verification process, not an ethical imperative; 2) it is still very much unpaid labor; and 3) its validation ultimately serves commercial publishers that facilitate and profit from this labor. Peer review is an essential part of a Western model of scholarly communication that is being exported globally as necessary and good,15 and this is important to publishers because, as several scholars have pointed out, there is a labor shortage in the peer review system, and what labor is being done is inequitably distributed.16 Like many inequities and inefficiencies, the pandemic surfaced this peer review labor shortage to wider awareness in the academic community. Journals rushed to turn around submitted COVID-19 research as the virus moved around the world, but it wasn’t fast enough for many who turned to preprint servers.17 This move to preprint servers allowed data scientists to determine that women were not contributing as authors and researchers in the deluge of pandemic papers.18 Presumably this means they were left out of the peer reviewer pool as well, and as we now know because of AGU, this does make a difference as to whether women are published. This is not a surprise to anyone following how the pandemic has impacted women inequitably.

This brings us back to the subject of unpaid labor, and how peer review depends on it. We teach scholars that being a good member of the scholarly community means participating in peer review, but people are unpaid, untrained, and unappreciated for it. It’s understandable that people are cranky in response (perhaps we have all, at some point, been Reviewer #2). In other publishing industries, this kind of review is the work of paid consultants, fact checkers, and readers. In academia, editors are encouraged to send thank you notes. According to a recent study, in 2020 peer reviewers around the world worked more than 100 million hours. “The estimated monetary value of the time US-based reviewers spent on reviews was over 1.5 billion USD in 2020. For China-based reviewers, the estimate is over 600 million USD, and for UK-based, close to 400 million USD.”19 It’s no wonder that the scholarly communication enterprise is invested in celebrating and encouraging “good” peer review. Its value is immense.

It’s also no wonder that we haven’t seen much change around the biases inherent in the process. Forthcoming research on academic library journals indicates that we as a field have not incorporated equity, diversity, and inclusion values in our policies, and this is particularly true for journals that are owned by commercial publishers.20 I have mentioned Lead Pipe, up//root, and AGU as examples of when things go well, but these are also examples that literal ownership of the publishing endeavor can be correlated with ownership of a journal’s inner workings. For most publications, trainings, and processes to improve peer review are rarely implemented because it is harder work and a greater investment to train someone to be a good reviewer than it is to increase the labor pool to include the necessary skillset.

Recently, the American Psychological Association decided to track the demographics of their peer reviewers in order to diversify their process. This is well-intentioned, and I will not argue that this isn’t important and necessary—it very much is. But the response was suspicion21 because surveillance and racism are very much a part of platform capitalism (i.e., platforms that are owned by commercial entities),22 and peer review is an integral part of scholarly communication capitalism. When we in the United States don’t own our processes and platforms, there is a greater risk of upholding and exporting our white supremacist capitalistic patriarchal colonialist structures.23 We have seen this with the open access movement, where well-intentioned goals and values have been easily co-opted to justify the expansion of Western companies and Western models—possibly because the new tools and structures of open access are often replicative of the traditional ones.

In summary, if we persist in peer review as the process by which research is verified, we need to own and acknowledge this volunteer labor more highly while surfacing the ways in which it fails.24 We should also acknowledge that one of the ways we have failed is in supporting a norm around unpaid work while other major parts of the scholarly ecosystem have changed—namely capture of the ecosystem by commercial publishers with large profit margins. I believe the solution is to both change the norms around payment for this work and to own the process and platforms for ourselves so that we are doing the work for ourselves.


  1. Kristiana Santiago, co-editor of up//root, has called peer review “just a way to get students to accept what they’re reading.” M. Abebe, K. Santiago, and S. Leung (October 26, 2021) Peer Review as Relationship, Open Access Week, Indiana University, https://events.iu.edu/libraries/event/232283-peer-review-as-relationship.
  2. C. J. Lee, C. R. Sugimoto, G. Zhang, and B. Cronin, “Bias in peer review,” Journal of the American Society for Information Science and Technology 64: 2-17, https://doi.org/10.1002/asi.22784.
  3. T. Ross-Hellauer, “What is open peer review? A systematic review,” F1000Research, 6, 588, https://doi.org/10.12688/f1000research.11369.2.
  4. For examples of poor behavior in peer review, I recommend Retraction Watch’s website and database, which includes retractions for plagiarism by peer reviewers, illegitimate/faked peer review, and abuses of power, https://retractionwatch.com/.
  5. S. van Rooyen, F. Godlee, S. Evans, N. Black, R. Smith, “Effect of open peer review on quality of reviews and on reviewers’ recommendations: A randomized trial,” BMJ 318:23, https://doi.org/10.1136/bmj.318.7175.23.
  6. E. Walsh, M. Rooney, L. Appleby, and G. Wilkinson, “Open peer review: A randomised controlled trial,” British Journal of Psychiatry 176(1), 47-51, https://doi.org/10.1192/bjp.176.1.47.
  7. u/legendofthekarma, Reviewer 2... Digital Image, https://www.reddit.com/r/labrats/comments/el60yo/reviewer_2/, January 12, 2021.
  8. Tweets: Search results with the term #Reviewer2 on Twitter, https://twitter.com/search?q=%23Reviewer2&src=hashtag_click.
  9. David A. M. Peterson, “Dear Reviewer 2: Go F’ Yourself,” Social Science Quarterly 101.4 (2020): 1648–1652.
  10. A disclaimer that I have a forthcoming publication with In the Library with the Lead Pipe and am an advisory board member for up//root.
  11. For more on this see the abstract of their recent talk at https://events.iu.edu/libraries/event/232283-peer-review-as-relationship.
  12. M. Giampoala, R. Townsend, and P. Wooden, Guest Post—“Why the ‘Who’ of Peer Review is Important, The Scholarly Kitchen, https://scholarlykitchen.sspnet.org/2021/09/22/why-the-who-of-peer-review-is-important/.
  13. “Anti-racist scholarly reviewing practices: A heuristic for editors, reviewers, and authors,” retrieved from https://tinyurl.com/reviewheuristic.
  14. See https://peerreviewweek.wordpress.com/.
  15. Publons holds a Global Peer Review Award during Peer Review Week, and I do recommend looking at their report on the global state of peer review: https://publons.com/community/awards/2019.
  16. M. Kovanis, R. Porcher, P. Ravaud, and L. Trinquart, “The Global Burden of Journal Peer Review in the Biomedical Literature: Strong Imbalance in the Collective Enterprise,” PLOS ONE, https://doi.org/10.1371/journal.pone.0166387.
  17. H. Else, “How a torrent of COVID science changed research publishing—in seven charts,” Nature, https://www.nature.com/articles/d41586-020-03564-y.
  18. P. Vincent-Lamarre, C. R. Sugimoto, and V. Larivière, “Monitoring women’s scholarly production during the COVID-19 pandemic,” http://shiny.initiativesnumeriques.org/monitoring-scholarly-covid/.
  19. B. Aczel, B. Szaszi, and A. O. Holcombe, “A billion-dollar donation: Estimating the cost of researchers’ time spent on peer review,” Research Integrity Peer Review 6, 14, https://doi.org/10.1186/s41073-021-00118-2.
  20. R. Borchardt, C. Roh, S.Bruce, and A. Click, “Are we walking the talk? A snapshot of how academic LIS journals are (or aren’t) enacting disciplinary values,” In the Library with the Lead Pipe, https://www.inthelibrarywiththeleadpipe.org/2022/walking-the-talk/.
  21. See the comments in K. Einhorn, S. Pollock, and N. Paolini, Guest Post—“Introducing Demographic Questions during Manuscript Submission at the American Psychological Association,” https://scholarlykitchen.sspnet.org/2021/10/14/guest-post-introducing-demographic-questions-during-manuscript-submission-at-the-american-psychological-association/.
  22. T. McMillan Cottom, “Where Platform Capitalism and Racial Capitalism Meet: The Sociology of Race and Racism in the Digital Society,” Sociology of Race and Ethnicity 6(4):441-449, http:// doi.org/10.1177/2332649220949473.
  23. L. Chan, Platform Capitalism and the Governance of Knowledge Infrastructure, (April 29-30, 2019), Digital Initiative Symposium, University of San Diego, https://zenodo.org/record/2656601#.YZPy4pDMJej.
  24. One recent example of this failure is how Code4Lib solicited Becky Yoose’s expert input and then disregarded it completely. You can see her comment at https://journal.code4lib.org/articles/16087#comment-2745444.
Copyright Charlotte Roh

Article Views (Last 12 Months)

No data available

Contact ACRL for article usage statistics from 2010-April 2017.

Article Views (By Year/Month)

January: 0
February: 0
March: 1906
April: 259
May: 83
June: 333
July: 118
August: 75
September: 155
October: 71
November: 51
December: 3