More than gatekeeping: Close-up on open access evaluation in the Humanities

Korey Jackson

* Contact series editors Zach Coble, digital scholarship specialist at New York University, and Adrian Ho, director of digital scholarship at the University of Kentucky Libraries, at E-mail: with article ideas

I had the privilege to join the SPARC-ACRL Forum at the 2014 ALA Annual Conference in Las Vegas, (viva!). This year’s forum theme was “Evaluating the Quality of Open Access Content,” and I was tasked with exploring new modes of evaluating humanities scholarship in a talk titled “Close-up on Open Access Evaluation in the Humanities.” What follows is a slightly expanded version of that talk.

A quick rumination (and perhaps provocation) to get us started: Since I’m a humanist, I want to take the humanist’s license and do a little close reading of this year’s theme title. Really, I just want to think critically about the last phrase in the title: Open Access (OA) Content. It seems pretty natural on the surface to talk about content distributed openly in this way. The trouble is that when we frame openly distributed content as something in need of specialized evaluation we end up talking about something other than a distribution or business model. We end up talking about a wholly separate scholarly genre. In effect, we affirm the peculiarity of OA content by insisting on referring to that content in terms of its distribution scheme. Researchers and other consumers of academic content don’t usually speak of “paywall” or “toll access” content; we don’t talk about books, manuscripts, or journals as “paid for.” They’re simply books, manuscripts, and journals.

And, yes, maybe it would do us good to think more overtly about the costs of scholarship to individuals and to institutions. But when OA proponents themselves talk about a separate genre of scholarly content, there’s a possibility that we do that content a disservice, especially given OA’s already hard fought (and ongoing) battle with valuation.

OA scholarship has long faced a “brand challenge,” in part because it disrupts the traditional (largely journal-driven) scholarly publishing marketplace. It’s become such a well-rehearsed part of the OA drama that it hardly needs rehashing. But here’s the thumbnail version: ideally, OA scholarship is revolutionary because its business model rejects the notion of “business” as shorthand for crass, consumer-side profiteering. In doing so, however, it also rejects some of the market risks that bestow the kind of prestige that can only be bestowed by a capitalist value system. Rejecting values that lead to overpriced products is a good thing (especially if you’re a consumer of those products). But there’s a baby in that bathwater: when things cost money, we tend to ascribe more than monetary value to those things because capital is cultural, too. So, OA scholarship has had it pretty tough, wanting to be valued in the same way its paywall counterparts are, but needing at the same time to disavow paywall value metrics.

And that challenge is evident in the amount of time we’ve spent advocating for OA’s parity with paywall content. Over ten years ago, Peter Suber faced down the issue of OA journal quality.1 “The rigor of peer review,” he writes, “is independent of the price, medium, and funding model of a journal. OA may threaten the profits and market position of some publishers, but it does not threaten the quality of published science.” While he’s speaking to the smaller circle of STEM publishers, his remarks apply beyond any strict disciplinary boundaries, and even beyond the specific context of article processing charges (APCs) addressed in Suber’s post. The practice that contributes to a journal’s selectiveness and prestige—peer review—need not change simply because a journal chooses to flip the script and freely distribute already-paid-for content.

Of course saying as much doesn’t make a thing true. Not only that, but for content that exists outside of the framework of the journal article, peer review is simply not the central (or even a very important) gatekeeping function. All of which leads me to think that it might just be time for OA to ditch the banner of paywall parity altogether and opt for a new standard, one that says something along the lines of: “content is content regardless of how it gets into our hands; peer review is one way to vet content, but peer review is largely a game of branding…not quite as gimmicky as, say, ‘As Seen on TV,’ but with similar fluctuations in reliability and scope.

It may be the only choice for vetting when content remains closed behind paywalls. But it’s not the only choice when content can be freely disseminated.” Okay, that particular slogan might have a hard time fitting on a banner, but you get the idea.

This freeing up of choice for how we want to vet content is where altmetrics come in. Altmetrics (a term coined by Jason Priem, a PhD candidate at University of North Carolina-Chapel Hill’s iSchool and cofounder of Impactstory)2 refer to any kind of measurement of impact outside of traditional metric types like citation count, h-index, etc. These measurements can take shape as blog post mentions, Twitter citations, use of an article within citation managers like Mendeley and Zotero, user downloads on data-sharing platforms like figshare—for the most part, any online arena that doesn’t have representation within traditional metric rubrics can potentially be tracked through various altmetric engines (provided that these arenas offer the right kinds of APIs to allow aggregation of this data).

Greg Tananbaum offers a concise account of altmetrics and the more specific subcategory of “article-level metrics” in his “Article-Level Metrics: A SPARC Primer,”3 where he writes that these alternative scholarly barometers “open the door to measures of both the immediacy and the socialization of an article.”

Some examples of altmetric applications and services include Priem’s own Impact-story, the aptly named Altmetric,4 and Plum Analytics.5 Each of these offers a slight variation on the theme, but all are focused on Tananbaum’s idea of the “socialization” of scholarship, tracking timely usage, mentions, and other signals of general circulation among an engaged, online audience.

It is true, however, that the humanities have been slow to adopt altmetrics at the level of even the individual scholar, let alone as a larger component of, say, promotion and tenure review. But this sluggishness has less to do with any particular resistance to the “alt” in altmetrics and more to do with a generalized suspicion of counting.

As Jason Baird Jackson remarks, “In many humanities fields, those scholars have intuitions and beliefs about the most important journals,” but specific measurements of impact are simply not the coin of the humanities realm. More succinctly: “They don’t know which to be more nervous about,” altmetrics or all metrics. “Any kind of metric entails the risk of promoting short-sightedness,” Jackson says. “I think the humanists are particularly sensitive to this.”6

This kind of skepticism has kept many from delving into the stats-laden world of metrics, but it hasn’t kept pioneering scholars from exploring and refining the landscape of new-model review and evaluation. In fact, there are quite a few exciting developments in OA humanities evaluation—developments that mirror those in the STEM fields, but also help to point out what some of this evaluation is also about: finding new readers and creating deeper (read: not superficial or crass) markets. Examples include:

  • Kathleen Fitzpatrick’s Planned Obsolescence, a seminal work on the history, present, and possible future of scholarly communication in higher education. Before publication by NYU Press, the book was available as a CommentPress manuscript,7 meaning that it could be commented on by anyone in the community who was willing to toss in a hat and provide commentary. The book received a great deal of attention when it was released, in no small part because of its status as an artifact of community editing.
  • Jack Dougherty and Kirsten Nawrotzki’s Writing History in the Digital Age,8 which used the same CommentPress platform and invited commentary from a hybrid of community readers and select editors from the University of Michigan Press, where the book was under contract. In this case, the edited collection provided contributing authors with feedback about their specific submission. Not only this, but it became a platform allowing authors to comment on each other’s work prior to publication—a strikingly novel practice in the often underappreciated world of edited collections.
  • Dougherty has recently completed a similar project, Web Writing: Why and How for Liberal Arts Teaching and Learning.9 Much like Writing History, Web Writing has been released as an OA, openly reviewable manuscript. With authors and community readers all working collaboratively on review, the book is again a standout example of how edited collections can benefit from embracing openness not only as a business model, but as an essential component of the craft of knowledge creation. Dougherty offers an insightful behind-the-scenes for those curious about the book’s evolution in a final section: Editorial Process and Intellectual Property Policy.10
  • DHCommons journal is an Alliance of Digital Humanities Organizations project designed to provide ongoing peer review services to digital humanities scholarship. The journal’s ambitions, according to its editorial statement, are “to bridge the ‘evaluation gap’ between the Digital Humanities and more traditional disciplinary scholarship.” As the editors explain, “Digital projects often continue for many years as a continuum of work. Rather than building to a single publication moment as monographs do, digital projects often mark progress through a series of significant milestones. DHCommons will provide a concrete way to certify the value of long-standing, influential, but unfinished projects to colleagues unfamiliar with the contours of digital scholarship.”11
  • Open Library of Humanities12 very deliberately borrows from the Public Library of Science (PLOS) and their flagship publication PLOS ONE, seeking to introduce this model of editorial-gatekeeping-plus-community-review to humanities scholarship. Rather than charging authors APC, the Open Library of Humanities is looking to sustain nonprofit business operations through what it calls “Library Partnership Subsidies.”13 In essence, it asks the library community to support content creation at the beginning of its lifecycle, rather than at the end, and to pay quite a bit less to do so. It’s a noble and provocative model that, if successful, has the potential to forge stronger ties between libraries, librarians, and the researchers who depend on library resources to produce scholarship.

All of these venues are concerned with providing effective evaluation for content, but they’re equally about community building and content amplification. And they signal a possible opportunity for libraries to begin encouraging and helping to develop better outlets for humanistic OA publishing and review. There’s been a lot of discussion lately about the library’s role in the growing field of digital humanities, and about what constitutes the “digital” in digital humanities. My answer is broad: that any online production, especially those that engage OA distribution models, counts; and that information delivery and access are decidedly where libraries have the most important role to play, whether that role is about educating scholars about options for publication, about things scholars will want and need to know before embarking on such publication, or about the many styles of metrics and altmetrics that can be marshaled to help showcase the quality and impact of their endeavors.

In the end, better and more varied evaluation is not merely a function of gatekeeping, but a step toward freeing content from profiteerism and allowing it to freely enter the terrain of real knowledge sharing.

1. Suber, P.. , “Objection-reply: Whether the upfront payment model corrupts peer review at open-access journals. ,” SPARC Open Access Newsletter, Issue #71,, accessed September 28, 2014.
2. Impactstory: about,, accessed September 28, 2014.
3. Tananbaum, G.. , “Article-Level Metrics: A SPARC Primer,”, accessed September 28, 2014.
4. Altmetric,, accessed September 28, 2014.
5. Plum Analytics,, accessed September 28, 2014. See also for a comprehensive overview of different metric types.
6. Howard, J.. , “Rise of ‘Altmetrics’ Revives Questions About How to Measure Impact of Research. ,” Chronicle of Higher Education, June.3. , 2013 , accessed September 28, 2014.
7. Fitzpatrick, K.. , “Planned Obsolescence: Publishing, Technology, and the Future of the Academy,”, accessed September 28, 2014.
8. Dougherty, J.. Nawrotzki, K.. , eds., Writing History in the Digital Age, open access version:, accessed September 28, 2014.
9. Dougherty, J.. , ed., Web Writing: Why and How for Liberal Arts Teaching and Learning,, accessed September 28, 2014.
10. Ibid,
11. DHCommons,, accessed September 28, 2014.
12. Open Library of Humanities,, accessed September 28, 2014.
13. Library Partnership Subsidies,, accessed September 28, 2014.
Copyright © 2014 Korey Jackson

Article Views (Last 12 Months)

No data available

Contact ACRL for article usage statistics from 2010-April 2017.

Article Views (By Year/Month)

January: 8
February: 15
March: 46
April: 26
April: 0
May: 1
June: 2
July: 3
August: 2
September: 5
October: 6
November: 2
December: 4