Internet Resources

Systematic reviews and evidence synthesis

Resources beyond the health sciences

Megan Kocher is science librarian, email: mkocher@umn.edu, and Amy Riegelman is social sciences librarian, email: aspringe@umn.edu, at the University of Minnesota Libraries

Asystematic review is a type of review that “seeks to systematically search for, appraise and synthesis research evidence,”1 including results published in grey literature. For decades, systematic reviews have been widely used to synthesize evidence in the health sciences. More recently, other disciplines, such as agriculture and the social sciences, have seen a rise in systematic reviews and related research methodologies. In response to this development, both Cornell University2 and the University of Minnesota Libraries3 have launched systematic review services that explicitly cater to non-health-sciences researchers at their institutions. Because it is recommended that librarians play a part on systematic review teams,4 there is a need for resources and skill development in this area.

While comprehensive and expert searching may be part of the traditional aspects of academic librarianship, systematic reviews also require transparency and reproducibility of search methodology. This work is supported by use of reporting guidelines and related librarian expertise.

This guide provides resources that are useful to librarians assisting with systematic reviews in a broad range of disciplines outside the biomedical sciences. Because the bulk of published literature on systematic reviews is concentrated in the health sciences, some resources are subject-specific in title, but have broader applications.

Research network organizations

  • Campbell Collaboration. Founded in 2000, Campbell Collaboration is an organization that supports research synthesis in education, criminal justice, and social welfare, whereas the Cochrane Collaboration supports health science research synthesis. The stated mission of Campbell includes advancing “positive social and economic change through the production and use of systematic reviews and other evidence synthesis for evidence-based policy and practice.” As of early 2018, 145 systematic reviews have been published via Campbell Collaboration. The Campbell Collaboration Online Library allows for advanced searching as well as browsing by coordinating group (e.g., education, nutrition) to locate systematic reviews, summaries, and protocols. Access: https://www.campbellcollaboration.org/.

Reporting guidelines

  • EQUATOR (Enhancing the Quality and Transparency of health Research) Network. The EQUATOR initiative is focused on improving reliability of published research literature. The EQUATOR Network hosts reporting guidelines in an effort to promote transparency and accuracy in scholarly research. As of March 2018, EQUATOR Network contained 398 searchable reporting guidelines. Librarians could discover the most appropriate reporting guidelines by selecting study type (e.g., systematic reviews/meta-analyses, study protocols), discipline, report section (e.g., data, appendix), or by keyword searching. When systematic reviews/meta-analyses is selected, 31 reporting guidelines appear in the results as of March 2018. Access: http://www.equator-network.org/.
  • PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) Statement. PRISMA Statement is an example of reporting guidelines that are discoverable in the EQUATOR Network. As noted in the name, PRISMA guidance is specific to systematic reviews and meta-analysis. The PRISMA flow diagrams are visible in many published systematic reviews and meta-analyses. The 2009 PRISMA Statement evolved from a previous iteration. In 2005, several entities gathered to revise and expand the QUOROM statement (published in 1999), and they relied on evidence from an international survey of review authors. The PRISMA Checklist includes 27 required parts of the title, abstract, introduction, methods, results, discussion, and funding. For example, #8 of the checklist refers to the search: “Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated.” The Flow Diagram is a transparent depiction of the studies discovered and screened at each stage. As of March 2018, there is a plan to update and revise the PRISMA Statement. Access: http://www.prisma-statement.org/.

Registering protocols

  • Open Science Framework (OSF) Registries. OSF was launched by the Center for Open Science. OSF gives any researcher the option to register a project, such as a systematic review protocol. Registry of a protocol on OSF creates a frozen, time-stamped record of the protocol, thus ensuring a level of transparency and accountability for the research. There are no limits to the types of protocols that can be hosted on OSF. Access: https://osf.io/registries/.
    Center for Open Science logo
  • Prospero: International Prospective Register of Systematic Reviews. This is the primary database for registering systematic review protocols and searching for published protocols. Prospero accepts protocols from all disciplines (e.g., psychology, nutrition) with the stipulation that they must include health-related outcomes. Access: https://www.crd.york.ac.uk/PROSPERO/.


  • Rayyan. Rayyan is a free web-based tool developed by the Qatar Computing Research Institute. It is designed to allow collaboration on systematic review projects through the article screening process. Rayyan supports blinding during screening to reduce bias, as well as coding and tagging. Data stored in Rayyan is held in a secure cloud location and is backed up daily. Access: https://rayyan.qcri.org/welcome.
  • SR Toolbox. This website is a searchable catalog of tools to support the entire systematic review process. Users can search by keyword, features, discipline, approach, and cost. Content is community-driven, but is edited by Chris Marshall at University of York. Access: http://systematicreviewtools.com/.

Transparency and reproducibility of search methodology

  • “Reproducibility of search strategies is poor in systematic reviews published in high-impact pediatrics, cardiology and surgery journals: A cross-sectional study.” Jonathan B. Koffel and Melissa L. Rethlefsen studied the reproducibility of systematic review search strategies in high-impact pediatrics, cardiology, and surgery journals. The article explains the need for transparent reporting of methods (e.g., PRISMA) and that many systematic reviews have incomplete reporting. They found that librarian involvement in systematic reviews did influence reproducibility. Access: https://doi.org/10.1371/journal.pone.0163309.
  • “Role of Expert Searching in Health Sciences Libraries.” In this 2003 policy statement by the Medical Library Association, the significance of expert retrieval and evaluation is explained. Especially helpful, this policy statement defines expert searching and lists the skills and knowledge needed for recognized experts. Expert searching plays an important role in systematic reviews and evidence-based decision making broadly. Access: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC545120/.
  • “Systematic Reviews and Librarians: A Primer for Managers. Partnership.” This 2015 article from The Canadian Journal of Library and Information Practice and Research by Genevieve C. Gore and Julie Jones references recommendations from various authoritative groups regarding the value of librarians contributing to systematic reviews. They quote the Campbell Collaboration, Cochrane Review Group, the Agency for Healthcare Research and Quality, and other entities. For example, this paper cites Campbell Collaboration, “[r]elevant methodological expertise includes: information/library science (searching and text retrieval).” This article would be helpful when making the case for librarian involvement on systematic review teams. Access: https://journal.lib.uoguelph.ca/index.php/perj/article/view/3343/3505#.WfCcM-mQzIU.
  • “Systematic Reviews need Systematic Searchers.” In this 2005 article published in the Journal of the Medical Library Association, authors Jessie McGowan and Margaret Sampson published on the methods, skills, and knowledge of expert searchers contributing to systematic reviews. While this article was written for a medical librarian audience, much of it translates to librarianship of other disciplines. This article also touches on negotiating authorship and refers to existing guidelines for authorship credit based on substantial contritions, intellectual content, etc. Access: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC545125/.
  • “Use of Recommended Search Strategies in Systematic Reviews and the Impact of Librarian Involvement: A Cross-Sectional Survey of Recent Authors.” This study by Jonathan Koffel published in PLOS ONE looked at the impact of librarian involvement on systematic review teams. A survey was disseminated to a sample of systematic review authors from the Database of Abstracts of Reviews of Effects between 2012 and 2014. Survey questions inquired about search methods and librarian involvement. The study aimed to measure recommended systematic review search methods and librarian involvement as well as “whether librarian involvement predicts the use of recommended methods.” Among other revelations, the survey results revealed that 51% of the systematic reviews had librarian involvement while only 64% of those acknowledged a librarian in the text of the paper (e.g., acknowledgements) or as a listed coauthor. Access: https://doi.org/10.1371/journal.pone.0125931.

Disciplinary resources

  • Collaboration for Environmental Evidence (CEE). CEE is a nonprofit organization that provides standards and guidelines for conducting systematic reviews that can impact environmental policy and practice. CEE assists organizations in commissioning systematic reviews and publishes syntheses in their journal, Environmental Evidence. Access: http://www.environmentalevidence.org/.
  • EPPI-Centre (Evidence for Policy and Practice Information). Started in 1993 and functioning as a specialist center in the United Kingdom, EPPI-Centre is based in the Social Science Research Unit of the Department of Social Science in University College London. Goals include informing evidence-based policy and practice in systematic reviews and research use (e.g., problem solving, decision making). They support projects of various disciplines (e.g., education and social policy, developing economies), and current projects can be explored on their website. Access: https://eppi.ioe.ac.uk/.
  • International Initiative for Impact Evaluation (3ie). This NGO supports and funds evidence synthesis in topics related to international development with the goal of influencing policy and improving effectiveness. Their outputs include systematic reviews as well as briefs, impact evaluations, and evidence gap maps. Their website includes a database to search for systematic reviews. Access: http://www.3ieimpact.org/en/.
  • What Works Clearinghouse (WWC). WWC is managed by staff of the Institute of Education Sciences of the Department of Education (IES). IES work includes statistics, research, and evaluation to “provide scientific evidence on which to ground education practice and policy and to share this information in formats that are useful and accessible.”5 Similarly the stated goal of WWC is “to provide educators with the information they need to make evidence-based decisions.” Navigating the interface includes filtering by topic (e.g., mathematics, literacy). After selecting filters, you are able to see effectiveness rankings by intervention. For example, the Early Childhood Education evidence review protocol assesses which interventions improve school readiness, emotional development, phonological processing, etc. WCC review protocols define the scope and parameters (e.g., population, outcomes) of a systematic review, and WCC provides authors with reporting guidelines to encourage clarity and transparency. Access: https://ies.ed.gov/ncee/wwc/.

Search tools

  • PRESS (Peer Review of Electronic Search Strategies). The PRESS Guideline includes recommendations for evaluating electronic search methodologies. The latest guideline from 2015 is hosted by the Canadian Agency for Drugs and Technologies in Health. Stated goals of PRESS include to 1) “formalize the peer-review process for librarians” and 2) provide “a second set of expert eyes . . . once a draft search strategy has been developed.” Guidance from a search strategy peer reviewer could include many things, such as recommendations for translating the research question into a search strategy, Boolean and proximity operators, subject headings, limits and filters, etc. Access: https://www.cadth.ca/resources/finding-evidence/press.
  • PRESS Forum. This closed wiki-based forum is a place for librarians to request reviews of systematic review search strategies, and to review the searches of others. Access: http://pressforum.pbworks.com/w/session/login.
  • Yale MeSH Analyzer. This tool allows users to enter up to 20 PubMed ID numbers, which it uses to aggregate the metadata from the associated articles into a spreadsheet. For systematic reviews, it is useful in search strategy development to quickly aggregate the Medical Subject Heading (MeSH) terms associated with relevant articles. While it only works for PubMed, it can be useful for developing searches in medical-adjacent fields, such as psychology, nutrition, and animal health. Access: http://mesh.med.yale.edu/.

Sources for grey literature

  • Open Grey. Open Grey is a venue for discovering grey literature that is produced in Europe. The types of documents discoverable in Open Grey include conference papers, dissertations, preprints, technical reports, etc. It’s multidisciplinary, and as of March 2018, there were more than 1 million citations encompassing the sciences and technology, social sciences, and humanities. Access: http://www.opengrey.eu/.
  • OSF Preprints. OSF Preprints is a discovery platform hosted by the Open Science Framework. OSF Preprints uses open source infrastructure and a public API. When grey literature is desired for a systematic review, OSF Preprints could be searched to discover preprints and other document types. OSF Preprints is able to cross-search OSF partner repositories, as well as several other repositories, including arXiv, bioRxiv, Cogprints, PeerJ, Preprints.org, and others. As of March 2018, the partner repositories hosted by OSF Preprints include AgriXiv, arabixiv, BITSS, Earth ArXiv, engrXiv, FocUS Archive, Frenxiv, INA-Rxiv, LawArXiv, LIS Scholarship Archive, MarXiv, MindRxiv, NutriXiv, Paleorxiv, PsyArXiv, SocArxiv, SportRxiv, and Thesis Commons. OSF Preprints demands no fees for deposits or access. Access: https://osf.io/preprints/.


  1. Maria J. Grant and Andrew Booth, “A typology of reviews: an analysis of 14 review types and associated methodologies,” Health Information and Libraries Journal 26, no. 2 (2009): 91-108, https://doi.org/10.1111/j.1471-1842.2009.00848.x.
  2. Cornell University Library, “Cornell University Library systematic review service,” https://www.library.cornell.edu/services/systematic-review (accessed March 29, 2018).
  3. Regents of the University of Minnesota, “Systematic review,” https://www.lib.umn.edu/researchsupport/systematic-review, accessed March 29, 2018.
  4. Genevieve C. Gore and Julie Jones, “Systematic reviews and librarians: a primer for managers,” Partnership: The Canadian Journal of Library and Information Practice and Research 10, no. 1 (2015), http://dx.doi.org/10.21083/partnership.v10i1.3343.
  5. See https://ies.ed.gov/aboutus/.
Copyright Megan Kocher, Amy Riegelman

Article Views (Last 12 Months)

No data available

Contact ACRL for article usage statistics from 2010-April 2017.

Article Views (By Year/Month)

January: 27
February: 26
March: 29
April: 35
May: 35
June: 17
July: 23
August: 13
September: 16
October: 26
November: 17
January: 44
February: 51
March: 54
April: 38
May: 49
June: 30
July: 39
August: 31
September: 36
October: 134
November: 27
December: 21
January: 75
February: 89
March: 67
April: 57
May: 87
June: 56
July: 41
August: 62
September: 53
October: 61
November: 54
December: 39
January: 70
February: 90
March: 124
April: 145
May: 119
June: 116
July: 96
August: 108
September: 91
October: 96
November: 95
December: 75
January: 0
February: 0
March: 0
April: 0
May: 1210
June: 265
July: 115
August: 103
September: 69
October: 72
November: 68
December: 73