Critical appraisal

The key to unlocking information literacy in the STEM disciplines

Kathryn Mercer is science and engineering librarian, email: kmercer@uwaterloo.ca, Kari D. Weaver is learning, teaching and instructional design librarian, email: kdweaver@uwaterloo.ca, Rachel Figueiredo is engineering and entrepreneurship librarian, email: rachel.figueiredo@uwaterloo.ca, and Caitlin Carter is pharmacy librarian, email: c8carter@uwaterloo.ca, at the University of Waterloo

Science is an essential social institution and a building block in advancing human societies. At the core of scientific discovery and problem solving is innovation to improve the human condition, within and beyond the institutions of higher education that shape scientific minds. Research in any field, beginning in undergraduate education, is fundamentally about information—finding, understanding, generating, discussing, influencing, contradicting, contextualizing, disproving, and communicating it; the ability to do so generally referred to as information literacy.1 Parallel to this, everyday life increasingly requires navigating an overwhelming amount of complex scientific information, and misinformation.

There has been a long-standing assumption by professors that many students enter into college tech-savvy, and, adjacent to those skills, there is an assumption they are able to navigate information seeking in the same way.2,3 Similarly students are overconfident in their abilities to find information, which has often been experienced by us hearing students say, “We’ll just Google it” as a benchmark for how they find what they perceive as good information. Parallel to this, in the applied sciences, like engineering, we have experienced a disconnect in perception around what we understand as necessary foundational information to practice, and what students perceive research necessary to make decisions.4

When students begin college in science disciplines, they often arrive without meaningful experience finding or using primary scientific literature and with engrained, yet underdeveloped, information-seeking behaviors built from prior experiences in high school classrooms and daily life.5 Many engineering students are focused on their professional future, and therefore do not understand the need to learn to research, since their goal is not to be an academic. Effective communication around professional responsibilities of finding and using information is equally important to communicating information literacy skills.

Training STEM students to properly navigate the current information landscape is foundational to their academic and professional success. The social and academic role of information seeking in science education is often taken for granted, and frequent assumptions are made around how, when, and where science, technology, engineering, and mathematics (STEM) students are finding scientific information. Receiving even less consideration is the learning required to accurately contextualize information within the current environment of overload, saturation, and misinformation. Professors often think about teaching the specific subject of their discipline, and often teaching students tenants around information-seeking and literacy is left out of traditional science courses.6

Information literacy, though a term used widely in the library context, has failed to effectively resonate within the academic discourse in the STEM fields.7,8 Within STEM, there is an abundance of terms used for the concept ranging from science communication and data information literacy9 to science media education10 and STEM literacy for learning.11 This indicates that while the ideals of information literacy are important to the STEM fields, the lack of consensus around terminology limits effective discussion and educational intervention. This absence of consensus is compounded by the concern that information literacy rarely resonates with STEM faculty, a gap STEM librarians face with communicating and supporting science faculties due to less than 30% of STEM librarians having a subject-specific degree, and the implicit connection of the term literacy exclusively with books.12 Science faculty members and students often come from academic cultures that rely heavily on in-lab or personal networks and professional associations for much of their own information seeking.13,14,15 Finding the right terminology to connect with faculty helps open the conversation to the library’s value in supporting STEM research by placing our skills within the scientific scope.

In the health disciplines, the use of information for clinical decision-making is core to professional practice and research. Called critical appraisal, health fields use a systematic process to delineate strengths and weaknesses of research while ascertaining the applicability and validity of the content to the research or clinical task.16 Critical appraisal at its core is the process of systematically assessing scientific evidence to judge its trustworthiness, value, and relevance.17 Traditional critical appraisal tools evaluate information by asking “Does the study address a clear question? Does it use valid methods? Are they applicable to the population in question?”18 Readers are further encouraged to examine whether the research question is important, analysis is appropriate, and conflicts of interest are transparent. This question-based evaluation approach aligns closely with information evaluation models used in information literacy instruction, such as RADAR19 or CRAAP.20,21 The process asks individuals to probe evidence using a questioning approach, including the need to establish the relevance of any information consulted to the task at hand, an issue that Michael J. Carlozzi22 establishes as lacking in student information synthesis.

Critical appraisal is taught to students in healthcare professions through curriculums that incorporate evidence-based practice.23 As an example, first-year pharmacy students at the University of Waterloo complete a required drug information fundamentals course, cotaught by the librarian and professor, dedicated to finding, interpreting, evaluating, and applying health research. This course was developed through librarian and faculty collaboration, and favors the use of the term critical appraisal versus the term information literacy, as the course places emphasis on the importance of acquiring the skills necessary to assess the validity of information found on the Internet and in the published research. In addition to critiquing the published research, students are tasked with critically evaluating the credibility of a website related to opioid prescribing, drawing from the RADAR24 approach.

Using the pharmacy course as a guideline, in 2017, engineering librarians at the University of Waterloo adapted key elements from this course into a single lesson for a required first-year engineering communications course, followed in 2018 with a similar course for first-year science students led by the science librarians. Likewise, the focus of these two lessons was on giving students tools to acquire the skills they need to assess the validity of information they find, regardless of the source. Through field-specific examples, we have successfully approached contextualizing research within the context of being a professional engineer, aligning it with expectations of engineering professional practice to find, use, and share ethically sound information. For the science students, the librarians focus on the students becoming effective scientists—you need to be able to understand not only the lab-based experiments, but also to understand the process through which scientific discovery has happened.

In the engineering courses, we introduce students to critical appraisal, through the RADAR framework and use this approach to evaluate both an online source and an academic article that they find, around an engineering topic of their choosing. In the science courses, framing the topic as critical appraisal gives students a framework for finding and contextualizing information as part of building their understanding of science. With this in mind, we introduce these skills through interactive lecture and discussion and then reinforce them through a RADAR activity, where students must appraise a librarian-provided peer-reviewed article. Both of the activities focus on critiquing the information students are using to demonstrate that ease of availability does not equate accuracy or credibility, while offering opportunity for the approach to be scaffolded throughout the curriculum. For example, in one of the engineering programs, the librarian works with professors through each year on increasing levels of critical appraisal through design projects in the curriculum.

As interventions are further developed, within one program we see significant potential to implement broadly across other programs, and is currently being adapted for mathematics students. By using the term critical appraisal and having students draw their own conclusions when working through the RADAR framework, we have seen some success in navigating students away from online search engines and towards scholarly materials with a more open mind.

Moreover, this strategy has been extremely effective in working with STEM faculty to integrate into the curriculum. It has opened conversations around the role and importance of the librarian and the value of active assessment of resources used by students. Because critical appraisal at its core is a process, it is readily adaptable for nonacademic literature that must be evaluated for use in the design process for engineering fields and technical documentation prevalent in science and technology. Furthermore, because the method of evaluation aligns with existing approaches used in the library field, there is a significant body of resources that can help librarians make the shift to critical appraisal without radically redesigning classroom activities and approaches.

For education to be effective, it must meet individuals where they are and build from their existing knowledge base. Using the terminology critical appraisal allows librarians, particularly those who support STEM disciplines, to more effectively open discussions around information literacy. Though seemingly insignificant, changing our rhetoric from information literacy to critical appraisal has had a huge impact on our ability to connect with STEM faculty and students. Using an established process that is validated in health research (and therefore aligns with the scientific method) moves STEM students away from literacy and its connotation with their ability to read books, and towards a critical mindset around information. As librarians seek to effectively communicate value in the modern academic landscape, we must align our efforts with established processes and perceptions of science faculty and learners to foster innovation and drive scientific discovery.


  1. L. E. Briggs and Skidmore, “Beyond the blended librarians: Creating full partnerships with faculty to embed information literacy in online learning systems” in Using Technology to Teach Information Literacy, edited by T. P. Mackey and T. E. Jacobson (New York, NY: Neal-Schuman, 2008), 87–109.
  2. S. F. McEuen, “How Fluent with Information Technology Are Our Students?” Educause Quarterly 24, no.4 (2001): 8–17.
  3. Leah Thompson and Lisa Ann Blankinship, “Teaching Information Literacy Skills to Sophomore-Level Biology Majors,” Journal of Microbology and Biology Education 16, no. 1 (2015): 29–33, accessed December 12, 2019, http://doi.org/10.1128/jmbe.v16i1.818.
  4. Kathryn Mercer, Ariel Stables-Kennedy, and Kari D. Weaver, “Understanding Undergraduate Engineering Student Information Access and Needs: Results from a Scoping Review,” paper presented at ASEE Annual Conference, Tampa, Florida, June 2019.
  5. Heather Brodie Perry, “Information Literacy in the Sciences: Faculty Perception of Undergraduate Student Skill,” College & Research Libraries 78, no. 7 (2017): 964–977, accessed December 16, 2019, https://doi.org/10.5860/crl.78.7.964.
  6. K. Bain, What the Best College Teachers Do (Cambridge, MA: Harvard University Press), 2004.
  7. Thomas P. Mackey and Trudi E. Jacobson, “Reframing Information Literacy as a Metaliteracy,” College & Research Libraries 72, no. 1 (2011): 62–78.
  8. Loanne Snavely and Natasha Cooper, “The Information Literacy Debate,” The Journal of Academic Librarianship 23, no. 1 (1997): 9–14.
  9. Jacob Carlson, Michael Fosmire, C. C. Miller, and Megan Sapp Nelson, “Determining Data Information Literacy Needs: A Study of Students and Research Faculty,” portal: Libraries and the Academy 11, no. 2 (2011): 629–657.
  10. Hans C. Schmidt, “Media Literacy Education from Kindergarten to College: A Comparison of How Media Literacy is Addressed across the Educational System,” Journal of Media Literacy Education 5, no. 1 (2013): 3.
  11. Alan Zollman, “Learning for STEM Literacy: STEM literacy for learning,” School Science and Mathematics 112, no. 1 (2012): 12–19.
  12. Marc Songini, “STEM Librarians Face Gap With Science Audiences, Survey Indicates,” Jove (Blog), August 10, 2018, accessed December 11, 2019, https://www.jove.com/blog/librarian-blog/librarians-career-advancement/stem-librarians-face-gap-with-audiences-survey-indicates/.
  13. Gloria J. Leckie and Anne Fullerton, “Information Literacy in Science and Engineering Undergraduate Education: Faculty Attitudes and Pedagogical Practices,” College & Research Libraries 60, no. 1 (1999): 9–29.
  14. J. Edmund Maynard, “A Case Study of Faculty Attitudes Toward Library Instruction: The Citadel Experience,” Reference Services Review 18, no. 2 (1990): 67–76.
  15. Leslie Ward and Miseon Kim,“Faculty Perception of Information Literacy at Queensborough Community College,” Community & Junior College Libraries 23.1-2 (2017): 13–27.
  16. Jane M. Young and Michael J. Solomon, “How to Critically Appraise an Article,” Nature Reviews Gastroenterology & Hepatology 6, no. 2 (2009): 82.
  17. Ibid.
  18. Ibid.
  19. andalois, “RADAR: An Approach for Helping Student Evaluate Internet Sources,” Journal of Information Science 39, no. 4 (2013): 470–478.
  20. Sarah Blakeslee, “The CRAAP test,” LOEX Quarterly 31, no. 3 (2004): 4.
  21. Dawn Emsellem Wichowski and Laura E. Kohl, “Establishing Credibility in the Information Jungle: Blogs, Microblogs, and the CRAAP Test,” in Online Credibility and Digital Ethos: Evaluating Computer-Mediated Communication, pp. 229-251. IGI Global, 2013.
  22. Michael J. Carlozzi, “They Found It—Now Do They Bother? An Analysis of First-Year Synthesis,” College & Research Libraries 79, no. 5 (2018): 659.
  23. Paul Glasziou, Amanda Burls, and Ruth Gilbert, “Evidence Based Medicine and the Medical Curriculum,” BMJ 337:a1253 (2008), accessed December 9, 2019, https://doi.org/10.1136/bmj.a1253.
  24. Songini, “STEM Librarians Face Gap With Science Audiences, Survey Indicates.”
Copyright Kathryn Mercer, Kari D. Weaver, Rachel Figueiredo, Caitlin Carter

Article Views (Last 12 Months)

No data available

Contact ACRL for article usage statistics from 2010-April 2017.

Article Views (By Year/Month)

January: 29
February: 24
March: 32
January: 34
February: 27
March: 26
April: 37
May: 44
June: 23
July: 32
August: 33
September: 38
October: 26
November: 23
December: 41
January: 28
February: 46
March: 58
April: 39
May: 30
June: 27
July: 26
August: 26
September: 35
October: 51
November: 42
December: 30
January: 0
February: 8
March: 2218
April: 147
May: 98
June: 60
July: 81
August: 43
September: 51
October: 42
November: 12
December: 21