07_Liu

Moving up the ladder of source assessment

Expanding the CRAAP test with critical thinking and metacognition

Grace Liu is assistant professor of business and faculty services librarian at West Chester University’s FHG Library, email: yliu@wcupa.edu

Information literacy is one of the fundamental skills to acquire to be able to navigate today’s complex information ecosystem, succeed as a lifelong learner, and make critical decisions as an active and informed citizen. Today’s information environment—saturated with misinformation, fake news, misleading information, propaganda, etc.—poses a great challenge to form unbiased views of the world and make sound judgment and decisions.

In spring 2020, two management professors from the School of Business at West Chester University shared their interest in developing an information literacy module for their students in an upper-level management class. The course required students to find quality articles on management topics and present the topic to the class. What concerned the faculty was the students’ ability to distinguish high-quality information from low-quality information as they described. They hoped the information literacy module would improve students’ ability to critically evaluate online sources.

Considering students’ needs and the faculty’s requests, I created a four-step Source Assessment Strategy infographic guide. One problem I encountered at the very beginning was whether to call it an “evaluation strategy” or an “assessment strategy.” Librarians often use evaluation and assessment interchangeably. However, from an educational context, evaluation focuses on making a judgment or determination concerning the quality of performance, work product, or skill, while assessment is not judgmental and focuses on the process of measuring performance, work product, or skill to seek further improvement.1 In this sense, source assessment is more accurate.

The four-step source assessment strategy encourages students to assess sources through a progressively in-depth process: from looking at the source’s appearance, doing investigations, and using lateral reading and critical thinking skills to reflecting on their own biases and interpretations. This strategy intends to expand the CRAAP test with critical thinking and metacognition. It responds to the affective learning and metacognition concepts in ACRL’s Framework for Information Literacy for Higher Education.2 This article will share some of my thoughts behind the development of the infographic guide and hopefully inspire others to further explore the ways to enhance source assessment strategies.

The ladder of inference and information evaluation

The idea of describing source assessment as moving up a ladder was inspired by the concept of the “ladder of inference.” The ladder of inference, as depicted in the image, was initially developed by the late Chris Argyris, former professor at Harvard Business School and pioneer in the field of organizational learning. The concept is used to describe a common mental pathway of increasing abstraction, which often leads to misguided beliefs.3

The ladder of inference reveals our natural mental process when encountering new information. From observing the information available to forming beliefs, we may not be consciously aware that we have taken several leaps of abstraction. We have filtered out some relevant information, added personal or cultural meanings to interpret the selected information, have made our assumptions based on interpreted meanings, and have drawn conclusions based on these assumptions. The possibly distorted conclusions may eventually lead us to form misguided beliefs and take wrong actions.4 The ladder of inference reminds us that instead of quickly jumping to conclusions, we need to analyze and test the assumptions, meanings, and selected data and observations that created them.

Information source assessment creates a new mental pathway that intervenes in the ladder of inference process and prevents us from forming misguided beliefs. Over the years, scholars have developed many source assessment methods. Sarah Blakeslee, library faculty from California State University-Chico, developed the CRAAP test, which stands for Currency, Relevance, Authority, Accuracy, and Purpose,5 and is widely used by librarians to teach information evaluation. John McManus, former journalist and communication professor, developed the SMELL test, which stands for Source, Motivation, Evidence, Logic, and Left-out.6 More recently, with the proliferation of fake news online, many researchers have developed checklists for spotting fake news. Researchers started to reflect the traditional ways of source evaluation and compare them with fact-checkers’ practices.

A 2017 Stanford working paper assessed the web source evaluation skills between students, faculty, and professional fact-checkers and found what made some fact-checkers perform better than others was their habit of “lateral reading” along with their robust knowledge on sources, search engines, and content structures.7 Learning from the Stanford study, Jennifer A. Fielding, in the article “Rethinking CRAAP,” encouraged librarians to go beyond the CRAAP test and bring lateral searching and reading to the library classroom.8 But still, the process of source evaluation is not complete until we consider the source content with our critical thinking and reflection.

Critical thinking and information evaluation

Librarians often consider critical thinking and information literacy are closely related and information evaluation is at the intersection.9 Critical thinking scholars to some extent also recognized such correlation. As defined by the National Council for Excellence in Critical Thinking,

“Critical thinking is the intellectually disciplined process of actively and skillfully conceptualizing, applying, analyzing, synthesizing, and/or evaluating information gathered from, or generated by, observation, experience, reflection, reasoning, or communication, as a guide to belief and action. In its exemplary form, it is based on universal intellectual values that transcend subject matter divisions: clarity, accuracy, precision, consistency, relevance, sound evidence, good reasons, depth, breadth, and fairness.”10

However, critical thinking and information literacy take different paths in information evaluation. Critical thinking focuses on recognizing basic logic concepts, evaluating arguments and logical fallacies, and examining deductive and inductive reasoning. Information literacy focuses on evaluating the authority, motivations, currency, relevancy, accuracy, consistency, and transparency of an information source.

We cannot conduct a real information evaluation until we look deeply into the source content and assess the arguments. Recently, we made progress in integrating the fact checker’s practices into the information evaluation process. But that is not enough. We must take a step further to merge information evaluation with more fundamental knowledge in logic and critical thinking. We cannot ignore this fundamental knowledge of information evaluation to reach our goal to educate an informed citizen.

It may not be a librarian’s responsibility to teach hardcore knowledge of critical thinking, but helping students bridge information literacy with the knowledge they learn in other disciplines, and encouraging students to take a deep dive into the information assessment is what librarians could do and should be doing.

Metacognition and information evaluation

The ACRL Framework has incorporated the ideas of metaliteracy, with a special focus on metacognition or critical self-reflection. It recognizes that “metacognition is crucial to becoming more self-directed in [a] rapidly changing ecosystem.”11 As described in the Framework, “metacognition is an awareness and understanding of one’s own thought processes. It focuses on how people learn and process information, taking into consideration people’s awareness of how they learn.”12

Metacognition or self-reflection is a critical step for information evaluation. It leads us to reflect our observations, interpretations, assumptions, conclusions, beliefs, and actions in the information assessment process. It prevents us from jumping quickly through the ladder of inference and going into a vicious cycle of the reflective loop.

In the Framework, metacognition is embedded in many knowledge practices and dispositions:

Authority Is Constructed and Contextual

Knowledge Practices

  • acknowledge they are developing their own authoritative voices in a particular area and recognize the responsibilities this entails.

Dispositions

  • develop and maintain an open mind when encountering varied and sometimes conflicting perspectives,
  • develop an awareness of the importance of assessing content with a skeptical stance and with a self-awareness of their own biases and worldview,
  • are conscious that maintaining these attitudes and actions requires frequent self-evaluation.

Information Creation as a Process

Knowledge Practices

  • develop, in their own creation processes, an understanding that their choices impact the purposes for which the information product will be used and the message it conveys.

Information Has Value

Knowledge Practices

  • make informed choices regarding their online actions in full awareness of issues related to privacy and the commodification of personal information.

Dispositions

  • see themselves as contributors to the information marketplace rather than only consumers of it,
  • are inclined to examine their own information privilege.

Research as Inquiry

Dispositions

  • maintain an open mind and a critical stance;
  • value persistence, adaptability, and flexibility and recognize that ambiguity can benefit the research process;
  • demonstrate intellectual humility (i.e., recognize their own intellectual or experiential limitations).

Scholarship as Conversation

Dispositions

  • see themselves as contributors to scholarship rather than only consumers of it,
  • suspend judgment on the value of a particular piece of scholarship until the larger context for the scholarly conversation is better understood,
  • understand the responsibility that comes with entering the conversation through participatory channels.

Searching as Strategic Exploration

Dispositions

  • exhibit mental flexibility and creativity;
  • understand that first attempts at searching do not always produce adequate results;
  • persist in the face of search challenges, and know when they have enough information to complete the information task.

Integrating the concept of metacognition into information assessment and information literacy instruction will build students’ self-awareness, nurture their habits of mind, and help them to develop mental flexibility, creativity, and intellectual humility.

A four-step source assessment strategy

The four-step source assessment strategy is an initial effort to enrich our current source evaluation practices with critical thinking and metacognition. It brings in some popular concepts related to information source, such as content farm, lateral reading, and filter bubble.

Below is a summarized structure of the four steps:

Step 1. Assess sources by its appearances including URL, format, ads, errors, and references.

Step 2. Assess sources by investigation with CRAAP test including currency, relevance, authority, accuracy, and purpose.

Step 3. Assess sources by lateral reading and critical thinking, asking questions related to single versus diverse sources, supported versus unsupported claims, logical reasoning versus fallacies, and anecdotes versus research.

Step 4. Assess sources by reflection, asking students to pierce the filter bubbles, examine their own biases, question their own interpretations, keep an open mind, and suspend their judgment.

This strategy guides students to go beyond appearance check and the CRAAP test, and use lateral reading skills to explore diverse or competing facts and opinions. It asks students to differentiate supported claims from unsupported claims, and evaluate if the statements are supported by evidence and if the opinions are supported by arguments. It encourages students to distinguish logical reasoning from fallacies. It has students ask if the source is based on personal accounts or serious research, if the research method is valid and reliable, and what the research funder’s perspectives are on the issue.

With an effort to integrate metacognition, the strategy guides students to pierce the filter bubble and seek different perspectives. It encourages students to examine their own biases and weigh the reasons from different sides to form unbiased views. It asks students to question their own interpretations, prior knowledge, and assumptions. It motivates students to keep an open mind and seek other possible explanations and reasoning. It helps students to recognize the disruptive nature of a simple fact and encourages them to suspend their judgment until they see the big picture.

The infographic was designed using Canva. It has two pages and can be printed double-sided on cardstock. The front side lays out the four-step source assessment strategy, and the backside lists common cognitive biases, logical fallacies, and further readings on logical reasoning and critical thinking. I made some changes to the prior business-focused version and, hopefully, it can accommodate different needs. The PDF version of the infographic is available at the ACRL Sandbox13 and can be used and distributed under a Creative Commons License.

Summary

The ladder of inference inspires us to make the best use of the information sources assessment strategies to help learners resist the natural mental tendency of leaps of abstraction. To reach to the core of source assessment, we need to expand the CRAAP test with critical thinking and metacognition. I am hoping my four-step sources assessment strategy can inspire other librarians to integrate critical thinking and reflection in source evaluation and further explore the ways to enhance our current source assessment approaches.

Notes

  1. Susan Starr, “Moving from evaluation to assessment,” Journal of the Medical Library Association: JMLA vol. 102,4 (2014): 227-9, https://doi.org/10.3163/1536-5050.102.4.001.
  2. ACRL, “Framework for Information Literacy for Higher Education,” (2016), www.ala.org/acrl/standards/ilframework.
  3. Peter M. Senge, The fifth discipline fieldbook: Strategies and tools for building a learning organization (New York: Crown Business, 2014).
  4. Jonathan Levene, “Solving the problem with problem-solving meetings,” https://www.extension.harvard.edu/professional-development/blog/solving-problem-problem-solving-meetings (accessed April 14, 2020).
  5. Sarah Blakeslee, “The CRAAP Test,” LOEX Quarterly 31, no. 3 (2004), https://commons.emich.edu/cgi/viewcontent.cgi?article=1009&context=loexquarterly.
  6. John McManus, “Don’t Be Fooled: Use the SMELL Test To Separate Fact from Fiction Online,” (2013), http://mediashift.org/2013/02/dont-be-fooled-use-the-smell-test-to-separate-fact-from-fiction-online038/.
  7. Sam Wineburg and Sarah McGrew, “Lateral Reading: Reading Less and Learning More When Evaluating Digital Information,” Stanford History Education Group-Working Paper (September 2017), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3048994.
  8. Jennifer A. Fielding. “Rethinking CRAAP Test: Getting students thinking like fact-checkers in evaluating web sources,” (2019), https://crln.acrl.org/index.php/crlnews/article/view/24195/32005.
  9. Helena Hollis, “Information literacy and critical thinking: different concepts, shared conceptions,” (2019), http://www.informationr.net/ir/24-4/colis/colis1921.html#author.
  10. The National Council for Excellence in Critical Thinking, “Defining Critical Thinking,” (1987), http://www.criticalthinking.org/pages/defining-critical-thinking/766.
  11. ACRL, “Framework for Information Literacy for Higher Education,” (2016), www.ala.org/acrl/standards/ilframework.
  12. Jennifer A. Livingston, “Metacognition: An Overview,” online paper, State University of New York at Buffalo, Graduate School of Education (1997).
  13. https://sandbox.acrl.org/library-collection/4-step-source-assessment-strategy.
Chris Argyris’s “Ladder of Inference.” Source: Wikimedia Commons.

Chris Argyris’s “Ladder of Inference.” Source: Wikimedia Commons.

Copyright Grace Liu

Article Views (Last 12 Months)

No data available

Contact ACRL for article usage statistics from 2010-April 2017.

Article Views (By Year/Month)

2021
January: 0
February: 900
March: 472
April: 357