Firouzeh Rismiller, Holly Cerney, Susan Shultz, Grace Spiewak, and Sveta Stoytcheva

Going beyond the source

A revised curriculum for source evaluation

Firouzeh Rismiller is an instructional services librarian at DePaul University, email: frismill@depaul.edu. Holly Cerney is a part-time reference librarian at DePaul University, email: hcerney@depaul.edu. Susan Shultz is a business and social sciences librarian (retired) at DePaul University, email: sshultz@depaul.edu. Grace Spiewak is a health sciences librarian at DePaul University, email: gspiewak@depaul.edu. Sveta Stoytcheva is a data literacy librarian at DePaul University, email: sstoytch@depaul.edu.

© 2023 Firouzeh Rismiller, Holly Cerney, Susan Shultz, Grace Spiewak, and Sveta Stoytcheva

The DePaul University Library has a longstanding relationship with the First-Year Writing Program to provide information literacy instruction to students in Writing, Rhetoric & Discourse (WRD) 104, a required course for all DePaul University undergraduates. Partnering in these courses is the foundation of the library’s information literacy program: we provide instruction to every section of WRD 104 to introduce these concepts to as many first-year students as possible. We have a standardized curriculum for the WRD 104 library instruction session. The curriculum is developed by the Instruction Working Group (IWG), a committee chaired by the instructional services librarian and made up of three or four additional instruction librarians who serve two-year terms.

Source evaluation is an integral part of the WRD 104 library instruction curriculum. In the past, we have used the CRAAP Test to teach students to determine credibility of a source.1 However, over the past few years, we noticed that the CRAAP Test wasn’t meeting students’ needs in our increasingly complex information landscape.

During Summer 2022, the IWG made it a priority to explore other options for teaching source evaluation to students during our one-shot library instruction sessions. While reviewing the literature, we came across other, more up-to-date evaluation methods that address the new challenges we face in teaching source evaluation, such as SIFT, ACT UP, and proactive evaluation.2 Although we found elements that we liked in each method, we didn’t find a method that fulfilled all our needs. Therefore we decided to come up with a redesigned approach that incorporated elements of these alternative source evaluation frameworks to meet our needs in the classroom.

The purpose of our redesign was to encourage students to evaluate sources in the context of the broader social and information landscape, to incorporate lateral reading, and to employ critical thinking skills. We designed an interactive, discussion-based source evaluation activity and replaced the basic checklist we previously used with a more thorough table and an engaging graphic that work together to help students learn to critically evaluate information.

We titled this approach the Source + Beyond the Source framework because it prompts students to consider aspects of an information resource that can be evaluated by engaging just with the source itself, as well as to think critically about the information landscape in which the source is situated.

The Source + Beyond the Source framework

To create our own framework for evaluating sources, we began brainstorming questions that we want students to consider when evaluating information. While many of our questions reflected those of other source evaluation methods, others surfaced in our discussions. As we examined our questions more closely, a binary began to emerge. Some of our questions were focused on aspects of the source that could be answered by examining the source on its own, such as What is the main point? and When was it published? Other questions required higher-level thinking skills, an awareness of the broader information landscape, and an ability to find additional information beyond the source, such as What other information can you find about this topic? and Who is missing from the conversation?

We decided to build on this insight by creating a Source + Beyond the Source graphic (figure 1) with two circles to divide our questions. The inner circle represents “The Source” and contains questions that a user can answer by looking at the source alone, without needing to do extensive outside research. The outer circle represents “Beyond the Source” questions that require critical thinking, introspective reflection, and lateral reading to answer. During the instruction session, we first display the graphic with only the inner circle, then reveal the outer circle to illustrate the distinction between the two types of questions. We emphasize that while all the questions in the graphic are important in determining a source’s credibility, the “Beyond the Source” questions require the user to conduct additional research and reflect on the context of how the information was created and shared.

Additionally, we designed a handout as a reference to use during the presentation and a takeaway for participants after the session. The handout has the Source + Beyond the Source graphic on one side and an accompanying chart (figure 2) on the other. The chart’s purpose is to elaborate on the questions presented in the graphic. It is organized in two sections, one for “The Source” and one to address “Beyond the Source.” The columns beneath each of those headings represent the questions: What . . . , Where . . . , When . . . , Who . . . , Why . . . , and How . . . shown in the graphic. Beneath each of those interrogative words are examples of potential answers to the question that students may encounter when examining a source. These are focused questions to “ask” a source, or factors to discover within a source when considering its credibility.

We particularly wanted the new framework to address evaluation from a social justice lens by asking questions like Who is missing from the conversation? and encouraging students to consider marginalized voices, global perspectives, and others when coming up with an answer. Similarly, we wanted to highlight that, given the prevalence of algorithmic search results, bots, and filter bubbles, the position of the researcher in today’s information landscape has changed and that information also has a type of agency. Borrowing from Bull et al., we ask students to consider: How did the information find you?3

Developing the Source + Beyond the Source framework has given us an opportunity to start considering how we want to integrate source evaluation with the other core information literacy skills we teach, including developing a research question, identifying and selecting databases, developing a search strategy, and more. In this context, we’ve begun to consider how our source evaluation module fits into the broader ACRL Framework for Information Literacy for Higher Education.4 The ACRL Framework allows us to connect source evaluation to other stages of the research process and present a more holistic view of the information lifecycle to students.

We’ve found that the source evaluation activity is an opportunity to introduce core concepts related to the frames Authority is Constructed and Contextual, Information Creation is a Process, Information Has Value, and Scholarship as Conversation by talking about the expertise and credibility of information creators and their potential motivations for creating and disseminating information. The activity also presents opportunities to dive deeper into different information-creation processes by talking about the kinds of publications and platforms where students may encounter information and what differentiates them.

Source evaluation module lesson plan

To incorporate this new source evaluation framework into our library instruction, we developed a Source Evaluation module as part of the 90-minute one-shot library instruction session that all students in WRD 104 attend. This module takes approximately 35 minutes, and many of us prefer to start off our instruction session with it because it sets a tone of interaction and student participation for the whole session.

The learning outcomes for this module are for students to

1. understand and apply the Source + Beyond the Source evaluation framework to evaluate sources of information;
2. articulate the evaluation criteria they already use when encountering new information; and
3. consider how the context of their information needs shapes the information they use.

To begin the activity, we show students a sample source on a pre-selected topic. This initial source is something they might encounter in daily life or through a quick Google search: a social media post, YouTube video, or website. We have prepared content on a few perennially popular topics, such as social media and mental health, but the activity works well with any subject matter. We ask students to share what criteria they would use to determine if the source is credible and discuss it together. This is a great opportunity to get a sense of the knowledge students bring with them into the classroom. Next, we introduce the Source + Beyond the Source framework, taking care to emphasize the criteria that didn’t organically come up in the initial conversation and the “How” and “Why” questions in the framework.

After this discussion, we show students a scholarly source on the same topic and ask them to work in groups to evaluate it using the Source + Beyond the Source framework. When we come back together as a class, we ask students to compare the two sources and tell us in what circumstances, if any, they would cite them in a research assignment or share them with friends, family, or on social media. To wrap up, we reiterate the importance of asking critical questions about the information we encounter and our own motivations for using it.

This activity integrates well with classroom technologies like Mentimeter, Poll Everywhere, and Google Jamboard (all of which we’ve experimented with in our classrooms), which makes it a good fit for hybrid or online classes. It’s also very easy to implement in a low-tech way, simply by asking students to raise their hands and contribute to the discussion.

We’ve compiled a detailed lesson plan that includes sample topics and talking points for discussing them in class, slide decks, and handouts. These resources are available online and accessible to anyone with the link.5 We wanted to ensure that it is as easy as possible for our colleagues to incorporate this curriculum into their instruction and to customize it for their own teaching style and the needs of the course they are supporting. Our intention is to update the guide regularly with additional sources on timely topics.

Implementing the new curriculum

The Source + Beyond the Source framework represents a significant departure from the way most DePaul instruction librarians taught source evaluation during the WRD 104 library instruction session. We aspired to be deliberate in the rollout of the new curriculum and to provide ample support for our colleagues.

Before beginning the curriculum redesign, the IWG hosted an informal brown bag discussion of a recent article from the Project Information Literacy “Provocation” series titled “Dismantling the Evaluation Framework.”6 The discussion was well attended and gave us an opportunity to learn about what our colleagues wanted to accomplish when talking about source evaluation in their own instruction. As we began the redesign, we provided regular reports back to our colleagues during our weekly department meeting and solicited feedback on the draft lesson plan and handout.

We began our implementation by piloting the new curriculum in a handful of WRD 104 sessions taught by IWG members in the fall quarter of 2022. This gave us an opportunity to become comfortable with the lesson plan and to get a sense of some of the practical questions raised by implementing the curriculum in a classroom setting: Does this module work best at the beginning or the end of the WRD 104 session? How do we facilitate the activity in a hybrid in-person and online classroom? How much student participation can we expect in the discussion?

At the end of the fall quarter, we offered a number of small group training sessions to our colleagues where we led a live demonstration of the lesson plan and answered questions. Every instruction librarian attended at least one training session, and we encouraged our colleagues to attend multiple sessions or shadow an IWG member in a classroom setting. We also offered one-on-one support to colleagues interested in implementing the activity using a digital polling tool like Mentimeter or Poll Everywhere. Our colleagues raised useful questions and helped us to better understand the kinds of resources that would make a full rollout of the curriculum as painless as possible. Some examples of this include a sample slide deck for the activity and more examples of non-scholarly sources to use in the activity beyond the Twitter posts we had originally included. (Some of these alternative sources—in particular the kinds of non-scholarly, search engine–optimized websites that tend to float to the top of Google search results—proved to be very good examples to use in the classroom.)

In winter quarter 2023, we rolled out the Source + Beyond the Source lesson plan in all 51 sections of WRD 104, taught by 15 individual instruction librarians.

Feedback and evaluation

After our experience piloting the curriculum, we were excited for the change and eager to introduce a new framework for source evaluation that was more representative of the current information climate. The revised lesson plan promoted increased engagement from students during the session and we received positive feedback from participating faculty, which we hope to assess more thoroughly in the future.

The initial introduction of the Source + Beyond the Source framework in training sessions was met, understandably, with a mix of enthusiasm and trepidation by librarians. In addition to the challenge of making a significant change to an already packed one-shot instruction curriculum, our colleagues expressed apprehension in navigating various forms of social media in a classroom setting. There was much discussion regarding the inclusion of a tweet as one of our examples in the lesson plan: Is this a source we want to promote? Will Twitter even be relevant in regards to its recent acquisition?

When the new curriculum went live in all WRD 104 sessions however, the general response from colleagues mirrored IWG members’ experiences from the pilot program. Instruction librarians were happy for a change and enjoyed more active engagement in the classroom. Instruction librarians had the freedom to choose which sources to use and whether to use technology (Poll Everywhere, Mentimeter, etc.) to cultivate conversation. Colleagues also expressed gratitude for the level of communication during the rollout and the variety of resources we provided. These, and other successes, challenges, and surprises were shared with us through Microsoft Teams, verbally, and during a scheduled informal debrief. Instruction librarians’ input, faculty responses, and our own reflections will be addressed as we continue to evolve the framework and finetune the activity.

Next steps

Continued communication with staff and faculty will be vital in moving forward to our next goal: a complete overhaul of the entire WRD 104 library instruction curriculum. Our experience developing and implementing the Source + Beyond the Source framework will guide us in this endeavor. We also were eager to share our new approach with our partners in the First-Year Writing Program, who coordinate the broader WRD 104 curriculum. Program faculty recently authored a custom textbook for use in WRD 104 and a couple of related courses at DePaul. One exciting development of working on this new curriculum is the opportunity to build on this relationship by authoring a chapter for the textbook based on the Source + Beyond the Source framework. As we begin the next phase of revising our own WRD 104 curriculum, we will also be translating the work we’ve already done into a resource that faculty can reference in class and that students can take with them into the next stage of their academic career.

The development of the Source + Beyond the Source framework has given our team new insights, and a refresh as to who we are, what we do, and how we teach. We hope to channel this momentum moving forward.

We are grateful for the contributions of our colleague, Nora Gabor, who helped immensely with developing the curriculum shared here.

Notes

1. Sarah Blakeslee, “The CRAAP Test,” LOEX Quarterly 31, no. 3 (2004): 6–7, https://commons.emich.edu/loexquarterly/vol31/iss3/4.
2. Alaina C. Bull, Margy MacMillan, and Alison J. Head, “Dismantling the Evaluation Framework,” In the Library with the Lead Pipe (July 21, 2021), https://www.inthelibrarywiththeleadpipe.org/2021/dismantling-evaluation/; Dawn Stahura, “ACT UP For Evaluating Sources: Pushing Against Privilege,” College & Research Libraries News 79, no. 10 (2018): 551–52, https://doi.org/10.5860/crln.79.10.551; Mike Caulfield, “SIFT (The Four Moves),” Hapgood, June 19, 2019, https://hapgood.us/2019/06/19/sift-the-four-moves/.
3. Bull, MacMillan, and Head, “Dismantling the Evaluation Framework.”
4. “Framework for Information Literacy for Higher Education,” Association of College & Research Libraries, adopted January 11, 2016, https://www.ala.org/acrl/standards/ilframework.
5. “Source + Beyond the Source Framework: Getting Started,” LibGuide, DePaul University Library, last updated April 27, 2023, https://libguides.depaul.edu/evaluation.
6. Bull, MacMillan, and Head, “Dismantling the Evaluation Framework.”