Perspectives on the Framework

Kevin Baggett, Virginia Connell, and Allie Thome

Frame by frame

Using the ACRL Framework for Information Literacy to create a library assessment plan

Kevin Baggett is instruction librarian, email: kbaggett@cord.edu, Virginia Connell is coordinator of library instruction, email: vconnell@cord.edu, and Allie Thome is systems and web librarian at Concordia College, email: athome@cord.edu

© 2018 Kevin Baggett, Virginia Connell, and Allie Thome

Inspired by the ACRL Framework for Information Literacy for Higher Education, librarians at the Carl B. Ylvisaker Library at Concordia College-Moorhead decided to scrap our old information literacy assessment plan and start from scratch. We began by digging into the library’s past assessment plans in an attempt to gain insight into how the organization’s priorities have changed over time. The earliest assessment notes we uncovered were from 2005 and featured a new student orientation pre- and post-test, administered to a random sample of library users. The focus then was on incoming students’ information literacy abilities and how much they learned during library orientation. A sketch of an abandoned assessment plan from 2010 marked a turn to assessing the library’s instruction program and featured the use of focus groups along with a survey in which alumni were asked how the instruction program may have impacted their life after college. With this pivot towards instruction, the library developed a list of learning outcomes and tracked them to the ACRL Information Literacy Competency Standards for Higher Education. This list was used in-house for designing assignments and linked on our library web page to provide faculty with an overview of the program.

In 2011, the library began a four-year trial with the Standardized Assessment of Information Literacy Skills (SAILS) test, which was based directly on the ACRL Information Literacy Competency Standards for Higher Education. Concordia students consistently tested higher than both the Baccalaureate-Liberal Arts group and across all institutions, and the two cohorts tested showed improvement in all categories of the standards during each year of the testing. While this investigation gave us some useful information, it did not provide a clear way forward for measuring student learning in the library.

Goals

In reimagining our student assessment, we found that several things needed changing. Our main assessment goal is to understand what our students are learning, what skills and dispositions do they come into college with, and how can we build on those to further their information literacy? We want specific information that will allow us to actively change our teaching in a timely and supportive manner. Secondly, we need to adapt to the Framework and make it a seamless part of our instruction and assessment of student learning. We value the Framework as a flexible means of responding to campus initiatives, and as an inspiration to look at what we do from multiple perspectives.

Methodology

Given that our old assessments were not providing us the information we needed and were ill-suited for the new ACRL Framework, we began our new approach to assessment planning by discussing the implications of the Framework for our teaching. This conversation coincided with the campus-wide adoption of an integrative learning model that requires all students to complete two PEAK (Pivotal Experience in Applied Knowledge) projects before graduation. Examples of PEAK projects include creating community digital assets, lesson plans for K–12 students, presentations, shareable reports and reflections, and short films screened at the local theater. We felt it imperative to include this campus initiative in building a new assessment plan, and the ACRL Framework made envisioning research assignments within the PEAK environment easier. One of our first library discussions resulted in a set of “deconstructed” assignments emphasizing research skills and research applications to be used in classes without a traditional research paper. Librarians tracked these research assignment options to the Framework, to Concordia’s Goals for Liberal Learning, and to PEAK goals and made them available for faculty in campus workshops and online.

In part, this idea of deconstructing, or breaking the whole into more manageable parts, grew from our history of scaffolding library instruction and informed our approach to the Framework. Deciding to start with one frame, we then unpacked the learning practices and dispositions within that frame to build lessons and develop assessments. This seemed a natural approach within the context of our teaching habits. To start, we decided to do more substantive curriculum mapping (see the description of our new statistics forms in the next section) and to build the Authority Frame into our required first-year library instruction, the Library Launch. The Launch involves hands-on learning in a face-to-face, librarian-led 70-minute session scheduled by every First Year Inquiry instructor. Activities include brainstorming keywords, class-topic relevant searching, physically locating books, and participating in a reference interview. Librarians have curricular control of this session, so we can build a consistent information literacy foundation for the entire first-year class.

Curriculum mapping all of our instruction sessions will allow us to expand our assessment methods to the other five frames and across all four years of a student’s time at Concordia. Beyond the Library Launch, mapping our curriculum will help us set priorities in first-year written and oral communication classes, so that we may introduce other frames before students move beyond core classes. We will also be learning what frames our librarians teach in upper-level classes by gathering this information using a new instruction statistics form. This should allow us to better scaffold students’ grasp of each frame and to grow those dispositional habits of mind for lifelong learning.

In addition, we are widening our understanding of “instruction,” finding more ways to emphasize the importance of information literacy across campus. Highlighting aspects of the research process in deconstructed ways has already strengthened our partnerships across campus. Examples include:

• providing workshops for Residence Life student employees on finding copyright-permitted images, fonts, and color palettes for the required posters and bulletin boards they build in the dorms;
• working with the Student Government Association to provide campus access to the New York Times; and
• planning opportunities for paid or for-credit internships that fulfill PEAK requirements.

While we have yet to decide the best assessment for these partnership instruction opportunities, if what we are experiencing in our traditional instruction holds true, we are likely to expand our ideas of what assessment means in these contexts.

Beyond the classroom

We are taking a holistic view of our assessment of student learning in the library, which means going beyond the classroom. One of the most common places for student learning to happen in the library is at the reference desk. We have redesigned our statistics form for reference transactions to include a question that asks whether the reference interaction addressed one of the frames. If answered in the affirmative, the reference librarian then selects one or more of the frames that the reference interaction addressed. How we understand and interpret the framework in determining if a question falls under one of the frames is an ongoing discussion at reference and instruction meetings.

Another place that we have begun to assess our incorporation of the Framework in the library is in our Technical Services and Circulation/ILL departments. Our library staff in those departments have extensive interaction with students, whether via a service desk or supervising student workers. We have held discussions on the Framework with these supervisors to help us understand how they may indirectly teach the frames to our student workers. We are currently working on a way to track these interactions.

We have also begun to assess how students apply the Framework in their research papers. Each year, the library awards the Exemplary Research Award, a prize for the best research paper written by a junior or senior. We receive 10 to 15 submissions each year, which are judged by a campus committee comprised of three librarians and two teaching faculty using a rubric to grade the student’s research skills. Because this award attracts some of the best upper-level undergraduate research work completed at Concordia, we want to measure how well students have applied and exhibited the concepts of the Framework in their papers. The rubric has always measured student’s research skills, but we have recently made changes to include the language of the Framework for spring 2018.

Current status and next steps

Our new forms for recording reference and instruction statistics have provided us with an excellent snapshot of how our librarians apply the ACRL Framework in their work. Of the 880 reference questions we answered between September 1 and November 27, 2017, our data shows that one third of those addressed at least one of the frames (34.43%, 303 questions). If we break this down further, we find that the majority of those 303 questions specifically address “Searching is Strategic” (Figure 1). We think this makes sense, and that the dispositions and knowledge practices associated with “Searching is Strategic” match best with the kinds of questions we typically answer at the reference desk.

In the future, we will continue to discuss how we can make the other frames more transparent during the reference interview.

Our instruction statistics show a slightly more balanced application of the frames, though “Information Creation as a Process” and “Scholarship as Conversation” were covered less frequently than the other frames (Figure 2). When we reviewed the knowledge practices and dispositions associated with these two, we found that we teach these concepts more in our upper-division classes than in our First Year-level courses. One of our library’s long-term strategic goals is to increase our instruction for upper-level classes, so hopefully we will teach these frames more as we achieve that goal. Based on this data, one of our new long-term goals will be to better incorporate “Information Creation as a Process” and “Scholarship as Conversation” into other required first-year courses on written and oral communication skills, and then to implement assessments in those library sessions.

Because we focused this year’s assessment on “Authority is Constructed and Contextual,” we included two additional questions in our instruction assessment form. If librarians indicated they taught the Authority frame, they were asked which dispositions and knowledge practices they taught during the session. We found this level of granularity manageable to track for one frame, but anticipate that it will be difficult to assess for all the frames because the form will become too long. Because of this, our assessment committee is currently amending the library’s assessment plan to make it simpler and less granular. Once this is finished, each frame will have two measurable learning outcomes we can use in our curriculum mapping, instead of trying to measure all the knowledge dispositions and practices.

During our Library Launch sessions, we had planned to test students on their use of the “peer-reviewed” check box in our discovery layer to check their understanding of the Authority frame. During instruction, students watched a video that defined peer-review and listed some of the markers of authority. In the class activity, we asked students to perform a search, limit to articles, then limit to “peer-reviewed” by checking a box. Most students in the Library Launch successfully limited to peer-reviewed journals, but then applied the concept inexpertly when choosing an article from the list of results.

Our data shows us that some students picked a nonscholarly article or review from within a peer-reviewed publication. In the future, we will walk students through an example of what a peer-reviewed article should look like, then ask them a question that more directly assesses their learning of “Authority is Constructed and Contextual.” Through this summative assessment, we will be able to track student learning from year to year.

Conclusion

Our assessment efforts have yielded interesting and fruitful information on how we use the ACRL Framework at the Carl B. Ylvisaker Library. This has helped us in our plans to refine our forms for Reference and Instruction with the aim of collecting less granular data. We also plan to collect more data on how well students are applying the frames in library sessions by including an assessment on “Authority is Constructed and Contextual” in our Library Launch session. In spring of 2018 we will grade our first batch of Exemplary Research Award papers using an updated rubric, which will provide data on how well our best upper-level students apply the ACRL Framework in their research and writing. Our long-term goal is to implement more assessments of student learning in our library sessions, and to incorporate these into a cohesive assessment plan.