College & Research Libraries News

Assessment of reference instruction as a teaching and learning activity

College & Research Libraries

An experiment at the University of Illinois-Springfìeld

by Denise D. Green and Janis K. Peach

Reference service happens when a librarian interacts with a library user to answer questions and assist in the research process. It can take place face-to-face at a traditional library reference desk or, increasingly, through e-mail, live-chat or virtual software programs, and by telephone. Ref- erence has been a key academic libraiy service for over 100 years, with librarians instructing stu- dents in the use of card catalogs, indexes, and now databases. The context of reference transac- tions usually differs from classroom libraiy or bib- liographic instruction (BI). In BI there is a planned instructional workshop or guest lecture with a specific time and often a specific outcome, usually an assignment or tenn paper. Reference service is more random, with the user choosing when to approach a librarian who usually has not had time to prepare specific information on the students’ course or research assignment. Reference transac- tions can be one-time or ongoing interactions, with patrons returning for help as their project devel- ops.

The literature on reference evaluation is extensive. Several review essays are available including Charles Bunge’s 1994 essay “Evaluating Reference Services and Reference Personnel: Questions and Answers from die Literature”1 as well as other views by Lisa Smith2 and Jerry Campbell-.

The traditional technique to measure and evaluate reference service was a simple tally of questions diat were sorted by length of question, time of day, and day of the week. Also notable among published reference evaluation case studies are user interviews', focus groups,’ and a mix of survey and observational techniques.6

In 1999, the authors, academic librarians at the University of Illinois-Springfield (UIS), experimented widi evaluaüng reference service as a teaching and learning activity. To do this, we designed an assessment instrument tìrat attempted to measure patrons’ attitudes about learning from reference interaction and applied it in a mediumsized academic library.

The three types of reference evaluation literature

Most of the reference evaluation literature falls into three categories. The first categoiy is die unobtrusive approach or “55 percent school,” best known from the studies of Peter Hernan and Charles McClure.7 K Typically, persons posing as library patrons ask a series of predetermined factual questions either in person or by telephone. The reference service is evaluated on the accuracy of the responses to these questions, which averaged only 55 percent correct. The unobtrusive reference evaluation process assumes a model of reference work as answering discrete inquiries with right and wrong solutions. This approach has certain advantages since providing accurate answers is one goal of high-quality reference service, but critics of Hernan and McClure have pointed out that often library patrons do not ask discrete inquiries with right and wrong answers. Also, as in any communication activity, how the answer is conveyed can be as important to the library patron as what information is delivered during the reference transaction.91011

About the authors

Denise D. Green is coordinator of reference and Janis K. Peach is collection development coordinator in Brookens Library at the University of Illinois-Springfield, e-mail: and peach.jan@uis. edu

The second category of reference evaluation literature focuses on the interpersonal communication process. This category is most widely known by the studies of Joan Durrance and her colleagues.1213 Durrance was reacting in part to the “55 percent school” focus on accuracy regardless of departmental or inteipersonal variables by focusing on the willingness of the patron to reaim to the same staff member in the future, implying a more complex model of reference than Heman and McClure’s studies. Critics of the inteipersonal communication model argue that wrong answers, however charmingly delivered, are still wrong answers and not high-quality reference.

The third category of literature on the evaluation of reference is based on the Wisconsin-Ohio Reference Evaluation Program developed by Charles Bunge and Marjorie Murfin. This widely used instniment assesses user satisfaction and the conditions of the reference transaction.14-15 The Wisconsin-Ohio Reference Evaluation Program uses a two-part evaluation form for each reference transaction, with part one being answered by the patron and part two by the librarian, to allow for such variables as the number of resources used by the patron, how busy the library was, die training of the librarian, and the subject area of the inquiry.

It is the only assessment instrument available today that is externally validated, allowing a reference department’s perfonnance to be compared to data aggregated from over 100 other libraries. The Wisconsin-Ohio Reference Evaluation Program was designed primarily for the evaluation of an entire department rather than an individual librarian’s performance, with the implied notion that the model of reference service is a complex activity that results in user satisfaction or dissatisfaction.

While each of these methods may be useful for evaluating reference performance in terms of accuracy and patron satisfaction, none serves to offer proof of the teaching activity of librarians. Since librarians are on the front lines of teaching, providing infonnation literacy and research skills to students seeking help at the reference desk, the authors set out to test an instrument to measure this area of performance.

Evaluation at UIS

At UIS, reference (or instructional services) librarians have full faculty status and are evaluated for tenure, reappointment, and promotion by the same criteria as classroom faculty. One evaluation requirement is proof of “teaching excellence” by student evaluations, letters, and other documentation. In 1995, we used die Wisconsin-Ohio program to evaluate the department; while gratifying to learn that we rated 73.5 percent versus the national average of 69 percent, this data did not significantly help individual library faculty in their personnel process. In 1997, the reference department again used the Wisconsin-Ohio Reference Evaluation, tallying the questions separately for each librarian. While results were somewhat useful, they still did not evaluate individual teaching.

When teaching credit courses, library faculty use the campus-wide evaluation instrument to document and assess our teaching. We also developed an evaluation instrument for BI sessions. However, the significant absence of an assessment of our main teaching activity was a problem for library faculty in the reappointment, tenure, and promotion process. Infonnation desk or reference teaching is 20 percent or more of a typical UIS instructional services librarian’s time and remains an under-documented teaching activity for us and, indeed, for most academic librarians.

The authors were therefore very motivated to document reference as a teaching and learning activity. We based our survey on our literature review and experience with the Wisconsin-Ohio forms; on advice from die campus Personnel Policies Committee, we also added questions similar to those on the classroom teaching evaluation form, using Likert scale responses. The form was given only to library patrons who had fairly complex reference questions. To ensure confidentiality, respondents placed the completed forms in a locked wooden box, and a support staff member in the reference department tabulated results.

The survey was administered by the authors in the Brookens Library Reference Department at UIS as a pilot project in February and March 2000 and again on a larger scale from September to the end of November 2000. Founded in 1970 as Sangamon State University, UIS is an upper-division undergraduate and graduate university with enrollment of approximately 3,800 students and 160 faculty in 2000. The average age of our student body is 33, and 65 percent are part-time students. Only about 10 percent of the UIS student body lives on campus. The UIS curriculum offers a variety of undergraduate majors, graduate programs, and certificates with an emphasis on public affairs and degree completion programs.

The results

The summary of data from both the pilot and fall surveys shows a high degree of satisfaction with the reference teaching process by students and other library users. Our respondents were 50 percent undergraduate and 30 percent graduate UIS students, with the balance made up of high school, community college, and other university students, plus members of the community. The percentages from questions designed to rate actual teaching were very high: 92 percent agreed that they learned something new about how to do research, 95 percent agreed that they learned more about using the library resources, 85 percent agreed that research skills increased, and 87 percent rated the librarians’ quality as a teacher as high.

Several of the questions were designed to rate the librarians’ communication skills and knowledge, and the “comfort level” of the patron: 93 percent agreed that the librarian had knowledge and communication skills to teach research, 89 percent agreed that the instruction would help them succeed with their research and writing, 91 percent agreed that they felt more comfortable using the library after this encounter, and 98 percent would definitely ask the librarian for help again.

The authors feel that these results show a promising method of evaluating individual teaching at the reference desk; however, the small amount of data collected at UIS is not a substantial test of this instrument. The authors offer other academic librarians the chance to use our instrument, and would appreciate feedback from any libraries collecting data. We welcome its use at a variety of academic libraries, especially since the size and curriculum of UIS is unique.

The dilemma of academic library faculty describing what we do as a teaching and learning activity is widespread. Certainly there is a great need for this type of assessment of academic reference and a need for the recognition of reference as a valuable teaching activity of library faculty. More research is needed to assess and document the teaching of research skills as a component of reference, to show it as a legitimate educational activity


  1. Charles A. Bunge, “Evaluating Reference Services and Reference Personnel: Questions and Answers from the Literature,” The Reference Librarian no. 43 (1994): 195-207.
  2. Lisa L. Smith, “Evaluating the Reference Interview: A Theoretical Discussion of the Desirability and Achievability of Evaluation,” RQ 31 (1991): 75-81.
  3. Jerry D. Campbell, “Clinging to Traditional Reference Services,” Reference and User Services Quarterly 39 (2000): 223-227.
  4. Jennifer Mendelsohn, “Perspectives on Quality of Reference Service in an Academic Library: A Qualitative Study,” RQ36 (1997): 544-57.
  5. Virginia Massey-Burzio, “From the Other Side of the Reference Desk: A Focus Group Study ‚” Journal ofAcademic LibrarianshiplA, no. 3 (1998): 208-15.
  6. Elaina Norlin, “Reference Evaluation: A Three-Step Approach—Surveys, Unobtrusive Observations, and Focus Groups,” College & Research Libraries 61 (2000): 546-53-
  7. Peter Hernon and Charles R. McClure, “Unobtrusive Reference Testing: The 55 Percent Rule,” Library Journal 111 (April 15,1986): 37-41.
  8. Peter Hernon and Charles R. McClure, “Library Reference Service: An Unrecognized Crisis-a Symposium,” Journal of Academic Librarianship 13 (1987): 69-80.
  9. Lori Goetsch, “Reference Service is More Than a Desk "Journal of Academic Librarianship 21 (1995): 15-16.
  10. Carolyn W. Jardine, “Maybe the 55 Percent Rule Doesn’t Tell the Whole Story: A User- satisfaction Survey,” College & Research Libraries 56 (1995): 477-85.
  11. LorieneRoy, “Reference Accuracy,” 73/4e ReferenceLibrarianno. 49/50 (1995): 217-27.
  12. Joan C. Durrance, “Reference Success: Does the 55 Percent Rule Tell the Whole Story?" Library Journal 11A (April 15, 1989): 31-36.
  13. Joan C. Durrance, “Factors the Influence Reference Success: What Makes Questioners Willing to Return?” The Reference Librarian no. 49/50 (1995): 243-65.
  14. Marjorie E. Murfin, “Evaluation of Reference Service by User Report of Success,” The Reference Librarian no. 49/50 (1995): 229- 241.
  15. John C. Stalker and Marjorie E. Murfin. “Quality Reference Service: A Preliminary Case Study "Journal of Academic Librarianship 22
Copyright © American Library Association

Article Views (Last 12 Months)

No data available

Contact ACRL for article usage statistics from 2010-April 2017.

Article Views (By Year/Month)

January: 2
January: 7
February: 13
March: 8
April: 18
May: 7
June: 1
July: 6
August: 5
September: 4
October: 5
November: 0
December: 8
January: 0
February: 0
March: 0
April: 0
May: 0
June: 0
July: 0
August: 19
September: 18
October: 16
November: 8
December: 16