Association of College & Research Libraries

RESEARCH FORUM: Collection assessment and verification studies: Two reports from the Research Libraries Group

Several years ago, RLG institutions embarked on a program of collection evaluation studies: these have come to be known as “verification” studies. A collection “universe” (such as English literature) is defined and identified; a sample is drawn from this universe using a variety of techniques. The resulting titles then form the corpus of the study, providing a statistically sound instrument for measuring retrospective collection strength. The studies that RLG institutions have designed and undertaken have added markedly to RLG’s understanding of its members’ collections and have enabled member libraries to compare themselves against a common standard. They have given RLG a concrete picture of its range of collection strengths and of the patterns of collection overlaps in several, widely varying disciplines.

The general library community’s interest in collection assessment techniques and verification studies suggests that the following brief reports may be welcome. The first, “The Nature and Uses of RLG Verification Studies,” looks at a few of the studies RLG has completed, and offers some conclusions about their uses and value. The final section discusses the question of evaluating current collecting, and describes RLG’s solution, supplemental guidelines for large groups or sub-groups of the Conspectus. The second, “Verification Studies—Design and Implementation,” was written to guide those in RLG developing collection assessment tools. It argues that clarity about assessment intention and standards is paramount in creating such tools, and offers a methodological framework for developers. While directed specifically at studies designed to verify Conspectus data values, it should nevertheless be easily adaptable for those not using the RLG Conspectus.—Jim Coleman, RLG.

The nature and uses of the RLG verification studies

By Paul Mosher Director of Research Services Stanford University

The RLG “verification studies” are comparative collection analyses, designed by small teams of expert bibliographers in member libraries, which allow comparative study of the strength of holdings of each member library in specific fields, or certain lines of a segment of the Conspectus. They permit comparison of collection strengths and distribution of titles among reporting libraries. Let me amplify a bit on these two purposes:

Verification of the comparability of collection value reporting to the Conspectus

Completion of the RLG Conspectus requires use of standardized codes to describe the collections of different libraries in specified Library of Congress classes. To do this it is necessary for the sake of comparability to assign a common, empirical content to the terms being used. Is a level 4 collection at one library comparable to a level 4 collection at another library? In an attempt to deal with this issue, the Collection Management and Development Committee of RLG undertook certain collection assessment or verification projects intended to confirm the collection levels reported to the Conspectus. They have been completed to date in a number of fields including English literature, French literature, Swiss history, food and agricultural economics, art and architecture, mathematical journals, and music. Other studies are in progress.

RLG central staff has now produced consistent reports on each of these verification or overlap studies which will allow the Conspectus Task Force of CMDC to assess the comparability of reports and make recommendations to ensure more standard assignment of collecting intensity codes for the RLG Conspectus, as well as clarifying understanding of the appropriateness of assigning primary collection responsibilities within the consortium. The studies also describe the distribution of the literature of a subject, and unique titles, among member libraries. This helps to demonstrate patterns of little-held materials among the membership, and the relative and comparative strengths of collections at member institutions. The RLG series of verification and overlap studies differs from other studies of this type in representing groups of major national research libraries scattered across the country rather than smaller groups of libraries in state or regional consortia.

Examples of findings of verification studies

A study carried out in English literature from the Cambridge Bibliography of English Literature produced some interesting results. Four test libraries carried out the initial verification study. The results of this study suggested that values reported to the Conspectus were on the whole correct. Interest in the findings and their significance, both consortial and local, prompted eleven other RLG institutions to replicate the study. This first verification study proved, not surprisingly, that English literature is an area of considerable concentration for all RLG libraries. There is a fairly high rate of absolute holdings and collection overlap. Of the four libraries with the most extensive holdings, the strongest (Cornell) held 88%, while the top four together held 97 % . Library holdings of the many smaller RLG libraries, however, demonstrate not only lower overlap rates than among these very large collections, but also a smaller proportion of titles. Nevertheless, the pooled total holdings of these smaller libraries still result in a surprisingly large proportion of the total sample. Among these smaller libraries, while one library held 51 % of the sample, four hold 85%, a gain of over thirty percentage points. This large increase in coverage among the combined holdings of smaller RLG libraries underscores the advantages of resource pooling to an even greater degree than among large institutions.

The area of Renaissance and Baroque art history is obviously a more specialized subject. What did the overlap study reveal in this instance? Of twenty reporting libraries, Yale and Rerkeley each hold slightly over 72%. The top four holding libraries, including the Library of Congress, which is not a member of RLG, hold 91 % of the total. The bottom four reporting libraries hold only 49.4 % of the total (the smallest of these holds but 18%, so the gain is substantial).

Thus it is clear that a few libraries in the consortium hold by far the greater number of titles, and that the combined collections represent for most members very much stronger holdings than any one, or even a combination, of all but the few libraries with the greatest concentration of holdings.

The composite holdings of these major research libraries are greater than the holdings of local, regional, or multitype consortia, and richer in materials important to research. Indeed, the combined holdings of the major national research libraries represent collectively the strongest research resource collection the world has ever known. The results of the studies to date underscore and demonstrate the benefits of resource sharing to scholarship. They also suggest that a number of options are available to developers interested in collaborating on ways to enhance the use of resources for both enriching the base of seldom used research materials, and for reducing, as local policies may dictate, areas of unnecessary redundancy among participating libraries.

Supplemental guidelines to the segments of the RLG Conspectus

Experiences of RLG and ARL test libraries in completing the Conspectus had demonstrated the need for prospective measures to guide librarians in reporting values to the Conspectus. This has led to the preparation of what are called “supplemental guidelines” for each appropriate segment or sub- segment of the Conspectus. These supplemental guidelines are prepared by small committees of expert bibliographers, and consist of appropriate percentages of holdings for the subject from certain standard reference, bibliographic, or citation guides to the literature of a specific field.

For example, the supplemental guidelines to the Natural History and Biology Conspectus contain a brief description of the nature of library literature supporting biology, and the suggestion of certain standard guides and periodical indexes to the literature. The definitions of levels 2, 3, 4, and 5 are then expanded to include certain proportions of holdings from these indexes or guides that should be held at each subsequent level.

For the basic information level 2, for example, a few indexes, such as the Biological and Agricultural Index, and a selection of general periodicals, including 30 % or more of the biological titles in the “Periodicals Indexed” section of Biological and Agricultural Index, are called for. At instructional support level 3, the collection should include a wide range of monographs, access to non- bibliographic databases, and general texts. This level should also include the major indexing and abstracting services in the field, and a wide range of basic serials, including 90% or more of the titles pertinent to the subject described in the “Periodicals Indexed” section of the Biological and Agricultural Index. Each successive level of intensity is similarly outlined in terms of holdings. These supplemental guidelines allow a reporting library to test reported values against some generally accepted standards, and may also provide the basis for useful comparative overlap studies to examine the distribution of the literature within RLG libraries.

Verification studies: Design and implementation

By Jim Coleman Program Assistant Research Libraries Group, Inc.

Study design

Verification study proposals should begin by clearly defining the purpose and goals of the instrument to be developed. At its most basic, such a statement will include: 1) the scope of the topic to be investigated (such as “French Literature since 1789”); 2) the Conspectus subject lines associated with the topic (if appropriate); 3) the purpose of the study (e.g., to test coverage of the field among institutions, to examine relative strength of collections at certain levels, to set a dividing line between collection levels); and 4) the method to be employed to attain these goals.

Previous studies at RLG have focused their efforts on testing 1) absolute strengths of existing collections (as measured against the percentage of total holdings of the sample universe); 2) relative strengths of existing collections (study participants measured against each other); 3) absolute and relative strengths of collections within such subgroupings as format and collection type (e.g., “basic” or “research” collections); and 4) measuring collection sizes (shelf list counts). Verification studies testing collection coverage, absolute and relative strengths, and overlap are interested in describing and comparing retrospective populations, while population size measurements are interested in quantifying these populations. In designing verification studies, it is important to remember that different goals and purposes will call for different instruments and sampling techniques.

Studies testing holdings and the associated Conspectus lines descriptively require particular attention to strategies that will support classification of sections (or titles) by type, e.g. “basic,” “informational,” “research.” That is, a verification study will be most helpful if its structure helps uncover the different collection characteristics associated with collection levels as defined within the Conspectus. This may, for example, be accomplished by devising sections verifying “core” materials. Studies that are interested in fine differentiations within levels, e.g. relative strengths of research level collections, can be of use in specialized situations, but are not of greatest need at present.

Where the potential range or size of a population is unknown or seriously in doubt, procedures to quantify will be useful as a measure of the size of the total universe, and the range within which institutional holdings may fall. However, shelf list measurements generally present greater methodological difficulties when used for Conspectus verifications. Simple shelf list measurements based strictly on call number ranges are generally inappropriate for testing Conspectus values, although they may be of use in assessing special formats, such as journals or multiple editions, or in subject fields overwhelmingly dominated by a single format.

Descriptive studies: Retrospective materials

General methodology for descriptive verification studies involves defining the universe of titles to be tested within the specified scope of the study and then deriving a sample from this universe. For retrospective materials, this generally means identifying the bibliographic sources whose range is closest to that of the desired universe. Studies stratified according to collection type will almost always require employing multiple sources, using one or more sources for each stratified section. While it may not always be possible to be certain that these sources both define and exhaust the universe actually, the closer one approaches such exhaustiveness the more accurately the study will reflect the constitution of institutional holdings. This is the point in the study design to resolve particularly difficult questions (see, for example, the comments under the “Editions” part of section II). If the subject area or sources are too narrowly defined, it is likely that no discernible differences among institutions, other than fluctuations in percentage of total, will emerge. If drawn too widely, collection weaknesses or differences may be indicated where none, in fact, exist. In both instances, the results may heighten the incredulity with which “scientific” measurements of collections are often met, and vitiate the results of the exercise.

When selecting bibliographic sources, consideration must be given to their strengths, weaknesses, and age. Supplemental sources may be necessary to overcome any difficulties with the main sources that would adversely affect the study. If appropriate, the study can include a discussion of the limitations or potential drawbacks of the design.

Descriptive studies: Current collecting

Verification of current collecting activities should rely, wherever possible, on existing RLG supplemental guidelines. While these sources need not in every case comprise the complete sample universe, they should be used as the basis. Additional sources may be used as appropriate.

Additional considerations

When using several bibliographic sources, citation overlaps are to be expected. A description of the methods used, if any, to eliminate these overlaps should be included. The proposal should give the estimated or exact total of the universe from which the sample is being drawn; the number of titles to be drawn for the sample; and a description of the sampling techniques employed. For most purposes, sample sizes ranging from 300 to 600 titles will be adequate. Only those studies covering very broad subject areas or stratifying sections over several subject areas should consider samples in excess of 600 titles.

Study implementation

The accuracy and confidence we can invest in any verification study is a direct result of our certainty that like things are being tested. The implementation of a verification study goes hand in hand with its conceptual design: it is therefore important to consider at the outset what circumstances will lead to a participating library counting a title as a “hit.”

Four areas have given rise to the most frequent questions for past studies: journals, editions, alternate formats, and alternate languages. These comments are presented as suggestions for handling the most usual conditions. Procedures that vary substantially from the guidelines presented below should be used only for compelling reasons. No matter what principles or procedures are used to determine sample size and hit determination, resist all temptations to deviate from them for the sake of expediency or convenience.

Journals. Depending on the universe designation methods, journals will generally be considered either as individual items or as runs. As individual titles, they should be counted as hits only if individually held. As runs, levels of acceptable holding should be indicated, e.g., 65% of total titles. In such instances, also providing the total number of titles expected will aid participants. The most stringent demands would require a current subscription and retrospective holdings of at least 75%.

Editions. In general, acceptable editions should be identified before sampling takes place. If only certain editions lie within the universe under consideration, all others should be eliminated. If the range of editions is considered of importance, then no editions should be excluded. In either case, a hit is attained only when an edition is matched exactly, or it can be determined that another edition contains material identical to that of the edition requested. A less stringent procedure would offer a choice of editions. The least stringent (and least preferable) would claim any edition as a hit.

Alternate formats. Reprints and microforms count as hits if they are identical to the requested material.

Alternate languages. As in the case of editions, language considerations should be resolved before sampling. Alternate languages (translations) are generally not counted as hits. Wherever the title to be checked, however, is itself a translation, the original material should be counted unless there are reasons that dictate otherwise (philological apparatus, etc.).

Citations should follow Library of Congress in most instances, and partial names or initials should be avoided. Citations should be verified before the study is distributed generally.

Lastly, study designers should include a preliminary reporting form in their proposal, and indicate the length of time they estimate necessary to complete the study.

Copyright © American Library Association

Article Views (Last 12 Months)

No data available

Contact ACRL for article usage statistics from 2010-April 2017.

Article Views (By Year/Month)

January: 2
February: 1
January: 0
February: 0
March: 0
April: 0
May: 0
June: 0
July: 0
August: 5
September: 5
October: 3
November: 1
December: 7