Assessing the Value and Impact of Digital Content Brinley Franklin Vice Provost for University Libraries University of Connecticut March 1, 2007

Measuring Print Collection Usage Did we ever know what really happened in those “miles of aisles?” “It is useless to tell the acquisitions librarian that half the monographs ordered will never be used, unless we can specify which 50% to avoid buying.” (Galvin and Kent, 1977)

The Brave New World of Digital Content

Digital Content is Truly Omnipresent

ARL New Measures Initiative In January, 1999, the ARL Statistics and Measurement Committee and ARL Research Library Leadership and Management Committee, under Carla Stoffle’s leadership, initiated the ARL New Measures Initiative in response to: 1.Increased demand for libraries to demonstrate outcomes/impacts (instead of inputs and outputs) in areas important to the institution 2.Increasing budgetary and political pressure to maximize efficient use of resources and to identify best practices

ARL New Measures Initiatives Higher Education Outcomes Research Review –Learning Outcomes –Information Literacy (SAILS)* –Research Outcomes Investigation of Cost Drivers Technical Services Cost Study Document Delivery/Interlibrary Loan Study E-Metrics (Electronic Resources Measures)* StatsQUAL TM LibQUAL +TM * DigiQUAL TM* MINES for Libraries TM* Adapted from: Peter T. Shepherd and Denise M. Davis, “Electronic Metrics, Performance Measures, and Statistics for Publishers and Libraries: Building Common Ground and Standards” portal: Libraries and the Academy, Vol. 2, No. 4 (2002)

Vendor Supplied Data and Transaction Based Usage E-metrics (aka ARL Supplementary Statistics), ProjectCOUNTER, the Penn Library Data Farm, SUSHI, etc.

Reliance on Vendor Statistics Vendor statistics, while more reliable than in the past, are still maturing.

Transaction Based Usage What do those big usage numbers mean? It takes some serious analysis to understand what 1,354,478 Science Direct uses really means

Drexel University Library D-Lib Magazine (2002) Medical Branch Library of the University Library Muenster, Germany Health Information and Libraries Journal (2003) Nonsubscription Periodicals Costs CLIR Study (June 2004) Unit Costs of Electronic vs. Print Journals

Drexel University Total Unit Cost per Use Adapted from: C.H. Montgomery and D.W. King, “Comparing Library and User Related Costs of Print and Electronic Journal Collections: A First Step Towards a Comprehensive Analysis,” D-Lib Magazine, October Print Journals Current Journals Bound Journals Total Print Journals Electronic Journals Individual Subscriptions Publisher’s Packages Aggregator Journals Full-Text Database Journals Total Electronic Journals Recorded Use 15,000 9,000 24,000 23, ,000 20, , ,000 Subscription Cost per Use $2.50 NA $2.50 $3.20 $2.25 $1.35 $0.40 $1.40 Operational Cost per Use $6.00 $30.00 $15.00 $0.45 Unit Cost per Use $8.50 $30.00 $17.50 $3.65 $2.70 $1.80 $0.85 $1.85

Medical Branch Library, University Library Muenster, Germany Adapted from: Oliver Obst, “Patterns and Cost of Printed and Online Journal Usage,” Health Information and Libraries Journal Unit Costs of Print and Online Journals (in Euros) Academic Blackwell Elsevier High Wire Springer Total 67,533 € 35,742 € 60,143 € 8,984 € 85,353 € 257,737 € 3,350 1, ,223 1,679 13,795 3,593 6,329 8,248 25,975 20,346 64,491 Print Subscription Cost Print Usage Online Subscription Cost Online Usage Online Unit Cost 54,241 € 30,380 € 54,139 € 8,086 € 76,081 € 223,647 € € € € 2.79 € € € Print Unit Cost € 4.80 € 6.56 € 0.31 € 3.77 € 3.47 €

Annual Nonsubscription Costs for Print and Electronic Journals Schonfeld, King, Okerson, and Fenton, The Nonsubscription Side of Periodicals: Changes in Library Operations and Costs between Print and Electronic Formats, Council on Library and Information Resources, June Print Journal Backfiles Cost Per Title ($US) $ $ $ $ $ 7.28 $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ 0.88 $ 1.10 $ 0.32 $ 0.63 $ 0.39 $ 1.80 $ 0.98 $ 0.18 $ 1.63 $ 1.15 Bryn Mawr Franklin and Marshall Suffolk Williams Drexel George Mason Western Carolina Cornell NYU Pitt Yale Current Print Journals Cost Per Title ($US) E-Journals Cost Per Title ($US)

University of Connecticut Libraries Database Unit Costs Special thanks to: Deborah Sanford, Stephanie Willen Brown and Nicholas Eshelman (UConn Libraries) $0.44 $0.22 $0.86 $1.40 $0.73 WorldCat JSTOR FirstSearch Web of Science Total 38, ,722 15, , ,674 $17,000 $30,000 $13,000 $165,000 $225,000 30,131 49,439 13,700 66, ,690 $0.37 $0.62 $0.47 $2.17 $1.20 FY 2005 Searches/ Downloads Approximate FY 2005 Cost FY 2003 Searches FY 2003 Cost per Search $11,160 $ 30,650 $ 6,375 $ 144,039 $192,224 FY 2003 Cost Approximate FY 2005 Unit Cost

Measuring Digital Content Use The most popular current method of measuring usage of electronic resources by libraries is not through web- based usage surveys, but through vendor supplied data of library patron usage or transaction usage. Web-based usage surveys are increasingly relevant in the collection of usage data to make collection development and service decisions, to document evidence of usage by certain patron populations, and to collect and analyze performance outputs. Brinley Franklin and Terry Plum, “Successful Web Survey Methodologies for Measuring the Impact of Networked Electronic Services (MINES for Libraries TM )” IFLA Journal 32 (1) March, 2006

An Infrastructure of Assessment Using web-based surveys to probe deeper into digital content use

An Assessment Gateway Setting up an authenticated entrance point to networked resources

An assortment of library assessment tools that collectively describe the role, character, and impact of physical and digital libraries. Consists of LibQUAL+™, DigiQUAL™, and MINES for Libraries™, as well as a growing dataset of survey results that offer insights into contemporary library use and user satisfaction. StatsQUAL™ operates under the leadership of the Association of Research Libraries (ARL), which has a long-standing role in the development, testing, and application of performance measures, statistics, and management tools.

LibQUAL+ TM Used to identify user perceptions of service quality and gaps between users’ desired, perceived and minimum expectations Consists of 22 core items and “a box” Can be classified into three subscales: Affect of Service; Library as Place, and Information Control Utilized by more than 1000 libraries in the last seven years A new LibQUAL+ study looks at responses from more than 225,000 LibQUAL+ participants in the U.S. between 2004 and 2006 The most desired core item was: Making electronic resources accessible from my home or office The second most desired core item was: Print and/or electronic journal collections I require for my work The third most desired core item was: A library web site enabling me to locate information on my own

A grant-funded project that represents a modification of the LibQUAL +TM protocol. In its initial implementation, DigiQUAL TM evaluated digital library services provided for the user communities of the National Science, Math, Engineering and Technology Education Digital Library (NSDL) program. DigiQUAL TM was pilot tested on the Digital Library for Earth System Education (DLESE), the Computational Science Education Reference Desk (CSERD), Utopia, The Math and the Multimedia Educational Resource for Learning and Online Testing (MERLOT).

A web-based transactional survey that collects data on users’ demographics and their purpose of use. It is administered in real time over the course of at least a year using a random moments sampling plan. MINES for Libraries TM has been administered at 40 North American universities in the last four years. More than 100,000 North American networked services users have been surveyed using a standard protocol.

Library User Survey Patron Status

Library User Survey Affiliation

Library User Survey Location

Library User Survey Purpose

Sample Survey Data File Generated Other UConnInstruction/Education/Departmental (Non-Funded) Researchhttp://newfirstsearch.oclc.org/done=referer;dbname=WorldCat;autho= ;FSIP12:36:5012/3/2004Off CampusUConn Faculty Family StudiesInstruction/Education/Departmental (Non-Funded) Researchhttp:// bin/jstor/gensearch12:37:4312/3/2004Off CampusUConn Undergraduate Student Non-UConnOther Activitieshttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First13:08:4112/3/2004Off CampusNon- UConn Non-UConnInstruction/Education/Departmental (Non-Funded) Researchhttp://magic.lib.uconn.edu/index_real.html13:31:2912/3/2004Off CampusNon-UConn Non-UConnOther Activitieshttp://magic.lib.uconn.edu/index_real.html12:11:0612/3/2004Off CampusNon-UConn Agriculture & Natural ResourcesInstruction/Education/Departmental (Non-Funded) Researchhttp://magic.lib.uconn.edu/index_real.html12:33:5712/3/2004 Off CampusNon-UConn EducationInstruction/Education/Departmental (Non-Funded) Researchhttp:// Off CampusNon-UConn Non-UConnInstruction/Education/Departmental (Non-Funded) Researchhttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First13:28:5212/3/2004 Off CampusNon-UConn Business AdministrationOther Activitieshttp://homerweb.lib.uconn.edu/cgi- bin/Pwebrecon.cgi?DB=local&PAGE=First12:56:4612/3/2004In the LibraryUConn Faculty Liberal Arts & SciencesOther Activitieshttp:// Campus - StorrsUConn Graduate Student EngineeringInstruction/Education/Departmental (Non-Funded) Researchhttp://homerweb.lib.uconn.edu/cgi- bin/Pwebrecon.cgi?DB=local&PAGE=First12:04:3112/3/2004On Campus - StorrsUConn Graduate Student Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://proquest.umi.com/pqdweb?RQT=31812:16:3312/3/2004On Campus - StorrsUConn Graduate Student Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp:// Campus - StorrsUConn Graduate Student Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://homerweb.lib.uconn.edu/cgi- bin/Pwebrecon.cgi?DB=local&PAGE=First12:29:5312/3/2004On Campus - StorrsUConn Graduate Student Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://homerweb.lib.uconn.edu/cgi- bin/Pwebrecon.cgi?DB=local&PAGE=First12:48:4112/3/2004On Campus - StorrsUConn Graduate Student Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://proquest.umi.com/login?COPT=SU5UPTAmVkVSPTImREJTPTE3MjErMysxNkJD&clientId= :04:2312/3/2004On Campus - StorrsUConn Graduate Student

MINES for Libraries TM U.S. Main Libraries Total Users n = 25,698 U.S. Medical Libraries Total Users n = 31,883 OCUL (Canada) Libraries Total Users n = 20,300 Location of Electronic Resources Users

Jonathan – the University of Connecticut Husky Mascot (circa 1991) Demographics

Demographics by Location of User U.S. Main Libraries On Campus, Not in the Library n = 6,391 Inside the Library n = 9,172 Off-Campus n = 4,953 MINES for Libraries™

Off-Campus n = 5,133 Demographics by Location of User U.S. Medical Libraries Inside the Library n = 6,819 On Campus, Not in the Library n = 19,582 MINES for Libraries™

MINES for Libraries TM Off-Campus n = 9,163 Demographics by Location of User Ontario Council of University Libraries Inside the Library n = 4,047 On Campus, Not in the Library n = 7,090

Purpose of Use Are users engaged in coursework, funded (or unfunded) research, public service, patient care, or other activities?

Purpose of Use By Location U.S. Main Campus Libraries 2003 – 2005 *72% of sponsored research usage of electronic resources occurred outside the library; 83% took place on campus. On-Campus, Not in the Library n = 9,460 In the Library n = 9,733 Off-Campus n = 7,790 Overall Use n = 26, %

Purpose of Use By Location U.S. Medical Libraries 2003 – 2005 *83% of sponsored research usage occurred outside the library. 92% of this use took place on-campus. On-Campus, Not in the Library n = 15,948 In the Library n = 6,590 Off-Campus n = 4,852 Overall Use n = 27,390

OCUL Scholars Portal Users Purpose of Use In a sample of 20,300 electronic resources uses at OCUL libraries, there were four uses outside the library for each use in the library.

Recent Advances in Capturing All Networked Services Use Through Web-based Surveys An advanced application of EZproxy at UTMB presents the MINES survey to networked services users as they initiate a session, thereby capturing almost all networked services usage both locally and remotely UCSD is capturing all on-campus and off-campus networked services usage through the proxy server or virtual private network (VPN) at the campus Internet router. This approaches catches all authorized users’ usage during the randomly selected two hours per month MINES surveys.

Look How Far Libraries Have Come (in less than a decade) For most of our history, evaluation methods for print collections yielded unreliable data for decision-making International projects have identified relevant data elements and set standards for commercial electronic resource usage; standards have also recently been developed for harvesting usage data Libraries, consortia (and publishers) routinely calculate unit costs for electronic resources and perform cost/benefit analyses LibQUAL+, and DigiQUAL and other web-based user surveys have measured user satisfaction with electronic resources and digital library services and reinforced users desires for digital content MINES and other web-based user surveys provide another level of insight into user behavior in the networked environment The coming years will see further developments that will even further refine data about users and how they utilize digital content

Questions? This presentation is available at: Contact the author at: Learn more about LibQUAL +TM, DigiQUAL TM, & MINES for Libraries TM at: