New measures for new services Indicators for quality, cost, and impact of electronic library services Roswitha Poll Münster LIDA 2007 LIDA 2007

The new electronic library services catalogue databases portals online ordering and delivery personalized services online reference digitized collections electronic publishing online self-paced training Internet access via the library LIDA 2007 LIDA 2007

Reasons for evaluation of quality and impact accountability special funding needed: evidence of positive effects results-based budgeting justifying investment into change resource management rising expenses for new services increasing demand for new services new allocation of resources necessary LIDA 2007 LIDA 2007 promotion of the library‘s role communication of benefits by using library services

What library standards want to achieve to spread the knowledge about existing tested methods and practices to standardize with the aim of comparing and aggregating results terminology and definitions methods of data collection methods of analyzing the data LIDA 2007 LIDA 2007

ISO Technical Information and committee 46: documentation Subcommittee 8:Quality, statistics and performance evaluation WG 2: International library statistics ISO 2789 (2006) International library statistics 4th edition WG 4: Performance indicators for libraries ISO DIS (2006) Library performance indicators 2nd edition WG 7: Quality measures for National Libraries ISO TR Performance indicators for National Libraries in preparation LIDA 2007 LIDA 2007

Stages of an ISO standard Approved New Work Item Working Draft Committee Draft (CD) Draft International Standard (DIS) Final Draft International Standard (FDIS) Standard voting required LIDA 2007 LIDA 2007

Problems of standardizing data for electronic services  q uick changes in media and information ways  changes not always predictable  no clear boundaries of documents and localities  data must be collected from many sources (vendors, suppliers, institutional servers) State of technical development varies in libraries LIDA 2007 LIDA 2007

In terms of collection size number and type of users use of services staff space equipment training LIDA 2007 LIDA 2007 Assessing quantity (ISO 2789)

problem of counting can be counted manually OPACOPAC web- site electronic collection Internet access via the library electronic document delivery online reference user training on electronic services Electronic library services: counting use LIDA 2007 LIDA 2007

Electronic collection databases electronic serials electronic serials digital documents full text abstract and indexing abstract and indexing other electronic patents electronic patents networked AV-docu- ments other eBooks ISO 2789 computer files computer files LIDA 2007 LIDA 2007

Forms of use sessions rejected sessions contents or records download contents or records download searches Internet sessions Internet sessions virtual visits virtual visits session time inside the library outside the library, but inside the institution/ population outside the library, but inside the institution/ population from outside the institution/ population from outside the institution/ population ISO 2789 LIDA 2007 LIDA 2007

ISO 2789: Definitions of electronic use LIDA 2007 LIDA 2007 search specific intellectual query session successful request of a database or the online catalogue rejected session unsuccessful request exceeding the simultaneous user limit session time period between log-in and implicit or explicit log-off content unit downloaded content unit being successfully requested record downloaded descriptive record being successfully requested Internet access by a user from inside the library Internet session

a user's request on the website from outside the library premises web browser visit web browser visit IP visit home page visit home page visit ISO 2789: virtual visits LIDA 2007 LIDA 2007

Counting electronic usage in German academic libraries: Percentage of libraries delivering data OPAC sessions45.3 %39.8 % Database sessions 28.0 %31.1 % E-journal sessions 25.8 %32.0 % Digital document sessions 21.8 %17.0 % Downloads from databases 7.1 %9.2 % Downloads from e-journals 9.3 %13.1 % Virtual visits36.0 %34.5 % LIDA 2007 LIDA 2007

In terms of user-orientation reliability currency usability/accessability competence responsiveness speed cost-effectiveness perceived quality LIDA 2007 LIDA 2007 Assessing quality (ISO 11620)

ISO 11620: Library performance indicators resources, access, infrastructure use efficiency potentials and development New structure: Balanced scorecard LIDA 2007 LIDA 2007

Indicators for "hybrid" services ISO resources, access, infrastructure percentage of required titles in the collection speed of interlibrary lending use library visits per capita (including virtual visits) percentage of population reached user satisfaction efficiency cost per library visit cost per user correct answer fill rate potentials, development attendances at training lessons per staff member LIDA 2007 LIDA 2007

Indicators for electronic services ISO resources, access, infrastructure percentage of rejected sessions public access workstations per capita use number of content units downloaded per capita percentage of information requests submitted electronically workstation use rate efficiency cost per database session cost per content unit downloaded potentials, development percentage of acquisitions expenditure spent on the electronic collection percentage of library staff providing and developing electronic services LIDA 2007 LIDA 2007

Possible performance indicator for website quality Direct access from the homepage Measured by number of clicks neccessary to find relevant information adequacy of terms used on the homepage Example: renewals Directly on the homepage: user account, my account Under broader terms: circulation, loan service LIDA 2007 LIDA 2007

Performance indicators for costs ISO costs include total recurrent expendituretotal recurrent expenditure cost per loan cost per database session costs include contractual costs of a databasecontractual costs of a database cost per content unit downloaded costs include contractual costs of an electronic resourcecontractual costs of an electronic resource LIDA 2007 LIDA 2007 Cost issues

Changing from print to electronic collections What changes for the library? decrease of traditional services (loans, visits, copying) increase of electronic use In use in costs less costs for storage, binding higher systems and staff costs better qualified staff needed lower cost per use overall: increase in use LIDA 2007 LIDA 2007 Cost issues

Changing from print to electronic collections What changes for the users? access from the working-place or home access at once 24-hours access searching options, links In use in costs time saved ! no ways, no fetching of materials, no waiting in queues no waiting in queues additional value academics spend half a day per week on seeking and procuring information LIDA 2007 LIDA 2007 Cost issues

Journals: cost per use Print only subscription costs usagecost per use E-only subscription costs usagecost per use € € € € Users accessed online versions 10 times more than print versions Only subscription costs are considered! Example 1: Medical library Müster LIDA 2007 LIDA 2007

Journals: cost per use Space Costs Cost type Electronic Journals Print Journals (current) Print Journals (bound) Space$ 5,000$40,000$205,000 Systems$ 10,000$ 2,500$ 2,400 Supplies & Services$ (2,000)$ 600$ 8,000 Staff $125,000$46,000$ 42,000 TOTAL OPERATIONAL COSTS$138,000$90,000$258,000 Operational costs of print and electronic journals Montgomery, Carol Hansen, Donald W. King: Comparing library and user related costs of print and electronic journal collections. and user related costs of print and electronic journal collections. D-Lib Magazine October 2002 Drexel University Philadelphia Example 2: Drexel University LIDA 2007 LIDA 2007

Space Costs Staff costs by function (Drexel) E- journals Print (current) Print (bound) Director/Administration$ 22,00000 Communications (among staff)$ 7,000$ 2,500$ 800 Collection development$ 18,000$ 6,0000 Acquisitions$ 8,000$ 4,0000 Physical handling: bindery, labeling, reshelving0$10,000$22,000 Record creation and maintenance$ 16,000$10,000$ 6,000 Public relations $ 11,000$ 1,000$ 600 Reference$ 37,000$ 7,000$ 4,000 Teaching (users)$ 6,000$ 1,000$ 600 Other0$ 4,000$ 6,000 $125,000$46,000$42,000 LIDA 2007 LIDA 2007

Cost per use by journal type (Drexel) recorded use subscription cost per use operational cost per use total cost per use print current $ 2.50$ 6.00$ 8.50 print bound $ total print24.000$ 2.50$ 15.00$ total E- journals $ 1.40$ 0.45$ 1.85 LIDA 2007 LIDA 2007

Journals:cost per use Problems in measuring cost per use package deals for journals consortia with "additional access" publishers insist on "print + electronic" For subscription prices For measuring use reshelving counts may under-estimate use E-journal use data may differ between publishers For assessing operational costs assessing staff time will be time-consuming long-term preservation costs for E-media still questionable High acceptance and use of E-media will evidently result in lower cost per use LIDA 2007 LIDA 2007

In terms of knowledge information literacy academic or professional success social inclusion individual well-being changes in skills, competences, attitudes, behaviour Assessing impact/outcome LIDA 2007 LIDA 2007

Methods for assessing impact Use statistics as measures of impact electronic media frequency of use forms of use (sessions, searches, downloads) range of resources used change of use in traditional services change in remote use number of new users that had not used the library that had not used the same service in traditional form percentage of the primary user group using E- services Users are familiar with the use of electronic resources and services LIDA 2007 LIDA 2007

Qualitative methods Surveys (postal, face-to-face, telephone, online) Interviews Focus groups, discussion groups Self assessment of users Data and „stories“ must be brought in a systematic order in order to make trends visible The "anecdotal evidence" supports the statistics Methods for assessing impact LIDA 2007 LIDA 2007

Quantitative methods Tests of user skills before and after training Observation by trained observers Data mining Analysis of bibliographies in publications of users Comparing library use to academic success Problem: Influences on individuals are manifold. The library‘s influence not always clear Methods for assessing impact LIDA 2007 LIDA 2007

Example: Example: Changing the journal collection to electronic form Statistical results Survey to faculty Assessing the outcome of electronic services  online versions were accessed 10 times more  usage of an online article 5.4 times cheaper  high decline in print usage  remote use increased  physical library visits decreased In 1998 a number of academics were sceptical In 1998 a number of academics were sceptical as to E-journals as to E-journals In 2002 it was exceptional for faculty staff not to use the electronic version In 2002 it was exceptional for faculty staff not to use the electronic version LIDA 2007 LIDA 2007 change of attitudes and behaviour

Projects of assessing outcome eVALUEd: in "evidence base", University of Central England: Toolkit for evaluating electronic information services IMLS (Institute of Museum and Library Services): outcome-based evaluation of projects ARL New Measures Initiative: mehrere Projekte - Learning outcomes - Higher education outcomes research review - MINES (Measuring the impact of networked electronic services) SCONUL and LIRG (Library and Information Research Group): impact initiative and mailing list LIDA 2007 LIDA 2007

Standardization of impact measures? projects still on the way IFLA Section Statistics and Evaluation has instituted a working group Bibliography of projects and literature: LIDA 2007 LIDA 2007

Experienceis the mother of science Traditional Proverb Traditional Proverb LIDA 2007 LIDA 2007