Measuring the impact of new library services Dr. Roswitha Poll Münster

Impact / outcome Outcomes are the results of library use as affecting the individual user Impact shows not the quality of the service, but the "quality" of individuals in consequence of using the service

Impact / outcome of cultural institutions knowledge information literacy academic or professional success social inclusion individual well-being

Impact / outcome of library services changes in skills and competences changes in attitudes and behaviour changes in the structure of the library's clientele higher social inclusion higher success in research, study, or profession

Impact of a library's services on the outcomes of its parent institution of its parent institution Impact of a library's services on the outcomes of its parent institution of its parent institution recruitment and retention of students recruitment and retention of excellent academic staff effective teaching - high graduation rates - high grades in examinations - high employment rates after examination effective research - high renown and use of research results and publications - renown of faculties and research groups - high amount of special grants - awards, honours Outcomes of universities The library can support nearly all of these goals

accountability Reasons for measuring impact of new services management of resources promotion of the library's role special funding needed: evidence of positive effects results-based budgeting justifying investment into change rising expenses for new services higher workload when introducing new services increasing demand for new services new allocation of resources necessary communication of benefits

New services New services (includes services replacing or complementing traditional services) (includes services replacing or complementing traditional services) New services New services (includes services replacing or complementing traditional services) (includes services replacing or complementing traditional services) Electronic services (examples) New non-electronic services (examples) regional/national catalogue databases portals online ordering and delivery personalized services online reference digitized collections electronic publishing online self-paced training Internet access via the library group work areas study landscapes for problem-based learning

. Methods of measuring impact of new services Problems of measuring Most tested methods are time-consuming Data not available because of data protection rules Results of projects not comparable because of different data collection methods Services have different outcome and value for different user groups It is difficult to prove that changes in competences or behaviour are indeed due to library use Influences on individuals on individuals are diverse

Methods of measuring impact of new services Use statistics as measures of impact new electronic media frequency of use location of use forms of use (sessions, searches, downloads) range of resources used before and after introducing a new service use statistics for a new service (e.g. online delivery) change of use in traditional services change in remote use number of new users that had not used the library that had not used the same service in traditional form percentage of the primary user group using electronic services Users are familiar with the use of electronic resources and services

Methods of measuring impact of new services Qualitative measures Asking users: print or online surveys, focus groups, interviews What knowledge have they of the new service? Use of services (frequency, location of use) Did they find the new service useful? What problems? Did they get training for a new service? useful? Have they improved skills by using a new service? Exit surveys most useful

Methods of measuring impact of new services Qualitative measures Asking academic teachers Do new services support teaching and research? Do they encourage students to use new services? For what purpose do they use new services? Asking library staff What help do users need for what service? Was there a change of skills after the introduction of a new service? Anecdotalevidence

Methods of measuring impact of new services Other methods tests: can assess user skills before and after training on a new service performance monitoring / data mining: can document changes in searching by a new service "mystery shoppers": can report on failures or success and on improvements they perceived in their skills analysis of documents and bibliographies compiled by users: Is there a change in - resources used - accuracy of citations - number of electronic resources cited? expected outcomes – according to goals unexpected outcomes

Practical examples Implementing a regional portal for academic libraries Methods used: use statistics, surveys of academics Statistical results Survey results: Academics said they Survey results: Academics said they Higher percentage of external users Higher use of most electronic resources saved time by the integrated search saved time by the integrated search changed their seeking ways changed their seeking ways found access easier found access easier found broader covering of relevant information found broader covering of relevant information became acquainted with new resources became acquainted with new resources used more interdisciplinary search used more interdisciplinary search integrated the portal into teaching integrated the portal into teaching change of competences,attitudes,behaviour

Practical examples Opening a group learning area Opening a group learning area Methods used: use statistics, survey to students Statistical results Survey results: Students said they Survey results: Students said they Physical visits to the library have increased work more frequently in the library work more frequently in the library work more frequently in groups work more frequently in groups learn from group members who attended library training on electronic services learn from group members who attended library training on electronic services find working in the library more attractive find working in the library more attractive change in attitudes, behaviour, personal well-being

Practical examples Changing the journal collection to electronic form Changing the journal collection to electronic form Methods used: use statistics, survey to faculty Statistical results Survey results:   online versions were accessed 10 times more In 1998 a number of academics were sceptical In 1998 a number of academics were sceptical as to E-journals as to E-journals In 2002 it was exceptional for faculty staff not to use the electronic version In 2002 it was exceptional for faculty staff not to use the electronic version   usage of an online article 5.4 times cheaper   high decline in print usage   remote use increased   physical library visits decreased change of attitudes and behaviour

Practical examples Implementation of an inquiry-based instruction program Method used: Analysis of bibliographies in students' research papers Results of the analysis   increase in citations to scholarly journals   no significant differenc in other topics   accuracy of citation   currency of resources   competence of judging higher awarenes of scholarly journals

Impact of new services on the library Changes in organization and resource allocation and resource allocation Changes in organization and resource allocation and resource allocation investment of funds, room, staff time staff training user training change of workflows changes in organizational structure Replacement of traditional by electronic services: possible improvements in Replacement of traditional by electronic services: possible improvements in speed of delivery accuracy of delivery ease of access relevance for users market penetration

Impact of new services on the library Impact on costs Probably transforming a service from traditional to electronic form will not reduce costs But: Higher use will reduce cost-per-use Impact on traditional services possible decrease in - circulation - copying - physical visits may be counterbalanced by - comfortable surroundings in the library - group working areas - good in-house IT equipment

Outcome projects eVALUEd: in "evidence base", University of Central England: Toolkit for evaluating electronic information services IMLS (Institute of Museum and Library Services): outcome-based evaluation of projects ARL New Measures Initiative: several projects - Learning outcomes - Higher education outcomes research review - MINES (Measuring the impact of networked electronic services)

Outcome projects IBEC: Information School of the University of Washington and University of Michigan School of Information: Toolkit for assessing the impact of information in communities SCONUL and LIRG (Library and Information Research Group): impact initiative and mailing list IFLA Section Statistics and Evaluation: working group on outcome/impact; bibliography of literature and projects worldwide

Don't throw away the old bucket until you know whether the new one holds water. Swedish Proverb whether the new one holds water. Swedish Proverb A new broom sweeps clean, but an old broom knows the corners. Virgin Islander Proverb but an old broom knows the corners. Virgin Islander Proverb Men learn little from success, but much from failure. Arabian Proverb but much from failure. Arabian Proverb To change and to improve are two different things. German Proverb are two different things. German Proverb No matter how much the world changes, cats will never lay eggs. Bambara Proverb changes, cats will never lay eggs. Bambara Proverb