Why assess? What is assessment? What do we mean by actionable data? Martha Kyrillidou Planning for Results: Making Data Work for You The Cato Institute, Washington, DC February 20, 2008

of Research Libraries What’s bringing us together …

of Research Libraries Overview 1. Initiate (Why do we assess?) 2. Plan (What do we assess?) 3. Design (How do we assess?) 4. Implement (Design in action) 5. Analyze and Interpret (What do the results mean?) 6. Reflect and Initiate (What do we do with the results?) “Why assess? What is assessment? What do we mean by actionable data?” - Martha Kyrillidou (1&2) “Survey Design” – Neal Kaske (2&3) “Performance Measures” – Rick Garcia (2&3) “Analysis and Interpretation” - Ray Lyons (5) LibQUAL+®, MINES™, and Other Tools; Packaging Results to Communicate to Stakeholders Gary Roebuck and Kristina Justh (2,3,4,&5)

of Research Libraries Clear and present danger Bangor University considers removing librarians posted by Blake on Thursday January 27, -753 hits Ms Information writes "News from the University of Wales Bangor in the UK. senior management no longer feel that subject librarians / academic liaison librarians are needed in the modern academic library. They have made restructuring proposals which include removing all but one of the subject librarians and a tier of the library management, including the Head of Bibliographic Services. The university management thinks that technology has 'deskilled' literature searching. As far as I know, this proposal is unprecedented in the United Kingdom. In essence, there will remain 4 professional librarians serving a 'research-led' university of 8,000 plus FTEs and with 8 library sites. These will be the university librarian, cataloguing librarian, acquisitions librarian and Law librarian.Blake Ms Information Has anything like this happened anywhere that you know of? If so, what have been the effects?

Assessment Librarian Assessment Librarian

of Research Libraries Library of the Future Will Also Need To have it’s own data collection and management personnel, individuals who constantly collect, analyze and prepare reports on data regarding what services are being used, which portions of the collection are getting the highest usage, what materials are being lent through interlibrary loan, and who patrons are. Documenting the libraries contributions to quality teaching, student outcomes, research productivity will become critical.

of Research Libraries Technical Skills needed Technical Skills needed Quantitative Analysis: SPSS Qualitative Analysis: Atlas.ti Data collection: Custom Web interface, Survey Monkey, FormSpring Data storage: SQL Server, Excel Reporting: Crystal Reports, SPSS, SQL Reporting Services, Excel, Word Web delivery: Dundas Charts Web development: ASP.NET, C#, ColdFusion

of Research Libraries Communication Skills needed Communicating organizational vision and service ethic Leadership in capturing the future and making it happen Ability to take initiative and willingness to take risks and learn from mistakes Troubleshooting and being an independent thinker Ability to work collaboratively and move forward team and organizational goals

of Research Libraries... chart a course for every endeavor that we take the people's money for, see how well we are progressing, tell the public how we are doing, stop the things that don't work, and never stop improving the things that we think are worth investing in. –President William J. Clinton, on signing the Government Performance and Results Act of 1993

of Research Libraries What’s driving the ‘assessment’ agenda What’s driving the ‘assessment’ agenda “Times are changing” –Exploding growth in use and applications of technology –Increased customer expectations for services, including quality and responsiveness Shrinking budgets –Justification for spending $$$ on resources, programs, and services –Increasing competition for resources –Cost savings and/or reallocation –Fighting to remain relevant

of Research Libraries The Internet Goes to College Early data from ethnographic interviews –“I use Google because I heard it searches for more things” (than other sources). –“I believe I can find anything on the Internet. There hasn’t been anything I haven’t been able to find.” –“Because I’m lazy.” –Books have “so much information that no one can go through it all.” –I use “the Internet first because it is more convenient.” –I go to the library “because that’s what teachers like.” –“Google has gotten me through college.” Source: Steve Jones, The Internet Goes to College, ARL Talk

of Research Libraries Libraries Remain a Credible Resource in 21 st Century 98% agree with statement, “My … library contains information from credible and known sources.” Note. Digital Library Federation and Council on Library and Information Resources. (2002). Dimensions and Use of the Scholarly Information Environment.

of Research Libraries OCLC Reports Sharing, Privacy and Trust in Our Networked World College Student’s Perceptions: Libraries and Information Resources Perceptions of Libraries and Information Resources »OCLC:

of Research Libraries ‘Universal’ library brand Amazing stability of library user’s desired expectations across –Types of users –Languages –Countries –Some differences across disciplines 297,158 LibQUAL+® survey respondents

of Research Libraries Towards what end The emerging library user sees the need for a different kind of library – a library that serves as a source of information ready to be accessed any time, from anywhere, using modern technologies. Can we effectively translate the comfort and assurance of human interactions into a virtual reality that is trustworthy, reliable, and ever present? And will learning be quite as effective or deep in this virtual world of never-ending interaction with the collective digital-knowledge base? Thompson, Kyrillidou, Cook “Library Users’ Service Desires” Library Quarterly (2008)

of Research Libraries Library Assessment and its Global Dimensions Markets and people exposed to economic and social frameworks unheard of before Competing internationally Library users exposed to global forces Libraries facing similar challenges Libraries as the Internet Libraries as Google Libraries as Collaborative Spaces

of Research Libraries Defining Success in a Digital Environment Crafting new measures of success. Moving from measuring inputs to outputs and outcomes Understanding impact of library roles and services. Agreeing on qualitative measures of success: user perceptions, user success, creating value, advancing organizational goals. Reallocating and managing capabilities to focus on new definitions of success.

of Research Libraries Thinking Strategically About Libraries’ Futures What is the central work of the library and how can we do more, differently, and at less cost? What important set of services does the library provide that others can’t? What new roles are needed? What advantages does the library possess? What will be the most needed by our community of users in the next decade? How is user behavior changing? What should our libraries aspire to be ten years from now? What are the implications of technology driven change? What are the essential factors responsible for the success of the library?

of Research Libraries Library Assessment Library assessment provides a structured process to learn about our communities, their work and the libraries connection to what they do The information acquired through library assessment is used in an iterative manner to improve library programs and services and make our libraries responsive to the needs of our communities. Academic libraries do not exist in a vacuum but are part of a larger institution. Assessment within the institution may take place in individual areas as well as at the broad institutional level.

of Research Libraries Assessment is… Assessment is… “…a structured process to learn about communities and evaluate how well the library supports them.” - Steve Hiller, Director of Assessment & Planning University of Washington Libraries

of Research Libraries Free speech wall, Charlottesville, Sept 2006

of Research Libraries Evaluation – foci by Peter Brophy InputsProcessesOutputsOutcomesImpacts Increasing difficulty ………… Increasing usefulness ………… We have 2 million records We have the technology to transform data into information- rich products We have delivered one million documents 26 students have read this paper As a result of our service, humankin d has beaten malaria

of Research Libraries IFLA: Measuring Quality Resources, infrastructure: What services does the library offer? Use: How are the services accepted? Efficiency: Are the services offered cost- effectively? Potentials and Development: Are there sufficient potentials for future development? (an adaptation of the Balanced Score Card)

of Research Libraries Assessment at ARL A gateway to assessment tools: StatsQUAL ®: –ARL Statistics -- E-Metrics –LibQUAL+ ® -- Service Quality and Satisfaction –DigiQUAL ® –MINES for Libraries ® Library Assessment Conferences – libraryassessment.org Library Assessment Blog – libraryassessment.info SPEC Kits Service Quality Evaluation Academy ESP Assessment (Making Library Assessment Work) –Effective, Sustainable, Practical

of Research Libraries Qualitative Profile Developing New Metrics (per Yvonna Lincoln) Uniqueness of collections Defining the value of consortia Administrative and budgetary efficiencies Student outcomes/student learning/graduate success Contributions to faculty/stakeholders productivity Social frameworks/intellectual networks Generating new knowledge Creating the collective good with reusable assets

of Research Libraries How do you describe a Library? Breadth and quality of collections and services Sustained institutional commitment to the library Distinctive resources in a variety of media Services to the scholarly community Preservation of research resources Contributions of staff to the profession Effective and innovative use of technology Engagement of the library in academic planning Association of Research Libraries ‘Principles of Membership’

of Research Libraries Group exercise Share important aspects of your –Services –Collections –Collaborative Relations

of Research Libraries Tell the story … Facilities: number and type (library buildings, classrooms, studios, computer labs, cafes, offices outside of the library, etc.) University centers, services and responsibilities (managed by the library or housed in the library): slide libraries, media centers, teaching and learning centers, education centers, writing centers, distance ed or online course development/instructional design, iTunesU, laptop support, technology help desk, escholarship centers, university press or other publishing entity, university archives, university records management, etc. Digital collections: institutional/digital repositories, images, articles, simulations, data, learning objects, etc. Grants and development: numbers, types, friends organizations, advisory boards/councils, etc. Partnerships: nature and number, on campus and external Consortial memberships and roles Outreach: web hits, computer use hours, learning hours/contact/impacts (in classrooms, Info Lit, etc.), visits, external teaching, volunteering, programs offerings, sponsored lectures, etc. Leadership: on campus and external committees, boards, initiatives, projects, etc. Publications/scholarship: numbers, types (articles, books, presentations, poster sessions, performances, etc.) » from ARL Library Director – February 13, 2008

of Research Libraries Measuring Impact Concerns Based Adoption Model (CBAM) –Stages of concern How engaged are subjects in the service /innovation? How much interest do they show? –Levels of use How much use do they make? Avoid dichotomy of user / non-user –Levels of Impact Draw together concern and use Assess effects on behaviour »Peter Brophy (South Africa, August 2007)

of Research Libraries Measuring Impact Surface learning –Skim, memorise, regurgitate for tests Little long-term impact Deep learning –Involved, questioning, interactive, integration with own knowledge Significant long-term impact A habit of critical thinking »Peter Brophy (South Africa, August 2007):

of Research Libraries “If you think you're too small to have an impact, try going to bed with a mosquito.” »Anita Roddick

of Research Libraries Actionable Data What is the desired outcome of assessment? –An organization that strives to be customer-centered and acts on information about environmental factors and especially on user input. »Raynna Bowlby and Martha Kyrillidou

of Research Libraries Align with mission, vision and goals of parent organization –Are there prominent themes/directions – research support, instruction, physical facilities; priority user groups; expansion and growth or conservation and reallocation of resources; emphasis on current technologies and methods or more traditional approaches –Are there strategically favored programs/initiatives and/or individual champions with key agendas? –Are there stated goals for the organization and the library?

of Research Libraries Address users’ top priorities Are you undertaking initiatives that users indicate are their highest wants and are you doing enough? What budget/resources are being re/allocated to these initiatives? Are enough budget/resources being directed to these initiatives? Have the departments/staff that most affect these initiatives been informed? Do these departments/staff understand the data (and comments/specifics) and have goals and improvement plans in place? Are there improvement targets; how is change being measured?

of Research Libraries Identify resources that can be reallocated Is the work that you are performing addressing areas of strong dissatisfaction? Are you doing too much of what you’ve always done, rather than the work that results in user wants & needs? What budget/resources are being allocated to legacy work that is not a user priority and is it too much? Have the departments/staff that are currently doing this work been informed of changing user wants and needs? Do these departments/staff understand the data (and comments/specifics) and can they redirect their time and resources? Are there improvement targets; is change being measured?

of Research Libraries Continuous improvement and assessment How will you incorporate the results of your analysis and findings into your Library’s strategic and annual plans? How can you develop some of your more resource- intensive improvements over several years? How can you insure that your assessment results are always available when and where decisions are being made? How and when will you be assessing the user improvements implemented after your user survey?

of Research Libraries The importance of accurate data in decision making

of Research Libraries Your library … Organization and use of information is focused on disciplines and increasingly interdisciplinary centers Subject-oriented staff provide information services required and integrate them in the research process Developing and maintaining relationships with key stakeholders in the disciplines/departments Acquainting specialized stakeholders with the broad range of information and information services available outside their specialty Assuring adequate performance of network and technology based information services Assuring adequate performance of the centralized basic support functions

of Research Libraries Exercise What are the key indicators of this library's work? How do you monitor each indicator? How do you record the data for each indicator? With what frequency? To whom and how do you communicate information about the library's performance?

of Research Libraries

of Research Libraries Making Library Assessment Work ARL project approved in 2004 and funded by participating libraries Site visits by Steve Hiller (U of Washington) and Jim Self (U of Virginia) –Presentation –Interviews and meetings –Report to the Library 24 libraries in U.S. and Canada visited in Effective, Sustainable and Practical Library Assessment in 2007 –Open to all libraries –6 libraries participated in 2007

of Research Libraries What We Found Strong interest in using assessment to improve customer service and demonstrate value of library Many libraries uncertain on how to establish, maintain, and sustain effective assessment Effectiveness of assessment program not dependent on library size or budget Each library has a unique culture and mission. No “one size fits all” approach works. Strong customer-focus and leadership support were keys to developing an effective and sustainable assessment

of Research Libraries What are the lessons learned? Understanding changes in users approach to information resources. Service quality improvement is a key factor. Understanding the impact of e-resources on library services - TRL. Learning how to compete or collaborate with Google. Upfront investment in design and development. Making the assessment service affordable, practical, & effective. Assessment needs to be satisfying and fun.

of Research Libraries Multiple methods Transactional surveys* Mystery shopping New, declining, and lost-customer surveys Focus group interviews Customer advisory panels Service reviews Customer complaint, comment, and inquiry capture Total market surveys* Employee field reporting Employee surveys Service operating data capture *A SERVQUAL-type instrument is most suitable for these methods Note. A. Parasuraman. The SERVQUAL Model: Its Evolution And Current Status. (2000). Paper presented at ARL Symposium on Measuring Service Quality, Washington, D.C.

of Research Libraries Assessment can… Assessment can… Support decision making Establish accountability Strengthen organizational mission, goals, and objectives Realign library goals and mission statements with the needs of users Identify trends in user behaviors and technology Measure learning outcomes and impact Improve programs and services

of Research Libraries Use Assessment Data for Good!

of Research Libraries

of Research Libraries What’s in a word? What makes a quality library ? “Quality much like beauty is in the eye of the beholder”

of Research Libraries What’s in a “Library”? A word is not crystal, transparent and unchanged; it is the skin of a living thought, and may vary greatly in color and content according to the circumstances and time in which it is used. --Justice Oliver Wendell Holmes

of Research Libraries Interpreting Data