Recognize Progress: Measuring Outcomes Not Outputs Yvonne Attard, Director, Customer Development, Oakville Public Library Rebecca Jones, Partner, Dysart Jones Associates

Agenda Basis for discussion Measurement System: On Overview Critical success factors Options Can you really measure impact? Measurement Framework – Theory and Reality: Oakville example Gaps and hurdles Libraries change lives

Basis for discussion There is no one magic measure There IS a strong correlation between an organizations success and: Its clarity of purpose Its ability to understand its culture and its stakeholders A well-designed performance measurement system that fits that culture. Value is client or stakeholder defined Definition is dynamic It is two-dimensional: economic & psychological It is relative to alternatives

Basis for discussion Measuring for Results: The Dimensions of Public Library Effectiveness by Joe Matthews, 2004 Asserts that few public libraries have a culture of assessment –Difficult and complex –Most measures indicate past performance –No cause-and-effect relationship between measures –Performance measures measure quantitatively, but library outcomes are largely qualitative

Measurement System: an overview Impact

Measurement System - Input Inputs are basically resources or capabilities: Financial resources******* –Content –Staff –Technology –Facilities Ultimately, who enables you to obtain these inputs?

Measurement System - Output The service or program created by using the Input.

Measurement System - Outcomes What the client is able to do with the service or product; how the client uses the Output.

Measurement System - Impact The effect or influence of the service or program. What changed for the client and/or the client organization. How the library changed a life or a group of lives.

Measurement System - Example Input$100K collection budget Output2500 new resources in collection Outcomes75% of students in the community schools used collection resources Impact % of students graduating rose 5% in the past 2 years; EQAO results improved in 3 of the 5 community schools

Measurement System - Oakville Input$20,000 spent on an outreach program to students budget Output2,500 students reached Outcomes1,500 new student library cards issued during outreach program and increased use in online resources ImpactStudents assignment grades increase, ease of research, teachers view marked improvement in studies

Systems View LibraryClients Stakeholders Feedback Outputs InputsInputs ImpactImpact

Critical Success Factors For defining, measure & communicating progress & impact? Alignment of stakeholder goals and your goals –Balance of government demands vs. business plan vs. library demands Your definitions, measures & communication style match those of your stakeholders, other service organizations, non-profits and others in the local community Doable capture and analysis of measures Measures communicated in a meaningful and understandable way for stakeholders Conversations outside the library

Some Options Balanced Scorecard LibQual Cost-benefit

Balanced Scorecard Aligns measures with strategies to track progress, reinforce accountability and prioritize improvement opportunities A system of measures based on 4 perspectives: –customer –internal –financial –Innovation Limits measures to those most critical

Customer Perspective How do we look to our clients? Goals Measures Financial Perspective How do we look to our funders or stakeholders? Goals Measures Innovation Perspective How can we improve & create value? Goals Measures Internal Perspective What must we excel at? Goals Measures Kaplan & Norton Balanced Scorecard

Balanced Scorecard: Example METRIC U.1.a. OVERALL RATING IN STUDENT AND FACULTY SURVEYS. Target1: A score of at least 4.00 (out of 5.00) from each of the major constituencies: undergraduate students, graduate students, humanities faculty, social science faculty, science faculty. Target2: A score of at least 3.90 from each of the major constituencies. Method: The University Library conducts extensive surveys of our clientele approximately every other year. A final question in each survey asks the respondent to "rate your overall satisfaction…" with the Library using a 1 to 5 scale. This metric will consider the two latest surveys on record.

Benefits of Scorecard …a clear understanding of what drives value within your area and what doesnt, greater insight into senior managements strategic plans, and a better knowledge not only of the strategic role you play within the organization but how you can enhance that role and sit at the decision-making table Joseph DeFeo, Measuring What Matters. Industrial Management, v.42,n.3, May 2000

LibQual+(TM) Based on ServQual Collaboration of Association of Research Libraries (ARL) & Texas A&M University Libraries Diagnostic tool for measuring library users' perceptions of service quality Identifies gaps between desired, perceived, & minimum service expectations Defined survey questions, dimensions, & data gathering processes for academic libraries

Benefits of LibQual+(TM) Provides better understanding of patrons perceptions & desires, & how services are meeting these Libraries using this have learned that they must keep narrowing categories & questions

Cost-benefit Does the benefit exceed the cost? St. Louis Public Library Measures the value people place on the consumption of a service in excess of what they pay to receive that service Found benefits received >$10 for each dollar of tax support Other public libraries place a fair market value on their output measures San Diego Library and Miami-Date Public Libraries; benefits exceed costs by 6:1

Benefits of cost-benefit Stakeholders may easily understand Difficult, however, to determine valid market prices

Understand the context Align objectives Define measures Manage collection Interpret data Communicate results Critical Success Factor: Measurement Framework Adapted from Measuring What Matters: A Library/LRC Outcomes Assessment Manual by Lindauer

Context for your framework Who are your key stakeholders? Who do you really need to prove your value to? How do they prove themselves? What measures/indicators do they use? Ask yourself, in our environment, what is it critical for us to we measure? Page 1 of Framing Template

Framework: Align This is where you invest the most time Requires conversations with councillors, or local government superiors Profile critical stakeholders What are their goals & objectives? What makes them look good to their superiors or constituents? Page 2 of Framing Template

Align: Profile Stakeholders Stakeholders are those who can put a stake of support under your organization or a stake of destruction through your organization Superiors Funders Constituents or market Clients & potential clients

Align: Know Your Stakeholders Who are your stakeholders How do they determine value? What comprises value to them? How do they express it? What are the most important questions they have? What are the most important decisions they make? What are they communicating to their clients? Which of their goals & objectives do you contribute towards?

Framework: Define Clearly define this alignment by articulating & documenting your goals & objectives Clarify how these contribute towards your stakeholders desired outcomes Test these goals & objectives with your stakeholders Test them with staff to ensure they are in keeping with your purpose & will aid with planning & decisions Page 2 of Framing Template

Framework: Identify How will you track your success/progress towards meeting these goals & objectives? What indicators or measures will you use? –Qualitative? Quantitative? –What data needs to be collected? How? –Keep it key (KISS principle) –Dont get mired Page 3 of Framing Template

Framework: Collect Do it Determine a collection schedule Are you currently collecting data or indicators that are no longer relevant? How long do you need to keep data? Who is responsible? Page 3 of Framing Template

Framework: Analyze & Interpret So…..what? –What does the data say? What doesnt it say? –Examine it from various angles –What is the progress towards the goals & objectives? Page 3 of Framing Template

Framework: Communicate Actually begins back at the define stage If your goals & objectives are meaningful for stakeholders, your measures will be too Your message to them If your goals & objectives are meaningful for your planning & decision-making, your measures will be indispensable Your message to you & to staff: we have to start, we have to stop & we have to continue… Page 3 of Framing Template

Oakville Public Library Background Oakville Library Board operates on 3 year business planning model Arms-length relationship with Town of Oakville Community Services Commission, operates on a rolling 3 year Integrated Business Planning model Brand focused business plan goals and KPIs

Context Key stakeholders: Municipality, board, donors, public, staff, other libraries Measures that are important –efficiency measures (ROI) and –effectiveness measures (usage, market penetration etc.)

ALIGN and DEFINE - Oakville Public Library Business Plan 2005 At A Glance The Centre for Learning – Your Gateway to Knowledge To provide the ideal environment for the customers learning experience. To support, educate, motivate and recognize staff and volunteers in providing the ideal environment for learning. Increase awareness Increase value in the community Increase usage Align organizational culture with external position in the community Increase awareness Increase value in the community Increase usage Align organizational culture with external position in the community Vision The ultimate goal, a future state, a picture of what we want to be. Mission The purpose of the organization, for all stakeholder groups. Objectives Measures how well we achieve our mission. Strategies The means used to achieve the mission. Customer Service Priorities Brand Integration Collections Management Outreach Services Early Childhood Literacy Corporate Priorities Strategic Planning Board Orientation & Advocacy Human Resources Strategies Town Integration Infrastructure Management E-services Development

IDENTIFY – KPIs used at Oakville ValueMaintain satisfaction level in Citizens Survey Maintain Cardholders per capita 70% UsageNumber of service transactions per capita AwarenessOverall awareness of breadth of programs & services improved Align cultureSatisfaction measure through informal customer satisfaction & staff awareness survey Number of service transactions > 1.0 Annual Expenditures

COLLECT Tools used for collection include: –Citizens survey – measures value and awareness every 3 years –Cardholders per capita (Dynix/Horizon) –Informal satisfaction surveys and feedback forms – focus/advisory groups, web surveys, print surveys –Collection satisfaction survey

ANALYZE AND INTERPRET Service transactions = circulation + people entering + program attendance + outreach + information queries answered + e- transactions Trends analysis Comparative analysis with other areas (libraries, town departments) Variance reporting

COMMUNICATION Look at frequency, depth and message and customize for specific stakeholder Yearly communication to municipality through budget process Yearly communication to Library board, with quarterly updates throughout the year Annual Report to the Community to donors and public Semi-annual updates to staff through semi-annual reports, intranet and staff newsletter

Gaps and Hurdles Consistency in measurement over the years Ability to measure Staff engagement and education Detail of measurement required for different stakeholders Meaningful communication Extrapolation errors – its part of the equation Cause and effect – direct or indirect? Advocacy plan Identify comparators across libraries, local community groups and other non profits etc.

Can you really measure impact? Are there times you cannot?

Can we measure impact? Sometimes what counts cant be counted, and what can be counted doesnt count. –Albert Einstein We should be a bit wary of the little library …For when it is good, it is very, very good and when it is bad, its a pretty good library for a town this size. - E. Rodger, New Zealand Libraries, March 1990

Measuring Impact - Libraries change lives Knowledge of alphabet letters at entry into kindergarten is a strong predictor of reading ability in 10 th grade – Every Child Ready to Read, West Bloomfield Township …it helped be improve my ability to help students in their essay and civic topics; all useful says a teacher-librarian of the Youth Online! program