Presentation on theme: "How to Identify Your Impact: The Value of Libraries Joe Matthews June 10, 2013."— Presentation transcript:
How to Identify Your Impact: The Value of Libraries Joe Matthews June 10, 2013
Outline Performance Measures Value Value of Information Value of Information Services
Outline Value of a Library – Personal value Direct measures Indirect measures – Organizational value – Financial value
Few libraries exist in a vacuum, accountable only to themselves. There is thus always a larger context for accessing library quality, that is, what and how well does the library contribute to achieving the overall goals of the parent constituencies? Sarah Pritchard
There is no systematic evidence collected which shows the value of academic libraries for teaching and research staff. Claire Creaser and Valerie Spezi
Performance Measures Input Process OutputOutcomes Library Services Individual Society Efficiency Effectiveness Cost Effectiveness Impact VALUE Cost benefit ResourcesCapabilityUseBeneficial effects
Start with the end in mind: work backwards Refocus from the activity to the impact
Library Control How much?How many?How economical?How prompt? Magnitude % of change last year % of overall change Cost Magnitude Change Resources used Units processed Cycle times Turnaround time Anticipatory
Customers Decide How well?How courteous?How responsive?How satisfied? Accuracy Promptness Courtesy Expertise Attentive Welcoming Anticipatory Helpful Empathetic Expectations met Materials obtained Personal interaction Ease of use Environment Comfort Willingness to return
Live by the numbers, ….
Challenges Lack of consensus about what should be measured and how Lack of understanding of performance measurement and metrics Organizational structural issues Lack of precision in measuring performance, and alignment issues Determining the “bottom line” is too far away Majority of stakeholders are too far away Library staff find it difficult to see the “big” picture
And the survey said ….
Lack of a Connection Budget and outputs (and outcomes) are separated No “bottom line” measure for libraries Decision-making process is bigger than the library Library has neither champions nor foes Library benefits are not widely self-evident
Orr’s Fundamental Questions How good is the library? What good does the library do? How well is the library managed?
We should be a bit wary of the “ little library ” …For when it is good, it is very, very good and when it is bad, it’s a “ pretty good library for a town this size.” Eleanor Jo Rodger
Levels of Assessment … Individual student Course Departmental/Program College or University
Types of Measures Direct – Provide tangible, visible and self- explanatory evidence of what students have & have not learned Indirect – Capture students’ perceptions of their knowledge & skills; supplement direct measures; sometimes called surrogates
Qualitative Tools Focus groups – open ended Biography Phenomenology – capture the “Aha!” moment Grounded theory Ethnography Case study
Qualitative Assessment Provides in-depth understanding of user responses and interactions Represents part of a long-term strategy of formative evaluative
Quantitative Tools Surveys Transaction logs Statistics from systems Observations (count)
Quantitative Assessment Analyses to determine library impacts on academic performance, retention rates Describe retention rates and GPAs in defined populations over semesters and users Compare users & non-users of library services while adjusting for academic preparation and background differences Conduct quasi-experimental designs employing multivariate analysis of covariance & hierarchical regression techniques
Be cautious about cause-and-effect relationships
The Issue Is it: Use library resources/services and you will get better grades. Or: I want to do well and so I work hard to achieve better grades - and one way I do that is to use library resources/services.
“Not surprisingly, librarians are keen to show that the use of expensive, scholarly materials positively correlates with higher grades, although they cannot prove that this is so.” Deborah Goodall & David Pattern
“There is growing pressure on all academic library managers to be more accountable for how they use limited resources and to achieve institutional outcomes perceived as important by college and university stakeholders….” Elizabeth Mezick
Value of Information Expect value-in-use Library’s collection reflects a “potential value” Collection also reflects a “future value” Value of local collection is declining
Valuable is not about our professional values; in the paradigm of the value of public libraries, we are the producers, not the consumers of services. Our sense of what is valuable really doesn’t matter much at all unless it matches that our our customers. Eleanor Jo Rodger
Fundamental Changes Libraries have changed more in the past two decades than in the prior two centuries. Technology is the major driver... We need to recognize that all this change has only begun, and that change is irreversible.
Increasingly it is important to remember that libraries provide few unique services.
Information is woven into our lives
Quality of Information This fast food approach to information consumption drives librarians crazy. “Our information is healthier and tastes better too” they shout. But nobody listens. We’re too busy Googling.” Peter Morville
Key Characteristics of Information UncertaintyKnowledge AmbiguityIndeterminacy RedundancySystem dependency SharingTimeliness CompressionPresentation StabilityMultiple life cycles LeakabilitySubstitutability
Criteria for Judging Value Customer CriteriaValue Added by the Service Ease of useBrowsing, formatting, mediation service, orientation service, ordering, physical accessibility Noise reductionAccess (item identification, subject description, subject summary), linkage, precision, selectivity QualityAccuracy, comprehensiveness, currency, reliability, validity AdaptabilityCloseness to problem, flexibility, simplicity, stimulatory Time savingsResponse speed Cost savings
Collections are disrupted Atoms to bits
Nature of Information is Changing Scare, controlled Expensive Shaped by elites One-way, mass consumption Slow moving External to our worlds All around us Cheap or free Shaped by consumers Designed for sharing, participation & feedback Immediate Embedded in our worlds Information was …. Information is ….
Value of the Academic Library
If the physical proximity of print collections had a demonstrable impact on researcher productivity, no university would hesitate to allocate prime real estate to library stacks.
Traditional Value Proposition Without a great library, there can be no great university. David Kinly, President of The University of Illiniois 1929
Universities Provide Private goods & services – Courses exchanged for tuition – Research completed for funding The value proposition The value to an individual or an organization determines whether payment is made for the service
Academic Libraries Provide Public goods and services Print and online resources are shared by all, usually without the exchange of payment Value proposition The collective value of all users must be estimated to determine if a good or service should be continued
Value is determined by the user and the use of information
Astin’s IEO Model InputOutput Environment Programs Institutional characteristics Library Fellow students Faculty Place of residence
Student Learning Outcomes Model Intelligence General Reasoning Broad Abilities Knowledge, Understanding, and Reasoning Abstract, Process Oriented Concrete, Content- Oriented
Define, develop, and measure outcomes that contribute to institutional effectiveness ACRL Standards for Libraries in Higher Education
Challenge It is not how much a library is used that matters, rather how does the library impact or benefit the customer?
Perspectives on Value Benefits Use Nonuse Direct Indirect Option – Preservation of option for future use by me Existence – Perceived value and significance to the community Legacy – Value of preservation for future generations Personal Organizational Financial Impacts
Why Use the Library? ReasonsInteractionsResults For a TASK For PERSONAL reasons To get an OBJECT or INFORMATION To perform an ACTIVITY Access RESOURCES Use of RESOURCES or SERVICES OPERATIONS ENVIRONMENTS COGNITIVE results AFFECTIVE results ACCOMPLISHMENTS EXPECTATIONS met TIME aspects MONEY estimates
Full-time students Live on campus Interact more with faculty Study more Collaborate with their peers Student Learning is Affected by …
NSSE 5 benchmarks of effective educational practice – Level of academic challenge – Active & collaborative learning – Student-Faculty interaction – Enriching educational experiences – Supportive campus environment Student self-reported gains in intellectual & personal development No overlap between self-reported data & standardized objective tests
NSSE – Use of the Academic Library 50% never used the library Use of libraries at small, academically challenging liberal arts colleges are correlated with other purposeful activities Library use less intensive at larger universities Students who work harder use library resources
Assessing Student Achievement Direct measures – Capstone experience – Use of a portfolio – Standardized exam (Collegiate Learning Assessment) Indirect measures
Gains in student performance are quite low Individual learning is characterized by persistence Notable variation within and across institutions Assessment of Higher Ed
Wabash National Study Different instrument – CAAP 2,212 students Nearly identical results to Academically Adrift 44 percent no gains in the first year 33 percent no gains in 4 years Students only study about 15 hours per week
Bibliographic Instruction Improvement in basic library skills is the means and not the end Yet the means is the focus for evaluation efforts Evaluation efforts focus on – Opinion surveys – Skills improvement – Pre-test & post-test knowledge Not the impact on student achievement
Library experiences do not seem to directly contribute to gains in information literacy, to what students gain overall in college, or to student satisfaction. Kuh & Gonyea
“One way to demonstrate the library’s contribution is to assess whether students’ experiences with the library directly or indirectly contribute to desired outcomes of college.” George D. Kuh & Robert M. Gonyea
How to Demonstrate Impact in … Student enrollment Student experiences Student learning Student grades (GPA) & achievement Student retention & graduation Student career success Faculty productivity Institutional reputation The environment
Recruitment of prospective students Matriculation of admitted students Recommendation of current students Student Enrollment
Meta-analysis Entering student characteristics – SES – High school GPA – ACT/SAT scores Environment - Psychosocial and study skill factors – Academic goals, skills and self-confidence – Social support & engagement (acculturation)
Student Learning Occurs … in the classroom in the laboratory with peers in the student union in the dorm in the library (for some) online and
Direct Measures Capstone Experience Portfolios Standardized Test Locally-developed test
Usage of Electronic Resources WAM = Weighted Average Marks (Grades)
Other Studies Hong Kong Baptist University Georgia State University
University of Minnesota Gym Bags and Mortarboards Use Campus Recreational Facilities At least 25 times, first-year retention increased 1% & 5-year graduation rates increased 2%
University of Minnesota Library 5,368 first-year non-transfer students Use of library was associated with a.23 increase in students GPA More use of the library, GPA also goes up
University of Minnesota Library
Library Instruction and GPA Surveys of student opinions & habits Assessing student work for specific skills Analysis of grade point average Mixed results
Library Instruction and GPA Hong King Baptist University 45 sample groups – N=31 to 1,223, study majors Pairs of data One-fourth (11) had a positive relationship Results: – 1 or 2 workshops – little impact on GPA – 3 or 4 workshops – ½ show a positive impact – 5 workshops (1 sample group) – 100% had a higher GPA
Library Instruction and GPA University of Wyoming Libraries Analysis of 4,489 transcripts Slight positive relationship between upper- level library instruction courses and GPA – 0.075 GPA difference – that’s less than 1/10 th of 1 percent
Research statement – 44% Evaluate Web site – objectivity – 52% – authority – 65% Presentation to persuade – 12%
Retention Concepts Institutional retention – Enrolling & graduating from the same institution Program retention – Enrolling & graduating with the same major/department/school System retention – Students who leave one university yet continue and complete post-secondary studies elsewhere
Measures of Retention Persistence (Continuation rate) – From first to second year? Entry to graduation? Completion rate – From entry to graduation (Student goals?) Graduation rates – Are transfers included? Time period? Attrition – Leaving university? Leaving higher ed?
Measures of Retention Stopout – Leave university with the intention (and action) of returning later to complete a program Dropout – Leave university with intention (and action) of NOT returning Transfer – Change institutions yet persist in higher education – May change type of institution Voluntary vs. involuntary attrition?
Why Students Leave? Students’ decision to leave University is influenced by many personal factors – Financial reasons – Family responsibilities – Lack of academic ability – Poor fit, etc. Foundational Theories from Education / Psychology: – Tinto’s “Model of Student Integration” – Bean’s “Model of Student Attrition”
Tinto’s Model of Student Integration Pre-entry Attributes Goals / Commitments Institutional Experiences IntegrationGoals / Commitments Outcome SES Skills & Abilities Quality of Education Intentions Institutional Commitments Academic System Performance Engagement Extra- Curricular Peer group Social System Academic Integration Social Integration Intentions Goal Institutional Match Stay or Leave
Bean’s Model of Student Attrition Grades Courses Educational Goals Major & Job Certainty Opportunity to Transfer Family Approval Organizational Variables Personal Variables Environmental Variables LoyaltyAttitudes Certainty Practical Value Intent Dropout
Student Retention & Graduation Important because … rankings, revenues, educational achievement, emotional well- being Many reasons for drop-outs are not under the control of the university Engagement is the key
Indicators Student goal attainment Course retention Subsequent course work Fall-to-fall persistence Time to degree Degree completion Grad school enrollment Transfer rate & success Employer assessment Academic value add Student satisfaction Professional growth Student involvement Citizenship & engagement
Australasian Survey of Student Engagement (AUSSE)
University of Huddersfield Dropping out!
“A high rate of attrition is indicative of a failure on the part of an institution to achieve its purpose.” Elizabeth Mezick
Student Engagement “many students don’t develop a personal connection with their institution. And when they don’t, they leave” Gonzales 2010 NSSE & CSEQ
Library Retention Studies Statistically significant relationships between library expenditures, or staffing levels and student retention E.g. Hiscock, 1986 Hamrick, Schuh, & Shelley, 2004 Mezick, 2007
Graduation Rates & Library Expenditures Used IPEDS data on institutional characteristics & resource allocations Library expenditures was strongly correlated with graduation rates – 1.77 percent increase in graduation rates Greatest payoff is attributable to enhanced library expenditures (+0.92) and instruction (+0.80) while increased non-library contributions were quite modest (+0.27) Hamrick, Schuh & Shelley
Library Retention Studies Relationships between library use (collections) and student retention – Student who borrowed books = more likely to persist E.g.Kramer & Kramer, 1968 Impact of instruction – Students involved in library skills programs showed lower attrition rates E.g.Knapp, 1966
Library Retention Studies University of Minnesota – 77% of undergrads made use of the libraries, 85% of grad students made use of the libraries – Students who used the library at least once were 1.54 times more likely to re-enroll
Library Retention Studies Some library involvement in first year experience programs; specific programs for “at risk” groups – NOT proven to have significant effect E.g. Hollis, 2001 Colton, et al, 2002 Aguilar & Keating, 2009 Love, 2009
Library Retention Studies Relationship between library employment & retention – Higher completion rate among library student workers E.g.Wilder, 1990 Rushing & Poole, 2002
“If strong linkages between libraries and student retention can be made, then the perceived value of the library may indeed rise.” Steven Bell
Some groups, some majors & seniors engage in more library-related activities Academic support expenditures tend to correlate with increased engagement Institutional academic challenge correlates with library use
Student Career Success Grad School Exams Time to First Job Average Salary of First Job Alumni Surveys Alumni Giving
Integration of library resources and services into course syllabi, Websites, lectures, labs, reserve readings, etc. Faculty/librarian collaborations; cooperative curriculum, assignment, or assessment design Faculty Teaching
Perceived Benefits for Teaching Savings – Of own time – Of own money – Of other resources Improvements – Teaching – Course-related materials – Student performance
Impact on Faculty Library is the source for most journal articles (individual subscriptions are way down) If library subscriptions were unavailable – productivity would decrease 17% Library is not the source of book readings 42% of reading material is library provided
Time Academics spend a lot of time reading Article reading inspires new thinking, improved results, changed focus Award-winning academics read more Academics who publish more use more library resources
Ithaka Studies Library services not understood Library services not valued The Library is disappearing
Number of publications, number of patents, value of technology transfer Tenure/promotion judgments Faculty Research Productivity Faculty Grants Number of grant proposals (funded or unfunded) Value of grants funded
Assessment of Research Payback model – form of ROI Research impact Research utilization ladder Lavis decision-making impact model Weiss logic model HTA organization assessment framework Societal impact framework Research assessment exercise Becker medical library model
For Most Impact Models Indicators of research output Indicators of knowledge transfer Indicators of implementation Indicators of community benefit
Institutional Reputation Changes in reputational rankings affects student & faculty recruitment University budget allocations to libraries have decreased Since the library absorbs a very small percentage of a university budget, the contribution of the library is disproportionately high relative to its cost to the institution. Sharon Weiner
University & the Library Can Attract outstanding faculty Retain outstanding faculty Foster innovative research Align library activities with university goals
Indispensable for their research Maintain a high-level overview of their field Value for money is good Library not available, costs would increase 40% Take 31% longer to locate same information
At the table or On the menu?
What is ROI?
ROI Applications Projects Services Organization
ROI in Library Contexts Demonstrating the value of libraries Evaluating existing services, collections, etc. Making the case for additional services or resources Recruiting support for a program or initiative
ROI Terminology Cost-Benefit Analysis (CBA) – Compares cost or purchase price with estimated value of variables that are difficult to measure Consumer Surplus – Value that consumers place on the consumption of a good or service in excess of what they paid for it Cost of Time and Effort – Measures time and effort expended by users Contingent Valuation – Measures value of use and non-use of non-priced goods and services, e.g., a library
Cost/Benefit Methodologies Maximize the benefits for given costs Minimize the costs for a given level of benefits Maximize the ratio of benefits over costs Maximize the net benefits (present value of benefits minus the present value of costs) Maximize the internal rate of return
Ratio of Benefits to Costs Value of benefits divided by Costs ROI = Benefit – Cost Cost
University of Pittsburgh ROI If the library’s journal collection (print & electronic) were not available, faculty would use 250,000 hours and $2.1 million to find alternative sources for the articles It would cost the university 4.38 times the cost of the current library journal collections for the same amount of information gathering to be carried out
University of Illinois, Urbana-Champaign Connected citations to resources in the library’s collection to successful grant proposals, and the income the grants generated ROI = 4.38:1 Other studies – 0.27:1 to 15.54:1
ROI may provide us with calculations that seek to document a financial relationship between action and benefit, but too often in the library community these studies are poorly constructed, ineffectively executed, and naïvely communicated. And in the final analysis, do not respond to the legitimate questions being raised by our administrators and funders, and do not advance the academic library as a critical factor in institutional success. James Neal
Comprehensive assessment of the library ROI of the journal collection & readership ROI for support of teaching & learning ROI of digitized special collections ROI of eBooks Value of library commons Bibliography
Bryant University – Faculty Access Article ROI = 3.2:1 Book ROI = 3.5:1 Other publications ROI = 3.2:1
Syracuse University – ROI 4.49:1 FacultyStudents Economic In Person$13.6$23.1 Remote19.014.5 TOTAL$32.6$37.6 Environmental Remote access$1.6$3.7 Read not-printed0.10.7 Social?? TOTAL$34.3$42.0 (Millions of Dollars)
Journal Collections & Reading
ROI for Support of Teaching & Learning Perceived Benefits – Savings … – Of own time – Of own money – Of other resources – printing, copier Improvements … – Teaching – Course-related materials – Student performance
ROI of Digitized Special Collections User What is the value to a user in terms of time and money spent? Prestige What is the prestige to the institution for high visibility digital special collection? Development What value accrues to the development effort of the institution?
ROI of Digitized Special Collections Environmental What is the value of the environmental savings from limited physical access to unique and often fragile material? Scholars What value accrues from the role of special collections in attracting graduate students? Collections What is the value of digital collection in attracting additional special collections?
ROI of eBooks
ROI = 26:1
Past Future Focus Structure Change Outreach Decision making Measures of Success Building & maintaining collections Engaging students & faculty Internal silos University priorities IncrementalTransformational Top down Periodic Shared Deliberative TraditionalImpact
Library Value How integral it is to the community How well it supports learning and teaching How well it supports research
What your library does well What your customers value Value Proposition
An offer, not a demand Not what you value Only valuable perspective is the customer’s Valuable in a competitive environment The Value Proposition
Or The promise that a library makes to its customers about what they can expect to receive in return for their time, their effort, their loyalty, and especially their dollars.
The library needs to … Focus on customers and what they want and need as well as how they want and need it.
Stories + Stats = Success
Is the value in the glass, the wine or the savoring?
The financial crisis is looking even worse, but you will pleased to know that the director reports that the library performance went up a half a point on the “library goodness scale” last week. Michael Buckland
Joe@JoeMatthews.Org www.joematthews.org Joe Matthews Library Consultant
What Are the Results Four Year College Full-time Four Year College Part-time Two Year College Stopout/Transfer College degree – 19% AA degree – 8% No college degree – 60%No college Graduate degree – 11%
Valuing the Collection Dewey Subclass Number of Titles 2010-2011 Avg ListTotal Value ($) 001 - Knowledge $76.71$0.00 002 - The book $62.45$0.00 003 - Systems $129.77$0.00 004 - Data processing. Computer science $89.82$0.00 005 - Computer programming, programs, data $69.14$0.00 006 - Special computer methods $83.60$0.00 010 - Bibliography $73.65$0.00 011 - Bibliographies $69.46$0.00 012 - Bibliographies of individuals $0.00 013 - Of works by specific classes of authors $0.00 014 - Of anonymous and pseudonymous works $55.95$0.00 015 - Of works from specific places $184.99$0.00 016 - Of works on specific subjects $134.87$0.00 017 - General subject catalogs $0.00 018 - Catalogs arranged by author, date, etc. $0.00 019 - Dictionary catalogs $0.00 020 - Library and information sciences $56.06$0.00 021 - Library relationships $62.60$0.00 022 - Administration of the physical plant $65.50$0.00 023 - Personnel administration $56.00$0.00 025 - Library operations $77.02$0.00 026 - Libraries for specific subjects $89.99$0.00 027 - General libraries $66.25$0.00