Presentation is loading. Please wait.

Presentation is loading. Please wait.

How to Identify Your Impact: The Value of Libraries Joe Matthews June 10, 2013.

Similar presentations

Presentation on theme: "How to Identify Your Impact: The Value of Libraries Joe Matthews June 10, 2013."— Presentation transcript:

1 How to Identify Your Impact: The Value of Libraries Joe Matthews June 10, 2013

2 Megan Oakleaf

3 Outline Performance Measures Value Value of Information Value of Information Services

4 Outline Value of a Library – Personal value Direct measures Indirect measures – Organizational value – Financial value

5 Few libraries exist in a vacuum, accountable only to themselves. There is thus always a larger context for accessing library quality, that is, what and how well does the library contribute to achieving the overall goals of the parent constituencies? Sarah Pritchard

6 There is no systematic evidence collected which shows the value of academic libraries for teaching and research staff. Claire Creaser and Valerie Spezi

7 Performance Measures Input Process OutputOutcomes Library Services Individual Society Efficiency Effectiveness Cost Effectiveness Impact VALUE Cost benefit ResourcesCapabilityUseBeneficial effects

8 Start with the end in mind: work backwards Refocus from the activity to the impact

9 Impact

10 Library Control How much?How many?How economical?How prompt? Magnitude % of change last year % of overall change Cost Magnitude Change Resources used Units processed Cycle times Turnaround time Anticipatory

11 Library & Customers Decide How valuable?How reliable?How accurate? Effort expended Cost Benefits obtained Dependability Access Accuracy Completeness Comprehensiveness Currency

12 Customers Decide How well?How courteous?How responsive?How satisfied? Accuracy Promptness Courtesy Expertise Attentive Welcoming Anticipatory Helpful Empathetic Expectations met Materials obtained Personal interaction Ease of use Environment Comfort Willingness to return

13 Live by the numbers, ….

14 Challenges  Lack of consensus about what should be measured and how  Lack of understanding of performance measurement and metrics  Organizational structural issues  Lack of precision in measuring performance, and alignment issues  Determining the “bottom line” is too far away  Majority of stakeholders are too far away  Library staff find it difficult to see the “big” picture

15 And the survey said ….

16 Lack of a Connection Budget and outputs (and outcomes) are separated No “bottom line” measure for libraries Decision-making process is bigger than the library Library has neither champions nor foes Library benefits are not widely self-evident

17 Orr’s Fundamental Questions How good is the library? What good does the library do? How well is the library managed?

18 We should be a bit wary of the “ little library ” …For when it is good, it is very, very good and when it is bad, it’s a “ pretty good library for a town this size.” Eleanor Jo Rodger


20 Levels of Assessment … Individual student Course Departmental/Program College or University

21 Types of Measures Direct – Provide tangible, visible and self- explanatory evidence of what students have & have not learned Indirect – Capture students’ perceptions of their knowledge & skills; supplement direct measures; sometimes called surrogates

22 Qualitative Tools Focus groups – open ended Biography Phenomenology – capture the “Aha!” moment Grounded theory Ethnography Case study

23 Qualitative Assessment Provides in-depth understanding of user responses and interactions Represents part of a long-term strategy of formative evaluative

24 Quantitative Tools Surveys Transaction logs Statistics from systems Observations (count)

25 Quantitative Assessment Analyses to determine library impacts on academic performance, retention rates Describe retention rates and GPAs in defined populations over semesters and users Compare users & non-users of library services while adjusting for academic preparation and background differences Conduct quasi-experimental designs employing multivariate analysis of covariance & hierarchical regression techniques

26 Useful Assessment

27 Be cautious about cause-and-effect relationships

28 The Issue Is it: Use library resources/services and you will get better grades. Or: I want to do well and so I work hard to achieve better grades - and one way I do that is to use library resources/services.

29 “Not surprisingly, librarians are keen to show that the use of expensive, scholarly materials positively correlates with higher grades, although they cannot prove that this is so.” Deborah Goodall & David Pattern

30 “There is growing pressure on all academic library managers to be more accountable for how they use limited resources and to achieve institutional outcomes perceived as important by college and university stakeholders….” Elizabeth Mezick




34 Value of Information Expect value-in-use Library’s collection reflects a “potential value” Collection also reflects a “future value” Value of local collection is declining

35 Valuable is not about our professional values; in the paradigm of the value of public libraries, we are the producers, not the consumers of services. Our sense of what is valuable really doesn’t matter much at all unless it matches that our our customers. Eleanor Jo Rodger

36 Fundamental Changes Libraries have changed more in the past two decades than in the prior two centuries. Technology is the major driver... We need to recognize that all this change has only begun, and that change is irreversible.

37 Increasingly it is important to remember that libraries provide few unique services.

38 Information is woven into our lives

39 Quality of Information This fast food approach to information consumption drives librarians crazy. “Our information is healthier and tastes better too” they shout. But nobody listens. We’re too busy Googling.” Peter Morville

40 Key Characteristics of Information UncertaintyKnowledge AmbiguityIndeterminacy RedundancySystem dependency SharingTimeliness CompressionPresentation StabilityMultiple life cycles LeakabilitySubstitutability

41 Criteria for Judging Value Customer CriteriaValue Added by the Service Ease of useBrowsing, formatting, mediation service, orientation service, ordering, physical accessibility Noise reductionAccess (item identification, subject description, subject summary), linkage, precision, selectivity QualityAccuracy, comprehensiveness, currency, reliability, validity AdaptabilityCloseness to problem, flexibility, simplicity, stimulatory Time savingsResponse speed Cost savings

42 Collections are disrupted Atoms to bits


44 Nature of Information is Changing Scare, controlled Expensive Shaped by elites One-way, mass consumption Slow moving External to our worlds All around us Cheap or free Shaped by consumers Designed for sharing, participation & feedback Immediate Embedded in our worlds Information was …. Information is ….


46 Value of the Academic Library

47 If the physical proximity of print collections had a demonstrable impact on researcher productivity, no university would hesitate to allocate prime real estate to library stacks.

48 Traditional Value Proposition Without a great library, there can be no great university. David Kinly, President of The University of Illiniois 1929

49 Universities Provide Private goods & services – Courses exchanged for tuition – Research completed for funding The value proposition The value to an individual or an organization determines whether payment is made for the service

50 Academic Libraries Provide Public goods and services Print and online resources are shared by all, usually without the exchange of payment Value proposition The collective value of all users must be estimated to determine if a good or service should be continued

51 Value is determined by the user and the use of information

52 Astin’s IEO Model InputOutput Environment Programs Institutional characteristics Library Fellow students Faculty Place of residence

53 Student Learning Outcomes Model Intelligence General Reasoning Broad Abilities Knowledge, Understanding, and Reasoning Abstract, Process Oriented Concrete, Content- Oriented

54 Define, develop, and measure outcomes that contribute to institutional effectiveness ACRL Standards for Libraries in Higher Education

55 Challenge It is not how much a library is used that matters, rather how does the library impact or benefit the customer?

56 Perspectives on Value Benefits Use Nonuse Direct Indirect Option – Preservation of option for future use by me Existence – Perceived value and significance to the community Legacy – Value of preservation for future generations Personal Organizational Financial Impacts

57 Personal

58 Why Use the Library? ReasonsInteractionsResults For a TASK For PERSONAL reasons To get an OBJECT or INFORMATION To perform an ACTIVITY Access RESOURCES Use of RESOURCES or SERVICES OPERATIONS ENVIRONMENTS COGNITIVE results AFFECTIVE results ACCOMPLISHMENTS EXPECTATIONS met TIME aspects MONEY estimates

59 Generic Learning Outcomes Knowledge & UnderstandingSkills AttitudesEnjoyment, Inspiration, Creativity Activity, Behavior, Progression

60 Organizational

61 Full-time students Live on campus Interact more with faculty Study more Collaborate with their peers Student Learning is Affected by …

62 NSSE 5 benchmarks of effective educational practice – Level of academic challenge – Active & collaborative learning – Student-Faculty interaction – Enriching educational experiences – Supportive campus environment Student self-reported gains in intellectual & personal development No overlap between self-reported data & standardized objective tests

63 NSSE – Use of the Academic Library 50% never used the library Use of libraries at small, academically challenging liberal arts colleges are correlated with other purposeful activities Library use less intensive at larger universities Students who work harder use library resources


65 Assessing Student Achievement Direct measures – Capstone experience – Use of a portfolio – Standardized exam (Collegiate Learning Assessment) Indirect measures

66 Gains in student performance are quite low Individual learning is characterized by persistence Notable variation within and across institutions Assessment of Higher Ed

67 Wabash National Study Different instrument – CAAP 2,212 students Nearly identical results to Academically Adrift 44 percent no gains in the first year 33 percent no gains in 4 years Students only study about 15 hours per week

68 Opinion surveys Skills testing Observed behaviors

69 Bibliographic Instruction Improvement in basic library skills is the means and not the end Yet the means is the focus for evaluation efforts Evaluation efforts focus on – Opinion surveys – Skills improvement – Pre-test & post-test knowledge Not the impact on student achievement

70 Library experiences do not seem to directly contribute to gains in information literacy, to what students gain overall in college, or to student satisfaction. Kuh & Gonyea

71 “One way to demonstrate the library’s contribution is to assess whether students’ experiences with the library directly or indirectly contribute to desired outcomes of college.” George D. Kuh & Robert M. Gonyea


73 How to Demonstrate Impact in … Student enrollment Student experiences Student learning Student grades (GPA) & achievement Student retention & graduation Student career success Faculty productivity Institutional reputation The environment

74 Student Enrollment

75 Recruitment of prospective students Matriculation of admitted students Recommendation of current students Student Enrollment

76 Student Learning

77 GPA Professional/educational test scores Learning assessments Faculty judgments Student Learning

78 Meta-analysis Entering student characteristics – SES – High school GPA – ACT/SAT scores Environment - Psychosocial and study skill factors – Academic goals, skills and self-confidence – Social support & engagement (acculturation)

79 Student Learning Occurs … in the classroom in the laboratory with peers in the student union in the dorm in the library (for some) online and


81 Direct Measures Capstone Experience Portfolios Standardized Test Locally-developed test

82 Capstone Projects

83 Indirect Measures Surveys Retention, graduation rates Grades Acceptance rates into graduate programs Job placement rates Exit interviews Alumni surveys

84 Student Grades



87 Non/Low Use


89 Usage of Electronic Resources WAM = Weighted Average Marks (Grades)

90 Other Studies Hong Kong Baptist University Georgia State University

91 Hope College

92 University of Minnesota Gym Bags and Mortarboards Use Campus Recreational Facilities At least 25 times, first-year retention increased 1% & 5-year graduation rates increased 2%


94 University of Minnesota Library 5,368 first-year non-transfer students Use of library was associated with a.23 increase in students GPA More use of the library, GPA also goes up

95 University of Minnesota Library

96 Library Instruction and GPA Surveys of student opinions & habits Assessing student work for specific skills Analysis of grade point average Mixed results

97 Library Instruction and GPA Hong King Baptist University 45 sample groups – N=31 to 1,223, study majors Pairs of data One-fourth (11) had a positive relationship Results: – 1 or 2 workshops – little impact on GPA – 3 or 4 workshops – ½ show a positive impact – 5 workshops (1 sample group) – 100% had a higher GPA

98 Library Instruction and GPA University of Wyoming Libraries Analysis of 4,489 transcripts Slight positive relationship between upper- level library instruction courses and GPA – 0.075 GPA difference – that’s less than 1/10 th of 1 percent

99 Research statement – 44% Evaluate Web site – objectivity – 52% – authority – 65% Presentation to persuade – 12%

100 How scalable is library instruction?

101 Student Retention & Graduation Attrition Retention Persistence Completion Graduation Rates

102 Fall-to-fall retention Graduation rates Student Retention

103 Retention Concepts Institutional retention – Enrolling & graduating from the same institution Program retention – Enrolling & graduating with the same major/department/school System retention – Students who leave one university yet continue and complete post-secondary studies elsewhere

104 Measures of Retention Persistence (Continuation rate) – From first to second year? Entry to graduation? Completion rate – From entry to graduation (Student goals?) Graduation rates – Are transfers included? Time period? Attrition – Leaving university? Leaving higher ed?

105 Measures of Retention Stopout – Leave university with the intention (and action) of returning later to complete a program Dropout – Leave university with intention (and action) of NOT returning Transfer – Change institutions yet persist in higher education – May change type of institution Voluntary vs. involuntary attrition?

106 Why Students Leave? Students’ decision to leave University is influenced by many personal factors – Financial reasons – Family responsibilities – Lack of academic ability – Poor fit, etc. Foundational Theories from Education / Psychology: – Tinto’s “Model of Student Integration” – Bean’s “Model of Student Attrition”

107 Tinto’s Model of Student Integration Pre-entry Attributes Goals / Commitments Institutional Experiences IntegrationGoals / Commitments Outcome SES Skills & Abilities Quality of Education Intentions Institutional Commitments Academic System Performance Engagement Extra- Curricular Peer group Social System Academic Integration Social Integration Intentions Goal Institutional Match Stay or Leave

108 Bean’s Model of Student Attrition Grades Courses Educational Goals Major & Job Certainty Opportunity to Transfer Family Approval Organizational Variables Personal Variables Environmental Variables LoyaltyAttitudes Certainty Practical Value Intent Dropout


110 Student Retention & Graduation Important because … rankings, revenues, educational achievement, emotional well- being Many reasons for drop-outs are not under the control of the university Engagement is the key

111 Indicators Student goal attainment Course retention Subsequent course work Fall-to-fall persistence Time to degree Degree completion Grad school enrollment Transfer rate & success Employer assessment Academic value add Student satisfaction Professional growth Student involvement Citizenship & engagement

112 Australasian Survey of Student Engagement (AUSSE)

113 Curtin University

114 University of Huddersfield Dropping out!

115 “A high rate of attrition is indicative of a failure on the part of an institution to achieve its purpose.” Elizabeth Mezick

116 Student Engagement “many students don’t develop a personal connection with their institution. And when they don’t, they leave” Gonzales 2010 NSSE & CSEQ

117 Library Retention Studies Statistically significant relationships between library expenditures, or staffing levels and student retention E.g. Hiscock, 1986 Hamrick, Schuh, & Shelley, 2004 Mezick, 2007

118 Graduation Rates & Library Expenditures Used IPEDS data on institutional characteristics & resource allocations Library expenditures was strongly correlated with graduation rates – 1.77 percent increase in graduation rates Greatest payoff is attributable to enhanced library expenditures (+0.92) and instruction (+0.80) while increased non-library contributions were quite modest (+0.27) Hamrick, Schuh & Shelley

119 Library Retention Studies Relationships between library use (collections) and student retention – Student who borrowed books = more likely to persist E.g.Kramer & Kramer, 1968 Impact of instruction – Students involved in library skills programs showed lower attrition rates E.g.Knapp, 1966

120 Library Retention Studies University of Minnesota – 77% of undergrads made use of the libraries, 85% of grad students made use of the libraries – Students who used the library at least once were 1.54 times more likely to re-enroll

121 Library Retention Studies Some library involvement in first year experience programs; specific programs for “at risk” groups – NOT proven to have significant effect E.g. Hollis, 2001 Colton, et al, 2002 Aguilar & Keating, 2009 Love, 2009

122 Library Retention Studies Relationship between library employment & retention – Higher completion rate among library student workers E.g.Wilder, 1990 Rushing & Poole, 2002

123 “If strong linkages between libraries and student retention can be made, then the perceived value of the library may indeed rise.” Steven Bell

124 Some groups, some majors & seniors engage in more library-related activities Academic support expenditures tend to correlate with increased engagement Institutional academic challenge correlates with library use

125 Student Career Success Grad School Exams Time to First Job Average Salary of First Job Alumni Surveys Alumni Giving

126 Job placement rates First-year job salaries Professional/graduate school acceptance Internship success Marketable skills Student Success


128 Integration of library resources and services into course syllabi, Websites, lectures, labs, reserve readings, etc. Faculty/librarian collaborations; cooperative curriculum, assignment, or assessment design Faculty Teaching

129 Perceived Benefits for Teaching Savings – Of own time – Of own money – Of other resources Improvements – Teaching – Course-related materials – Student performance


131 Impact on Faculty Library is the source for most journal articles (individual subscriptions are way down) If library subscriptions were unavailable – productivity would decrease 17% Library is not the source of book readings 42% of reading material is library provided

132 Time Academics spend a lot of time reading Article reading inspires new thinking, improved results, changed focus Award-winning academics read more Academics who publish more use more library resources

133 Ithaka Studies Library services not understood Library services not valued The Library is disappearing

134 Number of publications, number of patents, value of technology transfer Tenure/promotion judgments Faculty Research Productivity Faculty Grants Number of grant proposals (funded or unfunded) Value of grants funded

135 Assessment of Research Payback model – form of ROI Research impact Research utilization ladder Lavis decision-making impact model Weiss logic model HTA organization assessment framework Societal impact framework Research assessment exercise Becker medical library model

136 For Most Impact Models Indicators of research output Indicators of knowledge transfer Indicators of implementation Indicators of community benefit


138 Faculty Productivity

139 Faculty recruitment Institutional rankings Community engagement Institutional Reputation & Prestige

140 Institutional Reputation Changes in reputational rankings affects student & faculty recruitment University budget allocations to libraries have decreased Since the library absorbs a very small percentage of a university budget, the contribution of the library is disproportionately high relative to its cost to the institution. Sharon Weiner

141 University & the Library Can Attract outstanding faculty Retain outstanding faculty Foster innovative research Align library activities with university goals

142 Indispensable for their research Maintain a high-level overview of their field Value for money is good Library not available, costs would increase 40% Take 31% longer to locate same information

143 Financia l

144 At the table or On the menu?

145 What is ROI?


147 ROI Applications Projects Services Organization

148 ROI in Library Contexts Demonstrating the value of libraries Evaluating existing services, collections, etc. Making the case for additional services or resources Recruiting support for a program or initiative

149 ROI Terminology Cost-Benefit Analysis (CBA) – Compares cost or purchase price with estimated value of variables that are difficult to measure Consumer Surplus – Value that consumers place on the consumption of a good or service in excess of what they paid for it Cost of Time and Effort – Measures time and effort expended by users Contingent Valuation – Measures value of use and non-use of non-priced goods and services, e.g., a library

150 Cost/Benefit Methodologies Maximize the benefits for given costs Minimize the costs for a given level of benefits Maximize the ratio of benefits over costs Maximize the net benefits (present value of benefits minus the present value of costs) Maximize the internal rate of return

151 Ratio of Benefits to Costs Value of benefits divided by Costs ROI = Benefit – Cost Cost

152 University of Pittsburgh ROI If the library’s journal collection (print & electronic) were not available, faculty would use 250,000 hours and $2.1 million to find alternative sources for the articles It would cost the university 4.38 times the cost of the current library journal collections for the same amount of information gathering to be carried out

153 University of Illinois, Urbana-Champaign Connected citations to resources in the library’s collection to successful grant proposals, and the income the grants generated ROI = 4.38:1 Other studies – 0.27:1 to 15.54:1



156 ROI may provide us with calculations that seek to document a financial relationship between action and benefit, but too often in the library community these studies are poorly constructed, ineffectively executed, and naïvely communicated. And in the final analysis, do not respond to the legitimate questions being raised by our administrators and funders, and do not advance the academic library as a critical factor in institutional success. James Neal

157 15:1 ROI

158 Comprehensive assessment of the library ROI of the journal collection & readership ROI for support of teaching & learning ROI of digitized special collections ROI of eBooks Value of library commons Bibliography

159 Bryant University – Faculty Access Article ROI = 3.2:1 Book ROI = 3.5:1 Other publications ROI = 3.2:1

160 Syracuse University – ROI 4.49:1 FacultyStudents Economic In Person$13.6$23.1 Remote19.014.5 TOTAL$32.6$37.6 Environmental Remote access$1.6$3.7 Read not-printed0.10.7 Social?? TOTAL$34.3$42.0 (Millions of Dollars)

161 Journal Collections & Reading



164 ROI for Support of Teaching & Learning Perceived Benefits – Savings … – Of own time – Of own money – Of other resources – printing, copier Improvements … – Teaching – Course-related materials – Student performance

165 ROI of Digitized Special Collections User What is the value to a user in terms of time and money spent? Prestige What is the prestige to the institution for high visibility digital special collection? Development What value accrues to the development effort of the institution?

166 ROI of Digitized Special Collections Environmental What is the value of the environmental savings from limited physical access to unique and often fragile material? Scholars What value accrues from the role of special collections in attracting graduate students? Collections What is the value of digital collection in attracting additional special collections?

167 ROI of eBooks

168 The Commons

169 ROI = 26:1

170 Communicating Value

171 Past Future Focus Structure Change Outreach Decision making Measures of Success Building & maintaining collections Engaging students & faculty Internal silos University priorities IncrementalTransformational Top down Periodic Shared Deliberative TraditionalImpact

172 Library Value How integral it is to the community How well it supports learning and teaching How well it supports research


174 What your library does well What your customers value Value Proposition

175 An offer, not a demand Not what you value Only valuable perspective is the customer’s Valuable in a competitive environment The Value Proposition

176 Or The promise that a library makes to its customers about what they can expect to receive in return for their time, their effort, their loyalty, and especially their dollars.

177 The library needs to … Focus on customers and what they want and need as well as how they want and need it.




181 Stories + Stats = Success


183 Is the value in the glass, the wine or the savoring?

184 The financial crisis is looking even worse, but you will pleased to know that the director reports that the library performance went up a half a point on the “library goodness scale” last week. Michael Buckland



187 Joe@JoeMatthews.Org Joe Matthews Library Consultant

188 What Are the Results Four Year College Full-time Four Year College Part-time Two Year College Stopout/Transfer College degree – 19% AA degree – 8% No college degree – 60%No college Graduate degree – 11%

189 Valuing the Collection Dewey Subclass Number of Titles 2010-2011 Avg ListTotal Value ($) 001 - Knowledge $76.71$0.00 002 - The book $62.45$0.00 003 - Systems $129.77$0.00 004 - Data processing. Computer science $89.82$0.00 005 - Computer programming, programs, data $69.14$0.00 006 - Special computer methods $83.60$0.00 010 - Bibliography $73.65$0.00 011 - Bibliographies $69.46$0.00 012 - Bibliographies of individuals $0.00 013 - Of works by specific classes of authors $0.00 014 - Of anonymous and pseudonymous works $55.95$0.00 015 - Of works from specific places $184.99$0.00 016 - Of works on specific subjects $134.87$0.00 017 - General subject catalogs $0.00 018 - Catalogs arranged by author, date, etc. $0.00 019 - Dictionary catalogs $0.00 020 - Library and information sciences $56.06$0.00 021 - Library relationships $62.60$0.00 022 - Administration of the physical plant $65.50$0.00 023 - Personnel administration $56.00$0.00 025 - Library operations $77.02$0.00 026 - Libraries for specific subjects $89.99$0.00 027 - General libraries $66.25$0.00

Download ppt "How to Identify Your Impact: The Value of Libraries Joe Matthews June 10, 2013."

Similar presentations

Ads by Google