Creating value and maximising impact : issues, concepts and methods Presentation to 2013 AWHILES Conference The proof of the pudding. Providing the evidence of our value for money 26th & 27th June 2013, Wrexham by Dr Judith Broady-Preston Reader in Information Management Institute Director of Postgraduate Studies Department of Information Studies, Institute of Management, Law and Information Science, Aberystwyth University (copyright retained)

Context and Background Need to demonstrate value and impact = not new ACRL has long been concerned with accountability [and] assessment [since] the early 1980s (2012, 4) BUT - still searching for mechanisms to describe impact i.e. effectiveness accurately impacts...are extremely difficult to measure (Hernon & Altman, 2010, 50) Direct vs indirect – holy grail? Efforts to date focused on demonstrating economic value or impact library service activity (outcome) Need to assign economic value and measure more socially oriented outcomes or aspects library activity – the qualitative

Context and Background Is economic impact sole criterion? Need to assign economic value to the more socially oriented outcomes of library, museum and archive services Current methods fail to fully capture the wider economic value gained from this activity (ERS, 2011, p.35) Need to Capture social returns (ERS, 2011) ACRL 2012 Report – need to demonstrate, assess and communicate impact academic libraries vis-à-vis student learning and achievement Need to gather perceptions of value From a wide variety of stakeholders including current NON-users Use these design, deliver and measure service performance

Outline Context and background : drivers for impact Value Complexity – what are we measuring? Impact Concepts and semantics Definitions and perspectives Models and methodologies Where are we now?

Why is demonstrating value and impact so important? The Global Economic downturn

Global retrenchment in library services The Future of University Libraries: 2012 Midwinter Report: –continuously shrinking budgets are the new normal [sic] for university libraries (2012, 2) AND Research Information Network (RIN) and SCONUL 2010 Report: –After a decade of growth in budgets and services, academic librarians now expect a sustained period of cuts over the next three to five years (Challenge…2010, 4)

Drivers for Impact Economic recession = global –Results in need to target service activity to achieve greatest impact –Defining/determining essential (core) and additional (value-added) services (Broady-Preston and Swain, 2012) Open Access –Berlin Declaration (2003) creates possibility to increase markedly reach, and ergo impact academic research REF2014 – Research Excellence Framework –NEW = not only academic impact but also wider, societal impact OA and Research Council funding – new business models? –Finch Report (UK) June 2012 – gold vs green?? Accepted in full by UK Government 16 July 2012; RCUK Policy on Open Access and Guidance (updated 24 May –Stakeholder perspective

Concepts Two core constructs... VALUE and IMPACT in relation to INFORMATION in relation to Information and Library SERVICES

What is value? Is this the same as impact or quality or satisfaction? Perspectives: organisation vs customer economic vs societal Synonymous with: –Worth? –Fitness for purpose (quality)? –Exceptional/Perfection? –Utility Relationship to values – morality, ethics, culture Perceived value – whose perspective?

Intellectual capital and value/s Relationship between categorisations impact; value; and Intellectual Capital (IC) Conceptual links IC re: change, knowledge transfer and learning organisation (Kong, 2010) Deeper purpose IC to change individual behaviour and values Is IC suitable framework within which to place our understanding and evaluation models such as Balanced Scorecard (BSC) re: measuring progress towards fulfillment complex social objectives and purpose? Public sector reforms often carry with them values consistent with value for money and competition, causing threats to …traditional qualities such as fulfilling social objectives [ergo] IC becomes a valid strategic management conceptual framework within the …nonproft context in the knowledge economy (Kong, 2010, p.295).

Complexity: what are we measuring? Determining what should be measured, how, and why = fundamental to our understanding of service impact Significance Criteria employed Whose perspective? judgments of quality, worth and value Performance measurement = judged in relation to basic purpose. Difficulties: In defining basic purpose in library context Measuring performance in relation to basic purpose viz: in practice it is difficult to draw distinct lines between core business and value added services. Views of service priorities change over time, in relation to external pressures and developments and are…subject to individual interpretation, depending on perspective ( Broady-Preston & Swain, 2012, p. 117).

Complexity: what are we measuring? COMPLEXITY = key characteristic attempts to determine impact in the information sector. Need to take account of complex array multiple stakeholders beyond management and customers anticipate and satisfy stakeholders perception of value and worth acknowledge centrality of requirement that evaluating indirect impact on third parties such as Government = crucial to continued success or even existence Related issue = increasing use volunteer labour to deliver library services Difficulties evaluating performance = complicated when staff also customers or clients of the service Central challenge = identify methodologies which enable libraries to demonstrate their activity results in measurable change re: social objectives AND do so reliably and robustly

More than mere semantics? Terminology and relationships Impact –Variation + attempts to standardise (Poll + ISO 16439, 2012; 2013) –Definitions: all changes resulting from an activity, project or organisation...intended as well as unintended effects, negative as well as positive and long-term as well as short-term (Arvidson 2009 adapted from Wainwright 2002) An effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life beyond academia (REF2014) Difference or change in an individual or group resulting from contact with library services NOTE the change can be tangible or intangible (ISO cited in Poll 2012) NB Centrality of Change

What is Impact? Differing perspectives Organisation Customer Society –Implies measurement - the Q debate i.e. Qualitative vs Quantitative –BUT not everything worth knowing can be counted precisely and reduced to a rank, percentage or ratio; many aspects of information service are intangible, and must be evaluated in other ways (Hernon & Altman, 2010, 50) –Relationship to purpose – aims and objectives (SMART) Organisational Perspective –Measures need to be Meaningful/relevant and are often related to one another to give performance INDICATORS

What is Impact ? Comparative perspective - Benchmarking and Standards Comparisons with and to –other orgs - league tables –external standards – accreditation – e.g. ISO (Poll, 2012; 2013) CSE (Broady-Preston & Lobo, 2011) Process –Gap identification –Monitoring progress Encourages reflective and evidence-based practice Multiple benchmarking standards exist; however –basic principles behind the present QM [Quality Management] systems…are actually rather similar. They all emphasise the continuing improvement of services…[and] all adopt a customer oriented focus (Balague & Saarti, 2009, 227-8)

Types of Impact Determining meaning = key to demonstrating or measuring its existence Need to differentiate between: 1.Economic impact library services on a community 2. Economic value of social returns 3. Social impact = transformative aspect of library services (Broady-Preston, 2012) No. 3 = Most challenging!

Economic impact: models and theories Economic Impact Toolkit (ERS, 2011) –commissioned by the Archives Libraries and Museums Alliance UK (ALMA-UK) –Critique contemporary methodologies in 4 categories Multiplier analysis Contingent valuation –methodology of choice? (cf Poll, 2012) –Predictive vs actual Return On Investment (ROI) –Generic studies of University performance overall claim ROI amongst others = demonstrably inadequate (Fryer et al, 2009; Zangoueinezhad and Moshabaki, 2011) –critique of ROI by Jim Neal (March/April 2011) Stop the Madness: The Insanity of ROI and the Need for New Qualitative Measures of Academic Library Success Economic valuation

Social Enterprise methodologies Originating in voluntary or non-profit sector of economy Often categorised as third or tertiary sector. Defined as An intermediate space between business and government where private energy can be deployed for public good (BusinessDictionary.com, 2013) Closer resemblance to libraries, especially with increasing use of volunteers, multiple stakeholders and social objectives? Methodologies and toolkits Social Enterprise Balanced Scorecard (BSC) Adaptation of Kaplan & Nortons original BSC Designed to help social enterprises deliver multiple bottom lines Clarifies and articulates complexity arising from multiple stakeholders BUT – even modified BSC may focus on satisfying service funders rather than recipients cf Kong, No reports of use by LIS June 2013.

Social Enterprise Balanced Scorecard: Strategy map Sample strategy map Key: White circles reflect objectives Dotted arrows reflect cause and effect linkages

Social Return on Investment (SROI) Outcomes – based tool quantifying social, environmental and economic value created by organisations Narrative of how organizations create and destroy value whilst making changes in the world, producing a ratio stating how much social value (in ££s) created per 1£ spent 6 step methodology Primer = Libraries for Life: Wales aluation/?lang=en aluation/?lang=en Welsh Government used SROI to evaluate impact of strategic development programme for library services (2011) Demonstrated Government's investment in library services had significant impact AND excellent value for money Politically = very popular – profile raising BUT – limited value in evaluating large scale projects

Social/Societal Impact Methodologies Customer Relationship Management (CRM) Customer Value Discovery (CVD) Commonality and/or Core set of customer values? Do these change over time? (McKnight, 2010) Value, Values and Valuation - Values Scorecard transcendent value = holistic, recognising need to assess additional dimensions – human, structural, and relational capital (Town, 2011; Town & Kyrillidou, 2013)

Impact: measuring activity + behaviour JISC – Activity Data Programme –benefit from exploiting activity data (2011: 2013) ( (2013)) US and UK projects = statistical correlation between library service usage and student learning and achievement BUT – Statistical measures alone = limited perspective –University of Nottingham – systematic reviews(Barr, August 2012); IMLS LibValue project (Fleming-May, Kyrillidou, Walker, 2013) –Phase 2 Huddersfield University Library Impact Data Project Seeks to deepen our understanding…by investigating additional data such as gender, age, ethnicity, declared disability, retention, VLE and reading list use…to identify predictors for student outcomes…to understand better how library activity relates to student attainment, including causal relationships (Stone, Collins and Patmore, 2012)

Finally… Provided an overview of SOME ideas and concepts in relation to demonstrating the value and impact of information and library services NB – not definitive! Economic measures alone = limited perspective. However –promoting the value of the libraries in the community also through economic statements can be quite effective. Speaking in terms of dollars and cents seem to have a heavy impact on people, both politicians and ordinary citizens who may not register the value of library services otherwise (Aabø, 2009, 323) Need to look beyond economic impact alone. Money speaks but is not the sole language in terms of demonstrating impact! Measuring true impact library service activity clearly extremely important in turbulent and unpredictable environment.

Finally… In a global recession, service organisations need to focus on continuous service improvement and the stakeholder perspective BUT – a complex activity. Demonstrating social impact may be difficult but we need to look beyond methodologies used currently, and evaluate those used successfully elsewhere. reinforce the message to staff that excellence in the provision of our services should always be at the forefront of our minds (Professor John Lancaster, 2010). Active engagement and partnership with customers = imperative; not only to be valued but to be seen as vital and survive! ANY QUESTIONS? If you think of something later, please contact me – THANK-YOU!