February 14, 2016NAME OF EVENT1 What’s the Big Deal? Collection Evaluation at the National Level Clare Appavoo, Executive Director, Canadian Research Knowledge Network (CRKN)

February 14, 2016ALA Introduction to CRKN

Collection evaluation at the national level: questions to be answered Is there an affordable way to dismantle Big Deals while continuing to foster research and promote access? Does the value of journals and packages remain constant notwithstanding institution’s individual characteristics? Is there a benefit to evaluating collections collaboratively? February 14, 2016ALA 20133

Big Deal Exit Checklist Leaving a Big Deal – Assess potential institutional impacts Exiting a Big Deal Loss of Access Interlibrary Loan Needs for teaching Needs for research Collections Budget Collection Management (Workflows) Leaving a Big Deal – Exit implementation guidelines Exiting a Big Deal Define objectives Define resources for transition Define resources for title-level management Communicate with users and other stakeholders Consider compliance and accreditation needs Identify titles February 14, 2016ALA 20134

Big Deal Cost-Per-Use February 14, 2016ALA Gather COUNTER JR1 usage statistics Decide what constitutes “high-use” (e.g. 50, 100 uses) Divide the # of high-use titles by total package price to determine maximum “ideal price” Eg. if 300 titles are high use, and the package price was $100,000, the ideal price is $333. Individual subscriptions can be NO MORE than $333 Determine average list price for single subscriptions using vendor pricing data Compare ideal price with list price. If list price is lower than ideal price, a move to individual subscriptions may be warranted

Limits of usage and exploring the California Digital Library Journal Value Metrics Limits to the cost-per-use methodology as an assessment measure Favors resources in STM disciplines Favors resources used for teaching Does not consider value solely for research Can change with unpredictable spikes in use patterns Does not consider journal quality Does not consider content needs for accreditation The CDL Model “How much value does our institution derive from Journal X compared to other journals that we license in the same discipline?” Value is defined as quality, utility and cost effectiveness Evaluating 8,600 journals in 36 licensed packages A numerical score is assigned to each journal depending on whether it is above or below the benchmark in a subject area February 14, 2016ALA 20136

Journal Value Metrics - Method February 14, 2016ALA Utility Full text usage Faculty publications Quality Impact Factor SNIP (Source Normalized Impact per Paper) Cost Effectiveness Cost per use Cost per SNIP

CRKN Pilot – Adapted Journal Value Metrics InstitutionFTERegion Library A1,300Ontario Library B3,300Atlantic Library C4,200Prairie/Pacific Library D12,500Quebec Library E18,220Atlantic Library F22,000Ontario Library G27,000Prairie/Pacific Library H29,000Quebec February 14, 2016ALA 20138

Does the value of journals remain constant regardless of varied institutional characteristics? Top 100 Journals No. of titles in institution’s top 100 and overall top 100 Library A65 Library B62 Library C69 Library D68 Library E71 Library F74 Library G74 Library H74 Comparing Quartiles February 14, 2016ALA Top 100 journals overall were within the top quartile for each library Bottom quartile (340) for each institution was virtually identical Top quartile (340) for each institution was virtually identical

Combined individual list price of the top quartile of journals compared to total package price February 14, 2016ALA

Do the top 100 journals as determined by JVM differ from the top 100 journals as determined by use? DisciplineJournals in top JVM Medicine31 Life Sciences22 Social & Behavioral Sciences16 Psychology13 Nursing, Dentistry & Healthcare4 Agriculture, Aquaculture & Food Science3 Business, Economics & Finance3 Mathematics3 Earth & Environmental Science2 Chemistry1 Computer Science1 Law & Criminology1 11 DisciplineJournals in top Usage Medicine33 Life Sciences32 Chemistry7 Social & Behavioral Sciences5 Psychology5 Physical Science & Engineering5 Nursing, Dentistry & Healthcare4 Agriculture, Aquaculture & Food Science3 Veterinary Medicine3 Earth & Environmental Science2 Mathematics1

Do the top 100 journals as determined by JVM differ from the top 100 journals as determined by use? JVM – top 100 journalsUsage – top 100 journals February 14, 2016ALA

Benefits of evaluating collections collaboratively Determine where to commit consortia’s resources Shared resources for information gathering Data will determine negotiation priorities Shared management of journal data February 14, 2016ALA

Journal Value Metrics – Next Steps and Applications Findings: Without movement from the vendors on individual list prices, a centrally negotiated Big Deal continues to represent the best value for a comprehensive suite of content. When evaluating participation in a Big Deal, or when financial pressures necessitate cancellation, usage is not a sufficiently multifaceted unit of measure to determine key journals for research and teaching. Simplified JVM model for use at institutions All publishers asked to provide COUNTER JR1 stats for all members Evaluating all packages in order to make comparisons? Ensure the right Big Deals, containing the right titles, are being licensed for the right price February 14, 2016ALA Utility Full text usage Quality SNIP (Source Normalized Impact per Paper) Cost- effectiveness Cost per use Next Steps:

Questions? Clare Appavoo: Eva Jurczyk for study