1. Best Practices for Evaluating a New Electronic Resource Sponsored by The Committee on Relations with Information Vendors (CRIV)
Presented by: Brandi Ledferd Technical Services Librarian K&L Gates LLP Seattle, Washington With: Shaun Esposito Head of Public Services University of Arizona Law Library Tucson, Arizona Ed Edmonds Associate Dean for Library and Information Technology, Professor of Law Kresge Law Library Notre Dame Law School Notre Dame, Indiana
Prepared by: Rob Myers Associate Director for Collections, Acquisitions and Planning The Judge Ben C. Green School of Law Case School of Law Case Western Reserve University Cleveland, Ohio With assistance from: Judith Kaul Electronic Resources & IP Reference Librarian The Judge Ben C. Green Law Library

2. Best Practices Defined
Best practices is a concept that migrated from the business community to the academic world as part of knowledge management. 
Organizations gather information about the practices of other organizational units of comparable size and scope.  They compare the outside organizations with each other and with themselves to identify the best practices in their areas of interest. 
The term "best practices" is often used interchangeably with other terms such as "effective practices" or "guidelines" or "standards.“
From “Library Best Practice for Print and Online Periodicals.” Montgomery College Libraries, Montgomery County, Maryland, viewed July 4,

3. Focus of Presentation is on Evaluating Online Commercial Databases
Many of the principles also transfer to e-books, CD-ROMs, and other media.
The presentation will not address specifics of license agreements as they are covered in two other CRIV sponsored programs: G4 “Anatomy of a License Agreement” on Monday at 2:15 pm and H1 “Getting to Yes for Your Library: Negotiation Contracts in Your Favor” on Tuesday at 9:00 am and co-sponsored with PLL.
The presentation will not discuss at any length the design and development of an e-resource Collection Development Policy as this topic merits an entire program in itself and tends to be very library specific based on the subject specialties of the institution.
Discussion of electronic resource management is limited to only those aspects impacting the selection process. We will not discuss at any length organization and access to e-resources, Content Management Systems, Electronic Resource Management Systems, or preservation of e-resources.
While much of the criteria in this program can be applied to e-book collections, e-books are addressed in the program “E-books and the Future of Legal Publishing” on Tuesday, from 10:45 to 11:30.

4. What are the Best Practices for Evaluating a New Electronic Resource?
Develop an e-resource Collection Development Policy that fits the needs and goals of your institution and integrates electronic resources within the array of other collection formats in your library.
“Try before you Buy” – Always get a trial from the e-publisher before agreeing to purchase.
Obtain stakeholder involvement before deciding to purchase.
Use an “Electronic Resource Evaluation Checklist” to ensure you look at all facets of the particular e-resource.

5. Best Practices Continued
Perform a cost benefit-analysis.
Benchmark- Before, during and after the acquisition is made.
Monitor usage statistics regularly (and especially before renewing a subscription).
Develop and implement library workflows, policies, and procedures for evaluating, selecting, and accessing e-resources that are tailored to the goals and needs of your library and promote the use of your collection of e-resources.

6. Life Cycle of an E-Resource Seven Phases
Discovery- Awareness of a new or existing e-resource originates from an attorney, paralegal, faculty member, student, librarian, or publisher brochure. Awareness may also arise out of some recognized need and then researched in a directory of e-resources, other selection tool, or posed as a question to a listserv to find a fitting e-resource solution.
Trial- It is a good practice to always try out an e-resource before deciding to subscribe or purchase. A trial may include additional stakeholders other than just librarians. Nearly all publishers now accommodate trials of their e-resources before deciding to subscribe.
Evaluation- The process of gathering and analyzing information gained from the trial, completing an Electronic Resource Evaluation Checklist, reading/evaluating the license agreement, and performing a cost benefit analysis.

7. Life Cycle of an E-Resource Seven Phases Continued
Selection- Based on results from the trial, the E-Resource Evaluation Checklist, the cost benefit analysis, and satisfaction with the clauses in the license agreement, a decision is made whether to subscribe or purchase the e-resource. (You may be evaluating competing e-resources in which case you would select the one which best suits your needs and budget.)
Acquisition- The process of informing the publisher that you would like to purchase the database. While negotiation of price and clauses in the license agreement can take place at any point along the life cycle, it is at this point that final negotiation takes place. A publisher is most likely to relent somewhat on price and minor changes to the license agreement when they know that a deal is nearly in hand.

8. Life Cycle of an E-Resource Seven Phases Continued
Access- This is the phase at which you inform patrons of the availability of the new e-resource, provide points of access for patrons to use it, place information regarding the e-resource in the library’s ERM system or homegrown equivalent, and passwords if needed. The points of access you use depend on each library’s bibliographic access policies. Examples include placing bibliographic records and links in the library catalog, placement in an a A-Z list or subject list on the library’s webpage, and/or placement in a database of library databases. Examples of ways to inform the patron base of new resources include placing the e-resource on the library’s new acquisition list, notification in the library’s newsletter, broadcast , or simply by placing “new” next to the name of the e-resource in the library’s database list.

9. Life Cycle of an E-Resource
Decision to Renew or Cancel- As the time to renew an e-resource nears, a decision to renew or cancel the subscription must be made. Factors influencing the decision include usage, reliability of the interface (technical issues), receptivity of the patron base, responsiveness of the vendor in addressing issues, and whether better or more cost-effective competing e-resources have emerged during the subscription period
We will examine the first four phases of the life cycle during this presentation though not nearly in the strict order presented above.
Loosely adapted from Sadeh, Tamar and Mark Ellingsen. “Electronic Resource Management Systems: The Need and the Realization.” New Library World 106, no. 5/6 (2005).

10. Stakeholders in the Selection Process
Librarians – It is essential to have your co-workers onboard!
Reference/Public Services Librarians/Subject Specialists- They must use the resource for reference, for teaching, and for training purposes.
Electronic Services Librarians- They must answer and handle all resulting complaints and technical issues.
Catalogers/Bibliographic Access Librarians- They need to address how the database can be cataloged and presented on the library’s website. (e.g. top level linking v. title level deep-linking)
Patron base- Attorneys, paralegals, faculty members, students, and public patrons must like it and have a good enough experience obtaining relevant results to be repeat users.
IT Staff- IT may or may not need to be involved in the selection process though it is best to include them up front should something unforeseen happen such as firewalls blocking access.

11. Essential E-Resource Discovery Tools
Gale Directory of Databases, 34th ed. Detroit: Gale, Cengage Learning, (Vols. 1-3 are devoted to online databases.)
Law and Legal Information Directory, 23rd ed. Detroit, Gale Cengage Learning, (Contains section on Electronic Resources within chapter on Legal Publishers.)
Ken Svengalis’ Legal Information Buyer’s Guide and Reference Manual , N. Stonington, CT: Rhode Island LawPress, (Contains a chapter on computer-assisted legal research. Note: the 2011 edition has just been released.)
Websites- Both publisher/aggregator websites and other library/consortia websites.
Blog Posts - Reviews of e-resources appear on numerous blog posts (Nina Platt’s Strategic Librarian, LLRX.com, Three Geeks and a Law Blog, Law Librarian Blog, to name a few).
Library trade magazines (Choice, Booklist, College and Research Libraries News, Library Journal, Spectrum, etc.)
E-newsletters and e-zines (e.g. the Scout Report, and D-LibMagazine

12. The E-Resource Evaluation Checklist
Reasons for using an E-Resource Evaluation Checklist:
Ensures that you are making the best selection decision.
Serves as a guide so that all necessary questions regarding an e-resource get asked and answered.
Enables and fosters fiscal responsibility and aids in budgeting.
Keeps all the information about an e-resource in one location (if not using an ERMS).
Provides documentation that you are doing your job and fill a need within the institution.

13. The E-Resource Evaluation Checklist
E-Resource Evaluation Checklists Generally Focus on Five Areas:
Content and authority
Usability and design
Technical issues, training and support
License agreement issues
Cost and budgetary considerations
Ways in which E-Resource Evaluation Checklists are maintained:
Electronically (ERMS or other database, PDF form, Word document, Spreadsheet)
As paper forms

14. The E-Resource Evaluation Checklist Specific Criteria to Include
Content Appropriateness- originality, uniqueness, authority
Currency and Archives- also embargos and number of documents or records
User Interface- ease of use, intuitiveness
Search Capability- Basic, advanced, Boolean, full-text, sorting search results, etc.
Browsing Capability- index, table of contents, linking to previous/subsequent sections
Display Formatting- HTML, PDF, etc.
Printing & Downloading – PDF, copy and paste
Clipping and/or services
Compatibility with Mobile Devices and Mobile Apps

15. The E-Resource Evaluation Checklist Specific Criteria to Include
Copyright for ILL and Course Pack Purposes
Access- Passwords, IP Recognition, and VPN
End-User Support- phone support, online support availability
End-User Training- In person, telephone, online, online tutorials, user guides
Technical Support- hours of availability
User Statistics- availability and type provided
Bill Back Mechanisms- Hourly and/or transactional
Marc Records Availability
Deep Linking- Allows for cataloging at the title level
Cost and Type of Subscription: Annual Subscription, Perpetual Access, Digital Ownership, etc.

16. The E-Resource Evaluation Checklist Specific Criteria to Include
Bench Marking- What others have said about the resource (other firms, during library association meetings and lunches, literature reviews, blogs, guides, posts to law-lib and other lists, etc.)
And Lastly, the License Agreement- Discussed in depth in other programs but need to recognize that a license agreement is very influential in the decision-making process especially where it is overly restrictive (e.g. not allowing for ILLs or Course Pack copies).

17. The E-Resource Evaluation Checklist
A sample E-Resource Evaluation Checklist is provided in the handouts on AALL2GO Electronic Resource Evaluation Checklist Form.pdf

18. Selection and Coordination of an E-Resource Trial Focus Group
Reasons for using a Trial Focus Group:
Obtain stakeholder buy-in for the e-resource.
Due diligence:
End-users (Attorneys, Students, Faculty members) may discover something, good or bad, about the resource that might otherwise have been overlooked.
IT Staff might uncover a technical issue that otherwise might not have been found. Even if found after the purchase/subscription has been entered, IT Staff tend to be more receptive to helping if they were included on the front end rather than just when help is needed.
It has a marketing effect (albeit limited) of promoting use of the e-resource.

19. Selection and Coordination of an E-Resource Trial Focus Group
Reason for not using a Trial Focus Group or at least limiting it to just a few select librarians and close others:
You may find that certain demanding faculty, students, attorneys, etc. try the database and say they must have it even though, on balance, the resource is not very good and not worth the money.

20. Selection and Coordination of an E-Resource Trial Focus Group
It is a good idea to invite at least one member of each of the stakeholder groups. (They may decline, but at least they can’t complain they were not informed or invited.):
Librarians (especially reference and bibliographic access librarians)
End-users (attorneys, faculty members, students, judges, clerks)
May want to use attorneys on the firm library committee
May want to use students and faculty members on the law school library committee.
IT Department Staff

21. Selection and Coordination of an E-Resource Trial Focus Group
Types of E-Resource Trial Focus Groups:
Limit to just librarians.
Limit to librarians and select IT staff.
Limited to librarians, select IT staff, and select attorneys, select faculty members, and select students)
An example of this is to use the members of your existing library committee.
Another example is to create a standing committee made up of librarians and attorneys/faculty member/students whose purpose is to trial e-resources.
A final example is a standing committee of librarians with an interchanging mix of attorneys or faculty members (the liaison model in many academic libraries.)
Open the survey to entire localized library community via web and provide link to online survey. (Example: University of South Carolina Coleman Karesh Law Library)

22. Selection and Coordination of an E-Resource Trial Focus Group

23. Selection and Coordination of an E-Resource Trial Focus Group

24. Selection and Coordination of an E-Resource Trial Focus Group

25. Selection and Coordination of an E-Resource Trial Focus Group
Suggestions for devising questions for the e-resource focus group trial:
Have the questions available prior to running the trial. (SurveyMonkey or Zoomerang are useful tools for gathering and analyzing answers.)
Make the process simple for the members of the trial group so that they will want to continue in the role.
Use multiple choice and/or close-ended questions for simplicity.
Keep questions broad and simple. You are looking for the reactions of the average user.

26. Selection and Coordination of an E-Resource Trial Focus Group
A sample questionnaire for participants of an e-resource trial is provided in the handouts on AALL2GO Sample Questions for Participants of E-Resource Trials.pdf

27. The Cost-Benefit Analysis
Factors to consider:
The database you are interested in versus competitors’ offerings (In this case you may want to run similar database trials simultaneously.)
The database you are interested in and there are no competitors.
The database you are interested in versus print formats and other formats already in the library.
The database you are interested in versus group subscription rates through a formal consortium such as NELLCO or OhioLINK or an informal grouping of libraries.
The database you are interested versus free on the web (e.g. Free EDGAR versus a commercial version).

28. The Cost-Benefit Analysis
Additional Factors to consider:
Just in Case v. Just in time:
Subscribe just in case someone needs it (mainly Academics)
vs.
Being able to obtain an item just at the time someone needs it.
Examples of Just in time:
Pay per use subscriptions (i.e., PACER)
Pay via Credit Card (Lexis stopped this practice in May 2011, Westlaw and many public record websites continue to offer it.)
Interlibrary Loans

29. The Cost-Benefit Analysis
Subscription v. Perpetual Access v. Digital Ownership v. Purchase Plus Access
Subscription- This is the traditional license agreement to “lease” licensed material for a specified term at a specified cost. When the subscription term ends, access to all the material also ends. The typical subscription period is for one year with an annual recurring subscription fee.
Perpetual Access- Typically, when an e-resource license lapses, access to all content is lost. Perpetual access provides the ability to retain access to the e-resource materials after the contractual agreement for these materials has passed. It is similar to maintaining access to the back run of a print journal after cancelling the print journal subscription. Ownership of the content never passes to licensee but access to all the material published up to the time of cancellation remains in perpetuity. Perpetual access usually requires a lump sum payment in advance plus an annual access fee. The annual access fee is lower than a typical annual subscription fee thus making perpetual access attractive for some libraries.

30. The Cost-Benefit Analysis
Subscription v. Perpetual Access v. Digital Ownership v. Purchase Plus Access
Digital Content Ownership- This is a purchase agreement whereby the library obtains limited ownership rights to the contents of the e-resource. The content is often provided to the library on a storage device (hard drive, CD-ROM or Digital Linear Tape) and may or may not include metadata files. Ownership typically does not include the sale of the delivery platform or any method for access (i.e., user interface, search engine, etc.). Ownership is a large, one-time lump sum payment (though financing for a term of years is often available.)
Purchase Plus Access- This is an agreement for digital content ownership along with a subscription license agreement granting access to the content and delivery platform on the publisher’s servers. There is a large one-time lump sum payment for ownership and an annual “access” or “maintenance” fee for the subscribed access to the content on the publisher’s server. The annual access fee is usually substantially lower compared to the annual subscription fee a library would pay to access the material.

31. The Cost-Benefit Analysis
Misc. Observations on Subscriptions, Perpetual Access,
Digital Ownership and Purchase Plus Access
The Smith, Leers, Roncevich’s article titles “Database Ownership: Myth or Reality?” Law Library Journal 103, no. 2 (2011) provides an excellent case study on the difficulties of having to construct a database using the files supplied by a publisher upon the cancellation of the access portion of a Purchase Plus Access agreement.
Purchase Plus Access makes sense when you have a sizeable amount of one-time only money left toward the end of a fiscal year or when a donor provides a one-time, non-endowment donation to the library.
The traditional subscription license agreement allows for predictability in budgeting akin to the annual recurring costs of print serials.

32. The Cost-Benefit Analysis
Additional Observations on Subscriptions, Perpetual Access,
Digital Ownership and Purchase Plus Access
The cost of usage under a traditional subscription license agreement can be more readily ascertained and billed onto clients. Passing costs onto clients is a much more difficult endeavor with perpetual access, purchase plus access, or digital content ownership.
It may be worthwhile to subscribe to an e-resource for a limited time to measure usage before deciding to go forward with an ownership or perpetual access pricing model.
The big question is what happens to the perpetual access if an e-resource publisher goes out of business or is sold?

33. The Cost-Benefit Analysis
So what is the cost-benefit analysis?
The cost of the subscription license or purchase agreement plus annual access fees
balanced against
The need for and uniqueness of the content contained in the e-resource.
Whether the cost can be recovered through billing on to clients.
Whether print material can be cancelled in reliance of the e-resource.
Whether space will be saved by removing cancelled print.
Whether loose-leaf filing and other processing costs will be saved.
Whether there is an anticipated large audience of users to justify the cost.
Whether the e-resource can be used for instructional purposes.
Whether the e-resource will save researchers significant amounts of time.
Whether the e-resource can be accessed by multiple simultaneous users.
Whether the e-resource can be accessed from home or other offsite location.

34. Cost Recovery Through Billing Clients
This topic is only briefly covered here as a full program, “To Recover or Not Recover: Trends, Solutions, and Alternatives for Taming Online Research Costs,” is devoted to the topic on Monday, July 25th from 2:15 to 3:30.
ABA Comm. on Ethics and Professional Responsibility, Formal Op “authorizes lawyers to charge reasonable costs associated with online legal research as long as the charges are disclosed to clients. The guiding principle around ethical practice is that the client charge should be commensurate with the service charge and not be marked up for firm practice.”
In other words, firms should not make a profit through online legal research mark-ups.
Many states also have opinions concerning online legal research cost recovery.

35. Cost Recovery Through Billing Clients
Percentage of firms charging for online research:
According to a survey of 81 firms (50 responses of which were from firms with more than 500 attorneys) conducted by Mattern & Associates in 2010, 73% of respondents charge clients for online legal research. That is a drop from the 97% of firms that said they charged for online legal research when the study was conducted in (See: Mattern, Robert. “Cost Recovery Breeds Client Frustration in New Survey.” Law Technology News, April 21, 2011)
According to a study conducted by the ABA General Practice, Solo & Small Firm Division in 2008, 50% of small and solo firms recover all or part of their online legal research costs and 50% do not. According to the 2008 study, the cost recovery process varies widely among firms.
While cost recovery is on the decline, it can still be a very influential factor in e-resource selection for many firms.

36. Cost Recovery Through Billing Clients
Articles on low cost alternatives to Westlaw and Lexis:
Justiss, Laura K. “A Survey of Electronic Research Alternatives to LexisNexis and Westlaw in Law Firms.” Law Library Journal 103, no. 1 (2011):
Svengalis, Kendall F. “Chapter 25: Computer-Assisted Legal Research” in Kendall F. Svengalis. Legal Information Buyer’s Guide and Reference Manual N. Stonington, CT: Rhode Island LawPress, 2010,

37. A Bibliography for this Program is available on AALL2GO.
THANK YOU!
A Bibliography for this Program is available on AALL2GO.