1. Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE

2. Background AUS implemented Encore Discovery in 2009 Other products like Serials Solutions Summon and EBSCO’s EDS are actively marketing in the Middle East Decision in 2010 to evaluate all products Having Encore installed gave us opportunity to evaluate products against the same databases, and think about how an ILS-integrated discovery solution compares to a less integrated solution Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

3. Methodology Setting up trials of Summon and EDS Scheduling vendor presentations of Summon and EDS Testing all 3 products for ability to produce useful, relevant and accessible results Reviewing the features of each of the products Performing usability testing with all products Assessing the costs of implementing each product Assessing the impact of implementing each product on other library systems and operations Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

4. Usability Testing Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

5. Informal Usability Testing Think Aloud Protocol 5 participants per product Participants only told they would be testing a new discovery tool to find library resources Participants used their own research topic to complete realistic tasks Observers did not interact with participant except to give tasks and encourage thinking aloud Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

6. Testing Environment 1 test administrator 1 observer Laptop with built in microphone for recording voice Camtasia Studio with Powerpoint plug-in to capture screen and voice Browser of choice Other software, websites as requested – Google and Microsoft Word for spelling Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

7. Tasks Based on common assignments requirements seen at the Research Help Desk, participants were asked to find: an article an article from a peer reviewed/scholarly/academic journal a newspaper article a reference book a general collection book a book published in 2008 or 2009 Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

8. Observations Limit / facet confusion – “Format” or “Content Type” vs “Location” – Books, e-books, e-journal E-journal format was used to find articles Limits reset for each new search Click through to article – Expect to get article, but given a WebBridge or 360 Link page Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

9. System Usability Scale Participants filled out System Usability Scale questionnaire for perceived ease-of-use score 10 item Likert scale questionnaire used to determine user’s perception of the system for – Effectiveness – Efficiency – Satisfaction Resulting scores range from 0 to 100 – Average SUS score across 500+ studies is 68 Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

10. System Usability Scores System Usability ScaleEncoreEDSSummon 1. I think that I would like to use this system frequently43.24.4 2. I found the system unnecessarily complex2.43 3. I thought the system was easy to use43.44.2 4. I think that I would need the support of a technical person to be able to use this system 2.42.21.8 5. I found the various functions in this system were well integrated4.43.43.8 6. I thought there was too much inconsistency in this system22.61.4 7. I would imagine that most people would learn to use this system very quickly 43.24 8. I found the system very cumbersome to use2.22.41.8 9. I felt very confident using the system43.64 10. I needed to learn a lot of things before I could get going with this system 2.23.22 Average System Usability Score73.55877.5 Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

11. Overall Usability Summon – Pop-up abstract and number of results for limit in brackets were very popular Encore – Worked very well for books – Article search interface (Synergy) not well understood or used EDS – Limits not intuitively named, extra “Update” click to refine – Can set mutually exclusive limits giving 0 results Preference to use web for finding newspaper articles Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

12. Relevancy and Functionality Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

13. Relevancy Test Testers - AUS public service librarians Searches - Actual searches done this year in Encore Scoring - Scale 1 - 5 + comments Scorecards - One scorecard for each search in each discovery tool. Extensive comments Discovery tools tested – Encore – Summon – EDS Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

14. Actual Searches Selected from Google Analytics Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

17. Encore URLs /iii/encore/search/C|Scarbon+fiber+reinforced+p olymer|Orightresult|U1?lang=eng&suite=pearl /iii/encore/search/C|Sglobalization|Orightresult| U1?lang=eng&suite=pearl Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

18. Searches arranged+marriage carbon+fiber+reinforced+polymer Electronics+All-In-One+Desk+Reference+For+Dummies electronics+for+dummies globalization jean+luc+godard sex+education uae+federal+laws walt+disney "Artificial+intelligence"+And+"Manufacturing" Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

19. Types of Results Top 5 results – Number of: Books Articles Other (e-books, e-journals, DVDs, e-resources, print journals, videos, etc) Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

20. Types of Results BooksArticles Other EDS 1230 EDS 2005 reports EDS 3122 e-books EDS 4140 EDS 5050 Summon 1050 Summon 2032 newspapers Summon 3320 Summon 4131 e-journal Summon 5050 Encore 1320 Encore 2320 Encore 3122 e-books Encore 4221 e-journal Encore 5320 Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

21. Relevancy Assessment Overall reaction to top 5 results: 5Excellent (just what I was looking for) 4Very good (on target and useful) 3Fair (acceptable) 2Poor (not useful) 1Bad (unrelated to topic) Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

23. Results Tester 1Tester 2Tester 3Tester 4Tester 5AVGTotal Score Summon 1114111.68 Summon 2333433.216 Summon 3555544.824 summon 4455534.422 Summon 5454534.221 SUMMON TOTAL171921201418.291 EDS 1444433.819 EDS 2232322.412 EDS 344444420 EDS 4443333.417 EDS 5444433.819 EDS TOTAL181917181517.487 Synergy 1231332.412 Synergy 233.52353.316.5 Synergy 323352315 Synergy 4233232.613 Synergy 5442443.618 SYNERGY TOTAL1316.51117 14.974.5 Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

24. Functionality Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

25. Functionality Conclusions Basically, all 3 systems offer similar user functionality – sorting, limiting, emailing and “did you mean” are available. There is no superior feature that would make one system preferable to another, just as there is no missing feature that would eliminate a system. Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

26. Impact Assessment Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

27. Pieces & Configurations A-Z ListLink ResolverDiscoveryFederated Search 1Existing360 CoreWebBridgeEncore SynergyResearch Pro 2Summon360 Core360 LinkSummonna 3Summon360 CoreWebBridgeSummonna 4EDSEH A-ZLinkSourceEDS 5 360 CoreWebBridgeEDS New Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

28. Functional Equalization EDS Federated Search EDS & Summon – Millennium Integration Pieces III Mobile Interface (free if Airpac) III Encore Harvesting III Query API – in development Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

29. Costs Sunk Costs to write off – Encore Server – WebBridge License Ongoing Costs – Quotes for each component Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

30. Work Needed for any Change Implementation – Profiling of resources and catalog records – Interface customization where possible – Changes to website – Changes to electronic resources – Staff training – Modify instruction materials and curriculum Ongoing Work – Bibliographic Data Loads – Changes to Holdings Loads for EDS Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

31. Potential Lost Functionality Tag Clouds & Authorities for related searching Availability Facet WebBridge/Pathfinder for Collection Maps User tags Relevancy modification Millennium integration – Automatic update/syncing – Hold placement – Collection map links – My Lists – Saving RP Groups in My Millennium Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

32. Potential Added Functionality Administrative interface for searching & user interface Mobile Interface bundled in Repository Integration bundled in API’s bundled in 1-Click in 360 Link Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

33. Final Scoring FactorSummonEDSEncore Weight SummonEDSEncore Relevance321 0.3 0.90.60.3 Usability312 0.3 0.90.30.6 Financial Impact312 0.2 0.60.20.4 Implementation Work Impact123 0.1 0.20.3 Ongoing Work Impact213 0.1 0.20.10.3 12711 2.71.41.9 Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

34. Thank you Thomas Hodge thodge@aus.edu Kathleen Botter kbotter@aus.edu Mary Ann Nash mnash@aus.edu