1. MEASURING THE EFFECTIVENESS OF FACETS IN THE NEXT GEN OPAC 2008 LIBRARY ASSESSMENT CONFERENCE Kathleen Bauer and Alice Peterson-Hart Yale University Library Usability and Assessment 1

2. Usability process Assess current behavior Conduct usability protocol testing Share findings with staff Implement changes and assess effect 2

3. Collecting evidence from Yales Voyager- based catalog, called Orbis, using log file data. Assess current behavior 3

4. Log file line 4 12/2/07 20:0620071202200646 Title Index TALL JOHN DOGGET JNR 0

5. Log file sample descriptive data 5 Most common search: title (41.8%) followed by keyword (31.4%) Average search phrase is 2.5 words long Most common phrase searched is united states Most common number of hits: 0 (21.4% )

6. Examples of title searches 6 BOOKS ON PHOTOGRAPHY? ALABAMA 2004? PATHOPHYSIOLOGY CHILDREN? JOHNSON, JAMES? INTRODUCTION TO AFRICAN AMERICAN STUDIES: A READER? Topic Name Close

7. Why do title searches get 0 hits? 7 Title 0 hits sampleNumber in class Percent of all 0 hits (N=506) Subject or keyword10821.3% Misspelling9318.4% Author or person8717.2% Correct no holdings6913.6% Close6713.2% Foreign language326.3% Unknown275.3% Article142.8% Other91.8%

8. Decreasing search types 2007-2008 8

9. Some conclusions from log analysis 9 Patrons commonly use field specific searches unsuccessfully Majority of searches dont use catalog metadata beyond title (decreasing) Errors happen a variety of ways and current Voyager system is not helpful in avoiding those errors

10. Next Gen OPAC -- Vufind 10

11. After deciding to implement Vufind, use data from log files to help design usability tests Conduct usability tests 11

12. Creating usability tests for Yufind 12 Two opportunities for tests: medical implementation of subject-based ebooks display alternative interface for catalog (aimed at undergraduates) Objective: Would the participant see and choose facets? (more than 26% of the time) Objective: Would the participant use facets successfully?

13. Questions Find books within a set by a particular author (John Adams). In a set of books about healthcare, find some that are about healthcare reform. Within a subject-based display of ebooks find a topic-based subset. 13

15. Facets: key findings 15 Were seen and sometimes used but not always successfully for author and topic Subsets sometimes didnt make sense Hard to navigate (not alphabetical) Little use of call number facet Some preferred search refinement to facets Participants noted potential usefulness: format, language, and topic facets

16. Did facets increase metadata use? Opportunities for facet use Facet Use|Success Search Refinement with 2 nd search Use|Success 5 (question 2a undergraduate)3|1 5 (question 4c undergraduate)2|21|0 8 (question 5 medical)4|44|1 All opportunities=18 9|7 50.0% | 38.9% 8|1 44.4% | 16.7% 16 YES

17. Staff need to engage with findings of usability testing and current behavior in Orbis to decide best practices and to build empathy for user experience. Share findings with staff 17

18. Relevance ranking and facets 18

19. Review keyword searches with subject specialist librarians 19 Times square red times square blue hague conference on peace hague conference on peace 1907 second hague conference on peace 2nd hague conference on peace cuban revolution cuban revolution of 1959 … (list of 50 to start)

20. Set best practices for facet display Display facets on left Develop consistent terminology between different applications Consistent order of display 20

21. Implement changes and assess 21

22. Implement changes and assess 22 Move facets to the left side (standard) Demoted (in list order) topic facet Add a sort optionwill that increase successful use of facets? Can we promote use of call number facets?

23. Usability process Assess of current behavior Conduct usability protocol tests Share findings with staff Implement changes and assess effect

24. http://www.library.yale.edu/libepub/usability/yulstudies.html/arl.ppt Slides and findings available 24 Yale University Library Usability and Assessment Department Kathleen Bauer Alice Peterson-Hart Send questions to kathleen.bauer@yale.edu