Strategies LLC Taxonomy October 7, 2009Copyright 2009 Taxonomy Strategies LLC. All rights reserved. How to Validate a Taxonomy ASIST Workshop Preview

2 Taxonomy Strategies LLC The business of organized information What is a Taxonomy? v A categorization framework agreed upon by business and content owners (with the help of subject matter experts) that will be used to tag content. 6 broad, discrete divisions (called facets) 2-3 levels deep. Up to 15 terms at each level terms total. With some logichierarchical, equivalent and associative relationships between terms.

3 Taxonomy Strategies LLC The business of organized information Why taxonomy is important

4 Taxonomy Strategies LLC The business of organized information Why is taxonomy important? v Easier information management. v Flexibility to respond to changing needs. v Foundation for findability and usability.

5 Taxonomy Strategies LLC The business of organized information How can you know if a taxonomy will work? v Tagging content. v Publishing content. v Finding and using content in user-facing applications.

6 Taxonomy Strategies LLC The business of organized information What types of taxonomy validation can you do? v Validating taxonomy design. v Validating taxonomy usability. v Validating taxonomy use in a collection.

7 Taxonomy Strategies LLC The business of organized information Taxonomy design validation methods MethodProcessWhoRequiresValidation Web toolOpen Card sort Representative users Rough taxonomy Card sort analysis (emergent patterns) Cards & markup Delphi card sort Representative users Rough taxonomy End result (consensus) Walk-thruShow & explain Stakeholders Rough taxonomy Approach Appropriateness to task Walk-thruCheck conformance to editorial rules Taxonomist Draft taxonomy Editorial rules Consistent look & feel Matrix or cards Closed card sort End users (or surrogates) Rough taxonomy Top queries, etc. Consistent results User satisfaction Survey End users (or surrogates) Draft taxonomy Reaction to taxonomy Tagging samples Tag sample content with taxonomy Taxonomist Indexers Sample content Draft taxonomy (or better) Content fit Fills out content inventory Training materials for people & algorithms Basis for quantitative methods

8 Taxonomy Strategies LLC The business of organized information Usability validation methods MethodProcessWhoRequiresValidation Usability Testing Task-based scenario testing (Find it, Tag it, etc.) End users (or surrogates) Draft taxonomy Tasks & answers Tasks are completed successfully Time to complete task is reduced User Satisfaction Survey End users (or surrogates) Draft taxonomy UI mockup, Search prototype Reaction to taxonomy Reaction to new interface, Reaction to search results

9 Taxonomy Strategies LLC The business of organized information Collection analysis validation methods MethodProcessWhoRequiresValidation DistributionStatistical analysis Taxonomist Analyst Tagged collection Do categories follow Zipf distribution? Do % of categories equal % of documents? Identify candidates for merging & splitting. Query log & click trail examination Clustering & statistical analysis Taxonomist Analyst Search logs Web analytics Do (clustered) queries follow Zipf distribution? Are top (clustered) queries in taxonomy?

10 Taxonomy Strategies LLC The business of organized information Full-day taxonomy validation workshop agenda 9:00-9:3030 minIntroduction 9:30-10:0030 minTaxonomy & validation overview 10:00-10:3030 minTaxonomy design validation 10:30-10:4515 minCoffee Break 10:45-11:3045 minDelphi card sorting exercise 11:30-12:0030 minClosed card sorting 12:00-1:0060 minLunch 1:00-1:4545 minClosed card sorting exercise 1:45-2:1530 minOnline tools 2:15-2:4530 MinTask-based validation methods 2:45-3:0015 minCoffee break 3:00-3:4545 minTask-based validation exercise 3:45-4:1530 minCollection analysis overview 4:15-5:0045 minQ&A, Closing Hands-on ExercisesBreaksLectures

11 Taxonomy Strategies LLC The business of organized information Closed card sorting case study v Project: Develop Substance Abuse and Mental Health Services Administration (SAMHSA) taxonomy to be the backbone of a unified agency information service that collects, integrates, and reports on customer interaction data in ways that help the agency plan health communications products, messages and outreach. v Task: Sort popular queries (words and phrases) from search logs into the most likely Taxonomy facet.

12 Taxonomy Strategies LLC The business of organized information Term sorting data collection form

13 Taxonomy Strategies LLC The business of organized information Summary of term sorting results Frequently chosen related categoryFrequently chosen incorrect categoryCorrect category

14 Taxonomy Strategies LLC The business of organized information Percentage of popular search terms sorted correctly

15 Taxonomy Strategies LLC The business of organized information Blind sorting of popular search terms (n=12) 84% of terms were correctly sorted % of the time. Results: Excellent Difficulties v For Methadone, confusion when, in this case, a substance is a treatment. v For general terms such as Smoking, Substance Abuse and Suicide, confusion about whether these are Conditions or Research topics.

16 Taxonomy Strategies LLC The business of organized information Search terms sorting task user rating (n=12)

17 Taxonomy Strategies LLC The business of organized information Task-based case study v Project: Develop American Society of Civil Engineers (ASCE) taxonomy to draw from existing databases and content sources, and provision the next generation websites being built to support this non-profit, membership associations activities. v Task: Find specific content (web pages)

18 Taxonomy Strategies LLC The business of organized information Find web pages AAudiences CContent Types EEvent Types LLocations OOrganizations TTopics AAudiences CContent Types EEvent Types LLocations OOrganizations TTopics T.1Architectural Engineering T.2 Coasts & waterways T.3Construction T.4Cross-Cutting Topics T.5Disaster & Hazard Management T.6Education & Career Development T.7Engineering Mechanics T.8Energy T.9Environment T.10Geotechnical Engineering T.11People, Projects & Heritage T.12Planning & Development T.13Professional Issues T.14Project Management T.15Structural Engineering T.16Transportation T.17Water & Wastewater TTopics T.1Architectural Engineering T.2 Coasts & waterways T.3Construction T.4Cross-Cutting Topics T.5Disaster & Hazard Management T.6Education & Career Development T.7Engineering Mechanics T.8Energy T.9Environment T.10Geotechnical Engineering T.11People, Projects & Heritage T.12Planning & Development T.13Professional Issues T.14Project Management T.15Structural Engineering T.16Transportation T.17Water & Wastewater T.6Education & Career Development T.6.1Continuing Education T.6.2Engineering Education T.6.3Management & Professional Development T.6.4 Scholarships, Internships & Competitions T.6Education & Career Development T.6.1Continuing Education T.6.2Engineering Education T.6.3Management & Professional Development T.6.4 Scholarships, Internships & Competitions ASCE Continuing Education

19 Taxonomy Strategies LLC The business of organized information Summary of navigation results trial Frequently chosen related categoryFrequently chosen incorrect categoryCorrect category Gave up

20 Taxonomy Strategies LLC The business of organized information Overall navigation task performance (n=54) v 87% navigated as predicted or used a reasonable alternative. v In only 4% of the trials, did the subject give up.

21 Taxonomy Strategies LLC The business of organized information Overall user rating of navigation task (n=9) No one rated the overall task Difficult!

Strategies LLC Taxonomy October 7, 2009Copyright 2009 Taxonomy Strategies LLC. All rights reserved. Questions ASIST Annual Meeting - Full-Day Workshop November 7, Vancouver, BC Gilbane Conference - Half-Day Workshop December 1, Boston, MA Joseph A. Busch