1. An Introduction to Treejack
Out on a limb with your IA
Dave O’Brien
Optimal Usability

2. Welcome Dave O’Brien Optimal Usability 22 Jan 2010 36 attendees
Wellington, New Zealand
22 Jan 2010
36 attendees
USA, CA, UK, NZ, AU, BR, CO

3. Agenda Quickie Treejack tour What is tree testing?
Planning a tree test
Setting up Treejack
Running a test
High-level results
Detailed results
Lessons learned
(Q&A throughout)

4. Poll Have you used Treejack yet? No, haven’t tried it yet = 20%
Yes, but only a practice test = 60%
Yes, have run a "real" test = 20%

5. Tree testing - the 5-minute tour
Creating a medium or large website
Does your top-down structure make sense?

6. Does your structure work?
Can users find particular items in the tree?
Can they find them directly, without having to backtrack?
Could they choose between topics quickly, without having to think too much?
Which parts of your tree work well?
Which fall down?
The Krug Test – “Don’t make me think!”

7. Create a site tree

8. Write some tasks

9. Put this into Treejack

10. Invite participants

11. Participants do the test
Browsing down
Backing up
Skipping

12. You see the results

13. Live demo for participants*
Q: Can you control font size and color?
Q: it would be great if all the options had an 'i'd pick this ' button - when it's on just one, feels leading
Q: when you get to the deepest level, if all things are even, have the 'i pick this' button on every row

14. What is tree testing, really?
Testing a site structure for
Findability
Labeling

15. What’s it good for? Improving organisation of your site
Improving top-down navigation
Improving your structure’s terminology (labels)
Comparing structures (before/after, or A vs. B)
Isolating the structure itself
Getting user data early (before site is built)
Making it cheap & quick to try out ideas
Quick to get a test up and running
Quick/easy for users: 10 min for 10 tasks
Analysis is straightforward
Can test baselines and revisions quickly
By isolating the site structure - by removing other variables at this early stage of design - we can more clearly see how the tree itself performs, and revise until we have a solid structure. We can then move on in the design process with confidence. It’s like unit-testing a site’s organisation and labeling. Or as my colleague Sam Ng says, “Think of it as analytics for a website you haven’t built yet.”

16. What it’s NOT NOT testing other navigation routes
NOT testing page layout
NOT testing visual design
NOT a substitute for full user testing
NOT a replacement for card sorting

17. Origin Paper tree testing “card-based classification” – Donna Spencer
Show lists of topics on index cards
In person, score manually, analyse in Excel

18. Make it faster & easier Create a web tool for remote testing
Quick for a designer to learn and use
Simple for participants to do the test
Able to handle a large sample of users
Able to present clear results
Quick turnaround for iterating

19. But I already do card sorting!
Open card sorting is generative
Suggests how your users mentally group content
Helps you create new structures
Closed card sorting – almost not quite
Tree testing is evaluative
Tests a given site structure
Shows you where the structure is strong & weak
Lets you compare alternative structures
While closed card sorting mimics how users may file a particular item of content (e.g. where they might store a new document in a document-management system), it doesn’t necessarily model how users find information in a site. They don’t start with a document – they start with a task, just as they do in a usability test.
What we wanted was a technique that more closely simulates how users browse sites when looking for something specific. Yes, closed card sorting was better than nothing, but it just didn’t feel like the right approach.

20. A useful IA approach Run a baseline tree test (existing structure)
What works? What doesn’t?
Run an open card sort on the content
How do your users classify things?
Come up with some new structures
Run tree tests on them (same tasks)
Compare to each other
Compare to the baseline results

21. Planning a tree test Stakeholder interview
Find out who, what, when, etc.
fill in "planning questions" template
Get the tree(s) in digital format
use Excel tree-import template, etc.
Show “planning questions” template – a canned set of questions specific to tree testing

22. Getting the tree Import a digital format Or enter in Treejack Excel
Text file
Word
Or enter in Treejack
Show Excel template – multiple trees on worksheets

23. Poll How big are your trees? Small (less than 50 items) = 25%
Medium ( items) = 39%
Large ( items) = 22%
Huge (more than 250 items) = 14%

24. Tree tips Recommend <1000 items Bigger? Cut it down by:
Using top N levels (e.g. 3 or 4)
Testing subtrees separately*
Pruning branches that are unlikely to be visited
Remove “helper” topics
e.g. Search, Site Map, Help, Contact Us
Watch for implicit topics!
which assumes that users can navigate the top level easily – a big assumption
Helper topics - If we leave them in, it makes it too easy for users to choose them as alternatives to browsing the tree, and we don’t learn as much.
Q: We are testing some IA's which need supporting text for the links - not ideal, but are there any ways to factor this sort of support into a Treejack evaluation (if not that's fine)

25. Implicit topics
Create your tree based on the content, not just the page structure.
Home
Products
Support
Contact Us
South America
Europe
Contact Us
North America
Lorem ipsum dolor sit amet,
consectetur adipisicing elit
sed do eiusmod tempor incididunt
ut labore et dolore magna aliqua.
South America
Home
Products
Support
Contact Us
North America
South America
Europe
Beware content-management systems that spit out page trees. They need careful review!
Europe

26. User groups and tasks Identify your user groups
Draft representative tasks for each group
Tasks must be “real” for those users!
~10 tasks per participant
Beware the learning effect
Small tree ~8, large tree ~12
More tasks? Limit per participant
Randomise the task order
Example – Shimano website with sections for bike parts and fishing equipment.
Q: What’s the upper limit on # of tasks?
Q: Since you can only have 1 user test 10 or so paths, can you combine data sets so you can get a bigger sampling? So by running 100 people through test A, 100 through test B, etc..... I can combine results from A-Z tests to see a picture of something with 100's of paths

27. Drafting tasks What parts of the tree do you want to test?
Coverage should reflect importance
Each task must:
Be specific
Be clearly worded
Use the customer’s language
Be concise
Beware “give-away” words!
Review now, preview before the real test
Show Excel template – tasks and coverage
Customer’s language – “How would you upgrade a fixed asset, such as your laptop?”
Q: can task instructions include images (e.g. Where would you find this and show a picture of a sofa). (In ecommerce, it's much better to show an image since the text description for a product can be misleading, ambiguous, or biasing.)
A If you add images to this, please keep in mind that some may administering tests in a couple of languages, so images should not contain text (or should be available in a couple of languages) --- sorry, us Canadian government folk have to do things in Engliish and French!
Q: With the instruction " I smell garlic bread" this influenced my search because I thought it was an instore bakery item, was this what you are intended? Does this point to th eimportance of the instruction phrasing.

28. Setting up a Treejack project
Creating a Treejack project
Entering your tree
Entering the tasks and answers
Less on mechanics, more on tips

29. Creating a project New vs. Duplicate Survey name vs. address
Identification
The “Other” option
Passing an argument in the URL

30. Entering your tree Paste from Excel, Word, text file, etc.
“Top” – how to replace
Randomising
Not the same as randomising tasks
Changing the tree after entering answers
Lesson learned:
Edit/review/finalise the tree elsewhere before putting it into Treejack

31. Entering tasks and answers
Preview is surprisingly useful
Multiple correct answers
The “main” answer is usually not enough
Check the entire tree yourself
Must choose bottom-level topics
Workaround: Mark all subtopics correct
Workaround: Remove the subtopics
Choose answers LAST

32. Task options Randomising tasks – almost always Limiting the # of tasks
20-30 tasks = 10 per participant
Increase the # of participants to get enough results per task
Skip limit
Eliminate users who didn’t really try
Defaults to 50%
Q: does it 'randomize' or 'rotate' the order of the tasks (i.e. will the tasks be evenly distributed across the sample of participants)?

33. Testing the test Not previewing/piloting is just plain dumb
Spot mistakes before launch
Preview the entire test yourself
Pilot it with stakeholders and sample users
Launch it, get feedback, duplicate, revise
Look for:
Task wording (unclear, ambiguous, typos)
Unexpected “correct” answers
Misc. problems (e.g. instructions)

34. Poll How many participants do you get per test? 1 – 20 = 44%
21 – 40 = 20%
41 – 100 = 24%
Over 100 = 12%

35. Running the tree test Invite participants
Website-page invitations
invitations
Recommend >30 users per user group/test
Monitor early results for problems
low # of surveys started
invitation not clear? Subject = spam? Not engaging?
low completion rate
didn’t set expectations? Test too long? Too hard?
Generally less taxing than card sorting
Q: Is there any benefit in testing 100 people rather than 5-7? Traditional user testing doesn't need statistical significance in sample size, so what benefit does this add?

36. Skimming high-level results
10/100/1000 level of detail
Middling overall score
Often many highs with a few lows
Inspect tasks with low scores (low total or low sub-scores)
Inspect the pie charts
Show sample results
Q: Is there any way to export these high-level views into Excel as well? They can be useful in presentations when reporting back results.

37. Success % who chose a correct answer (directly or indirectly)
low Success score
check the spreadsheet to see where they went wrong
Destinations tab
Path tab

38. Directness % of successful users who did not backtrack
Coming soon: making this independent of success
low Directness score
check the spreadsheet for patterns in their wandering
Paths tab

39. Speed
% who completed this task at about the same speed as their other tasks
% who completed task within 2 standard deviations of their average task time for all tasks
70% Speed score
7/10 users went their “normal” speed
3/10 users took substantially longer than normal for them
Low Speed score
indicates that user hesitated when making choices
e.g. choices are not clear or not mutually distinguishable
Wish: add the raw times to the spreadsheet, so you can do your own crunching as needed.
Overall score uses a grid to combine these scores in a semi-intelligent fashion

40. Detailed results – destinations
Where did people end up?
# who chose a given topic as the answer
Wrong answers
High totals - problem with that topic (perhaps in relation to its siblings)
Clusters of totals – problem with the parent level
Ignore outliers
For >30 sessions, ignore topics that get <3 clicks.
Show sample results from food shopping – Destinations tab

41. Detailed results – destinations
Look for high “indirect success” rates (>20%)
Check paths for patterns of wandering
Look for high “failure” rates (>25%)
Check the wrong answers above
Look for high skip rates (> 10%)
Check paths for where they bailed out.
Look for "evil attractors"
Topics that get clicks across several seemingly unrelated tasks.
Usually a vague term that needs tightening up
Q: Need to split out the values: How many skipped before even clicking a level; how many quit after truly despairing?
Evil attractors - We saw this happen to a consumer-review site that had a “Personal” category; they meant personal-care products like electric shavers, but participants also went there for “personal” items like cell phones, watches, etc.

42. Detailed results – first clicks
Where they went on their first click
Important for task success
Which sections they visited overall
Did they visit the right section but back out?
Show sample results from food shopping – First Click tab
The first click is the one most highly correlated with eventual success – if we can get users to the right section of a site, they’re much more likely to find the right topic.
Perhaps the participants completely ignored the correct section, or maybe they visited it, but backed out and went somewhere else instead.

43. Detailed results – paths
Click-by-click paths that they took through the tree
Useful when asking:
How the heck did they get way over there?
Did a lot of them take the same detour?
No web UI for removing participants.
Support and we’ll fix you up.
Show sample results from food shopping – Paths tab

44. Some lessons learned Test new against old
Revise and test again – quick cycles
Test a few alternatives at the same time
Cover the sections according to their importance
Analysis is easier than for card sorting
Use in-person testing to get the “why”
Paper is still effective (and free!) for this
Tree testing is only part of your IA work
New vs. old - we promised a client we would deliver a better IA, not just a different one? Tree testing proved to be a great way to demonstrate this. In our baseline test, the original structure notched a 31% success rate. Using the same tasks, the new structure scored 67% - a solid quantitative improvement.
Q: Can you do A/B testing in the tool?
Q: We've used it in face-to-face sessions as we get other data too. We've never run it as remote-only as we're nervous of losing the rich insights. Any experiences or thoughts?
Tree testing not enough - A good tree still needs to be married up to an effective navigation system, content design, and visual treatment before we can say we’ve built a good information architecture.

45. What’s coming Better scoring for Directness, Speed
Improved results (10/100/1000)
General enhancements across Treejack, OptimalSort, and Chalkmark
Whatever you yell loudest for… 
GetSatisfaction lets you “vote” for issues

46. Tree testing – more resources
Boxes & Arrows article on tree testing
Donna Spencer’s article on paper tree testing
Treejack website Webinars, slides, articles, user forum

47. Getting your input Specific issues/questions Feature requests
Feature requests
Check the support forum (GetSatisfaction)
“Feedback” button
Show GetSatisfaction

48. Thanks!