1. Rapid Iterative Testing and Evaluation RITE
Dennis Wixon
Michael Medlock
Melissa Federoff
I manage the Games User Research Group at Microsoft
Michael is the inventor of RITE and is currently the chief usability for Internal HR tools
Melissa user researcher in MGS and works with the FASA studio – Mechassault

2. Copyright - Microsoft Corporation
Outline of Talk
What’s wrong
What’s RITE
Creation of the RITE method
Running a RITE study
Conclusions
Q/A
Dennis What’s wrong with the past research on usability methods
Dennis And what is an alternative to that approach
Michael will give you an overview of the development of the RITE method –
We’ve been running RITE studies for years at Microsoft –
Melissa developed our toolkit to systematize them and will talk about a case study
Dennis Conclusions
2/23/2019
Copyright - Microsoft Corporation

3. Editorial: Research on Methods
While I admire the research that was done – I think that as practitioners working in a business they have led us in the wrong direction
It compared methods according to which one was the most efficient at finding problems – which at best is only half the problem.
2/23/2019
Copyright - Microsoft Corporation

4. Copyright - Microsoft Corporation
Why
It focuses on the failures not on how can we improve products.
It also failed as research untimely end with a critique by Gray and Saltzman – Damaged merchandise
2/23/2019
Copyright - Microsoft Corporation

5. Copyright - Microsoft Corporation
What’s wrong
Purpose: the goal of a method is to find problems
Cost: determine the minimal number of participants
Evaluation: we should use traditional research methods to determine the “best” method
Assumption: there is a best method
Goal is to get problems or to improve the interface as much as possible
It’s good to keep costs low – but it’s better to look at efficiency – how much effect do you have for investment. It’s part of an unfortunate trend ot discount methods – which looks a user research as a cost
Gray and Salzman argue for an experimental approach to evaluate methods – not likely in the real world – it’s too expensive and there are two many factors to evaluat e
Methods have purposes and a method needs to be evaluated against its purpose – benchmark methods establish baselines, usability engineering done properly sets the right context for tradeoffs
What is the ultimate goal of iterative testing?
2/23/2019
Copyright - Microsoft Corporation

6. Copyright - Microsoft Corporation
What’s right?
Purpose of an iterative test
Rapid improvements
Create a collaborative environment
Measure cost and benefit:
Percentage of problems addressed
Percentage of effective fixes
Evaluation: use case studies
Match method to purpose
Position user research as a business asset
As many improvements as possible – as quickly as possible (calendar time) at the lowest cost (person hours)
Collaborative environment sets you up for the next test
How would we evaluate a method –
How many did we try to address given the number we found
How many worked – RITE allows you to test fixes and determine that they work – it reduces your risk
Case studies – look a software development of real products and follow them through to conclusion – did they sell, how expensive was support, look at a total cost of development –
Match method to purpose – covered
We want to develop methodology that makes user research a tactical and strategic advantage – no business function ever became truly important by emphasizing how little they cost
2/23/2019
Copyright - Microsoft Corporation

7. What is a Rapid Iterative Test and Evaluation test (RITE)?
Just like a “standard” usability test except:
Changes made as soon as a problem is identified and a potential solution is clear
The changed interface is tested with subsequent users
This is the Essence of RITE
Many people did this but did not want to admit it – Teams want to fix problems instead of telling them not to harness that energy and get them to do a good job at it
2/23/2019
Copyright - Microsoft Corporation

8. Copyright - Microsoft Corporation
“Standard” vs. RITE
Notice – that we make changes we can
Find out if the changes worked
Get further into the interface to make more changes
Is it a real problem
You don’t decide that by running more pariticipants – you would do that if you didn’t know anything users or the design
Instead you ask the following questions as a group
‘What were the problems
Do we agree on the cause
Can we agree on and do a fix
If no then keep testing – for cases you don’t understand it becomes a standard usability test
2/23/2019
Copyright - Microsoft Corporation

9. What is the business case for RITE?
Return on investment
Faster detection of problems
More issues found
More issues fixed
Fixes are verified
Better team dynamics
The method degrades gracefully
What has been our experience?
Teams love this – because they want the best product
Degrades to a standard test
2/23/2019
Copyright - Microsoft Corporation

10. Case Study: Age of Empires II
What is Age of Empires II (AOE II)
Historically themed war game
Progress through “Ages”: dark, feudal, castle, imperial Civilizations have different capabilities
Develop technologies
Rock/paper/scissors strategy
Real Time Strategy game (RTS)
Chess = turn based strategy game
Complex divided attention task, build an economy & an armed force
Defeat opponents with good combination of both
Original case study for the RITE method
Case studies are a good way to evaluate methods in a business context
Real world case study
Focus on outcome –Melissa will focus on process
2/23/2019
Copyright - Microsoft Corporation

11. Copyright - Microsoft Corporation
Interactive Tutorial
Age of Empires (AOE) was a successful game – over 1 million users
AOE II should capture new users
Make it easy to learn without reducing the complexity which makes the game fun
The decision was made to create a tutorial
Decided to bring in novice RTS players
Used the RITE method to iterate
2/23/2019
Copyright - Microsoft Corporation

12. Record of errors & failures over time
Changes made after 1 participant
Failures and errors decrease over time
Things had to be “re-fixed”
Success likelihood can be calculated
How to read graph
X axis = 16 participants total, all novice RTS interested in some other game
Y axis = Failures/Errors
Define failure and error
Vertical lines indicate where tutorial was iterated
There were 0 failures for the last 6 participants, there was 1 error on one thing (meaning 0 for most things)
2/23/2019
Copyright - Microsoft Corporation

13. Change made after 1 participant
In AOE II users need to gather resources (wood, food, gold and stone).
1st participant was instructed to gather wood by tutorial, but there were no trees in view.
Issue and solution were clear
Place a tree within view & teach users to explore off-screen
Issue never reappeared with final 15 participants
2/23/2019
Copyright - Microsoft Corporation

14. Likelihood that problems were fixed
Final 6 participants with no previously fixed issue reoccurring (one issue that turned up never received a fix).
90% confident that between 61 – 100% likely to have fixed issues addressed (binary confidence interval).
Many issues had many more participants and a higher fix probability.
2/23/2019
Copyright - Microsoft Corporation

15. Copyright - Microsoft Corporation
Re-fix example
Re-fix = solution that did not work
In AOE II users need to make militia at the “Barracks” building.
Train vs. Create, first 6 of 8 participants
Text inconsistency confused participants –sometimes “train”, sometimes “create”
Conceptually confusing as well –participants thought villagers would become militia
Fixed text inconsistency by changing all labels to “train”
Conceptual issue still remained after next 2 participants, changed all labels to “create”.
2/23/2019
Copyright - Microsoft Corporation

16. AOE II: Reviews, Awards, Sales
AOE II’s tutorial received “Excellence” award from Society of Technical Communication in
AOE II generated excellent reviews, many focusing on the tutorial.
Aug Oct 2000 AOE II never left the top 10 in games sales.
AOE I sold 456,779 units between Oct 97 – Dec 98.
AOE II sold almost double in similar time frame (916,812 copies between Oct 99 – Dec 00).
2/23/2019
Copyright - Microsoft Corporation

17. Copyright - Microsoft Corporation
Getting it RITE
Key components to running a successful RITE study:
UE familiar with product and target user
Product architected for iteration
Scheduling allows time for debriefing & iteration
Resources available to make changes
Decision makers present in the lab
Documentation of changes and fixes
Can’t run a RITE test cold. Need to be able to help develop good solutions to problems.
Prototype tools that are easy to change on the fly; database driven values rather than hard coding
Too many participants in one day can actually decrease from the productivity and value of the test
Dev/design need to be available to identify fixes and do them. Scope decided upon, areas assigned.
Team should be in the lab. Best if decision makers and product fixes. Minimum is decision maker(s).
Documenting is hard to do, easy to drag your feet. Great to have to show impact – great for product team to see progress.
2/23/2019
Copyright - Microsoft Corporation

18. Copyright - Microsoft Corporation
Scheduling RITE
Here’s what a typical RITE can look like. What you should note:
You are always analyzing the data – either after each participant or at the end of each day
Fixes are constantly happening
Some fixes will get verified, some won’t – in standard usability none of them would be
A day is left for fixes between each day of participants on this template, BUT time for iteration and number of participants each day is up to you, based on:
Resources available for changes
Amount of time required for changes
When team can be with you in the lab
Whether you want to analyze & debrief after each participant or after each day
2/23/2019
Copyright - Microsoft Corporation

19. Copyright - Microsoft Corporation
Documenting RITE
Key:
X = issue observed with this participant
Yellow highlight = 1st time fix attempted
Blue highlight = 2nd time fix attempted
Heavy vertical lines = denote each time product changed
Black shading = Participant did not get to task where issue could have been experience
This tool can be used to document study in real time.
Hardest part is getting devs to keep track of each fix and when it was made.
Walk through: The small table below displays a lot of information quickly. When you skim it, here's what you can quickly discern: There were 6 issues total and 10 participants including 1 pilot. The build was iterated 3 times and 4 fixes were made. The first two issues had one attempted fix each that did not resolve the problems found. The third and fourth issues had no attempted fix. The fifth issue had an attempted fix after P1 that did not work (changing only the text), but the severe problem was resolved after a second fix attempt (adding a button and changing location of option) after P2. The sixth issue appeared for the first time with P1 and no fix was attempted and P8 didn't engage in a task where they could have encountered this issue.
2/23/2019
Copyright - Microsoft Corporation

20. Copyright - Microsoft Corporation
Conclusions
RITE WORKS
Problems get fixed
Fixes get verified
Progress can be tracked
Teams feel a sense of accomplishment
RITE has variants; e.g. Oracle example
Oracle – used rite combined with X
2/23/2019
Copyright - Microsoft Corporation

21. Copyright - Microsoft Corporation
Cautions
RITE has limits
Requires team to be present
Will not
Discover a new product concept
Help you understand how users work in the real world
Generate product requirements
Prioritize usability issues
2/23/2019
Copyright - Microsoft Corporation

22. Conclusions about Methods
The basics
Study methods in context
Be clear about the goals of the method
Report method effectiveness as case studies
The future
Refine the case study approach
Institutionalize methods that have business impact
Build a new discourse community around methodology and design
2/23/2019
Copyright - Microsoft Corporation

23. Copyright - Microsoft Corporation
How do we see the future?
With HOPE
2/23/2019
Copyright - Microsoft Corporation