Presentation is loading. Please wait.

Presentation is loading. Please wait.

Rapid Iterative Testing and Evaluation RITE

Similar presentations


Presentation on theme: "Rapid Iterative Testing and Evaluation RITE"— Presentation transcript:

1 Rapid Iterative Testing and Evaluation RITE
Dennis Wixon Michael Medlock Melissa Federoff I manage the Games User Research Group at Microsoft Michael is the inventor of RITE and is currently the chief usability for Internal HR tools Melissa user researcher in MGS and works with the FASA studio – Mechassault

2 Copyright - Microsoft Corporation
Outline of Talk What’s wrong What’s RITE Creation of the RITE method Running a RITE study Conclusions Q/A Dennis What’s wrong with the past research on usability methods Dennis And what is an alternative to that approach Michael will give you an overview of the development of the RITE method – We’ve been running RITE studies for years at Microsoft – Melissa developed our toolkit to systematize them and will talk about a case study Dennis Conclusions 2/23/2019 Copyright - Microsoft Corporation

3 Editorial: Research on Methods
While I admire the research that was done – I think that as practitioners working in a business they have led us in the wrong direction It compared methods according to which one was the most efficient at finding problems – which at best is only half the problem. 2/23/2019 Copyright - Microsoft Corporation

4 Copyright - Microsoft Corporation
Why It focuses on the failures not on how can we improve products. It also failed as research untimely end with a critique by Gray and Saltzman – Damaged merchandise 2/23/2019 Copyright - Microsoft Corporation

5 Copyright - Microsoft Corporation
What’s wrong Purpose: the goal of a method is to find problems Cost: determine the minimal number of participants Evaluation: we should use traditional research methods to determine the “best” method Assumption: there is a best method Goal is to get problems or to improve the interface as much as possible It’s good to keep costs low – but it’s better to look at efficiency – how much effect do you have for investment. It’s part of an unfortunate trend ot discount methods – which looks a user research as a cost Gray and Salzman argue for an experimental approach to evaluate methods – not likely in the real world – it’s too expensive and there are two many factors to evaluat e Methods have purposes and a method needs to be evaluated against its purpose – benchmark methods establish baselines, usability engineering done properly sets the right context for tradeoffs What is the ultimate goal of iterative testing? 2/23/2019 Copyright - Microsoft Corporation

6 Copyright - Microsoft Corporation
What’s right? Purpose of an iterative test Rapid improvements Create a collaborative environment Measure cost and benefit: Percentage of problems addressed Percentage of effective fixes Evaluation: use case studies Match method to purpose Position user research as a business asset As many improvements as possible – as quickly as possible (calendar time) at the lowest cost (person hours) Collaborative environment sets you up for the next test How would we evaluate a method – How many did we try to address given the number we found How many worked – RITE allows you to test fixes and determine that they work – it reduces your risk Case studies – look a software development of real products and follow them through to conclusion – did they sell, how expensive was support, look at a total cost of development – Match method to purpose – covered We want to develop methodology that makes user research a tactical and strategic advantage – no business function ever became truly important by emphasizing how little they cost 2/23/2019 Copyright - Microsoft Corporation

7 What is a Rapid Iterative Test and Evaluation test (RITE)?
Just like a “standard” usability test except: Changes made as soon as a problem is identified and a potential solution is clear The changed interface is tested with subsequent users This is the Essence of RITE Many people did this but did not want to admit it – Teams want to fix problems instead of telling them not to harness that energy and get them to do a good job at it 2/23/2019 Copyright - Microsoft Corporation

8 Copyright - Microsoft Corporation
“Standard” vs. RITE Notice – that we make changes we can Find out if the changes worked Get further into the interface to make more changes Is it a real problem You don’t decide that by running more pariticipants – you would do that if you didn’t know anything users or the design Instead you ask the following questions as a group ‘What were the problems Do we agree on the cause Can we agree on and do a fix If no then keep testing – for cases you don’t understand it becomes a standard usability test 2/23/2019 Copyright - Microsoft Corporation

9 What is the business case for RITE?
Return on investment Faster detection of problems More issues found More issues fixed Fixes are verified Better team dynamics The method degrades gracefully What has been our experience? Teams love this – because they want the best product Degrades to a standard test 2/23/2019 Copyright - Microsoft Corporation

10 Case Study: Age of Empires II
What is Age of Empires II (AOE II) Historically themed war game Progress through “Ages”: dark, feudal, castle, imperial Civilizations have different capabilities Develop technologies Rock/paper/scissors strategy Real Time Strategy game (RTS) Chess = turn based strategy game Complex divided attention task, build an economy & an armed force Defeat opponents with good combination of both Original case study for the RITE method Case studies are a good way to evaluate methods in a business context Real world case study Focus on outcome –Melissa will focus on process 2/23/2019 Copyright - Microsoft Corporation

11 Copyright - Microsoft Corporation
Interactive Tutorial Age of Empires (AOE) was a successful game – over 1 million users AOE II should capture new users Make it easy to learn without reducing the complexity which makes the game fun The decision was made to create a tutorial Decided to bring in novice RTS players Used the RITE method to iterate 2/23/2019 Copyright - Microsoft Corporation

12 Record of errors & failures over time
Changes made after 1 participant Failures and errors decrease over time Things had to be “re-fixed” Success likelihood can be calculated How to read graph X axis = 16 participants total, all novice RTS interested in some other game Y axis = Failures/Errors Define failure and error Vertical lines indicate where tutorial was iterated There were 0 failures for the last 6 participants, there was 1 error on one thing (meaning 0 for most things) 2/23/2019 Copyright - Microsoft Corporation

13 Change made after 1 participant
In AOE II users need to gather resources (wood, food, gold and stone). 1st participant was instructed to gather wood by tutorial, but there were no trees in view. Issue and solution were clear Place a tree within view & teach users to explore off-screen Issue never reappeared with final 15 participants 2/23/2019 Copyright - Microsoft Corporation

14 Likelihood that problems were fixed
Final 6 participants with no previously fixed issue reoccurring (one issue that turned up never received a fix). 90% confident that between 61 – 100% likely to have fixed issues addressed (binary confidence interval). Many issues had many more participants and a higher fix probability. 2/23/2019 Copyright - Microsoft Corporation

15 Copyright - Microsoft Corporation
Re-fix example Re-fix = solution that did not work In AOE II users need to make militia at the “Barracks” building. Train vs. Create, first 6 of 8 participants Text inconsistency confused participants –sometimes “train”, sometimes “create” Conceptually confusing as well –participants thought villagers would become militia Fixed text inconsistency by changing all labels to “train” Conceptual issue still remained after next 2 participants, changed all labels to “create”. 2/23/2019 Copyright - Microsoft Corporation

16 AOE II: Reviews, Awards, Sales
AOE II’s tutorial received “Excellence” award from Society of Technical Communication in AOE II generated excellent reviews, many focusing on the tutorial. Aug Oct 2000 AOE II never left the top 10 in games sales. AOE I sold 456,779 units between Oct 97 – Dec 98. AOE II sold almost double in similar time frame (916,812 copies between Oct 99 – Dec 00). 2/23/2019 Copyright - Microsoft Corporation

17 Copyright - Microsoft Corporation
Getting it RITE Key components to running a successful RITE study: UE familiar with product and target user Product architected for iteration Scheduling allows time for debriefing & iteration Resources available to make changes Decision makers present in the lab Documentation of changes and fixes Can’t run a RITE test cold. Need to be able to help develop good solutions to problems. Prototype tools that are easy to change on the fly; database driven values rather than hard coding Too many participants in one day can actually decrease from the productivity and value of the test Dev/design need to be available to identify fixes and do them. Scope decided upon, areas assigned. Team should be in the lab. Best if decision makers and product fixes. Minimum is decision maker(s). Documenting is hard to do, easy to drag your feet. Great to have to show impact – great for product team to see progress. 2/23/2019 Copyright - Microsoft Corporation

18 Copyright - Microsoft Corporation
Scheduling RITE Here’s what a typical RITE can look like. What you should note: You are always analyzing the data – either after each participant or at the end of each day Fixes are constantly happening Some fixes will get verified, some won’t – in standard usability none of them would be A day is left for fixes between each day of participants on this template, BUT time for iteration and number of participants each day is up to you, based on: Resources available for changes Amount of time required for changes When team can be with you in the lab Whether you want to analyze & debrief after each participant or after each day 2/23/2019 Copyright - Microsoft Corporation

19 Copyright - Microsoft Corporation
Documenting RITE Key: X = issue observed with this participant Yellow highlight = 1st time fix attempted Blue highlight = 2nd time fix attempted Heavy vertical lines = denote each time product changed Black shading = Participant did not get to task where issue could have been experience This tool can be used to document study in real time. Hardest part is getting devs to keep track of each fix and when it was made. Walk through: The small table below displays a lot of information quickly. When you skim it, here's what you can quickly discern: There were 6 issues total and 10 participants including 1 pilot. The build was iterated 3 times and 4 fixes were made. The first two issues had one attempted fix each that did not resolve the problems found. The third and fourth issues had no attempted fix. The fifth issue had an attempted fix after P1 that did not work (changing only the text), but the severe problem was resolved after a second fix attempt (adding a button and changing location of option) after P2. The sixth issue appeared for the first time with P1 and no fix was attempted and P8 didn't engage in a task where they could have encountered this issue. 2/23/2019 Copyright - Microsoft Corporation

20 Copyright - Microsoft Corporation
Conclusions RITE WORKS Problems get fixed Fixes get verified Progress can be tracked Teams feel a sense of accomplishment RITE has variants; e.g. Oracle example Oracle – used rite combined with X 2/23/2019 Copyright - Microsoft Corporation

21 Copyright - Microsoft Corporation
Cautions RITE has limits Requires team to be present Will not Discover a new product concept Help you understand how users work in the real world Generate product requirements Prioritize usability issues 2/23/2019 Copyright - Microsoft Corporation

22 Conclusions about Methods
The basics Study methods in context Be clear about the goals of the method Report method effectiveness as case studies The future Refine the case study approach Institutionalize methods that have business impact Build a new discourse community around methodology and design 2/23/2019 Copyright - Microsoft Corporation

23 Copyright - Microsoft Corporation
How do we see the future? With HOPE 2/23/2019 Copyright - Microsoft Corporation


Download ppt "Rapid Iterative Testing and Evaluation RITE"

Similar presentations


Ads by Google