5 April 2005 Is ROI an Effective Approach for Persuading Decisionmakers on the Value of User-Centered Design? Organizer: Susan Dray, Dray and Associates, USA Clare-Marie Karat, IBM T. J. Watson Research Center, USA, Daniel Rosenberg, Oracle Corporation, USA, David Siegel, Dray and Associates, USA, Dennis Wixon, Microsoft, USA Tuesday, 16:30-18:00, Hall B This panel examines the utility and effectiveness of various ways of making the business case for user-centered design (UCD). Most of the discussion in our field has assumed that measuring and demonstrating ROI for usability is the key to this effort. However, experience shows that the most brilliant ROI analysis may not win the day in the real world of business. Our panelists range from people who claim that ROI is an important persuasive tool as long as the communication about ROI is happening within a healthy business relationship, to people who claim that a focus on ROI can actually be destructive. We also explore the idea that there are important business contexts where ROI simply does not fit. Through the presentations by the panelists and through discussion of a business case scenario, we explore some alternatives to ROI in making the business case for user-centered design.

Strategic ROI – A commercial Dennis Wixon Manager Games User Research Microsoft

5 April 2005 Building a team with strategic ROI Games User Research: Group started in 1997 with one contractorGroup started in 1997 with one contractor Now 2 nd largest user research team at MicrosoftNow 2 nd largest user research team at Microsoft 23 employees, 3 contingent staff, and 10 moderators.23 employees, 3 contingent staff, and 10 moderators. ScopeScope Last year 10,794 playtest participantsLast year 10,794 playtest participants 56 Microsoft and non-Microsoft games.56 Microsoft and non-Microsoft games. Over 1200 usability participantsOver 1200 usability participants Since 1998, over 1250 reports on gamers’ experiences with gamesSince 1998, over 1250 reports on gamers’ experiences with games Largest consumer research effort by any game developer or publisher for the purposes of improving games.Largest consumer research effort by any game developer or publisher for the purposes of improving games.

5 April 2005 Tenets of strategic ROI  Understand the business context  Meet management needs  Embed methods in case studies  Partner with creative power (design)

5 April 2005 Big opportunity: Movie box office was $8.4 billionMovie box office was $8.4 billion $9.4 billion spent on games$9.4 billion spent on games Tripled in 7 yearsTripled in 7 years Game consoles in 30-40% of U.S. householdsGame consoles in 30-40% of U.S. households Strategic ROI: Games business context

5 April 2005 Strategic ROI: business context (contd) The business context:The business context: Prolific: 600+ game products in N. AmericaProlific: 600+ game products in N. America “Hit” driven: rule“Hit” driven: rule  High cost of development  Success of platform depends on appeal of software Halo 2:Halo 2: Over $100 million in sales on day 1 (Shrek 2: 75 million in sales first weekend)Over $100 million in sales on day 1 (Shrek 2: 75 million in sales first weekend) At last count 6 million soldAt last count 6 million sold

5 April 2005 Strategic ROI: management need-response  Improve games quickly and effectively - RITE method  Measure fun - playtest assessment  Help sign up partners: playtest and usability labs and case studies

5 April 2005 Strategic ROI: RITE

5 April 2005 Strategic ROI: RITE  More issues fixed  Fixes are verified  More issues found  Truly collaborative effort – usability & design “This was a fantastic experience. Whenever we get asked "how do you like working with Microsoft," two of the first things out of our mouths are "the people there are AWESOME" and "usability ROCKS!"

5 April 2005 Strategic ROI: playtest lab

5 April 2005 Strategic ROI: playtest lab

5 April 2005 Strategic ROI example: Halo 1 Won “Oscar” for Game of the YearWon “Oscar” for Game of the Year 58% “attach rate” at launch58% “attach rate” at launch Designer quote “Without a doubt it is one of the most valuable resources in MGS [Microsoft Games Studios] and can be a critical factor in the success of the games we publish. (I only wish UT [User-testing] had greater bandwidth so we could use you more frequently!)”Designer quote “Without a doubt it is one of the most valuable resources in MGS [Microsoft Games Studios] and can be a critical factor in the success of the games we publish. (I only wish UT [User-testing] had greater bandwidth so we could use you more frequently!)”

5 April 2005 Strategic ROI: Acknowledgements  Playtest:  Bill Fulton  Eric Schuh  RITE:  Michael Medlock  Melissa Federoff  Dan Gunn  Halo 1 and 2:  Keith Steury  Randy Pagulayan  Jamie Griesemer – Bungie

5 April 2005 Tenets of strategic ROI  Understand the business context  Meet management needs  Embed methods in case studies  Partner with creative power (design)

5 April 2005 Apply Strategic ROI: you can win too /