1. Wake up and smell the metrics! Metrics Driven Development in online games Larry Mellon Darius Kazemi AGC ‘08

2. Data mining is pure gold! What stops people from reaching it?

3. Cultural collision is our Smaug! Boxed goods mentality Online service reality Wrong assumptions lead to painful decisions!

4. Larry Mellon ( Consultant) Alberta Research Council & Jade Simulations – Distributed computing, 1982+ – Optimistic computing, 1000+ CPU virtual world – Fault-tolerant cluster computing Synthetic Theatre of War: virtual worlds for training – DARPA: 50,000+ entities in real-time virtual worlds – ADS, ASTT, HLA & RTI 2.0, interest management EA (Maxis): The Sims Online, The Sims 2.0 Scalable server architecture Automated testing to accelerate production and QA Player, pipeline & performance metrics Emergent Game Technologies (CTO) Architect for scalable, flexible MMO platform Research era Wife era

5. Darius Kazemi Orbus Gameworks darius@orbusgameworks.com

6. Boxed goods industry is evolving online and needs to adapt to service industry requirements!

7. Why is it hard to get funding for something as clearly useful as metrics? Nobody wants to pay for it, because no one has traditionally paid for it! FeaturesQAMetrics, CS, … $$$$$$$$$$ $$ 

8. Diasporas kill off the home-rolled systems

9. Business Intelligence has driven the success of many other industries for years! Las Vegas Strip

10. Metrics help manage complexity “When you can measure what you are speaking about and can express it in numbers, you know something about it. But when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind." - Lord Kelvin Institution of Civil Engineers, 1883 “The general who wins the battle makes many calculations in his temple before the battle is fought. The general who loses makes but few calculations beforehand.” -- Sun Tzu

11. The people who led the scientific revolution depended on metrics! Pasteur: A science is as mature as its measurement tools. Maxwell: To measure is to know.

12. “The three largest factors that will influence gaming will be […] and metrics (measuring what players do and responding to that)” -- Will Wright The Secret of The Sims", PC Magazine, 2002. http://www.pcmag.com/article2/0,1759,482309,00.asp

13. – GIGO – Avoid false causality by correlating data!

14. Multiple views of data provides a deeper understanding and fewer analysis errors Time AI data Player and game actions Minute 1 1.AI: open door 2.AI: cook food Minute 2 1.Game: fire breaks out Screenshots Minute one Minute two

15. 3 – Final road block: Metrics are role-specific and hard to move across boundaries

16. The “3P's” model of game metrics Player Performance Process

17. 17 Next Gen Games Increased Complexity Increased Complexity of Analysis Art from “Fun Meters for Games”, Nicole Lazzaro and Larry Mellon

18. 18

19. 19 Next Gen Games

20. Tuning imbalances or exploits can throw your entire economy out of kilter, but remember to triangulate!

22. Know Your Players

23. Comparing groups is another no-brainer…

24. Process metrics Find the leaks that are slowing you down or costing you money! Another cultural problem – Process = evil – Tools != game feature Not ‘fun’ to build No ‘status’ – Thus, worst possible programmers are assigned to most critical systems…

25. Fixing development leaks is like adding free staff!

26. Process: Measurable targets & projected trends give you actionable progress metrics, early enough to react Any metric (e.g. # clients) Target Time Any Time (e.g. Alpha) First sample Now Risk!

27. Real data gives you faster focus, less debate! Question for the audience: If you have 5 engineers in the room, how many opinions do you have on what the team needs to do next? Numbers cut the debate from “what is next?” to “how do we fix the thing we found?” You QUICKLY go from finding the key data and fixing the problem, not searching the entire data space, just to get started!

28. Performance: engine metrics (Dev & Ops) Debugging & optimizing large-scale, complex system – With hackers, non-determinism & constant change – Tick-tock!! Users (and VPs) are waiting… Most engineering metrics tools are for one person, one machine, one debugger – Need system-level view to deal with MMOs Studio head metric: only team-wide, red-ink email – GET THIS METRICS SYSTEM IN THE LIVE SERVERS, BY TOMORROW (tired of ‘black box debugging’ across 20 servers and 200 clients)

29. Lower dev & ops costs: FREE MONEY! Profit… New Content Regression Customer Support Operations ~ $10 per customer

30. Profit… Regression Customer Support Operations ~ $10 per customer Lower New Content Cost

31. Profit… Customer Support Operations ~ $10 per customer Lower New Content Cost Lower Testing Cost

32. Profit… Operations ~ $10 per customer Lower New Content Cost Lower Testing Cost Happy Customers Don’t Call

33. Process (Dev & Ops): Lower recurring costs that are worth the cost to lower [makes you more money with the same number of players!] Profit… Operations ~ $10 per customer Lower New Content Cost Lower Testing Cost Happy Customers Don’t Call Lower bandwidth & CPU

34. Process: Rapid Iteration is how you find fun! Project Start Launch

35. Implementation options (both are hard) 1.Grassroots movement & scale as required 2.DEFINE metrics as a fully funded feature needed for success in an online service!

36. 1.a How to start your own Mini metrics project Image by Flickr user RaeA; Creative Commons (BY-NC-SA)

37. 1.b How to scale your own Mini metrics project Image by Flickr user Dredrk; Creative Commons (BY-NDW)

38. Culture Shift option: Treat metrics as a critical feature from day one! Fund everything that helps both team and customers, not just game play! FeaturesQAMetrics $$$$$$$$$$ $$$$ $$!!!

39. Metrics are your first-pass tool : they quickly find you a starting point that would take hours/days for you to find by yourself

40. Find oddities automatically for you, then you decide what action to take

41. Scaling the metrics system as data scales Tip: make your system self- maintaining, and critical to programmers and/or management (they’ll fix it the moment it breaks )

42. Train ‘bots to look for patterns of data when the datasets get even larger

43. Takeaways Grassroots or formal feature : either way, get started with metrics across your entire game service Find something that will help an obviously important question to be answered with facts rather than guesses To get executive and funding support, translate your day-to-day need for metrics into the metrics the executives understand to justify the funding You will either make more cash with this change! Or you’ll save more cash with this change! Presentation and aggregation are key: use the computer to do the hard parts People should see high-level data, with easy search and correlate functionality, and access to the raw data

44. The migration online is a Darwinian moment for our industry Boxed goods mentalities must shift to online service realities, or die… Complexity: people need metrics to be able to do their jobs Player Retention: It’s not just about game features anymore!

45. Question: How would you rather live your life? OR MeasureChangeMeasure GuessChangeHope Slides are online at http://www.maggotranch.com/biblio.htmlhttp://www.maggotranch.com/biblio.html www.orbusgameworks.com/blog