1. How data rule the world Kristoffer Benjaminsson CTO

2. Easy? Easy is a studio within EA Games that do free to play games Originates from DICE 3 live titles – Battlefield Heroes – Battleforge – Lord of Ultima

3. Trailer

4. Telemetry - Wikipedia ” Telemetry (synonymous with telematics) is a technology that allows remote measurement and reporting of information. The word is derived from Greek roots tele = remote, and metron = measure.”

5. Telemetry - Easy Data that helps us make decisions to provide a better service for our players Provides answers to specific questions Multiple sources – Financial data – Game statistics – Web behaviour data – Game telemetry Vertical slice of the player population

6. Why bother? We prefer to know what is going on instead of guessing We can measure results from actions Data do not (usually) lie Forums not representative – Less than 20% read forum posts – Less than 5% post in forums

7. Game telemetry examples Latency data Punkbuster kicks Matchmaking Time to level Hardware profiles Play time

8. Architecture Game Client Game Server Magma DB Reporting

9. Implementation TelemetryManager sends data to our backend for database storage Hooks in the game code collects data Hooks are records packed as name value pairs Each record will create its own table in the database All telemtry includes a common identifer so we can cross reference data between telemetry hooks

10. Example record class PingRecord : public ITelemetryRecord { public: PingRecord(const std::string& clientIp, const std::string& serverIp, int ping, const std::string& dataCenter, const std::string& community); public: virtual bool hasType( int typeId ) const { return typeId == RTPing; } virtual bool assimilate( const ITelemetryRecord* other ) { return false; } virtual IDataCollector::DataRef buildData() const; private: std::string m_clientIp; std::string m_serverIp; int m_ping; std::string m_dataCenter; std::string m_community; };

11. Example usage player->addTelemetry(new backend::telemetry::PingRecord(clientIp, serverAddressAndPort, conn->getAveragePing(), serverSettings->getProdDataCenter(), serverSettings->getServerCommunity()));

12. Real life examples

13. High latency - before

14. Why were people in Europe being matchmade to Australia and North America?

15. High latency - after

16. How did we do it? Generated an HTML page – GeoIP City – A bit of C# – Google Maps API – Telemetry data – A bit of SQL

17. Example page function initialize() { var latlng = new google.maps.LatLng(45.5735931396484, 12.1060943603516); var myOptions = { zoom: 2, center: latlng, mapTypeId: google.maps.MapTypeId.ROADMAP }; var map = new google.maps.Map(document.getElementById("map_canvas"), myOptions); var marker = null; var coords = null; var path = null; marker = new google.maps.Marker({ position: new google.maps.LatLng(59.433, 24.728), map: map, title:"gva, 62.65.243.167", icon: "client.png" }); marker = new google.maps.Marker({ position: new google.maps.LatLng(52.350, 4.916), map: map, title:"gva", icon: "server.png" }); coords = [new google.maps.LatLng(59.433, 24.7283), new google.maps.LatLng(52.350, 4.916)]; path = new google.maps.Polyline({ path: coords, strokeColor: "#ff0000", strokeOpcaity: 0.5, strokeWeight: 1}); path.setMap(map); }

18. Google Maps

19. Punkbuster - before

20. Punkbuster - after

21. How did we do it? Microsoft SQL Server 2008: Reporting Services – Standard BI tool – Easy to create charts and tables – Updates on a daily basis

22. During development

23. Asserts We collect all asserts from both client and server We differentiate between content asserts and code asserts We have leaderboards and summaries – Top 10 asserts – Total content/code asserts per branch Provides guidance to what we need to fix

24. Fragalyzer Level design helper tool Collects information about how a map is played – Kill position – Killed from position – Player/Vehicle movement – Etc Currently collected locally on a game server

25. Summary Telemetry is awesome! Collect data to answer specific queries Build a flexible system to make it super easy to add new telemetry hooks Chose the right reporting tools to mine and report on the data

26. Questions?