//Building scalable web sites with tight game integration Johan Mjönes & Joakim Bodin

What is battlelog? Battlefield 10 year anniversary soon Battlelog becoming the community hangout Integrate game launching for PC players

Features Friend-centric Easy communication Current stats and history Activity log PC Menu News / Forums

ESN PLANET

ESN PLANET OVERVIEW

Languages Developing Battlelog means developing in Python, Java, C++ The Battlelog Web is written primarily using Python Plugin, game components in C++ Python allows for rapid web development (e.g. remote access console)

Horizontal (sharding) Battlelog User DBs (slices and shards)... Vertical Battlelog UserLab Feature Different processes Partitioning

Persistence & Index MySQL No joins Used as indexed KVS Apache Solr Full text search for forums Other uses in the future Fast!

Server to client push Uses ESN Beaconpush Allows delivery of messages from web server to client browser. Uses long polling or websockets (or flash) AV / anti-malware software + push = Blocks all websockets Injects Javascript into the DOM Etc...

Caching with Memcached Great framework support Service methods cached via annotations Invalidation using method signature instead of key Allows populating cache directly after modification Dog pile prevention

EA Services Nucleus Users No user info in Battlelog DB! Success! Personas Soldiers Entitlements Licenses Blaze Game Servers reports to Blaze Events to Battlelog Game Server information Asynchronous responses (e.g. matchmaking) REST Stats

Nucleus Details Nucleus Accounts Entitlements Web Backend Web Soldiers Licenses HTTP

Blaze Details Uses Web Access Layer (WAL) WAL client generated from TDF TDF is a API definition language Blaze events (XML over HTTP) Blaze DB Web BackendWeb Frontend Blaze Game Servers WAL Real-time Events Thrift Events AJAX Web Browser

Friends Started out normalized Unmanagable amount of rows Ended up as one blob per user Packed user ids A lot easier to cache properly

Server Browser Custom search server (Java) Custom query language, minimal message overhead Fast update, fast search

Joining a game

Joining a game: plugin details Web Browser Game Client Plugin Game UI JS Pipe

Battle Reports Game report from Blaze (HTTP XML event) Parse and divide per player One report per player via internal message queue Look for unlocks, rankups, medals, awards etc Add to feed, send real time updates Invalidate player stats caches Compile Battlereport (& pre-cache it) Send notification to involved (logged in) users

Background Load tested using Locust Open Source (MIT License) Built by ESN (& others) Battlelog tested with 2.7 million PSU with 36 million players Thats 40,000 requests per second The tests was successful

Pre-pass Planet dev bar Timing Queries No point in load testing if there are apparent issues with a single user

Realistic load testing Some testing tools simply hammer certain URLs Locust allows for realistic usage scenarios Many players go away for 20 minutes while playing, receiving notifications Updated tests using actual usage data from alpha trial / open beta Coordinated testing Same user spans Overpopulate tables to simulate fragmented MySQL indices

What did we find? Bugs Bottlenecks Never assume where your bottlenecks are Test to find out Network related Saturated network, not CPU Keep the traffic within the rack! Software related Best practices be damned, cheat if neccessary Lazy loading data in controllers Client side rendering Surface + History API + JSON =

Being a part of the pipeline Data driven games Battlelog has its own pipeline Written in Python Imports data from the game pipeline Weapons, vehicles, levels, dog tags, etc Assets include images, text, data Resolving unlocks and dependencies Generates python structs and javascript structs Generates many helper maps. Some might only be used once. DLC friendly! Result of many iterations

User Interface Separate UI from consoles / in game menu Having the same wouldnt make sense, different UX Adapt to different devices Web features Easily accessible, e.g. Sharing Extendability with browser extensions, user styles etc Access on any platform (play on console, have Battlelog on iPad) Easy to iterate over UI changes

Battlelog will evolve Agile: Easy to update Big things: DLC Public APIs Other things we cant talk about Small things Improvements based on community feedback

Johan Joakim The Battlelog team is hiring! Technical Director Development Director Backend Frontend