Presentation is loading. Please wait.

Presentation is loading. Please wait.

Guided Content Optimization for Gears of War 3 Michael Noland Epic Games

Similar presentations

Presentation on theme: "Guided Content Optimization for Gears of War 3 Michael Noland Epic Games"— Presentation transcript:


2 Guided Content Optimization for Gears of War 3 Michael Noland Epic Games



5 Overview Our iterative process and some tips/tricks Memory feedback loop Performance feedback loop Texture streaming improvements Wrapup


7 Information flow Capture and process data Generate action items Fix issues and retrigger captures Keep iterating – Streamline ID / capturing process for QA – Change information captured as new areas of interest are identified

8 Current process Daily memory/perf standup ~ Weekly reports on all maps Quick turnaround on maps in flux Periodic analysis of other captures – Continuous play-thru runs – MP soak tests – Cooked data report mining – Always loaded content packages

9 Implementation tips/tricks Classify dark memory – Ideally traceable all the way back to a placed actor in the editor or a defined subsystem Play thru maps on target as early as possible – Use artificial memory savers if necessary reducepoolsize, nosound Report adjusted numbers to keep your eyes on the goal – Establishes an ongoing baseline BugItGo – Quick way to express a game location

10 Other data sources Always loaded content pulled in by script – Periodically scrape; savings here help everything Automatic cooked data reports – Top 10 lists showing large content items – Per level audio and animation breakdowns – ‘Content on-disc’ tags fed back to editor Quickly narrow down which version is referenced – Fresh data available every day


12 Memory feedback loop 1.QA plays thru individual levels capturing data 2.QA runs a tool to process the raw data into CSVs 3.QA generates and sends out a hotspot report showing the overall trend and specific dips 4.Content or code makes changes and requests an updated report from QA

13 1. Capturing raw data memleakcheck – Logs out: – Loaded objects – Streamed in levels – Various other stats / metadata – Can be done fast or full domemleakchecking 30 – Does a fast memleakcheck every 30 seconds

14 1. Capturing raw data (cont.) – Reboot console – Turn on memleakchecking and on-screen level streaming report – Load a level and play thru it – Start playing the next level until all of the test level finishes streaming out

15 2. Parsing raw data MemLeakCheckDiffer – Extracts information from unstructured.memlk – Loads a grouping script that defines how to bucket objects together and in what order to report them – Two click processing for QA – Generates.CSV reports

16 3. Example Hotspot Graph

17 3. Hotspot Report Lists lowest memory dips, with context – Loaded streaming levels – Commonly important bucket sizes Static meshes Skeletal meshes Animation data Sound data – BugItGo location

18 4. Action Items Sent to a fairly broad list: – Level LDs / artists, lead LD – Strike team (memory/perf programmers and QA) – Producers and game team If dips are not well explained by an increase in LD buckets, someone will investigate further – Generated CSVs list all loaded assets – If still not accounted for, it’s dark memory that needs to be fixed

19 4. Content Fixes Loaded levels list is often enough – Cinematic levels left loaded too long – Levels not streamed in/out at the right time – Expensive assets in a geographically large submap Transitions between maps can be problematic Can look at a list of loaded assets – Merge/replace unique assets with library items – Stop non-visible assets from being pulled in – Actual cuts of noticeable content are fairly rare

20 4. Code issues May end up being a code problem or leak – Track genuine memory leaks with Malloc Profiler – Grow-only caches and pools are more common Characterizing the expected amount is important – Round-trip thru an exhaustive level and back to the menu Can factor into desired memory bar and ignore Reset / purge when possible but plan for worst case

21 4. Code improvements Assets that are larger than expected – Re-evaluate your assumptions – A new game means new usage patterns Some real world examples – Static mesh collision data – Blocking volume brushes – Improving animation compression – Non-uniform light environment samples

22 Balancing Act Always consider memory savings in concert with performance Trading off memory for speed (in either direction) only helps if you have room in the other

23 Lessons from the past Went back to Gears 2 and retroactively pulled some stats from it to guide us thru Gears 3 Full memlk runs for all SP maps – Gives approximate budgets, per-map slack Used as approximations, not hard budgets – New code optimizations give us more bang / MB – New features and fixed costs take away some too

24 It’s working!


26 Performance Feedback Loop Gather appropriate profiles – Run once to identify problematic areas and then perform the appropriate profile captures – stat unitmax shows the time broken down into: Render thread Game thread GPU time Generate action items

27 Performance Feedback Loop – Capturing data Capture on same run if the problem is steady-state Otherwise, reload a checkpoint or start the level over – Files generated depend on the bottleneck Render thread or game thread – Native CPU trace – Stats capture Game thread – Gameplay Profiler GPU – Native GPU trace (PIX)

28 Game thread Associated profiles – Gameplay Profiler captures – Stats captures – Native CPU traces Attack strategy – Depends on the actual bottleneck Animation, Line checks, Pathfinding, etc… – Turn off expensive / low impact features – Optimize game code – Cut or modify triggering content

29 Stats Viewer Counters or timers Lots already in place Easy to add new or temporary ones

30 Stats Viewer Stats are hierarchical Relatively low overhead to record

31 Gameplay Profiler Allows seeing script and native game code in context; great for tracking down hitches

32 Render thread Associated profiles – Native CPU trace – Stats Capture Most commonly an issue in split screen Attack strategies – Reduce draw calls Turn off bad shadow casters Merge sections – Code fixes for unexpected cases

33 GPU time PIX – Works like magic with a solid marker setup – We mark each drawcall with: Material name Mesh / Skelmesh name Actor name Hierarchy of rendering – Prepass, Shadows, Opaque, Transparent, etc… – Enabling shader PDBs


35 Texture Streaming Eliminating texture streaming artifacts Two main kinds: – Insufficient streaming time Enough memory but not enough time to fill it (level loads / transitions) – Totally over budget Content uses too many unique textures

36 Texture streaming stats Desired texture memory is updated constantly Log out stats via memlk or ustats to look for over budget cases QA also bugs locations with noticeable blurriness or mip popping

37 Isolating streaming issues BugItGo to the bad spot on console or editor Texture stats view shows which textures are loaded and how many mips – Can be connected to a remote console Streaming system has a command to dump info on a particular texture – Lists everything it knows about a texture Which objects reference it Desired mip level calculation


39 What’s next Continuing to combat dark memory Automatic capture file management – DB upload – Associated metadata Automated reporting after content checkins Empowering users to look at lower level data

40 Empowering users: Memory Analyzer

41 Conclusion Gather the information as early as possible Filter it down into actionable information Tighten the feedback loop Keep iterating

42 Any questions?


44 References Many of the tools mentioned are documented in more detail on UDN Hotspot reporting process UE3 performance and memory hub

Download ppt "Guided Content Optimization for Gears of War 3 Michael Noland Epic Games"

Similar presentations

Ads by Google