1. The Tech Behind the Tools of Insomniac Games
Geoff Evans – Insomniac Games

2. Welcome!
An overview of how Insomniac’s tools get current generation games out the door
There are no slick demos of our game engine
This talk is focused tools internals, pipelines, and best practices we have found

3. Tools used to ship Ratchet and Clank Future: Tools of Destruction were a Sea change in how the company organizes its game assets
That game truly earned the name TOOLS OF DESTRUCTION 
To understand why we made the choices we did, one must get a primer on what was wrong with what we had

4. Outline Areas of improvement: Revision control Asset organization
Flexible Asset definition
Efficient Asset Processing

5. Warning: Open Source Ahead
Nocturnal Initiative is Insomniac's Open Source project for tools code & more
Portions available for download under BSD-style license, long term initiative
Wiki:
Perforce:
nocturnal.insomniacgames.com:1666

6. Problem Proprietary revision control system Implicit file retrieval
Files synced before open
Per-file database overhead
Written files queued for check-in at process exit
Heavy code instrumentation (iopen, iclose)
Damage spreads immediately
All processed data under revision control
Based around live assets, emulated shared drive
Data errors could stop everyone’s work
Phantom bugs would crop up and disappear before programming team could fix them
We have shipped games with computed data in revision control that we could not easily regenerate!
Proprietary database was hard to support
Downtime killed productivity

7. Solution Move all revision control to Perforce
Handles branching of binaries efficiently
Label code and data together
Explicit asset retrieval
Users decide when to download
Big speed improvement
Better stability while getting a unit of work done
Remove built data from revision control

8. Branching We branch all files for milestones
Work continues in the main line during milestone stabilization
Milestone fixes brought back piecemeal
Milestone hacks stay in branch forever

9. Continuous Integration
Custom system implemented in Perl
Populates a ‘Known Good’ label in Perforce
Failure notifies users based on recent checkins (changes between testing and last known good changelist numbers)
Moving forward we intend to sync to ‘Known Good’ by default (code and assets)
Code and data validated separately but labeled together

10. Problem Problem: Asset tree organized by engine type
Tools and engine expected numbered assets
Moby 500: Wooden Breakable Crate
Level 9000: My Test Level
Storage location of each asset was fixed or had a natural home in the tree:
/Data/Mobys/Moby500/actor.xml (fixed location)
/Art/Mobys/Moby500/crate.mb (natural home)

11. Problem Numbered assets are simple to program for
Reference any asset using type and id
Lacked descriptive names, hard to find, hindered re-use of assets
Users tracked assets with pen and paper
Changing the type of an asset lead to complex copy logic that had to copy files around

12. Solution Tools shouldn’t force asset organization
Users should be able to put files anywhere
Assets are defined in files with a reasonable names (BreakableCrate.entity vs. moby500)
Files should be able to be moved or renamed without a lot of work
Renaming a single file is simple, but references to that file are troublesome
We wanted to make a system where content organization is completely independent of engine requirements

13. Asset Referencing Historically references are file path strings
Makes renaming content files difficult and error prone
Asset A references B, renaming B to C breaks A
Housecleaning before a sequel game causes pain due to broken references
Typically fixed up by brute force

14. Asset Referencing Don’t rely on string paths for referencing files
Use a unique IDs to indirect the file location
IDs are assigned and shared via events
Event files are retrieved with asset files
Event files contain simple event logic:
File <foo> was added with id <blah>
File <foo> was moved to <bar>
File <bar> was deleted

15. Event System
Each event is written to a file per-user, per-computer, per-project, per-branch
“geoff-geoff_xp-rcf-devel.event.dat”
Avoid contention for file checkout
Avoid conflict resolution during integrate
EventSystem collates events, sorts by time
Events handled only once

16. File Resolver File Resolver maps unique ID -> File path
Consumes events from Event System
Populates a local machine SQLite DB
Native formats store only the unique ID
Maya stores ID in dynamic attributes
String paths are fixed up at load time by plug-in
TUID – 64 bit unique ID
Open Source via Nocturnal

17. Asset Referencing
Processed data is output to folders named for the file id of the Asset definition file
Even though assets are renamed, processed data location remains the same
Helps efficiency on the back end of pipeline
It’s possible to reference assets in game by the file ID by hand, but we try to avoid this

18. Here be Dragons
Since databases are built on each computer, it can have tricky machine-specific bugs (revert)
We commit Add and Rename operations immediately to fight missing asset references (dangling TUIDs), data files may be valid stubs
Moving forward we think its possible to store resolver data as meta-data and eliminate collation of Events altogether

19. Problem
Problem: Assets defined by engine type specific data structures
Our assets were defined in terms of our engine, not in terms of the content our engine was consuming
This made changing the engine type arduous since the data could be very different

20. Solution
Asset data should be modular, and that modularity should drive the engine type
Asset definition should serve the content, not specific engine types (wherever possible)
Similar data about Assets should be reusable between engine types

21. Asset Fundamentals
What is an Asset: A discrete unit of content that needs to be processed by one or more builders to be loaded by the game
An Asset itself has only basic properties
An Asset is a collection of Attributes
Asset : public AttributeCollection

22. Assets (C++ Classes) AttributeCollection Asset EntityAsset LevelAsset
ShaderAsset

23. Attributes Attribute: A block of related information
AnimationAttribute: what animations to use
PhysicsAttribute: physics subsystem info
VisualAttribute: render information
Attributes store the bulk of the complexity
Stored in ‘slots’ using class type id (int32)

24. Attributes (C++ Classes)
AnimationAttribute
PhysicsAttribute
VisualAttribute

25. Asset Example EntityAsset – content backed placeable object
ArtFileAttribute – reference Maya content file
VisualAttribute – render information
EntityAsset
Art File
Visual

26. Asset Example LevelAsset – top level chunk of game data
WorldFileAttribute – placed objects in the level
DependenciesAttribute – list of assets to pack
LevelAsset
World File
Dependencies

27. Asset Example ShaderAsset – texture and render properties
ColorMapAttribute – texture, compress settings
NormalMapAttribute – processed differently
TextureMap
ColorMap
NormalMap
ShaderAsset
ColorMap
NormalMap

28. Attributes Simplifies sharing structures between Assets
Attributes only exist for features that are configured by content creators (auditing)
Easy to copy and paste between Assets
Prior examples are Exclusive attributes

29. Validating Attributes
If we aren’t careful users could easily create non-sensical Assets
LevelAsset with a TextureMapAttribute?
Shader with a PhysicsAttribute?
Must be able to validate Attributes
Attribute may not work with some Asset classes
Attributes may not work in conjunction
Prior examples are Exclusive attributes

30. Attribute Behavior Behavior defines default compatibility
Exclusive: ColorMapAttribute
Validated by ShaderAsset, not by LevelAsset
Inclusive: DependenciesAttribute
References to miscellaneous resources
Useful for spawn-only types like weapons, HUD
Loosely related assets like weapons and ammo
Prior examples are Exclusive attributes

31. Attribute Usage Usage implies Class/Instance relationship
Class – Used in classes only
Instance – Used in instances only
Overridable – Works in both class and instance
Allows us to easily override class settings per instance on an Attribute level

32. Attribute Usage
Our SceneNode class also subclasses AttributeCollection:
AttributeCollection
Asset
EntityAsset
SceneNode
Entity

33. Attribute Usage
BakedLightingAttribute defines light map dimension and compression settings
Instances of that EntityAsset can override the light map information (if its scaled way up or down)
EntityAsset
BakedLightingAttribute
Default light map settings
Entity
BakedLightingAttribute
Override light map settings

34. Attribute Sibling Validation
Some Attributes inherently conflict
Attributes can reject potentially invalid siblings
Ex: BakedLightingAttribute will reject the addition of PhysicsAttribute because baked lighting doesn’t work with dynamic objects

35. Engine Type Classification
Assets map to one or more Engine Types
If Asset map to one type, it’s a natural match
ShaderAsset -> ShaderBuilder
LevelAsset -> LevelBuilder
If Asset class maps to more than one
Attributes determine Engine Type
EntityAsset -> ShrubBuilder, TieBuilder, etc…

36. Entity Assets Shrub Shrub – light weight static art
ArtFileAttribute – Maya content file
VisualAttribute – render information
EntityAsset
Art File
Visual
Shrub

37. Entity Assets Tie Tie – more expensive static art
CollisionAttribute – makes it collidable
BakedLightingAttribute – lightmap size
EntityAsset
Art File
Draw
Collision
Baked Lighting
Tie

38. Entity Assets Moby Moby – animated character
PhysicsAttribute – Physics subsystem settings
AnimationAttribute – Animation assets to use
EntityAsset
Art File
Draw
Physics
Animation
Moby

39. Too many Attributes! Plethora of Attributes is confusing to artists
Keeping Attributes simple is key
Attributes are cumbersome because knowing which Attributes yield each Engine Type its costly to learn

40. Just make it a <Engine Type>!
Sometimes users know what Engine Type they want, and need the ability to conform the Attributes to that Engine Type
Attributes have Enable flag which can be used to disable offending Attributes so data is not lost, just dormant
Changing the type back re-enables the disabled Attributes

41. What Attributes do I need again?
Setting up Assets from scratch can still be non-trivial… basic Attributes are needed to get an Asset up and running
We created a Wizard with simple presets for each Engine type
Moving forward we are going to select creation preset based on export data contents

42. Problem
Problem: .NET was not very productive for generating UI for editing our Assets
We don’t want to give up writing Native code, so .NET in general was not a good fit for us
Marshalling data structures between Native and Managed was time consuming
Windows::Forms::PropertyGrid was hard to deal with to get good custom property editing

43. Solution Drop .NET and develop our own solutions
Our C++ reflection was getting feature-rich
We already had a data-driven properties GUI
Switched to C++ / wxWidgets
Back up and running after 1 man month
Simplified everything

44. Asset Editor UI
Assets and Attributes are loaded into a single tree view
References to other Assets are expanded in-place
Properties displayed for each node in the tree
Property interaction can update information in real-time on the target platform

45. Asset Editor UI

46. What’s under the hood
We generate property GUI for Asset/Attributes from C++ reflection information
Adding a field to an Asset or Attribute shows up automatically
UI-script can add sliders, drop-downs, color pickers, etc…

47. Reflect Provides class and field level meta-data
All meta-data created at runtime
No compile time parsing required
Adding a field is usually a single line of code
Type Registry assigns type id’s at runtime
Automated persistence, cloning, visitor API
Handles data format changes & legacy files
Open Source via Nocturnal

48. Reflect Classes
class Version : public Reflect::ConcreteInheritor<Version, Element> { public: std::string m_Source; static void EnumerateClass( Reflect::Compositor<Version>& comp ) comp.AddField( &Version::m_Source, "m_Source" ); } }; void Init() Reflect::RegisterClass<Version>( "Version" );
Basic cost of entry is a single function, but more information can be added via the pointer from the result of AddField()
User friendly name, extensible properties (map<string,string>)
The data we really need:
Type, offset, and size

49. Reflect Enumerations
namespace BlendTypes { enum BlendType { Normal, Additive }; static void EnumerateBlendType ( Reflect::Enumeration* info ) info->AddElement(Normal, "Normal"); info->AddElement(Additive, "Additive"); } } typedef BlendTypes::BlendType BlendType; void Init() Reflect::RegisterEnumeration<BlendType>( "BlendType”, &BlendTypes::EnumerateBlendType );
Enum and bitfield serialization done by string
Safe to add/remove elements

50. Inspect System for data-driven property GUIs Control class hierarchy
Wrapper classes for wxWidgets UI elements
Can be factory created from UI script
Interpreter class hierarchy
Intimate with data formats
Automates the creation of the UI
Handles callbacks from user interactions
Open Source via Nocturnal

51. ReflectInspect Interpreter classes that generate Inspect UI
Binds control state to in memory structures
Basic support for STL containers + Math types
Support for custom UI handling via plug-ins
Easy to integrate custom property elements
Open Source via Nocturnal

52. Holy crap that’s a lot of Properties!
This system is awesome for drowning our users in lots of properties
Agility is good, not the most important thing
Cull properties that are used very infrequently
Adding support for filtering properties
Adding support for categorizing properties
Adding support for dependencies to hide other properties when not applicable

53. Problem Users spend a lot of time waiting for Assets to build
No longer getting built data from revision control
Building major animated characters took 15+ min
Building levels took 30+ min

54. Solution Build system must share data between users
Information about an output file is hashed into a single signature for the output:
Hash of input file paths, format version strings
Hash of input file contents or folder contents
Hash of dependency relationships
Hash of output file paths, format version strings

55. Cache Files
This signature used to build a file path to a output cache file on the network
That file is downloaded instead of built
Locally built data is uploaded automatically when the build phase completes
Downloading updates the modification time of the cache file on the network
Script trims down cache based modification time

56. Format Version Format version is a string compiled into tools
Changed a programmer changes a data format
“15 – geoff: improved shader flossiness”
Very efficient because users only reprocess data when data formats or assets change
Takes discipline when checking in changes to libraries that affect lots of output formats

57. Build Process Given list of assets to build
Create builder objects for assets
Builders compute output signatures
Attempt download of output files from cache
Build what could not be downloaded
Upload built files to cache

58. Cache Files Started with Distributed Cache System (DCS)
Query for files on other workstations around the office, central server for tracking files
DCS was complex, wasn’t well suited
Moved to a Central Cache System (CCS)
Now we have files sharing throughput issues with lots of concurrent users
Windows 2008 / Linux are possible solutions

59. Cache Files Cache file storage is just an optimization
Any built data can be regenerated at any point
VPN users have separate cache for their site if they have multiple machines
Continuous iteration system is fantastic for seeding build cache files

60. Summary Major wins for our tools: Perforce File Resolver
Assets & Attributes
Built File Cache

61. Thanks!