The Tech Behind the Tools of Insomniac Games Geoff Evans – Insomniac Games

Welcome! An overview of how Insomniac’s tools get current generation games out the door There are no slick demos of our game engine This talk is focused tools internals, pipelines, and best practices we have found

Tools used to ship Ratchet and Clank Future: Tools of Destruction were a Sea change in how the company organizes its game assets That game truly earned the name TOOLS OF DESTRUCTION  To understand why we made the choices we did, one must get a primer on what was wrong with what we had

Outline Areas of improvement: Revision control Asset organization Flexible Asset definition Efficient Asset Processing

Warning: Open Source Ahead Nocturnal Initiative is Insomniac's Open Source project for tools code & more Portions available for download under BSD-style license, long term initiative Wiki: http://nocturnal.insomniacgames.com Perforce: nocturnal.insomniacgames.com:1666

Problem Proprietary revision control system Implicit file retrieval Files synced before open Per-file database overhead Written files queued for check-in at process exit Heavy code instrumentation (iopen, iclose) Damage spreads immediately All processed data under revision control Based around live assets, emulated shared drive Data errors could stop everyone’s work Phantom bugs would crop up and disappear before programming team could fix them We have shipped games with computed data in revision control that we could not easily regenerate! Proprietary database was hard to support Downtime killed productivity

Solution Move all revision control to Perforce Handles branching of binaries efficiently Label code and data together Explicit asset retrieval Users decide when to download Big speed improvement Better stability while getting a unit of work done Remove built data from revision control

Branching We branch all files for milestones Work continues in the main line during milestone stabilization Milestone fixes brought back piecemeal Milestone hacks stay in branch forever

Continuous Integration Custom system implemented in Perl Populates a ‘Known Good’ label in Perforce Failure notifies users based on recent checkins (changes between testing and last known good changelist numbers) Moving forward we intend to sync to ‘Known Good’ by default (code and assets) Code and data validated separately but labeled together

Problem Problem: Asset tree organized by engine type Tools and engine expected numbered assets Moby 500: Wooden Breakable Crate Level 9000: My Test Level Storage location of each asset was fixed or had a natural home in the tree: /Data/Mobys/Moby500/actor.xml (fixed location) /Art/Mobys/Moby500/crate.mb (natural home)

Problem Numbered assets are simple to program for Reference any asset using type and id Lacked descriptive names, hard to find, hindered re-use of assets Users tracked assets with pen and paper Changing the type of an asset lead to complex copy logic that had to copy files around

Solution Tools shouldn’t force asset organization Users should be able to put files anywhere Assets are defined in files with a reasonable names (BreakableCrate.entity vs. moby500) Files should be able to be moved or renamed without a lot of work Renaming a single file is simple, but references to that file are troublesome We wanted to make a system where content organization is completely independent of engine requirements

Asset Referencing Historically references are file path strings Makes renaming content files difficult and error prone Asset A references B, renaming B to C breaks A Housecleaning before a sequel game causes pain due to broken references Typically fixed up by brute force

Asset Referencing Don’t rely on string paths for referencing files Use a unique IDs to indirect the file location IDs are assigned and shared via events Event files are retrieved with asset files Event files contain simple event logic: File <foo> was added with id <blah> File <foo> was moved to <bar> File <bar> was deleted

Event System Each event is written to a file per-user, per-computer, per-project, per-branch “geoff-geoff_xp-rcf-devel.event.dat” Avoid contention for file checkout Avoid conflict resolution during integrate EventSystem collates events, sorts by time Events handled only once

File Resolver File Resolver maps unique ID -> File path Consumes events from Event System Populates a local machine SQLite DB Native formats store only the unique ID Maya stores ID in dynamic attributes String paths are fixed up at load time by plug-in TUID – 64 bit unique ID Open Source via Nocturnal

Asset Referencing Processed data is output to folders named for the file id of the Asset definition file Even though assets are renamed, processed data location remains the same Helps efficiency on the back end of pipeline It’s possible to reference assets in game by the file ID by hand, but we try to avoid this

Here be Dragons Since databases are built on each computer, it can have tricky machine-specific bugs (revert) We commit Add and Rename operations immediately to fight missing asset references (dangling TUIDs), data files may be valid stubs Moving forward we think its possible to store resolver data as meta-data and eliminate collation of Events altogether

Problem Problem: Assets defined by engine type specific data structures Our assets were defined in terms of our engine, not in terms of the content our engine was consuming This made changing the engine type arduous since the data could be very different

Solution Asset data should be modular, and that modularity should drive the engine type Asset definition should serve the content, not specific engine types (wherever possible) Similar data about Assets should be reusable between engine types

Asset Fundamentals What is an Asset: A discrete unit of content that needs to be processed by one or more builders to be loaded by the game An Asset itself has only basic properties An Asset is a collection of Attributes Asset : public AttributeCollection

Assets (C++ Classes) AttributeCollection Asset EntityAsset LevelAsset ShaderAsset

Attributes Attribute: A block of related information AnimationAttribute: what animations to use PhysicsAttribute: physics subsystem info VisualAttribute: render information Attributes store the bulk of the complexity Stored in ‘slots’ using class type id (int32)

Attributes (C++ Classes) AnimationAttribute PhysicsAttribute VisualAttribute

Asset Example EntityAsset – content backed placeable object ArtFileAttribute – reference Maya content file VisualAttribute – render information EntityAsset Art File Visual

Asset Example LevelAsset – top level chunk of game data WorldFileAttribute – placed objects in the level DependenciesAttribute – list of assets to pack LevelAsset World File Dependencies

Asset Example ShaderAsset – texture and render properties ColorMapAttribute – texture, compress settings NormalMapAttribute – processed differently TextureMap ColorMap NormalMap ShaderAsset ColorMap NormalMap

Attributes Simplifies sharing structures between Assets Attributes only exist for features that are configured by content creators (auditing) Easy to copy and paste between Assets Prior examples are Exclusive attributes

Validating Attributes If we aren’t careful users could easily create non-sensical Assets LevelAsset with a TextureMapAttribute? Shader with a PhysicsAttribute? Must be able to validate Attributes Attribute may not work with some Asset classes Attributes may not work in conjunction Prior examples are Exclusive attributes

Attribute Behavior Behavior defines default compatibility Exclusive: ColorMapAttribute Validated by ShaderAsset, not by LevelAsset Inclusive: DependenciesAttribute References to miscellaneous resources Useful for spawn-only types like weapons, HUD Loosely related assets like weapons and ammo Prior examples are Exclusive attributes

Attribute Usage Usage implies Class/Instance relationship Class – Used in classes only Instance – Used in instances only Overridable – Works in both class and instance Allows us to easily override class settings per instance on an Attribute level

Attribute Usage Our SceneNode class also subclasses AttributeCollection: AttributeCollection Asset EntityAsset SceneNode Entity

Attribute Usage BakedLightingAttribute defines light map dimension and compression settings Instances of that EntityAsset can override the light map information (if its scaled way up or down) EntityAsset BakedLightingAttribute Default light map settings Entity BakedLightingAttribute Override light map settings

Attribute Sibling Validation Some Attributes inherently conflict Attributes can reject potentially invalid siblings Ex: BakedLightingAttribute will reject the addition of PhysicsAttribute because baked lighting doesn’t work with dynamic objects

Engine Type Classification Assets map to one or more Engine Types If Asset map to one type, it’s a natural match ShaderAsset -> ShaderBuilder LevelAsset -> LevelBuilder If Asset class maps to more than one Attributes determine Engine Type EntityAsset -> ShrubBuilder, TieBuilder, etc…

Entity Assets Shrub Shrub – light weight static art ArtFileAttribute – Maya content file VisualAttribute – render information EntityAsset Art File Visual Shrub

Entity Assets Tie Tie – more expensive static art CollisionAttribute – makes it collidable BakedLightingAttribute – lightmap size EntityAsset Art File Draw Collision Baked Lighting Tie

Entity Assets Moby Moby – animated character PhysicsAttribute – Physics subsystem settings AnimationAttribute – Animation assets to use EntityAsset Art File Draw Physics Animation Moby

Too many Attributes! Plethora of Attributes is confusing to artists Keeping Attributes simple is key Attributes are cumbersome because knowing which Attributes yield each Engine Type its costly to learn

Just make it a <Engine Type>! Sometimes users know what Engine Type they want, and need the ability to conform the Attributes to that Engine Type Attributes have Enable flag which can be used to disable offending Attributes so data is not lost, just dormant Changing the type back re-enables the disabled Attributes

What Attributes do I need again? Setting up Assets from scratch can still be non-trivial… basic Attributes are needed to get an Asset up and running We created a Wizard with simple presets for each Engine type Moving forward we are going to select creation preset based on export data contents

Problem Problem: .NET was not very productive for generating UI for editing our Assets We don’t want to give up writing Native code, so .NET in general was not a good fit for us Marshalling data structures between Native and Managed was time consuming Windows::Forms::PropertyGrid was hard to deal with to get good custom property editing

Solution Drop .NET and develop our own solutions Our C++ reflection was getting feature-rich We already had a data-driven properties GUI Switched to C++ / wxWidgets Back up and running after 1 man month Simplified everything

Asset Editor UI Assets and Attributes are loaded into a single tree view References to other Assets are expanded in-place Properties displayed for each node in the tree Property interaction can update information in real-time on the target platform

Asset Editor UI

What’s under the hood We generate property GUI for Asset/Attributes from C++ reflection information Adding a field to an Asset or Attribute shows up automatically UI-script can add sliders, drop-downs, color pickers, etc…

Reflect Provides class and field level meta-data All meta-data created at runtime No compile time parsing required Adding a field is usually a single line of code Type Registry assigns type id’s at runtime Automated persistence, cloning, visitor API Handles data format changes & legacy files Open Source via Nocturnal

Reflect Classes class Version : public Reflect::ConcreteInheritor<Version, Element> { public: std::string m_Source; static void EnumerateClass( Reflect::Compositor<Version>& comp ) comp.AddField( &Version::m_Source, "m_Source" ); } }; void Init() Reflect::RegisterClass<Version>( "Version" ); Basic cost of entry is a single function, but more information can be added via the pointer from the result of AddField() User friendly name, extensible properties (map<string,string>) The data we really need: Type, offset, and size

Reflect Enumerations namespace BlendTypes { enum BlendType { Normal, Additive }; static void EnumerateBlendType ( Reflect::Enumeration* info ) info->AddElement(Normal, "Normal"); info->AddElement(Additive, "Additive"); } } typedef BlendTypes::BlendType BlendType; void Init() Reflect::RegisterEnumeration<BlendType>( "BlendType”, &BlendTypes::EnumerateBlendType ); Enum and bitfield serialization done by string Safe to add/remove elements

Inspect System for data-driven property GUIs Control class hierarchy Wrapper classes for wxWidgets UI elements Can be factory created from UI script Interpreter class hierarchy Intimate with data formats Automates the creation of the UI Handles callbacks from user interactions Open Source via Nocturnal

ReflectInspect Interpreter classes that generate Inspect UI Binds control state to in memory structures Basic support for STL containers + Math types Support for custom UI handling via plug-ins Easy to integrate custom property elements Open Source via Nocturnal

Holy crap that’s a lot of Properties! This system is awesome for drowning our users in lots of properties Agility is good, not the most important thing Cull properties that are used very infrequently Adding support for filtering properties Adding support for categorizing properties Adding support for dependencies to hide other properties when not applicable

Problem Users spend a lot of time waiting for Assets to build No longer getting built data from revision control Building major animated characters took 15+ min Building levels took 30+ min

Solution Build system must share data between users Information about an output file is hashed into a single signature for the output: Hash of input file paths, format version strings Hash of input file contents or folder contents Hash of dependency relationships Hash of output file paths, format version strings

Cache Files This signature used to build a file path to a output cache file on the network That file is downloaded instead of built Locally built data is uploaded automatically when the build phase completes Downloading updates the modification time of the cache file on the network Script trims down cache based modification time

Format Version Format version is a string compiled into tools Changed a programmer changes a data format “15 – geoff: improved shader flossiness” Very efficient because users only reprocess data when data formats or assets change Takes discipline when checking in changes to libraries that affect lots of output formats

Build Process Given list of assets to build Create builder objects for assets Builders compute output signatures Attempt download of output files from cache Build what could not be downloaded Upload built files to cache

Cache Files Started with Distributed Cache System (DCS) Query for files on other workstations around the office, central server for tracking files DCS was complex, wasn’t well suited Moved to a Central Cache System (CCS) Now we have files sharing throughput issues with lots of concurrent users Windows 2008 / Linux are possible solutions

Cache Files Cache file storage is just an optimization Any built data can be regenerated at any point VPN users have separate cache for their site if they have multiple machines Continuous iteration system is fantastic for seeding build cache files

Summary Major wins for our tools: Perforce File Resolver Assets & Attributes Built File Cache

Thanks! geoff@insomniacgames.com