Presentation is loading. Please wait.

Presentation is loading. Please wait.

The Tech Behind the Tools of Insomniac Games

Similar presentations

Presentation on theme: "The Tech Behind the Tools of Insomniac Games"— Presentation transcript:

1 The Tech Behind the Tools of Insomniac Games
Geoff Evans – Insomniac Games

2 Welcome! An overview of how Insomniac’s tools get current generation games out the door There are no slick demos of our game engine This talk is focused tools internals, pipelines, and best practices we have found

3 Tools used to ship Ratchet and Clank Future: Tools of Destruction were a Sea change in how the company organizes its game assets That game truly earned the name TOOLS OF DESTRUCTION  To understand why we made the choices we did, one must get a primer on what was wrong with what we had

4 Outline Areas of improvement: Revision control Asset organization
Flexible Asset definition Efficient Asset Processing

5 Warning: Open Source Ahead
Nocturnal Initiative is Insomniac's Open Source project for tools code & more Portions available for download under BSD-style license, long term initiative Wiki: Perforce:

6 Problem Proprietary revision control system Implicit file retrieval
Files synced before open Per-file database overhead Written files queued for check-in at process exit Heavy code instrumentation (iopen, iclose) Damage spreads immediately All processed data under revision control Based around live assets, emulated shared drive Data errors could stop everyone’s work Phantom bugs would crop up and disappear before programming team could fix them We have shipped games with computed data in revision control that we could not easily regenerate! Proprietary database was hard to support Downtime killed productivity

7 Solution Move all revision control to Perforce
Handles branching of binaries efficiently Label code and data together Explicit asset retrieval Users decide when to download Big speed improvement Better stability while getting a unit of work done Remove built data from revision control

8 Branching We branch all files for milestones
Work continues in the main line during milestone stabilization Milestone fixes brought back piecemeal Milestone hacks stay in branch forever

9 Continuous Integration
Custom system implemented in Perl Populates a ‘Known Good’ label in Perforce Failure notifies users based on recent checkins (changes between testing and last known good changelist numbers) Moving forward we intend to sync to ‘Known Good’ by default (code and assets) Code and data validated separately but labeled together

10 Problem Problem: Asset tree organized by engine type
Tools and engine expected numbered assets Moby 500: Wooden Breakable Crate Level 9000: My Test Level Storage location of each asset was fixed or had a natural home in the tree: /Data/Mobys/Moby500/actor.xml (fixed location) /Art/Mobys/Moby500/crate.mb (natural home)

11 Problem Numbered assets are simple to program for
Reference any asset using type and id Lacked descriptive names, hard to find, hindered re-use of assets Users tracked assets with pen and paper Changing the type of an asset lead to complex copy logic that had to copy files around

12 Solution Tools shouldn’t force asset organization
Users should be able to put files anywhere Assets are defined in files with a reasonable names (BreakableCrate.entity vs. moby500) Files should be able to be moved or renamed without a lot of work Renaming a single file is simple, but references to that file are troublesome We wanted to make a system where content organization is completely independent of engine requirements

13 Asset Referencing Historically references are file path strings
Makes renaming content files difficult and error prone Asset A references B, renaming B to C breaks A Housecleaning before a sequel game causes pain due to broken references Typically fixed up by brute force

14 Asset Referencing Don’t rely on string paths for referencing files
Use a unique IDs to indirect the file location IDs are assigned and shared via events Event files are retrieved with asset files Event files contain simple event logic: File <foo> was added with id <blah> File <foo> was moved to <bar> File <bar> was deleted

15 Event System Each event is written to a file per-user, per-computer, per-project, per-branch “geoff-geoff_xp-rcf-devel.event.dat” Avoid contention for file checkout Avoid conflict resolution during integrate EventSystem collates events, sorts by time Events handled only once

16 File Resolver File Resolver maps unique ID -> File path
Consumes events from Event System Populates a local machine SQLite DB Native formats store only the unique ID Maya stores ID in dynamic attributes String paths are fixed up at load time by plug-in TUID – 64 bit unique ID Open Source via Nocturnal

17 Asset Referencing Processed data is output to folders named for the file id of the Asset definition file Even though assets are renamed, processed data location remains the same Helps efficiency on the back end of pipeline It’s possible to reference assets in game by the file ID by hand, but we try to avoid this

18 Here be Dragons Since databases are built on each computer, it can have tricky machine-specific bugs (revert) We commit Add and Rename operations immediately to fight missing asset references (dangling TUIDs), data files may be valid stubs Moving forward we think its possible to store resolver data as meta-data and eliminate collation of Events altogether

19 Problem Problem: Assets defined by engine type specific data structures Our assets were defined in terms of our engine, not in terms of the content our engine was consuming This made changing the engine type arduous since the data could be very different

20 Solution Asset data should be modular, and that modularity should drive the engine type Asset definition should serve the content, not specific engine types (wherever possible) Similar data about Assets should be reusable between engine types

21 Asset Fundamentals What is an Asset: A discrete unit of content that needs to be processed by one or more builders to be loaded by the game An Asset itself has only basic properties An Asset is a collection of Attributes Asset : public AttributeCollection

22 Assets (C++ Classes) AttributeCollection Asset EntityAsset LevelAsset

23 Attributes Attribute: A block of related information
AnimationAttribute: what animations to use PhysicsAttribute: physics subsystem info VisualAttribute: render information Attributes store the bulk of the complexity Stored in ‘slots’ using class type id (int32)

24 Attributes (C++ Classes)
AnimationAttribute PhysicsAttribute VisualAttribute

25 Asset Example EntityAsset – content backed placeable object
ArtFileAttribute – reference Maya content file VisualAttribute – render information EntityAsset Art File Visual

26 Asset Example LevelAsset – top level chunk of game data
WorldFileAttribute – placed objects in the level DependenciesAttribute – list of assets to pack LevelAsset World File Dependencies

27 Asset Example ShaderAsset – texture and render properties
ColorMapAttribute – texture, compress settings NormalMapAttribute – processed differently TextureMap ColorMap NormalMap ShaderAsset ColorMap NormalMap

28 Attributes Simplifies sharing structures between Assets
Attributes only exist for features that are configured by content creators (auditing) Easy to copy and paste between Assets Prior examples are Exclusive attributes

29 Validating Attributes
If we aren’t careful users could easily create non-sensical Assets LevelAsset with a TextureMapAttribute? Shader with a PhysicsAttribute? Must be able to validate Attributes Attribute may not work with some Asset classes Attributes may not work in conjunction Prior examples are Exclusive attributes

30 Attribute Behavior Behavior defines default compatibility
Exclusive: ColorMapAttribute Validated by ShaderAsset, not by LevelAsset Inclusive: DependenciesAttribute References to miscellaneous resources Useful for spawn-only types like weapons, HUD Loosely related assets like weapons and ammo Prior examples are Exclusive attributes

31 Attribute Usage Usage implies Class/Instance relationship
Class – Used in classes only Instance – Used in instances only Overridable – Works in both class and instance Allows us to easily override class settings per instance on an Attribute level

32 Attribute Usage Our SceneNode class also subclasses AttributeCollection: AttributeCollection Asset EntityAsset SceneNode Entity

33 Attribute Usage BakedLightingAttribute defines light map dimension and compression settings Instances of that EntityAsset can override the light map information (if its scaled way up or down) EntityAsset BakedLightingAttribute Default light map settings Entity BakedLightingAttribute Override light map settings

34 Attribute Sibling Validation
Some Attributes inherently conflict Attributes can reject potentially invalid siblings Ex: BakedLightingAttribute will reject the addition of PhysicsAttribute because baked lighting doesn’t work with dynamic objects

35 Engine Type Classification
Assets map to one or more Engine Types If Asset map to one type, it’s a natural match ShaderAsset -> ShaderBuilder LevelAsset -> LevelBuilder If Asset class maps to more than one Attributes determine Engine Type EntityAsset -> ShrubBuilder, TieBuilder, etc…

36 Entity Assets Shrub Shrub – light weight static art
ArtFileAttribute – Maya content file VisualAttribute – render information EntityAsset Art File Visual Shrub

37 Entity Assets Tie Tie – more expensive static art
CollisionAttribute – makes it collidable BakedLightingAttribute – lightmap size EntityAsset Art File Draw Collision Baked Lighting Tie

38 Entity Assets Moby Moby – animated character
PhysicsAttribute – Physics subsystem settings AnimationAttribute – Animation assets to use EntityAsset Art File Draw Physics Animation Moby

39 Too many Attributes! Plethora of Attributes is confusing to artists
Keeping Attributes simple is key Attributes are cumbersome because knowing which Attributes yield each Engine Type its costly to learn

40 Just make it a <Engine Type>!
Sometimes users know what Engine Type they want, and need the ability to conform the Attributes to that Engine Type Attributes have Enable flag which can be used to disable offending Attributes so data is not lost, just dormant Changing the type back re-enables the disabled Attributes

41 What Attributes do I need again?
Setting up Assets from scratch can still be non-trivial… basic Attributes are needed to get an Asset up and running We created a Wizard with simple presets for each Engine type Moving forward we are going to select creation preset based on export data contents

42 Problem Problem: .NET was not very productive for generating UI for editing our Assets We don’t want to give up writing Native code, so .NET in general was not a good fit for us Marshalling data structures between Native and Managed was time consuming Windows::Forms::PropertyGrid was hard to deal with to get good custom property editing

43 Solution Drop .NET and develop our own solutions
Our C++ reflection was getting feature-rich We already had a data-driven properties GUI Switched to C++ / wxWidgets Back up and running after 1 man month Simplified everything

44 Asset Editor UI Assets and Attributes are loaded into a single tree view References to other Assets are expanded in-place Properties displayed for each node in the tree Property interaction can update information in real-time on the target platform

45 Asset Editor UI

46 What’s under the hood We generate property GUI for Asset/Attributes from C++ reflection information Adding a field to an Asset or Attribute shows up automatically UI-script can add sliders, drop-downs, color pickers, etc…

47 Reflect Provides class and field level meta-data
All meta-data created at runtime No compile time parsing required Adding a field is usually a single line of code Type Registry assigns type id’s at runtime Automated persistence, cloning, visitor API Handles data format changes & legacy files Open Source via Nocturnal

48 Reflect Classes class Version : public Reflect::ConcreteInheritor<Version, Element> { public: std::string m_Source; static void EnumerateClass( Reflect::Compositor<Version>& comp ) comp.AddField( &Version::m_Source, "m_Source" ); } }; void Init() Reflect::RegisterClass<Version>( "Version" ); Basic cost of entry is a single function, but more information can be added via the pointer from the result of AddField() User friendly name, extensible properties (map<string,string>) The data we really need: Type, offset, and size

49 Reflect Enumerations namespace BlendTypes { enum BlendType { Normal, Additive }; static void EnumerateBlendType ( Reflect::Enumeration* info ) info->AddElement(Normal, "Normal"); info->AddElement(Additive, "Additive"); } } typedef BlendTypes::BlendType BlendType; void Init() Reflect::RegisterEnumeration<BlendType>( "BlendType”, &BlendTypes::EnumerateBlendType ); Enum and bitfield serialization done by string Safe to add/remove elements

50 Inspect System for data-driven property GUIs Control class hierarchy
Wrapper classes for wxWidgets UI elements Can be factory created from UI script Interpreter class hierarchy Intimate with data formats Automates the creation of the UI Handles callbacks from user interactions Open Source via Nocturnal

51 ReflectInspect Interpreter classes that generate Inspect UI
Binds control state to in memory structures Basic support for STL containers + Math types Support for custom UI handling via plug-ins Easy to integrate custom property elements Open Source via Nocturnal

52 Holy crap that’s a lot of Properties!
This system is awesome for drowning our users in lots of properties Agility is good, not the most important thing Cull properties that are used very infrequently Adding support for filtering properties Adding support for categorizing properties Adding support for dependencies to hide other properties when not applicable

53 Problem Users spend a lot of time waiting for Assets to build
No longer getting built data from revision control Building major animated characters took 15+ min Building levels took 30+ min

54 Solution Build system must share data between users
Information about an output file is hashed into a single signature for the output: Hash of input file paths, format version strings Hash of input file contents or folder contents Hash of dependency relationships Hash of output file paths, format version strings

55 Cache Files This signature used to build a file path to a output cache file on the network That file is downloaded instead of built Locally built data is uploaded automatically when the build phase completes Downloading updates the modification time of the cache file on the network Script trims down cache based modification time

56 Format Version Format version is a string compiled into tools
Changed a programmer changes a data format “15 – geoff: improved shader flossiness” Very efficient because users only reprocess data when data formats or assets change Takes discipline when checking in changes to libraries that affect lots of output formats

57 Build Process Given list of assets to build
Create builder objects for assets Builders compute output signatures Attempt download of output files from cache Build what could not be downloaded Upload built files to cache

58 Cache Files Started with Distributed Cache System (DCS)
Query for files on other workstations around the office, central server for tracking files DCS was complex, wasn’t well suited Moved to a Central Cache System (CCS) Now we have files sharing throughput issues with lots of concurrent users Windows 2008 / Linux are possible solutions

59 Cache Files Cache file storage is just an optimization
Any built data can be regenerated at any point VPN users have separate cache for their site if they have multiple machines Continuous iteration system is fantastic for seeding build cache files

60 Summary Major wins for our tools: Perforce File Resolver
Assets & Attributes Built File Cache

61 Thanks!

Download ppt "The Tech Behind the Tools of Insomniac Games"

Similar presentations

Ads by Google