Presentation is loading. Please wait.

Presentation is loading. Please wait.

The Tech Behind the Tools of Insomniac Games Geoff Evans – Insomniac Games.

Similar presentations

Presentation on theme: "The Tech Behind the Tools of Insomniac Games Geoff Evans – Insomniac Games."— Presentation transcript:

1 The Tech Behind the Tools of Insomniac Games Geoff Evans – Insomniac Games

2 Welcome! An overview of how Insomniac’s tools get current generation games out the door There are no slick demos of our game engine This talk is focused tools internals, pipelines, and best practices we have found


4 Outline Areas of improvement: Revision control Asset organization Flexible Asset definition Efficient Asset Processing

5 Warning: Open Source Ahead Nocturnal Initiative is Insomniac's Open Source project for tools code & more Portions available for download under BSD- style license, long term initiative Wiki: – Perforce: –

6 Problem Proprietary revision control system Implicit file retrieval – Files synced before open – Per-file database overhead – Written files queued for check-in at process exit – Heavy code instrumentation (iopen, iclose) Damage spreads immediately All processed data under revision control

7 Solution Move all revision control to Perforce – Handles branching of binaries efficiently – Label code and data together Explicit asset retrieval – Users decide when to download – Big speed improvement – Better stability while getting a unit of work done Remove built data from revision control

8 Branching We branch all files for milestones Work continues in the main line during milestone stabilization Milestone fixes brought back piecemeal Milestone hacks stay in branch forever

9 Continuous Integration Custom system implemented in Perl Populates a ‘Known Good’ label in Perforce Failure notifies users based on recent checkins (changes between testing and last known good changelist numbers) Moving forward we intend to sync to ‘Known Good’ by default (code and assets)

10 Problem Problem: Asset tree organized by engine type Tools and engine expected numbered assets – Moby 500: Wooden Breakable Crate – Level 9000: My Test Level Storage location of each asset was fixed or had a natural home in the tree: – /Data/Mobys/Moby500/actor.xml (fixed location) – /Art/Mobys/Moby500/crate.mb (natural home)

11 Problem Numbered assets are simple to program for Reference any asset using type and id Lacked descriptive names, hard to find, hindered re-use of assets Users tracked assets with pen and paper Changing the type of an asset lead to complex copy logic that had to copy files around

12 Solution Tools shouldn’t force asset organization Users should be able to put files anywhere Assets are defined in files with a reasonable names (BreakableCrate.entity vs. moby500) Files should be able to be moved or renamed without a lot of work Renaming a single file is simple, but references to that file are troublesome

13 Asset Referencing Historically references are file path strings Makes renaming content files difficult and error prone – Asset A references B, renaming B to C breaks A Housecleaning before a sequel game causes pain due to broken references Typically fixed up by brute force

14 Asset Referencing Don’t rely on string paths for referencing files Use a unique IDs to indirect the file location IDs are assigned and shared via events Event files are retrieved with asset files Event files contain simple event logic: – File was added with id – File was moved to – File was deleted

15 Event System Each event is written to a file per-user, per- computer, per-project, per-branch – “geoff-geoff_xp-rcf-devel.event.dat” – Avoid contention for file checkout – Avoid conflict resolution during integrate EventSystem collates events, sorts by time Events handled only once

16 File Resolver File Resolver maps unique ID -> File path Consumes events from Event System Populates a local machine SQLite DB Native formats store only the unique ID Maya stores ID in dynamic attributes – String paths are fixed up at load time by plug-in TUID – 64 bit unique ID – Open Source via Nocturnal

17 Asset Referencing Processed data is output to folders named for the file id of the Asset definition file Even though assets are renamed, processed data location remains the same Helps efficiency on the back end of pipeline It’s possible to reference assets in game by the file ID by hand, but we try to avoid this

18 Here be Dragons Since databases are built on each computer, it can have tricky machine-specific bugs (revert) We commit Add and Rename operations immediately to fight missing asset references (dangling TUIDs), data files may be valid stubs Moving forward we think its possible to store resolver data as meta-data and eliminate collation of Events altogether

19 Problem Problem: Assets defined by engine type specific data structures – Our assets were defined in terms of our engine, not in terms of the content our engine was consuming – This made changing the engine type arduous since the data could be very different

20 Solution Asset data should be modular, and that modularity should drive the engine type – Asset definition should serve the content, not specific engine types (wherever possible) – Similar data about Assets should be reusable between engine types

21 Asset Fundamentals What is an Asset: A discrete unit of content that needs to be processed by one or more builders to be loaded by the game An Asset itself has only basic properties An Asset is a collection of Attributes – Asset : public AttributeCollection

22 Assets (C++ Classes) AttributeCollectionAssetEntityAssetLevelAssetShaderAsset

23 Attributes Attribute: A block of related information – AnimationAttribute: what animations to use – PhysicsAttribute: physics subsystem info – VisualAttribute: render information Attributes store the bulk of the complexity Stored in ‘slots’ using class type id (int32)

24 Attributes (C++ Classes) AttributeAnimationAttributePhysicsAttributeVisualAttribute

25 Asset Example EntityAsset – content backed placeable object – ArtFileAttribute – reference Maya content file – VisualAttribute – render information EntityAsset Art FileVisual

26 Asset Example LevelAsset – top level chunk of game data – WorldFileAttribute – placed objects in the level – DependenciesAttribute – list of assets to pack LevelAsset World FileDependencies

27 Asset Example ShaderAsset – texture and render properties – ColorMapAttribute – texture, compress settings – NormalMapAttribute – processed differently ShaderAsset ColorMapNormalMap TextureMap ColorMapNormalMap

28 Attributes Simplifies sharing structures between Assets Attributes only exist for features that are configured by content creators (auditing) Easy to copy and paste between Assets

29 Validating Attributes If we aren’t careful users could easily create non-sensical Assets – LevelAsset with a TextureMapAttribute? – Shader with a PhysicsAttribute? Must be able to validate Attributes – Attribute may not work with some Asset classes – Attributes may not work in conjunction

30 Attribute Behavior Behavior defines default compatibility Exclusive: ColorMapAttribute – Validated by ShaderAsset, not by LevelAsset Inclusive: DependenciesAttribute – References to miscellaneous resources – Useful for spawn-only types like weapons, HUD – Loosely related assets like weapons and ammo

31 Attribute Usage Usage implies Class/Instance relationship – Class – Used in classes only – Instance – Used in instances only – Overridable – Works in both class and instance Allows us to easily override class settings per instance on an Attribute level

32 Attribute Usage Our SceneNode class also subclasses AttributeCollection: AttributeCollectionAssetEntityAssetAttributeCollectionSceneNodeEntity

33 Attribute Usage BakedLightingAttribute defines light map dimension and compression settings – Instances of that EntityAsset can override the light map information (if its scaled way up or down) EntityAsset BakedLightingAttribute Default light map settings Entity BakedLightingAttribute Override light map settings

34 Attribute Sibling Validation Some Attributes inherently conflict Attributes can reject potentially invalid siblings – Ex: BakedLightingAttribute will reject the addition of PhysicsAttribute because baked lighting doesn’t work with dynamic objects

35 Engine Type Classification Assets map to one or more Engine Types If Asset map to one type, it’s a natural match – ShaderAsset -> ShaderBuilder – LevelAsset -> LevelBuilder If Asset class maps to more than one – Attributes determine Engine Type – EntityAsset -> ShrubBuilder, TieBuilder, etc…

36 Entity Assets Shrub – light weight static art – ArtFileAttribute – Maya content file – VisualAttribute – render information EntityAsset Art FileVisual Shrub

37 Entity Assets Tie – more expensive static art – CollisionAttribute – makes it collidable – BakedLightingAttribute – lightmap size EntityAsset Art FileDrawCollision Baked Lighting Tie

38 Entity Assets Moby – animated character – PhysicsAttribute – Physics subsystem settings – AnimationAttribute – Animation assets to use EntityAsset Art FileDrawPhysicsAnimation Moby

39 Too many Attributes! Plethora of Attributes is confusing to artists Keeping Attributes simple is key Attributes are cumbersome because knowing which Attributes yield each Engine Type its costly to learn

40 Just make it a ! Sometimes users know what Engine Type they want, and need the ability to conform the Attributes to that Engine Type Attributes have Enable flag which can be used to disable offending Attributes so data is not lost, just dormant Changing the type back re-enables the disabled Attributes

41 What Attributes do I need again? Setting up Assets from scratch can still be non- trivial… basic Attributes are needed to get an Asset up and running We created a Wizard with simple presets for each Engine type Moving forward we are going to select creation preset based on export data contents

42 Problem Problem:.NET was not very productive for generating UI for editing our Assets – We don’t want to give up writing Native code, so.NET in general was not a good fit for us – Marshalling data structures between Native and Managed was time consuming – Windows::Forms::PropertyGrid was hard to deal with to get good custom property editing

43 Solution Drop.NET and develop our own solutions – Our C++ reflection was getting feature-rich – We already had a data-driven properties GUI – Switched to C++ / wxWidgets – Back up and running after 1 man month – Simplified everything

44 Asset Editor UI Assets and Attributes are loaded into a single tree view – References to other Assets are expanded in-place – Properties displayed for each node in the tree – Property interaction can update information in real-time on the target platform

45 Asset Editor UI

46 What’s under the hood We generate property GUI for Asset/Attributes from C++ reflection information Adding a field to an Asset or Attribute shows up automatically UI-script can add sliders, drop-downs, color pickers, etc…

47 Reflect Provides class and field level meta-data All meta-data created at runtime – No compile time parsing required Adding a field is usually a single line of code Type Registry assigns type id’s at runtime Automated persistence, cloning, visitor API Handles data format changes & legacy files Open Source via Nocturnal

48 Reflect Classes class Version : public Reflect::ConcreteInheritor { public: std::string m_Source; static void EnumerateClass( Reflect::Compositor & comp ) { comp.AddField( &Version::m_Source, "m_Source" ); } }; void Init() { Reflect::RegisterClass ( "Version" ); }

49 Reflect Enumerations namespace BlendTypes { enum BlendType { Normal, Additive }; static void EnumerateBlendType ( Reflect::Enumeration* info ) { info->AddElement(Normal, "Normal"); info->AddElement(Additive, "Additive"); } } typedef BlendTypes::BlendType BlendType; void Init() { Reflect::RegisterEnumeration ( "BlendType”, &BlendTypes::EnumerateBlendType ); }

50 Inspect System for data-driven property GUIs Control class hierarchy – Wrapper classes for wxWidgets UI elements – Can be factory created from UI script Interpreter class hierarchy – Intimate with data formats – Automates the creation of the UI – Handles callbacks from user interactions Open Source via Nocturnal

51 ReflectInspect Interpreter classes that generate Inspect UI Binds control state to in memory structures Basic support for STL containers + Math types Support for custom UI handling via plug-ins Easy to integrate custom property elements Open Source via Nocturnal

52 Holy crap that’s a lot of Properties! This system is awesome for drowning our users in lots of properties Agility is good, not the most important thing Cull properties that are used very infrequently Adding support for filtering properties Adding support for categorizing properties Adding support for dependencies to hide other properties when not applicable

53 Problem Users spend a lot of time waiting for Assets to build – No longer getting built data from revision control – Building major animated characters took 15+ min – Building levels took 30+ min

54 Solution Build system must share data between users Information about an output file is hashed into a single signature for the output: – Hash of input file paths, format version strings – Hash of input file contents or folder contents – Hash of dependency relationships – Hash of output file paths, format version strings

55 Cache Files This signature used to build a file path to a output cache file on the network That file is downloaded instead of built Locally built data is uploaded automatically when the build phase completes Downloading updates the modification time of the cache file on the network – Script trims down cache based modification time

56 Format Version Format version is a string compiled into tools Changed a programmer changes a data format – “15 – geoff: improved shader flossiness” Very efficient because users only reprocess data when data formats or assets change Takes discipline when checking in changes to libraries that affect lots of output formats

57 Build Process 1.Given list of assets to build 2.Create builder objects for assets 3.Builders compute output signatures 4.Attempt download of output files from cache 5.Build what could not be downloaded 6.Upload built files to cache

58 Cache Files Started with Distributed Cache System (DCS) Query for files on other workstations around the office, central server for tracking files DCS was complex, wasn’t well suited Moved to a Central Cache System (CCS) Now we have files sharing throughput issues with lots of concurrent users Windows 2008 / Linux are possible solutions

59 Cache Files Cache file storage is just an optimization Any built data can be regenerated at any point VPN users have separate cache for their site if they have multiple machines Continuous iteration system is fantastic for seeding build cache files

60 Summary Major wins for our tools: Perforce File Resolver Assets & Attributes Built File Cache

61 Thanks!

Download ppt "The Tech Behind the Tools of Insomniac Games Geoff Evans – Insomniac Games."

Similar presentations

Ads by Google