Presentation is loading. Please wait.

Presentation is loading. Please wait.

Mathias Westerdahl - System Architect, Tools The Asset pipeline for Just Cause 2: Lessons Learned.

Similar presentations


Presentation on theme: "Mathias Westerdahl - System Architect, Tools The Asset pipeline for Just Cause 2: Lessons Learned."— Presentation transcript:

1 Mathias Westerdahl - System Architect, Tools The Asset pipeline for Just Cause 2: Lessons Learned

2 2 Overview Background Just Cause 1 Just Cause 2 Current pipeline Build system and compilers Central code Take aways Q & A

3 3 The Asset pipeline for Just Cause 2: Lessons Learned What is Just Cause? Large open world Sand box game play Dynamic game play No Levels, continuous streaming Multiplatform Very flexible editing and testing of content Needs a fast and flexible pipeline

4 4 The Asset pipeline for Just Cause 2: Lessons Learned JustEdit Our main editor is called JustEdit Edits the terrain, missions, locations, roads, etc. Each mission/location contains entities An entity contains game objects or other entities This highly flexible system led to very many assets to export (~1500 locations and ~4000 entities) Runs the game No turnaround times for testing

5 5 The Asset pipeline for Just Cause 2: Lessons Learned JustEdit Maya Perforce Damage Editor Terrain Locations Missions Particles Models Physics Animations Skeletons Physics Workflow Overview for Just Cause 1 Game

6 6 The Asset pipeline for Just Cause 2: Lessons Learned JustEdit Maya Terrain Locations Missions Particles Models Physics Animations Skeletons Physics Workflow Overview for Just Cause 1 Many manual steps Game ready data in version control Editing of game ready data!!! Perforce Damage Editor Game

7 7 The Asset pipeline for Just Cause 2: Lessons Learned Lessons learned from Just Cause 1 Remove as many manual steps as possible Put export steps in separate tools (i.e. compilers) Decrease code complexity Increase robustness Easier to read and debug Introduce intermediate data formats Tools code broke easily

8 8 The Asset pipeline for Just Cause 2: Lessons Learned Actions Taken after Just Cause 1 We decided on creating our own content build system Compilers in C++, derived from previous export steps Took existing code and put into use directly The Project Compiler was the heart of the system Triggered all other compilers Dependency checking was implemented separately by each compiler (time stamps) Users can right click on any file and compile that dependency tree The editor and other tools use this functionality

9 9 The Asset pipeline for Just Cause 2: Lessons Learned Actions Taken after Just Cause 1 Auto Builders were setup Code All compilers, exporters, plug-ins, editor Content Full nightly builds for all platforms Builds ISO’s Cruise Control + NAnt was used to create the auto builders CCTray is a system tray icon tool Alerts users when a build fails

10 10 The Asset pipeline for Just Cause 2: Lessons Learned Just Cause 2 – New Tools A lot of new tools were ordered by the content creators Detailed design docs were written Each tool was assigned a “customer” The programmers and the customer kept close communication The customer was responsible to test the tool and approve it Tools as plug-ins to JustEdit WTL was used for GUI in editors/plug-ins

11 11 The Asset pipeline for Just Cause 2: Lessons Learned JustEdit Maya Perforce Compilers Terrain Locations Missions Particles Models Physics Skeleto ns Compile Game Workflow Overview for Just Cause 2 MotionBuilder Animations Cutscenes Constraints Plug-ins Vehicles Characters Cutscenes Auto Builder Sync

12 12 The Asset pipeline for Just Cause 2: Lessons Learned Lessons Learned from Just Cause 2 – New Tools 10 new tools at start of project!! Code not up in the 90%-100% complete where it should be Flaws in workflows Not all tools were used in the extent that was intended Content that was easy to edit was not very suitable in game with respect to performance and memory Design docs were accepted “as-is” They were essentially a wish list from content creators

13 13 The Asset pipeline for Just Cause 2: Lessons Learned Lessons Learned from Just Cause 2 – New Tools Customers They were not part of the dev-team, communication suffered Often planned full time on other tasks Unclear responsibilities Plug-ins Unstable C++ interfaces Should have gone with network communications Too complex GUI: WTL and C++ WTL is too bare bone and C++ slows down iterations

14 14 The Asset pipeline for Just Cause 2: Lessons Learned Lessons Learned from Just Cause 2 – Compilers Compilers in C++ Using C++ for scripting isn’t always very convenient Long build times, making it awkward to tweak Some compilers included all game code Very platform specific Win32 heavy with DirectX amongst other things Couldn’t run on auto builders with no desktop

15 15 The Asset pipeline for Just Cause 2: Lessons Learned Lessons Learned from Just Cause 2 – Compilers Programmers implemented things slightly differently Causing anomalies in overall behavior Breaking dependency checking Hard coded parameters in the compilers File formats weren’t stable enough If something went wrong, compiler probably crashed Or worse, the broken data made it into the game

16 16 The Asset pipeline for Just Cause 2: Lessons Learned Reluctance to work with compilers The pipeline was considered somewhat magic Lack of documentation The overall flow wasn’t easily overviewed Debugging broken data often done in game code Programmers already had all code & data set up This sometimes led to runtime fixes being done rather than fixing the bugs in the compilers Lessons Learned from Just Cause 2 – Compilers

17 17 The Asset pipeline for Just Cause 2: Lessons Learned No central code Not using central code meant problems when integrating Fixes went missing, you had to find them again Built compilers were version controlled in Perforce Not automatically rebuilt and used when needed You could still use old buggy code Lessons Learned from Just Cause 2 – Compilers

18 18 The Asset pipeline for Just Cause 2: Lessons Learned Data formats A lot of different file types (~30) Many of the formats were Xml based Mapping “property name” to “value” Formats were more optimal for the editor than for the game Read/write errors weren’t detected No version numbers No proper error messages could be logged Lot of things to improve! Lessons Learned from Just Cause 2 – Compilers

19 19 The Asset pipeline for Just Cause 2: Lessons Learned Content build system A bit naïve solution Slow dependency checking Incomplete dependency tree Time stamps are not good enough Due to bugs & anomalies in the behavior, users didn’t trust it to 100% “It went wrong! Did I do it in the wrong order?” “I’d better resort to my old ways without asking anyone…” No trust meant full clean and full rebuild Lessons Learned from Just Cause 2 – Compilers

20 20 The Asset pipeline for Just Cause 2: Lessons Learned Statistics from Just Cause 2 3 platforms: Win32, PS3, Xenon ~ source assets in version control ~80000 of those files pass through our compiler pipeline Source data amount to ~13GB Compiled data amount to ~7GB uncompressed ~4GB compressed A full rebuild took about 4h45m for all 3 platforms Building all tools code took ~1h

21 21 The Asset pipeline for Just Cause 2: Lessons Learned Current pipeline What was done to address the retrospective from JC2? Fresh goals Rebuild the trust in the pipeline Incremental builds, no magic workflows, easy to configure 100% accuracy using the dependency checking Simplify the development Move common code to a central repository Decrease turn around times when creating tools

22 22 The Asset pipeline for Just Cause 2: Lessons Learned Current pipeline Big decisions We needed a build/deployment system for central code The content build system also needed a major push Needed a simple to use and yet powerful generalized data format Development had to be done alongside JC2 Needed to commit to the task to actually get anywhere soon

23 23 The Asset pipeline for Just Cause 2: Lessons Learned Current pipeline - Needy Dependency resolver and deployment system Written in house using python, very light weight Handles “packages” Built libraries (.h +.lib/.so) Built executables Bundles of python scripts (or whatever you want) 3rd party libraries / executables Deploys packages to a central repository Fetches packages to local system Dependencies.txt Python ;buildtool Needy ;buildtool Buildsystem-6.1.*;buildtool Platform-7.*;buildtool Debug-8.*;buildtool Adf-8.* havok-55x Product.txt PRODUCT=animcompiler VERSION=1.0.0 PLATFORMS="Win32 Linux" CONFIGURATIONS="Debug Release" CATEGORY=compilers

24 24 The Asset pipeline for Just Cause 2: Lessons Learned Current pipeline - Needy Big change in the coding process Code base evolves towards smaller libraries Really fast compile times Helps keeping track of version conflicts Allows for very rapid development and testing

25 25 The Asset pipeline for Just Cause 2: Lessons Learned Current pipeline – WAF Light weight build system for code and data Open source (BSD), source is ~80kb Dependency checking Multi core support Easily to maintain and extend Replaces our in house Project Compiler system Python A small snippet from a task definition using waf: cls = Task.simple_task_type(‘.anim’, ‘${PYTHONC} ${ANIMC} –i ${SRC} –o ${TGT} –p ${PLATFORM}’) cls.ext_in = (‘.anim’, ) cls.ext_out = (‘.ban’, ) Task.TaskBase.classes[‘.anim’].scan = FindDependenciesFromAnimation

26 26 The Asset pipeline for Just Cause 2: Lessons Learned Current pipeline - ADF Avalanche Data Format Serialization framework C/C++, Python support Xml Binary support Hash based versioning for binary files Error handling The ability to access data both from Python and C/C++ enables seamless cross language interchange Cross platform support E.g. pass data from tools to/from consoles

27 27 The Asset pipeline for Just Cause 2: Lessons Learned Current pipeline - ADF TM2VertexData = DefStruct( "TM2VertexData", ( "PositionIndex", Array( uint32 ) ), ( "TangentSpaceIndex", Array( uint32 ) ), ( "ColorIndex", Array( uint32 ) ), ( "UVIndices", Array( uint32 ) ) ) TM2Model = DefStruct( "TM2Model", ( "Name", String ), ( "Triangles", Array( uint32 ) ), ( "Vertices", TM2VertexData ), ( "MorphTargets", Array( TM2MorphTarget ) ), ( "Positions", TM2PositionData ), ( "TangentSpaces", Array( float ) ) ) Type definitions:

28 28 The Asset pipeline for Just Cause 2: Lessons Learned Current pipeline - ADF from Adf import Context ctx = Context() ctx.ReadTypeLibrary('mytype.adf') objs = ctx.ReadInstancesFromXml(‘infile.xml') objs[0].Name = 'newvalue' ctx.WriteInstancesToFile( ‘outfile.bin', objs, BIG_ENDIAN ) Read / Write mytype.py AdfPack.exe mytype.adf mytype.h compiler.py Infile.xml Outfile.bin

29 29 The Asset pipeline for Just Cause 2: Lessons Learned Current pipeline - Compilers C++ Shared libraries, loaded from Python Removed dependencies on old game code Replaced in-house code with standard code fopen() rather than CFile() Decrease complexity, increase portability Scripting and logic in python Data processing in Python or C++ E.g. read/write/process Havok files using C++

30 30 The Asset pipeline for Just Cause 2: Lessons Learned Current pipeline - Compilers Python Extreme turn around times Vast collection of built in modules We use these a lot: optparse, ctypes, md5, numpy, cStringIO… Many of the modules have the backend implemented in C++ Most compilers were completely rewritten Cut a lot of dependencies Code size down to a 1/10 th

31 31 The Asset pipeline for Just Cause 2: Lessons Learned Current pipeline – Dependency Checking Waf handles the mechanics Scanners find the dependencies Waf caches the result MD5 checksums Command lines arguments Source/Dependency file contents A new compiler path will trigger a compilation Very handy when updating packages with Needy

32 32 The Asset pipeline for Just Cause 2: Lessons Learned Current pipeline – Dependency Checking A small example of scanning a file for other file references # Reading a text file containing file references def FindDependencies(path): f = open(path, "rb" ) lines = f.readlines() f.close() return [ x.strip() for x in lines ]

33 33 The Asset pipeline for Just Cause 2: Lessons Learned Current pipeline Platform independence Possible to run on win32, linux, ps3, xenon etc. Helps keeping the code clean Running on linux We wanted to run on linux for the file system performance Run win32 exe’s through WINE Wine is a Win32 implementation in linux. It’s not an emulator. Intermediate solution before all code was rewritten

34 34 The Asset pipeline for Just Cause 2: Lessons Learned Current pipeline Incremental auto builder Syncs from version control Only builds the things necessary Full auto builder Compares incremental build with full build Finds dependency problems Find bugs in data formats

35 35 The Asset pipeline for Just Cause 2: Lessons Learned Current pipeline Change of mindset Keep it simple! Each component should do one thing only …and do it well!

36 36 The Asset pipeline for Just Cause 2: Lessons Learned Current pipeline To sum things up so far We rewrote ~20 compilers (out of ~30) in 6 man weeks Using old code as blue print Remaining compilers took longer due to more complex work flows Code size decreased to 1/10 th Performance increased in all cases 10% to 1000%, 100% was quite common Possible to use compilers in the previous pipeline with little effort Early benefits of the new pipeline

37 37 The Asset pipeline for Just Cause 2: Lessons Learned Future improvements Reloading of assets in run time (in progress) Virtual File System Can load data from hard drive, archives or web server Distributed builds Looking into using AFS, a distributed file system ssh builder01 modelcompiler.exe -i mymesh1.tm … ssh builder02 modelcompiler.exe -i mymesh2.tm …

38 38 The Asset pipeline for Just Cause 2: Lessons Learned Take Aways A few thing we’ve learned so far Some are quite trivial but also easy to overlook

39 39 The Asset pipeline for Just Cause 2: Lessons Learned Take Aways Hardware Defrag the hard drives Try using other file systems, ext3, ext4, afs. Our pipeline is a lot faster on linux than on win32 Two separate discs that are designated source/target Ram discs/SSD are getting cheaper

40 40 The Asset pipeline for Just Cause 2: Lessons Learned Take Aways How to implement the compilers No assumptions on working directory or any other variables Keep the compiler data driven, it’s quite easy Project specific options could be read from a file or command line Create a simple plugin system Extremely easy in python

41 41 The Asset pipeline for Just Cause 2: Lessons Learned Example of loading python files. I.e. creating a plug-in system: for root, dirs, files in os.walk(‘c:/myapp/plugins’): if not root in sys.path: sys.path.append(root) for filename in files: if not filename.endswith('.py'): continue # We do an import() instead of an execfile() so that # the loaded code can act as a regular python file module = __import__(filename.replace('.py', '') ) Take Aways

42 42 The Asset pipeline for Just Cause 2: Lessons Learned Take Aways How to implement the compilers Write portable code E.g. remove DirectX, Win32 API ‘-e’ for endianess rather than ‘-p’ for platform Beware of case sensitiveness from the start Also, beware of ‘\\’ versus ‘/’

43 43 The Asset pipeline for Just Cause 2: Lessons Learned Take Aways How to organize the data Keep source data separate from compiled data When creating new file formats Avoid doing it “by hand”. Create a generalized file type. Include a version number Store file dependencies easily accessible Write the game ready format with efficiency in mind Memory, performance

44 44 The Asset pipeline for Just Cause 2: Lessons Learned Take Aways How to optimize the compilers Write compilers with parallelism in mind No “global variables” such as reused temp files on disc Disc cache thrashing It comes down to the access patterns Do as few reads/writes as possible Use tools like Process Explorer or vmstat

45 45 The Asset pipeline for Just Cause 2: Lessons Learned Take Aways How to optimize the compilers Do reads and writes in memory Lowers the disc usage E.g allocate large buffer and write into it Avoid temp files altogether import cStringIO memfile = cStringIO.StringIO() MultipleWritesToAFile( mydata, memfile) f = open(options.output, "wb" ) f.write( memfile.getvalue() ) f.close() memfile.close()

46 46 The Asset pipeline for Just Cause 2: Lessons Learned Take Aways Dare to do stuff! If the compiler code is unreadable/inefficient Will it be faster/more productive to rewrite it? Rewriting will probably introduce new bugs But you will also… Probably throw away many old bugs as well Increase your understanding of the work flow Get a chance to improve design/performance more directly

47 47 The Asset pipeline for Just Cause 2: Lessons Learned Take Aways Dare to do stuff! Rewriting doesn’t mean you throw the code away You will use the code as a blue print of what, or what not, to do It’s easy to rewrite something that does the same thing, only better!

48 48 The Asset pipeline for Just Cause 2: Lessons Learned Take Aways Separate code Use one code base to read/write data in compilers Use another code base for loading the data in game Keeps down code dependencies Keeps the compilers small and simple Less code to maintain in the compiler Decreased code size down to 1/10 th Less code to compile, faster turnaround times Suitable for file formats that rarely change

49 49 The Asset pipeline for Just Cause 2: Lessons Learned Take Aways Keep the build working! Only replace old code with new working code Down time is both costly and stressful Keep it simple!

50 50 The Asset pipeline for Just Cause 2: Lessons Learned Links Python – waf – wine – jqplot – protobuf – Download slides and source code on our site!

51 51 The Asset pipeline for Just Cause 2: Lessons Learned Q & A Download slides and source code on our site!

52 52 The Asset pipeline for Just Cause 2: Lessons Learned


Download ppt "Mathias Westerdahl - System Architect, Tools The Asset pipeline for Just Cause 2: Lessons Learned."

Similar presentations


Ads by Google