1. I Can Slack Off, My Code’s Compiling Comic and title care of XKCD.com

2. Jeff Ward Lead Architect Orbus Gameworks Who am I? IGDA Tools SIG Chair Author: The Toolsmiths www.thetoolsmiths.com

3. What this talk is about

4. The Big Question How much time and money is lost just waiting for a build?

5. To answer this question, you need to know:

6. How long is your build? On average? For a complete rebuild?

7. Our build (includes unit tests) CPP Libraries (vs2003, 2005, 2008): 00:36:24 C# Libraries: 00:00:27 Java Libraries: 00:00:04 Python Libraries: 00:00:01

8. Now that you know that… How long is your average incremental build? How many times, per day, is your programmer going to perform that build? By how much can you improve that? The difference is wasted time!!!

9. Why know that? It’s all about the Benjamins.

10. The Money Assumptions A 52 week year A 5 day work week A $62k average salary (~$30/hr) This does not include other overhead for a programmer.

11. The Programmer Burn Chart Number of Programmers Hours lost Per Day Money Lost Per Day WeekYear 1.5$10$50$2600 2.5$30$150$7800 51$150$750$39000 101$300$1500$78000 301$900$4500$234000 302$1800$9000$468000

12. …you can lose approximately $468,000, the cost of 2 lead programmers, to build times.

13. You Programmer Burn (cont.) Number of Programmers Hours lost Per Day Money Lost Per Day WeekYear 51$150$750$39000 101$300$1500$78000 Two way to look at these numbers: Money Lost (in programmer productivity time) Money Saved (by increasing productivity time)

14. Saving Money, Saving Time Faster builds save more than just build time ▫Keeps programmers from task switching ▫Keeps you in the groove Faster builds encourage quick iteration ▫Which encourages more testing

15. Fixing Programmer Burn

16. Fixing Programmer Burn (Easy) Parallelize your build (Incredibuild, distcc etc) ▫Very Little investment, can see huge returns ▫Little to no programmer involvement (unless you’re switching build systems)

17. Fixing Programmer Burn (Easy) Dedicate more hardware ▫2 programmers wasting 30 minutes per day pays for a beefy server. ▫Little to no programmer involvement, instant returns ▫2 programmers wasting 30 minutes per day costs you $7800, which buys one beefy server

18. Story Time

19. Fixing Programmer Burn (Medium) Fix dependencies, modularize code ▫Modularized code is faster to compile and easier to test ▫Reduce dependencies between modules. ▫Minimize unnecessary includes, use forward declarations. ▫Make your inheritance chains shallow.

20. Fixing Programmer Burn (Medium) Use Precompiled Headers (when you can) ▫Don’t put every header into precompiled headers. It’s silly and it annoys programmers more than anything else ▫Only place header that are significantly large, used very frequently, and unlikely to change ▫Put in guards to avoid PCH dependence

21. Fix Programmer Burn (Hard) Use a fast compiling or interpreted language in high level code, one that doesn’t require includes or linking. ▫C#, Lua, Python, JavaScript, Scheme, UnrealScript?

22. Fix Programmer Burn (Hard) Refactor code into individually testable libraries ▫Makes much of your code reusable ▫Allows testing of single library units ▫Can shorten iteration time on that library / unit ▫Going too crazy with this can increase link times

23. Fix Programmer Burn (Hard) Move libraries that don’t change out of your build process ▫Goes with modularizing your libraries. ▫Allows testing of individual modules to be issolated. ▫Beware this suggestion. Frequently teams that do this mistakenly think “rarely edited” means “never edited”

24. You’re so vain… Bad builds aren’t just about lost programmer time

25. Deployed Builds Broken Builds vs. Bad Builds Important to consider when they affect the productivity of the team ▫If a team can’t work because of a because the build is bad, this is all lost time.

26. Your Team Burn Chart Number of Team Members Hours lost Per Day Money Lost Per Day Month 301$600$18000 602$2400$72000 308$4800$144000

27. How do we prevent this?

28. Jeff’s Tenets of Good Builds Be fast Be automated Keep the team informed Leave little room for human error Prevent “bad builds” Anyone’s

29. … for some of them, you’ll never get there.

30. Be fast Use continuous integration to build on each check-in, unit test all builds, and analyze code. Create up to date and optimized assets on each check-in Build assets incrementally, use source control to trigger only specific builds in assets Use dependency trees to figure out what needs to be rebuilt as a result of asset changes.

31. Be Automated Have check-ins occur at the lowest possible point, automate the rest. ▫This means max, maya, psd. Automate informing the team of changes Automate testing ▫Unit, functional, approval, monkey Automate approval and deployment of good builds ▫Where it makes sense UNLIKELY!

32. Corollary: Remove things that are slow Integration tests ▫CPP: 00:09:14 ▫CS: 00:00:13 ▫Generator: 00:16:26 ▫Java: 00:15:44 Smoke Tests Developer Approval

33. Keep the Team Informed How long until the next build? What changes does the next build include? Do I need this build? The team should know builds are bad BEFORE they get it.

34. Human Error Automate export and build. ▫This reduces human interaction, and prevents build inconsistencies. Isolate mistakes from the rest of the team ▫We all make mistakes, we all check in bad code or bad assets ▫Keep these changes from dispersing in the wild, which would cause hours of wasted time.

35. Prevent Bad Builds

36. Create a Pipeline A build is not one step! ▫Get from source control ▫Build ▫Automated Unit Tests ▫Automated Integration Tests ▫Deploy to team ▫QA Smoke Tests ▫User Tests ▫Deploy to testers ▫Deploy to beta ▫Deploy to production

37. The Pipeline

38. Per Project vs. Per Company Most code optimizations are project based. Pipelines can be reused

39. No no no, nothing about this adds up at all!

40. Your Programmer Burn Number of Programmers Hours lost Per Day Money Lost Per Day WeekYear 51$150$750$39000 101$300$1500$78000 Creating your build system takes time. Time saved > Time spent This makes sense only on large projects losing lots of time to builds Don’t slack

41. Build Complete http://www.jeffongames.com http://www.thetoolsmiths.com jeff@fuzzybinary.com