Now that you know that… How long is your average incremental build? How many times, per day, is your programmer going to perform that build? By how much can you improve that? The difference is wasted time!!!
The Money Assumptions A 52 week year A 5 day work week A $62k average salary (~$30/hr) This does not include other overhead for a programmer.
The Programmer Burn Chart Number of Programmers Hours lost Per Day Money Lost Per Day WeekYear 1.5$10$50$2600 2.5$30$150$7800 51$150$750$39000 101$300$1500$78000 301$900$4500$234000 302$1800$9000$468000
…you can lose approximately $468,000, the cost of 2 lead programmers, to build times.
You Programmer Burn (cont.) Number of Programmers Hours lost Per Day Money Lost Per Day WeekYear 51$150$750$39000 101$300$1500$78000 Two way to look at these numbers: Money Lost (in programmer productivity time) Money Saved (by increasing productivity time)
Saving Money, Saving Time Faster builds save more than just build time ▫Keeps programmers from task switching ▫Keeps you in the groove Faster builds encourage quick iteration ▫Which encourages more testing
Fixing Programmer Burn (Easy) Parallelize your build (Incredibuild, distcc etc) ▫Very Little investment, can see huge returns ▫Little to no programmer involvement (unless you’re switching build systems)
Fixing Programmer Burn (Easy) Dedicate more hardware ▫2 programmers wasting 30 minutes per day pays for a beefy server. ▫Little to no programmer involvement, instant returns ▫2 programmers wasting 30 minutes per day costs you $7800, which buys one beefy server
Fixing Programmer Burn (Medium) Fix dependencies, modularize code ▫Modularized code is faster to compile and easier to test ▫Reduce dependencies between modules. ▫Minimize unnecessary includes, use forward declarations. ▫Make your inheritance chains shallow.
Fixing Programmer Burn (Medium) Use Precompiled Headers (when you can) ▫Don’t put every header into precompiled headers. It’s silly and it annoys programmers more than anything else ▫Only place header that are significantly large, used very frequently, and unlikely to change ▫Put in guards to avoid PCH dependence
Fix Programmer Burn (Hard) Refactor code into individually testable libraries ▫Makes much of your code reusable ▫Allows testing of single library units ▫Can shorten iteration time on that library / unit ▫Going too crazy with this can increase link times
Fix Programmer Burn (Hard) Move libraries that don’t change out of your build process ▫Goes with modularizing your libraries. ▫Allows testing of individual modules to be issolated. ▫Beware this suggestion. Frequently teams that do this mistakenly think “rarely edited” means “never edited”
You’re so vain… Bad builds aren’t just about lost programmer time
Deployed Builds Broken Builds vs. Bad Builds Important to consider when they affect the productivity of the team ▫If a team can’t work because of a because the build is bad, this is all lost time.
Your Team Burn Chart Number of Team Members Hours lost Per Day Money Lost Per Day Month 301$600$18000 602$2400$72000 308$4800$144000
Be fast Use continuous integration to build on each check-in, unit test all builds, and analyze code. Create up to date and optimized assets on each check-in Build assets incrementally, use source control to trigger only specific builds in assets Use dependency trees to figure out what needs to be rebuilt as a result of asset changes.
Be Automated Have check-ins occur at the lowest possible point, automate the rest. ▫This means max, maya, psd. Automate informing the team of changes Automate testing ▫Unit, functional, approval, monkey Automate approval and deployment of good builds ▫Where it makes sense UNLIKELY!
Corollary: Remove things that are slow Integration tests ▫CPP: 00:09:14 ▫CS: 00:00:13 ▫Generator: 00:16:26 ▫Java: 00:15:44 Smoke Tests Developer Approval
Keep the Team Informed How long until the next build? What changes does the next build include? Do I need this build? The team should know builds are bad BEFORE they get it.
Human Error Automate export and build. ▫This reduces human interaction, and prevents build inconsistencies. Isolate mistakes from the rest of the team ▫We all make mistakes, we all check in bad code or bad assets ▫Keep these changes from dispersing in the wild, which would cause hours of wasted time.
Create a Pipeline A build is not one step! ▫Get from source control ▫Build ▫Automated Unit Tests ▫Automated Integration Tests ▫Deploy to team ▫QA Smoke Tests ▫User Tests ▫Deploy to testers ▫Deploy to beta ▫Deploy to production
Your Programmer Burn Number of Programmers Hours lost Per Day Money Lost Per Day WeekYear 51$150$750$39000 101$300$1500$78000 Creating your build system takes time. Time saved > Time spent This makes sense only on large projects losing lots of time to builds Don’t slack