1. Automating Software Production Processes Kevin Jameson, Founder

2. History of Multics, Unix, Emacs, Codefast 1965-1969 Multics –MIT Project MAC, GE, Bell Labs, Naming culture of the time

3. History of Multics, Unix, Emacs, Codefast 1965-1969 Multics –MIT Project MAC, GE, Bell Labs, Naming culture of the time 1971 Unix –Was another joke name (“GNU is Not Unix”, “EINE Is Not Emacs”) –Unix = Eunuchs = “Multics without balls”

4. History of Multics, Unix, Emacs, Codefast 1965-1969 Multics –MIT Project MAC, GE, Bell Labs, Naming culture of the time 1971 Unix –Was another joke name (“GNU is Not Unix”, “EINE Is Not Emacs”) –Unix = Eunuchs = “Multics without balls” 1976 TECO Emacs (Tape/Text Editor COrrector) –[1 J^P$L$$ J $$ –Richard Stallman, “Editor Macros” package -> Emacs

5. History of Multics, Unix, Emacs, Codefast 1965-1969 Multics –MIT Project MAC, GE, Bell Labs, Naming culture of the time 1971 Unix –Was another joke name (“GNU is Not Unix”, “EINE Is Not Emacs”) –Unix = Eunuchs = “Multics without balls” 1976 TECO Emacs (Reconceptualized what an editor was) –Richard Stallman, “Editor Macros” package -> Emacs 1978 Multics Emacs –Bernie Greenberg, Multics Emacs in MacLisp

6. History of Multics, Unix, Emacs, Codefast Multics Emacs -> Codefast –1978 Bernie Greenberg, Multics Emacs in MacLisp –1979 U of Calgary buys a Honeywell Multics System –1982 Jameson, Codefast v0001 (Multics Emacs Lisp), U of Calgary Multics Emacs -> Unix Emacs -> GNU Emacs –1978 Bernie Greenberg, Multics Emacs in MacLisp –1979 U of Calgary buys a Honeywell Multics System –1981 Gosling, Gosling Emacs (Unix C, MockLisp), U of Calgary –1985 Stallman, GNU Emacs (Unix C, GNU Lisp), MIT –2006 GNU Linux, Emacs, Java, Modern IDE Editors (Eclipse, etc)

7. Initial Codefast Concept Main Concept – A Program to Write a Program –Possibility of perfect quality from generated code – no bugs! –Possibility of perfect speed – computer speeds, not human speeds Main Blocking Point – Software Build Processes –Could generate code, but could not build it automatically –Software builds, especially multiplatform builds, were big blocks 1994 O’Reilly Book – Multiplatform Code Management –Showed how to automatically generate build scripts, makefiles 2004 Software Lifecycle Automation Book –Applied principles to software lifecycle processes, not just builds

8. The Innovation—Production Cycle

9. Death by Variations

10. Codefast Focus

11. Typical Production Pains Constant Changes of All Kinds Break Dependencies Increase Iterations Use Up Human Labor Slow Down Cycle Times

12. What Codefast Does Puts the Production Cycle in a “Black Box” Automates the Internals Gives It a User Interface Removes Human Labor Increases Quality of Cycle Processes

13. Codefast User Interface – 2 Modes 1. Give Orders 1. Get Files 2. Modify Files 3. Put Files

14. Codefast Solution Structure

15. So What Happened? ? Production Cycle + Codefast = What Happened?

16. Pause to Reflect On The Possible Outcomes… Automation chaos worse than existing human chaos? Limited scope of application? Only good for build function, but useless for test function? Only good for Function X? Language X? Platform X? Only good for Situation X? Variation X? It works, but has low quality success rates? 50%? 75%? It works, but is too slow? No speedup? No punch? So many ways to fail, or to be limited in speed or quality A tough game, right?

17. How Do You Measure Quality Success? σ

18. Perfect Quality – Our Best Moment 5 Months, 1 MLOC, 7688 Checkins, Zero Build Failures

19. Our Sigma Graph Over Time (~2 Years)

20. Codefast Perpetrator Report Reason Count (22 failures in about 1 year) Compile-dep7(one piece depends on another) Compile4(compile failure on other platform) Sys Admin3(bad shutting down machines) Bug-rollover2 (special case defect in system) Unknown2(network - existing file “not there”) Missing-coll1(forgot to checkin a collection) Compile-set1(forgot a piece of the change set) Config files1(bad knowledge base change) Hardware1(machine/disk/network crash)

21. Our Track Record 1.75 Years, 1+ MLOC, 31541 Checkins, Never Below 99.99%

22. Technical Issues and Solutions “BIG” in Software Development Is Hard to Do –BIG doesn’t work well --- not BIG SIZE, not BIG PROCESS –BIG doesn’t scale well for speed, manageability, or quality –BIG is too slow, too complex, too many dependencies, too hard Learn to Work with Pieces – of Structure, Process, Responsibility –Must break down big code tree structures into smaller parts –Must break big recursive make processes into smaller parts –Must break human responsibilities into smaller parts –Then process parts in automated, parallel, distributed, incremental ways Structure, Process, and Responsibility Pieces –All three areas must be broken into pieces and combined together

23. Structure Pieces The Structural Granularity Problem –Big granules (such as big code trees) don’t work very well –Too big, too complex, too slow, too many dependencies –Too much surface area and exposure to failures –Can’t do parallelism or incrementalism easily on one big piece –Each granule is custom – it is irregular in size, language, processes A Granularity Solution – Collections –A generic “container” for code, files, knowledge of all types –Collections are like code objects, but are stored on a hard disk –They have a data type that determines permissible operations –Collections can do per-instance overrides of various kinds too –Internal collection structure is flexible –Multi-language, Multi-platform, Multi-Policy, Multi-Site, Multi-everything

24. Structure Pieces and Typical Examples Structure Pieces – Examples –Split builds by platform? No, that is just 1 big granule per platform –Directories in recursive make structures – ok, but weak pieces –Multiple IDE project files – ok, but weak pieces –Codefast collections map on to make directories or IDE trees – good Good Indicators of Structure Pieces –Pieces have individual names (identity by location vs identity by name) –Pieces have individual properties and custom overrides –Pieces can be relocated without breaking processes (abs pathnames) –Pieces can be shared and aggregated into different projects –Good pieces are like good objects, inherit, share, override, etc

25. Process Pieces The Process Pattern Problem –Scope of pattern is usually local in nature, not-sharable, not scalable Pattern usually belongs to a single program (make, ant, IDE) –Quality of overall process is limited by the person that writes the code Complexity – Every instance or combo of patterns is different –Responsibility for process pattern application is limited by human abilities Humans usually code every line in every make / ant tree structure A Process Pattern Solution – Knowledge Base + Generators –Scope -- Globally shared outside all project makefiles or IDE boundaries –Quality -- Automated application of global patterns to structural pieces –Responsibility – Human labor vs perfect parallel, incremental processes

26. Process Pieces and Typical Examples Process Pieces - Examples –Local make/ant rule templates – ok, but weak local, often language specific, usually tied to a particular program –Table-of-process-steps GUI tools – ok, but weak apply patterns to micro steps in a serial build – Best indicator is global process patterns that are automatically applied across N processes, N platforms, N machines, N configurations Good Indicators of Process Pieces –Pieces have individual names (identity by location vs identity by name) –Pieces have individual properties and custom behaviors –Pieces can be relocated without breaking processes (abs pathnames) –Pieces can be shared among multiple aggregated processes –Good pieces are like good objects, inherit, share, override, etc

27. Dependencies Among Pieces The Three Big Dependency Problems 1.Dependencies among structure pieces (structure  structure) Code A depends on code B 2.Dependencies among process pieces (process  process) Process A puts a file in location B, process C picks file up from B 3.Dependencies among structure & process pieces (structure  process) Change a structure piece, break the build process Change a process piece, and it won’t fit the old structure A Dependency Solution – Extraction and Generation –Automatically extract dependencies among sets of structure pieces –Automatically generate custom processes to fit the current structure pieces –This way, processes always fit structures, regardless of ongoing change

28. Build Speed Solution #1 The Speed Problem 1.Must have structure pieces 2.Must have process pieces 3.Must calculate dependencies among pieces 4.Must coordinate multiple computers to work on pieces Speed Solution #1 – PerfectBuild Express – (CF-1) –Modify or extend existing code tree to make better structure pieces –Use existing processes (makefiles, antfiles, IDE project files) –Automatically extract dependencies from pieces –Automatically calculate parallel / incremental build order among pieces –Automatically calculate, schedule, and execute parallel builds –Benefit is faster time to value (parallel / incremental speed) –Less migration work required – Migration is simpler and faster

29. Build Speed Solution #2 The Speed Problem 1.Must have structure pieces 2.Must have process pieces 3.Must calculate dependencies among pieces 4.Must coordinate multiple computers to work on pieces Speed Solution #2 – PerfectBuild Enterprise – (CF-2) –Modify or extend existing code tree to make better structure pieces –Automatically calculate custom process pieces –Automatically extract dependencies from pieces –Automatically calculate parallel / incremental build order among pieces –Automatically calculate, schedule, and execute parallel builds –Benefit is maximum value (parallel, incr speed, + Six Sigma quality) –More migration work is required – but more value is gained

30. Software Build Approaches 1.No structure pieces, no process pieces, serially on N machines More throughput, same speed, same quality 2.No structure pieces, no process pieces, parallel on N machines More throughput, more speed, same quality Only available for particular platform, language, tooling scenarios 3.Yes structure pieces, Your process pieces on N machines More throughput, more speed, same quality General solution, platform, language and tooling independent Parallel (10x) and incremental (100x+) builds 4.Yes structure pieces, Yes process pieces on N machines Maximum throughput, speed (parallel, incremental) Maximum quality (Six Sigma)

31. Vendor Matrix StructureProcessDepSpeedSpeedSigma VendorPiecesPiecesCPUsTputMgmtParaIncrQuality Baseline--1----- Vendor-1--N---- Vendor-2--N-- CF-1YoursN- CF-2N 1x10x100x

32. Build Speed Measurements Difficult to Measure in a Standard Way –Under What Conditions? –Specific to Particular Customer Trees, Tools, Situations, Policies CF-1 PerfectBuild Express –Use Codefast structure pieces, Use Customer Process Pieces –Best incremental speed measured at customer site –About 75x (7h 40min -> 6 minutes) CF-2 PerfectBuild Enterprise –Use Codefast structure pieces, Codefast process pieces –Best incremental speed measured at Codefast site –About 486x

33. Preferred First Implementation StructureProcessDepSpeedSpeedSigma VendorPiecesPiecesCPUsTputMgmtParaIncrQuality Baseline--1----- Vendor-1--N---- Vendor-2--N-- CF-1YoursN- CF-2N 1x10x100x Starting with CF-1 Is the Easiest Way

34. Some Closing Thoughts Learning to work with pieces, parallelism, incrementalism –Small, Lightweight, Extremely Fast, but requires coordination Learning to work with Six Sigma failure rates (~ zero) –Must learn to trust the system more –Like retirement—you must learn to do other things with your time 31,541 checkins over 1.75 years = ~20,000 checkins per year –20,000 checkins means 20,000 builds / test runs per year –If you can save 1 or 2 hours on each one, with 100x speedup, –That’s 20,000 hours saved out of your production cycle –With 99.99% perfect results

35. You Can Do This Too 1.75 Years, 1+ MLOC, 31541 Checkins, Never Below 99.99%

36. Our Product Name