Software Development The Good, The Bad (and Ugly) David Kaminsky Dave Ogle

What We’ll Talk About Some job experiences that we’ve had –Startup (internet technology) –Storage product –Networking product Why we’re telling you this –Diane had nothing better to fill the time … –Or, to relate some real-world experiences … –You decide…

The Good - Startup What was done: –Generated running server and client software in short period of time (< 6 months) –Server code ran on production site, 24x7 for 12 months with no outages (due to code) –Client code ran on 1000’s of client machines –Updates were done to both code bases with minimal disruption –Small team of 10 or so developers/architects/DBAs/etc. Started with a very small team and grew

The Good - Startup What worked: –Architecture/Design Had an architecture and a design Spent time up front understanding what the problem was, and what the design/architecture should be – good interaction with marketing team Spent time as development was going on to review and tweak the design/architecture Open and frank conversation between the team Closed room architecture resolution policy Open evaluation of skills of team members Hired a project manager early on – structure and discipline Willing to throw out code that was hard to understand and structured ‘uniquely’ –Willing to start over when necessary. Second time is usually easier What didn’t work –The ‘harold’ factor: toss code that doesn’t work –Airclic factor -- no market

The Bad: Storage Product What was done: –Large network storage project: Goal was to generate code for storage servers Code was to be done in conjunction with HW development Large team of 60+ developers/architects/testers/etc. But: code quality was poor … really poor What would you do?

The Bad: Storage Product What worked –Assigned a tiger team to solve a specific sub- problem (once) –Team had good design skills –Good communication skills among team –Small team – 5 developers/architects –Code delivered on time, ran well –Team did prototype first, threw that out and started over

The Ugly Warning signs that a project may be in trouble: –Company buys dinner for team every night for months on end Some developers arrange their schedule to maximize free dinners, not code output or quality. –When you ask the chief architect a simple architecture question and get a 30 minute soliloquy that makes no sense. If you ask him again, you get a different soliloquy –When you ask how a key component/module works you get a the answer “my job is just to make it work, not to understand how it works” from the person responsible for the module Developers don’t ask the architect – fear and boredom. –When you join the project and the lead developer comes into your office to tell you to stay out of his way –When the build breaks on a nightly basis due to programmer errors –When every component in the system duplicates the same code (eg, parsing config XML) … and all do it poorly.

Why things failed What didn’t work –No real design for the system –No real understanding of customer –No time spent up front to design –Team members didn’t understand the goals (large or small) –Problems addressed as they came up – band aid approach used –Key lead people were in over their heads Early turnover in technical leadership –General belief that putting in more hours coding solves basic design flaws

Networking example What was done: –Implementation of a networking protocol on PCs –Developed by a small team from a reference codebase What worked: –Architecture/Design Had an architecture and a design – see below Some skilled developers Good team dynamics What didn’t work –Key component assigned to an uninterested person (wanted to be writing academic papers, not code) All components worked approx on schedule … except that one –Limited organizational support: had to re-write modules that existed elsewhere –Bad architecture assumptions: algorithm required a 1ms clock; OS provided only 20ms clock. Algorithm became unstable.

Conclusions Time spent in design saves time in development and test –By analogy, when building a house, is your first stop: 1.to the architect to get the plans? 2.to the hardware store to buy supplies? Coding can’t solve design issues Don’t be afraid to toss out what you have and start all over –Patching broken code only makes it worse –Starting from scratch saves times in the long run, think about the long run! Admit what you don’t know –Bravado is great, but it machines don’t recognize it…..