5 Interview: Jim Johnson, Standish Group Chairman - Aug 2006 InfoQ: Your work on project failure is really a search for “how to succeed”, isn't it? I noticed your list of Project Success Factors includes #5: Agile Development. As in, Agile Software Development?JJ: Oh, absolutely! I'm a big believer in Agile, having introduced the iterative process in the early 90s and followed with the CHAOS reports on quick deliverables. We're a real flag waver for small projects, small teams, Agile process. Kent Beck has talked at CHAOS University, and I have talked at his seminars. I am a real fan of XP. My new book “My Life is Failure” looks at Scrum, RUP, XP and talks about those methods.GD: Agile has helped by chunking projects into small pieces. Even if you start to get it wrong, you know it early on. Not like the old waterfall projects where you go into a cave and find out the bad news two years later...JJ: I think that's the secret - incremental. I think that's why we're seeing improvement. A big problem was project bloat, causing projects to go over time, over budget, and creating features and functions not required. Especially in government. Agile really helps this - small increments. You know: they say “I need this”, but by next sprint: “it's not as important as I originally thought it was.”GD: Yes, prioritizing features helps... always asking: what's the business advantage? Identifying low value items and putting them at the end... and then maybe the project never gets there, so you didn't need them after all.InfoQ: Agile must bring in new issues: how do you say when “planned” scope is accomplished, for a project using adaptive planning?JJ: That's a good question. With companies like Webex, Google, Amazon, eBay, it's a challenge - they're doing something called “pipelining” instead of releases. “Whatever is done in 2 weeks, we'll put that up.” They're successful because users only get small, incremental changes. And their implementation is proven - remember, some projects only fail after the coding is finished, at the implementation step.[Gordon Divitt, Executive Vice President of Operations for the Acxsys Corporation]
7 Principles behind the Agile Manifesto Principles behind the Agile ManifestoOur highest priority is to satisfy the customer through early and continuous delivery of valuable software.Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.Business people and developers must work together daily throughout the project.Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.Working software is the primary measure of progress.Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.Continuous attention to technical excellence and good design enhances agility.Simplicity--the art of maximizing the amount of work not done--is essential.The best architectures, requirements, and designs emerge from self-organizing teams.At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.7
8 Agile methods are a family of development processes …not a single approach to software development. In 2001, 17 prominent figures in the field of agile development (then called “light-weight methodologies”) came together at the Snowbird ski resort in Utah to discuss ways of creating software in a lighter, faster, more people-centric way. They created the Agile Manifesto, widely regarded as the canonical definition of agile development, and accompanying agile principles.Some of the principles behind the Agile Manifesto are:Customer satisfaction by rapid, continuous delivery of useful softwareWorking software is delivered frequently (weeks rather than months)Working software is the principal measure of progressEven late changes in requirements are welcomedClose, daily, cooperation between business people and developersFace-to-face conversation is the best form of communicationProjects are built around motivated individuals, who should be trustedContinuous attention to technical excellence and good designSimplicitySelf-organizing teamsRegular adaptation to changing circumstancesThe publishing of the manifesto spawned a movement in the software industry known as agile software development.In 2005, Alistair Cockburn and Jim Highsmith gathered another group of people — management experts, this time — and wrote an addendum, known as the PM Declaration of Interdependence.From Wikipedia 5/28/078
9 The Known Agile Processes Spiral – Barry Boehm (1988) Evo – (Evolutionary Project Management) – Tom Gilb (1988) RAD (Rapid Application Development) – James Martin (1991) DSDM (Dynamic Systems Development Method) – DSDM Consortium (1995) SCRUM – Ken Schwaber (1996) RUP (Rational Unified Process) – Booch/Jacobson/Rumbaugh (1998) XP (Extreme Programming) – Kent Beck (1999) ASD (Adaptive Software Development) – Jim Highsmith (1999) FDD (Feature Driven Development) – Jeff DeLuca (1999) (Agile Manifesto – 2001) Lean Software Development – Mary and Tom Poppendieck (2003) Crystal Methodologies – Alistair Cockburn (2004) AUP (Agile Unified Process) – Scott Ambler (20??) Pipelining / Perpetual Beta – ??? (???)MostpopularName shown is strongly associated with the concept and writes on itsemployment extensively; often, not always, is considered the “inventor/founder”
10 “A Spiral Model of Software Development and Enhancement,” Barry W “A Spiral Model of Software Development and Enhancement,” Barry W. Boehm, Computer, IEEE, 1988WaterfallVee LeftVee BaseVee RightVee Model“The spiral model was defined by Barry Boehm in his 1988 article A Spiral Model of Software Development and Enhancement. This model was not the first model to discuss iterative development, but it was the first model to explain why the iteration matters. As originally envisioned, the iterations were typically 6 months to 2 years long. Each phase starts with a design goal and ends with the client (who may be internal) reviewing the progress thus far. Analysis and engineering efforts are applied at each phase of the project, with an eye toward the end goal of the project. [Wikipedia, May 28, 2007]
11 The Spiral ModelThe steps in the spiral model can be generalized as follows:The new system requirements are defined in as much detail as possible. This usually involves interviewing a number of users representing all the external or internal users and other aspects of the existing system.A preliminary design is created for the new system.A first prototype of the new system is constructed from the preliminary design. This is usually a scaled-down system, and represents an approximation of the characteristics of the final product.A second prototype is evolved by a fourfold procedure: (1) evaluating the first prototype in terms of its strengths, weaknesses, and risks; (2) defining the requirements of the second prototype; (3) planning and designing the second prototype; (4) constructing and testing the second prototype.At the customer's option, the entire project can be aborted if the risk is deemed too great. Risk factors might involve development cost overruns, operating-cost miscalculation, or any other factor that could, in the customer's judgment, result in a less-than-satisfactory final product.The existing prototype is evaluated in the same manner as was the previous prototype, and, if necessary, another prototype is developed from it according to the fourfold procedure outlined above.The preceding steps are iterated until the customer is satisfied that the refined prototype represents the final product desired.The final system is constructed, based on the refined prototype.The final system is thoroughly evaluated and tested. Routine maintenance is carried out on a continuing basis to prevent large-scale failures and to minimize downtime.Applications: The spiral model is used most often in large projects. For smaller projects, the concept of agile software development is becoming a viable alternative. The US military has adopted the spiral model for its Future Combat Systems program.Advantages:Estimates (i.e. budget, schedule, etc.) get more realistic as work progresses, because important issues are discovered earlier.It is more able to cope with the (nearly inevitable) changes that software development generally entails.Software engineers (who can get restless with protracted design processes) can get their hands in and start working on a project earlier.From Wikipedia 5/28/07
12 RUP Graphic Fairly Indicative of Agile Processes Fine print per Wikipedia: This is a screenshot of copyrighted computer software, and the copyright for its contents is most likely held by the author(s) or the company that created the software. It is believed that the use of a limited number of web-resolution screenshots qualifies as fair use under United States copyright law. The use of this image in educational material describing the Rational Unified Process does not diminish nor infringe upon the potential of IBM to profit from the use of this image or other copyright components of the Rational Unified Process through the sale of software, educational material or consulting services.From Wikipedia 5/28/07
13 CMMI Capability Maturity Model Integration Wikipedia 5/28/07:A growing body of software development organizations implement process methodologies. Many of them are in the defense industry, which in the U.S. requires a rating based on 'process models' to obtain contracts. The international standard for describing the method of selecting, implementing and monitoring the life cycle for software is ISOThe Capability Maturity Model (CMM) is one of the leading models. Independent assessments grade organizations on how well they follow their defined processes, not on the quality of those processes or the software produced. CMM is gradually replaced by CMMI.…The CMMI is the successor of the CMM. The CMM was developed from 1987 until In version 1.1 of the CMMI was released: v1.2 followed in August The goal of the CMMI project is to improve usability of maturity models for software engineering and other disciplines, by integrating many different models into one framework.
14 CMMI Maturity Levels“Introduction to the Capability Maturity Model Integration (CMMI),” Eric Buchholtz and Andy Cordes, SPIN Meeting, 2003,
15 CMMI Practices vis-à-vis Agility “LEVEL 1”Identify scope of workPerform the work“LEVEL 2”Organizational policy for plan, performRequirements, objectives and plansAdequate resourcesTrain the peopleAssign responsibility and authorityCM for designated work productsIdentify and involve stakeholdersMonitor and control to plan and take action if neededObjectively monitor adherence to process and QA products/servicesReview with upper management and resolve issuesKEY GREEN : Supportive, Black: Neutral, RED: Rough EdgesAppurva Jain, Annual Research Review & Executive Workshop Post Workshop Progress Report, March 14, 2002,
16 CMMI Practices vis-à-vis Agility “LEVEL 3”Maintain as a defined processMeasure the process performance to support environment“LEVEL 4”Establish and maintain quantitative objectives for the processStabilize the performance of one or more sub-processes to determine its ability to achieve“LEVEL 5”Ensure continuous improvement to support business goalsIdentify and correct root causes of defectsKEY GREEN : Supportive, Black: Neutral, RED: Rough EdgesAppurva Jain, Annual Research Review & Executive Workshop Post Workshop Progress Report, March 14, 2002,
17 Ken Schwaber's letter to IEEE Computer I read with dismay “The Agile Methods Fray”, where two of the luminaries of software processes discuss traditional (defined) and agile approaches. The discussion was irrelevant to those attempting to understand the distinction. A sentence characterized the apparent purpose of the article, “ …found a sensible middle ground and identifying some baby to be saved and some bathwater to be replaced.” There is no middle ground between traditional and agile processes. The practices of traditional software development processes are inadequate to control projects with complex technology and sophisticated requirements. Agile processes are based on empirical process control, a technique widely adapted by competitive manufacturing and development environments over the last twenty years. I quote from the bible of process control (Process Dynamics, Modeling, and Control, Ogunnaike and Ray, Oxford University Press, 1992), “It is typical to adopt the defined (theoretical) modeling approach when the underlying mechanisms by which a process operates are reasonably well understood. When the process is too complicated for the defined approach, the empirical approach is the appropriate choice.” Empirical process control relies on frequent inspection and continuous adaptation to minimize risk and produce quality product. Agile processes implement empirical process control through iterations, frequent increments of working, tested functionality, emergence of requirements and architecture, self-organization of multiple small teams, and collaboration. These are not spot practices shared by defined and agile processes since the underlying theory is different … using a defined approach for a complex problem is like using algebra to solve complex, non-linear problems. We indeed are in the middle of a revolution, as we shed traditionally weak inadequate practices and adopt agile processes. The issue isn’t merging the two, but successfully managing the change. Those who wish to tinker with either to reach a middle ground tread a dangerous path toward misleading those who rely on them for informed advice. Ken Schwaber- June 16, 2002
18 Are SE tools in conflict with reality? ProblemSolutionAnalyze dataFormulate solutionImplement solutionWaterfall MethodActual DesignerGather dataAre SE tools in conflict with reality?"..the MCC Elevator Study is significant because, for the first time, we have a model of the process that people actually follow when they tackle hard problems. And it is not the orderly, linear process we have assumed is proper."The non-linear pattern of activity that expert designers follow gives us fresh insight... It reveals that in normal problem-solving behavior, we may seem to wander about, making only halting progress towards the solution."This non-linear process is not a defect, not a sign of stupidity or lack of training, but rather the mark of a natural learning process. It suggests that humans are oriented more toward learning (a process that leaves us changed) than toward problem solving (a process focused on changing our surroundings).Wicked Problems: Naming the Pain in Organizations, E. Jeffrey Conklin & William Weil,
19 ControversyAgile development has been widely documented as working well for small (<10 developers) co-located teams. Agile development is expected to be particularly suitable for teams facing unpredictable or rapidly changing requirements.Agile development's applicability to the following scenarios is open to question:Large scale development efforts (>20 developers), though scaling strategies and evidence to the contrary have been described.Distributed development efforts (non-co-located teams). Strategies have been described in Bridging the Distance and Using an Agile Software Process with Offshore DevelopmentMission- and life-critical effortsCommand-and-control company culturesIt is worth noting that several large scale project successes have been documented by organizations such as BT which have had several hundred developers situated in the UK, Ireland and India, working collaboratively on projects and using Agile methodologies.While questions undoubtedly still arise about the suitability of some Agile methods to certain project types, it would appear that scale or geography, by themselves, are not necessarily barriers to success.From Wikipedia 5/28/07
20 Agile Software Development – The Process “A Practical Guide to Seven Agile Methodologies,” Rod Coffin and Derek Lane You know that Agile methodology is the right thing to do, but trying to parse all the different methodologies is a major research endeavor. How to know which one is right for your organization? In this two-part article, you'll learn all the ins and outs of the seven most popular methodologies so you can pick the one that's best for you. In part 1, we cover XP, Scrum, Lean, and FDD. In part 2, we cover AUP, Crystal, and DSDM.