1. Software Life Cycles: Evolution, Maintenance, and Perpetual Development Dror Feitelson Software Engineering Hebrew University

2. David Lorge Parnas “A sign that the Software Engineering profession has matured will be that we lose our preoccupation with the first release and focus on the long term health of our products.” Software Aging, ICSE 1994 Major contributor to information hiding and modularization Advocate of software development as an engineering discipline Including good documentation! Opponent of “star wars” Fellow of ACM, IEEE

3. Software Lifecycle Req'sAnalysisDesignConstructionTesting Transfer Maintenance Textbook view:

4. Software Lifecycle Req'sAnalysisDesignConstructionTestingTransfer Maintenance Textbook view: More realistic view: Req'sAnalysis Design Construction Testing Transfer EVOLUTION, MAINTENANCE, AND ADDITIONAL RELEASES

5. Software Lifecycle Req'sAnalysisDesignConstructionTestingTransfer Maintenance Textbook view: More realistic view: Req'sAnalysis Design Construction Testing Transfer EVOLUTION, MAINTENANCE, AND ADDITIONAL RELEASES “75% of the effort in a software project is spent on maintenance”

6. Meir (Manny) Lehman On team Building first computers in Israel in 1950s Worked at IBM studying OS/360 Professor at Imperial College London Received Harlan Mills award Passed away 29.12.10 in Jerusalem “Father of software evolution” Defined “E-type” systems that are ingrained with their environment Lehman's 8 Laws describe how such systems evolve

7. Lehman's Laws 1)Continuing change (adaptation) 2)Increasing complexity (unless refactored) 3)Self regulation (of rate of change) 4)Invariant work rate (inertia) 5)Conservation of familiarity (of users and developers) 6)Continuing growth (more features) 7)Declining quality (unless maintained) 8)Feedback system (at multiple levels)

8. All projects TerminatingEvolutionary Project defined by its goals Requirements map to acceptance test Develop-deliver- maintain trajectory Goals are uncovered as project progresses Continue indefinitely as long as useful Multiple releases along the way

9. All projects TerminatingEvolutionary Repeated cycles Each one is planned with well-defined goals Staged model Perpetual development Plan as you go Concurrent sub-projects Fuzzy long- term goals

10. All projects TerminatingEvolutionary Staged model Perpetual development Single delivery Multiple releases Continuous deployment Possible updates Feasibility Architecture Operational Three milestones after maintenance

11. Timescales and Terminology Waterfall / unified process Evolutionary development Agile development Facebook Continuous deployment – Once – Months – Weeks – One day – < Hour – delivery } Release } Deployment Maintenance  Evolution Requirements  Feature requests

12. Perpetuity Part of evolutionary development (and many agile and open source projects) New mindset: project will go on forever Useless to make detailed long-range plans Needs to be sustainable Normal work rate Initial development becomes progressively smaller part of the project “Maintenance is 75% of investment” is meaningless

13. Evolution and Agile All software projects TerminatingEvolutionary Agile Perpetual Agile can be terminating Agile implies a process Evolutionary/ perpetual may be process- less e.g. Linux Staged

14. Staged Model Each version passes conventional phases E.g. full cycles of Unified Process New versions branch in parallel to maintenance Rajlich & Bennett, Computer 2000

15. Perpetual Development Lifecycle time size initial development

16. Perpetual Development Lifecycle time size initial development release Production use & maintenance

17. Perpetual Development Lifecycle time size initial development release continued development Production use & maintenance feedback & requests

18. Perpetual Development Lifecycle time size initial development release continued development Production use & maintenance feedback & requests users upgrade

19. Perpetual Development Lifecycle time size initial development release continued development continued development Production use & maintenance feedback & requests feedback & requests users upgrade

20. Perpetual Development Lifecycle time size initial development release continued development continued development Production use & maintenance Production & maint. feedback & requests feedback & requests users upgrade users upgrade

21. Perpetual Development Lifecycle time size initial development release continued development continued development continued development Production use & maintenance Production & maint. feedback & requests feedback & requests users upgrade users upgrade

22. Linux Example

23. “Linux is evolution, not intelligent design” -- Linus Torvalds

24. Eric Raymond: The Cathedral and the Bazaar Planning and management Evolution

25. Raymond’s Main Insights Open-source developers are “scratching a personal itch”, so they know the requirements Having real users leads to real feedback “with enough eyeballs, all bugs are shallow” (Linus’s Law) Users are co-developers, may come up with good ideas “Release early, release often” leads to the most rapid progress

26. Linux Old Release Model

27. Distinction between versions Production is stable Development is current Distinction between releases Minor production release for bug/security fixes Minor development release every few days A major release is a major operation taking months Interval between major releases is long Two years of delaying new features

28. Linux New Release Model (2.6)

29. Only production versions are released Development done separately in parallel New version every 2-3 months Merge window of 2 weeks ~2 months of stabilization and preparation Few minor releases as needed New merge window opened at time of release Maintain some long-term versions for stability

30. Software Growth Lehman’s Law I: Continuing change Adapt to needs Lehman’s Law VI: Continuing growth Add features Code is rarely removed Are you sure it is never needed? Hard to figure out exactly what is not needed any more Reluctance to throw away what took long to produce Change is achieved by growth

31. Software Growth Lehman and Turski: A system's complexity grows with time It is harder to modify a more complex system Assuming a constant effort per release, This leads to a diminished growth rate:

32. Software Growth Godfrey and Tu (2000): Linux (and other systems) is growing at a super-linear rate

33. More Data 1994 to 2011 78x growth (29% annual) Best models: Quad.-exponential Piecewise linear No theory 660 l/d 1385 l/d 3541 l/d

34. Growing Growth Rate Possible explanation: Positive feedback Successful project attracts more developers More developers produce more code Suggests exponential growth Contradicts Brooks’s Law (“adding manpower to a late project makes it later”) System is modular so don’t need so much communication Good information dissemination via mailing lists etc. New developers are typically already experts, not novices

35. Complexity and Quality Lehman’s Law II: Increasing complexity  Interactions between more and more modules Lehman’s Law VII: Declining quality  Harder to improve  less satisfying The tradeoff: Do it fast vs. do it right The compromise: Technical debt  Fast-and-dirty = accumulate debt  Refactoring = pay off debt (with interest)  Cost-benefit decision of when (and if) to pay

36. Continuous Deployment Variant New software released in timescales of minutes, not days or weeks Each developer immediately deploys whatever he works on Based on passing a battery of automatic tests (unit + integration + regression) Requires strong framework to control releases and roll them back if needed Makes the whole notion of “a version” meaningless

37. Continuous Deployment Variant Popular mainly in web-based companies and applications Deploy to web site No need to download and install Immediately used by all Can be done at different granularities Facebook deploys front-end twice a day Amazon deploys update to some service every 11 seconds on average

38. Perpetual Development Benefits Lead time to first working version is short, and a working version is always available  No danger of the project coming to nothing Real users doing real work are effectively brought into the development cycle  Helps to test system functionality and find problems  Used to prioritize further development according to what is really needed Ability to use new technology as it becomes available

39. Implications for Development No fixed goal that has to be reached Goal is to continually improve the system and maintain is usefulness  Monitor system usage to identify inadequacies  Prioritize according to user needs  Don't plan too far ahead (YAGNI protection) Use contracts that take longevity into account  Support for continued evolution  Access to code if company becomes insolvent

40. Implications for Architecture Can't decide on architecture based on analysis of all the requirement Need architecture that accommodates change  Two tiers: stable core and evolving libraries  Open system like e-commerce site based on web services Use refactoring May need to abandon project eventually  But may still salvage parts for a followup project

41. Conservation of Familiarity Lehman's Law V of software evolution Limits the rate of progress that can be sustained Need specialized tools to help new team members to become familiar with the system  Newbies are at a disadvantage because they didn't see how the system developed  Need to capture the history of design decisions

42. Explaining Monumental Failures Failures caused by "feature creep"  Developers made elaborate and beautiful plans  But these plans were obsolete by the time they were completed  Do exactly what is most needed at each instant Failures caused by successful maintenance  Delivered system was good for a very long time  But when it is to be replaced, an attempt is made to do too much at once  Make improvements continuously all the time

43. Experimental Evidence Experiment conducted by the World Wide Consortium for the Grid (W2COG) The goal: develop a secure service-oriented architecture system Traditional approach: standard government acquisition process Alternative: use a "Limited Technology Experiment" based on evolutionary methods Both start with same government supplied software baseline

44. 18 Months Later... Traditional: A concept document with no functional architecture Cost $1.5M No concrete deployment plan or timeline Evolutionary: Delivered open architecture prototype addressing 80% of requirements Cost of $100K Plan to complete in 6 months Denning, Gunderson, & Hayes-Roth, CACM 12/2008

45. Bottom Line Expect to see many more projects using evolutionary and agile methods Especially in environments challenged by rapid technological progress and rapid change These ideas are actually not new ― However, not articulated well till recently (except for Gilb) ― Contradict traditional engineering approach ― Nevertheless work well in practice

46. New Laws 1.Continuing growth o Super success leads to super-linear growth o Independent sub-projects / modularity 2.Short release cycles (or continuous deployment) o Facilitate reorientation and acting on feedback 3.Individually sustainable work rate 4.Managing technical debt 5.Conservation of familiarity