(C) 2005 Václav Rajlich 1 Software Evolution and Agile Development Václav Rajlich Department of Computer Science Wayne State University Detroit, Michigan.

(C) 2005 Václav Rajlich2 Confusing situation New recommendations how to develop software New recommendations how to develop software –Rational Unified Process –Extreme Programming –SCRUM –… They often contradict what has been considered good software engineering They often contradict what has been considered good software engineering –how to make a sense of this?

(C) 2005 Václav Rajlich3 Paradigm Thomas S. Kuhn Thomas S. Kuhn –“The Structure of Scientific Revolutions” “Coherent tradition of scientific research” “Coherent tradition of scientific research” –includes knowledge, techniques, research agenda, culture of the field… –(more profound meaning than the usage common in software engineering)

(C) 2005 Václav Rajlich4 Paradigm change Discontinuity in the development of the discipline Discontinuity in the development of the discipline Kuhn collected extensive historical data on paradigm change Kuhn collected extensive historical data on paradigm change –old paradigm cannot explain compelling new facts –new paradigm is a genuine revolution –paradigm change is often contentious and traumatic –many examples –phlogiston - > oxygen in 1770’s

(C) 2005 Václav Rajlich5 Resistance to paradigm change Knowledge that was accumulated up to that point may lose its significance Knowledge that was accumulated up to that point may lose its significance –some knowledge may be completely lost Advantages of the new paradigm in dispute Advantages of the new paradigm in dispute –attempts are made to extend old paradigm to accommodate new facts –band-aid approaches try to fix old paradigm Final victory of the new paradigm guaranteed by a generation change Final victory of the new paradigm guaranteed by a generation change

(C) 2005 Václav Rajlich6 My claim Software evolution research Software evolution research Agile + iterative processes Agile + iterative processes Represent paradigm change! Represent paradigm change! –Third paradigm change in 50 years of software engineering

(C) 2005 Václav Rajlich7 Paradigm change of 1950’s Original programmers recruited from the ranks of hardware engineers and mathematicians Original programmers recruited from the ranks of hardware engineers and mathematicians –used ad-hoc techniques from their former fields Software separated from the hardware in 1950’s Software separated from the hardware in 1950’s –emerged as a distinct technology –became independent product Programming became a new discipline Programming became a new discipline –developed a specialized set of skills

(C) 2005 Václav Rajlich8 Paradigm change of 1970’s Previous techniques did not scale up Previous techniques did not scale up –Brooks: “Mythical Man-Month” –demands of the new operating system OS/360 taxed the limits of the programmers, project managers, and the resources of the IBM corporation Paradigm change established discipline of software engineering Paradigm change established discipline of software engineering –dealt with complexity of software –next step goes beyond programming –introduced the waterfall metaphor

(C) 2005 Václav Rajlich9 Waterfall metaphor Used in construction and manufacturing Used in construction and manufacturing –collect the requirements –create a design –follow the design during the entire construction –transfer finished product to the user –solve residual problems through maintenance Intuitively appealing metaphor Intuitively appealing metaphor –good design avoids the expensive late rework –waterfall became the dominant paradigm

(C) 2005 Václav Rajlich10 Problems of waterfall Requirements volatility (Cusumano and Selby) Requirements volatility (Cusumano and Selby) –30% or more requirements may change during development –this is the direct result of the team’s learning process and software interoperability Maintenance phase is not uniform Maintenance phase is not uniform –Lehner, Pigoski –frequency of the changes in long lived systems peaks, then declines

(C) 2005 Václav Rajlich11 Standish group report In 1995 In 1995 –31% of all software projects were cancelled –53% were “challenged” (completed only with great difficulty, with large cost or time overruns) –only 16% could be called successful. Obviously, the waterfall metaphor did not solve the problems of software development Obviously, the waterfall metaphor did not solve the problems of software development

(C) 2005 Václav Rajlich12 Anticipation of changes If changes can be anticipated at design time, they can be controlled by a parameterization, encapsulations, etc. If changes can be anticipated at design time, they can be controlled by a parameterization, encapsulations, etc. –waterfall model still can be used Experience confirms: Experience confirms: – many changes are not anticipated by the original designers – inability to change software quickly and reliably means that business opportunities are lost –only a band-aid solution

(C) 2005 Václav Rajlich13 Prototyping Another attempt to extend waterfall Another attempt to extend waterfall Create a prototype to capture requirements Create a prototype to capture requirements Problem: volatility continues after prototype has been completed Problem: volatility continues after prototype has been completed Another band-aid solution Another band-aid solution

(C) 2005 Václav Rajlich14 Paradigm change of 2000’s New metaphor based on self-directed learning New metaphor based on self-directed learning –Learner controls the process Self-directed learning can be planned only to a limited degree Self-directed learning can be planned only to a limited degree –Needs a happy middle between rigid plan and chaos –One semester at a time Iteration is the analogy of a semester Iteration is the analogy of a semester –Ends with a clear feedback (grade) –Continuation is based on that feedback

(C) 2005 Václav Rajlich15 Instances of the new paradigm New life-cycle models emphasize software evolution New life-cycle models emphasize software evolution –Staged model of software lifecycle –Based on data from long-lived systems Iterative development Iterative development –Rational Unified Process Agile development Agile development –SCRUM –Extreme programming

(C) 2005 Václav Rajlich16 Initial development Evolution first running version evolution changes Servicing code decay servicing patches Close-down Phase-out servicing discontinued switch-off Staged model Staged model

(C) 2005 Václav Rajlich17 New research agenda Focus on the evolution Focus on the evolution –Modifications in the existing software Incremental change Incremental change –adds a new functionality or a new properties to the existing software Refactoring Refactoring –cleans up software structure without changing functionality Redocumentation, reengineering, migration, … Redocumentation, reengineering, migration, … Exciting and rich agenda Exciting and rich agenda

(C) 2005 Václav Rajlich18 Incremental change (IC) IC was supposed to happen only rarely under the old paradigm IC was supposed to happen only rarely under the old paradigm IC was largely ignored by the researchers and educators IC was largely ignored by the researchers and educators –elephant in the living room The current programmers still must learn IC on their own The current programmers still must learn IC on their own –No presence in textbooks, curricula Large backlog of research issues Large backlog of research issues

(C) 2005 Václav Rajlich19 IC minicycle Initiation Initiation –New feature request from users, programmers, managers, marketing, … –Problem report Design Design Implementation Implementation Validation Validation Release Release

(C) 2005 Václav Rajlich21 Concept location Concept location finds where a change is to be made Concept location finds where a change is to be made Change requests are most often formulated in terms of domain concepts Change requests are most often formulated in terms of domain concepts –Example: “Correct error that arises when trying to paste a text” –the programmer must find in the code the locations where concept “paste” is located –this is the start of the change

(C) 2005 Václav Rajlich22 Concept location assumptions The programmer understands domain concepts better than the code The programmer understands domain concepts better than the code –The knowledge of domain concepts is based on program use and is easier to acquire All domain concepts map onto the fragments of the code All domain concepts map onto the fragments of the code –finding that fragment is concept location –“software engineer’s query”

(C) 2005 Václav Rajlich23 Concept location methodologies Human knowledge Human knowledge Traceability tools Traceability tools Dynamic search (execution traces) Dynamic search (execution traces) Static search Static search –"grep" (pattern matching) –Search of dependency graph –Information retrieval techniques –…

(C) 2005 Václav Rajlich24 Dependency Graph Search Programmer follows the dependencies among the modules. Programmer follows the dependencies among the modules. Local functionality Local functionality –All concepts that are actually implemented in the module and are not delegated to others. Composite functionality Composite functionality –All concepts of the module combined with all its supporting modules.

(C) 2005 Václav Rajlich25 Search Process The variant of the depth first search The variant of the depth first search Programmer investigates the module's local functionality. Programmer investigates the module's local functionality. If the concept is not present in the local functionality, but is present in the composite functionality, programmer continues search with one of the supporting modules. If the concept is not present in the local functionality, but is present in the composite functionality, programmer continues search with one of the supporting modules. In case of wrong choice programmer backtracks and begins the search with another supporting module. In case of wrong choice programmer backtracks and begins the search with another supporting module.

(C) 2005 Václav Rajlich27 Information Retrieval Technique Latent Semantic Indexing (LSI) Latent Semantic Indexing (LSI) –Preprocessing of the source code and documentation –Executing of queries formulated in natural language –Retrieving and analyzing the results –Results are returned as a list, ranked by relevance (similarity) to the search query

(C) 2005 Václav Rajlich28 Set of concepts is not fixed New concepts emerge during evolution New concepts emerge during evolution –set of concepts changes during the lifecycle –difficulty for traceability tools Origin of the concepts Origin of the concepts –Domain concepts familiar to an end user: "credit card payment" familiar to an end user: "credit card payment" –Design concepts "iterator pattern” "iterator pattern” –Coding CONCEPTS "network error while validating credit card” "network error while validating credit card”

(C) 2005 Václav Rajlich29 Primitive and latent concepts Primitive concepts have to be enriched Primitive concepts have to be enriched –Point-of-Sale: introduce credit card payment –old code: payment represented as just one number Latent concepts have to be implemented Latent concepts have to be implemented –Student Registration: introduce prerequisite check –old code assumption: prerequisites are satisfied

(C) 2005 Václav Rajlich30 Impact analysis Assesses full extent of change Assesses full extent of change –Impact set Selects different implementation alternatives Selects different implementation alternatives Programmers often underestimate impact set Programmers often underestimate impact set

(C) 2005 Václav Rajlich31 Change implementation Prepare software for the change Prepare software for the change –Opportunistic refactoring, prefactoring Actualization Actualization –Implementing new functionality Incorporation Incorporation –Plug-in the new into the old Change propagation Change propagation Improving software based on new knowledge Improving software based on new knowledge –Postfactoring: Bad smells -> change

(C) 2005 Václav Rajlich32 Change propagation Visit components one-by-one Visit components one-by-one If the visited component is modified, it may no longer fit If the visited component is modified, it may no longer fit –secondary changes must be made in interacting (“neighboring”) components –secondary changes may trigger additional changes (“ripple effect”) Software is inconsistent during propagation Software is inconsistent during propagation

(C) 2005 Václav Rajlich33 Model of change propagation Let C be a set of classes Let C be a set of classes For b,c  C, b  c, For b,c  C, b  c, {b,c} is interaction is mark is mark b was visited (changed), c will be visited Evolving interoperation graph (eig) is a set E of interoperations and marks Evolving interoperation graph (eig) is a set E of interoperations and marks  E implies {b,c}  E.  E implies {b,c}  E.

(C) 2005 Václav Rajlich34 Neighborhood All interactions, incoming, outgoing marks All interactions, incoming, outgoing marks G(b) = { {b,c} |  c {b,c}  E} interactions M(b) = { |  c  E} incoming M(b) = { |  c  E} outgoing E(b) = G(b)  M(b)  M(b)

(C) 2005 Václav Rajlich35 Visit Couple of eigs Couple of eigs E' = (E - E(b))  E'(b') E' = (E - E(b))  E'(b') E'(b') = G'(b')  M'(b')  M'(b') Marks point away from b’ Marks point away from b’ M'(b')  { |  c {b',c}  E'(b') }

(C) 2005 Václav Rajlich36 Model of propagation sequence Sequence of eigs E 1, E 2, … E n Sequence of eigs E 1, E 2, … E n Starts and ends with unmarked graphs Starts and ends with unmarked graphs marked graphs in the middle is either is either –a visit –or a backtrack: for some k, 0 < k < i, E i+1 = E k

(C) 2005 Václav Rajlich37 Finality Non-final propagation Non-final propagation –after visiting b and c, there may be an immediate need to revisit b again: M'(b')  { |  c {b',c}  E'(b')} M'(b')  { |  c {b',c}  E'(b')} Final propagation Final propagation M'(b')  { |  c {b',c}  E'(b')} - M'(b')  { |  c {b',c}  E'(b')} - { |  d  M(b)} { |  d  M(b)}

(C) 2005 Václav Rajlich38 Case study Subject: application framework Drawlets Subject: application framework Drawlets –adds graphical display to a host application Drawing canvas Drawing canvas –lines, free-hand lines, rectangles, rounded rectangles, triangles, pentagons, polygons, ellipses, text boxes, images

(C) 2005 Václav Rajlich39 Drawlets More than 100 classes, 35 interfaces and 40,000 lines of code More than 100 classes, 35 interfaces and 40,000 lines of code Originally implemented in Smalltalk by Kent Beck, Ward Cunningham Originally implemented in Smalltalk by Kent Beck, Ward Cunningham –later ported into Java –“perfect API” Applet SimpleApplet runs in a browser Applet SimpleApplet runs in a browser

(C) 2005 Václav Rajlich41 Top classes SequenceOfFigures ShapeTool EllipseToolRectangleToolRectangularCreationToolPG_RectImageTool Locator CanvasTool AbstractFigure SimpleDrawingCanvas DrawingCanvas ToolPalette SimpleApplet ToolBar LocatorConnectionHandle PrototypeConstructionTool SelectionToolConstructionTool StylePalette LabelTool Figure

(C) 2005 Václav Rajlich42 Change Request Implement an "owner" for each figure Implement an "owner" for each figure –owner put that figure onto the canvas –only owner is allowed to move that figure –each session declares a session owner –this session owner will own all new figures created –no other owner will be allowed to manipulate them This change will make SimpleApplet more versatile and useful This change will make SimpleApplet more versatile and useful –support for cooperative work

(C) 2005 Václav Rajlich43 Concept The concepts relevant to the change The concepts relevant to the change –figure owner and session owner Both concepts are latent Both concepts are latent –old code assumes that there is just one owner who owns both sessions and figures –look for “similar” concepts in the code

(C) 2005 Václav Rajlich44 Location SequenceOfFigures ShapeTool EllipseToolRectangleToolRectangularCreationToolPG_RectImageTool Locator CanvasTool AbstractFigure SimpleDrawingCanvas DrawingCanvas ToolPalette SimpleApplet ToolBar LocatorConnectionHandle PrototypeConstructionTool SelectionToolConstructionTool StylePalette LabelTool Figure

(C) 2005 Václav Rajlich45 Actualization SequenceOfFigures ShapeTool EllipseToolRectangleToolRectangularCreationToolPG_RectImageTool Locator CanvasTool AbstractFigure SimpleDrawingCanvas DrawingCanvas ToolPalette SimpleApplet ToolBar LocatorConnectionHandle PrototypeConstructionTool SelectionToolConstructionTool StylePalette OwnerIdentity SimpleListener LabelTool Figure

(C) 2005 Václav Rajlich46 Incorporation SequenceOfFigures ShapeTool EllipseToolRectangleToolRectangularCreationToolPG_RectImageTool Locator CanvasTool AbstractFigure SimpleDrawingCanvas DrawingCanvas ToolPalette SimpleApplet ToolBar LocatorConnectionHandle PrototypeConstructionTool SelectionToolConstructionTool StylePalette OwnerIdentity SimpleListener LabelTool Figure

(C) 2005 Václav Rajlich47 Propagation SequenceOfFigures ShapeTool EllipseToolRectangleToolRectangularCreationToolPG_RectImageTool Locator CanvasTool AbstractFigure SimpleDrawingCanvas DrawingCanvas ToolPalette SimpleApplet ToolBar LocatorConnectionHandle PrototypeConstructionTool SelectionToolConstructionTool StylePalette OwnerIdentity SimpleListener LabelTool Figure

(C) 2005 Václav Rajlich54 Refactoring Modifies the structure, not functionality Modifies the structure, not functionality –after repeated incremental changes, the architecture become disorganized –refactoring re-introduces the order Many different refactoring transformations Many different refactoring transformations –simple renaming of an entity of the program –refactoring a base class –refactoring design patterns, … Numerous articles, books, refactoring tools Numerous articles, books, refactoring tools –many software environments have some refactoring transformations

(C) 2005 Václav Rajlich55 Shortening change propagation Each class that creates new figures has the function basicNewFigure(...) Each class that creates new figures has the function basicNewFigure(...) –two main parts: –Create a new figure –If the figure was created at wrong location, move it. We refactored a new method called moveFigure(...) We refactored a new method called moveFigure(...) –made it a member of the base class ConstructionTool

(C) 2005 Václav Rajlich56 Refactoring SequenceOfFigures ShapeTool EllipseToolRectangleToolRectangularCreationToolPG_RectImageTool Locator CanvasTool AbstractFigure SimpleDrawingCanvas DrawingCanvas ToolPalette SimpleApplet ToolBar LocatorConnectionHandle PrototypeConstructionTool SelectionToolConstructionTool StylePalette LabelTool Figure

(C) 2005 Václav Rajlich57 Splitting the roles Another technique that shortens change propagation Another technique that shortens change propagation The same method used in two different roles The same method used in two different roles –the same code can do both jobs The propagating change highlights the differences The propagating change highlights the differences –only one of the roles needs to be updated –the other one can stay unchanged

(C) 2005 Václav Rajlich58 Splitting the roles function move(...) in AbstractFigure used in two ways function move(...) in AbstractFigure used in two ways –to move the figure as requested by the user must check user identity must check user identity –as a part of the new figure creation does not need to check user identity does not need to check user identity We split move(...) and secureMove(…) We split move(...) and secureMove(…)

(C) 2005 Václav Rajlich59 Code decay Causes transition from evolution to servicing Causes transition from evolution to servicing Large changes in the functionality are no longer possible Large changes in the functionality are no longer possible Substantially lowers the value of the software Substantially lowers the value of the software –software managers should be aware of this phenomenon –it can happen by accident –very little research into what causes code decay and how to prevent it

(C) 2005 Václav Rajlich60 Servicing Changes are limited to patches and wrappers Changes are limited to patches and wrappers –less costly, but they cause further deterioration Process is very different from evolution Process is very different from evolution –no need for senior engineers

(C) 2005 Václav Rajlich61 From servicing back to evolution in the current practice: in the current practice: –very hard, very rare the knowledge has to be rebuilt the knowledge has to be rebuilt for all practical reasons, the transition from evolution to servicing is irreversible for all practical reasons, the transition from evolution to servicing is irreversible worthy research goal worthy research goal

(C) 2005 Václav Rajlich62 Close-down The software use is disconnected The software use is disconnected –What is the current life of successful software Cost of changing to another system Cost of changing to another system What to do with long-lived data? What to do with long-lived data?

(C) 2005 Václav Rajlich63 Cognitive aspects The software evolution requires the programmers to learn The software evolution requires the programmers to learn –This learning will determine the success of the project New programmers must absorb project knowledge New programmers must absorb project knowledge –Current documentation systems fail to capture this knowledge adequately –entrance of the new programmers into an old project is difficult.

(C) 2005 Václav Rajlich64 Conjecture Code decay is in fact the situation where complexity of the code overwhelms the cognitive capabilities of the programmers Code decay is in fact the situation where complexity of the code overwhelms the cognitive capabilities of the programmers Positive feedback between Positive feedback between –loss of structure –knowledge of the system

(C) 2005 Václav Rajlich65 Knowledge construction Absorption Absorption –the new facts are added to the existing knowledge Denial Denial –the new facts are rejected Reorganization Reorganization –learners reorganize their knowledge to aid absorption of new facts Expulsion Expulsion –part of the knowledge becomes obsolete or provably incorrect and the learners reject it

(C) 2005 Václav Rajlich67 Early results Experienced programmers use reorganization and expulsion more often Experienced programmers use reorganization and expulsion more often –Inexperienced programmers keep things the way they are Experienced programmers spend more time clarifying domain concepts before coding Experienced programmers spend more time clarifying domain concepts before coding Top-down comprehension uses higher Bloom’s levels than bottom-up Top-down comprehension uses higher Bloom’s levels than bottom-up Aim: recording project knowledge Aim: recording project knowledge

(C) 2005 Václav Rajlich68 Conclusion The old waterfall paradigm tried to freeze requirements for the duration of software development The old waterfall paradigm tried to freeze requirements for the duration of software development –led to too many project failures The new paradigm emphasizes software evolution The new paradigm emphasizes software evolution –embrace the new paradigm! –long backlog of research problems –exciting topics for software engineering research –these topics have been neglected by the researchers

(C) 2005 Václav Rajlich69 References 1. Beck, K. Extreme Programming Explained. Addison Wesley, Reading, MA, 2000. 2. Brooks, F.P. The Mythical Man-Month. Addison-Wesley, Reading, MA, 1982. 3. Cusumano, A.M. and Selby, R.W. How Microsoft Builds Software. Communications of ACM (June 1997). 53-61. 4. Eick, S.G., Graves, T.L., Karr, A., Marron, J.S. and Mockus, A. Does Code Decay? Assessing the Evidence from Change Management Data. IEEE Transactions on Software Engineering, 27 (1) (2001), 1-12. 5. Fowler, M. Refactoring: Improving the Design of Existing Code. Addison Wesley, Reading, MA, 1999. 6. Jacobson, I., Booch, G. and Rumbaugh, J. The Unified Software Development Process. Addison Wesley, 1999. 7. Johnson, J.H. Micro Projects Cause Constant Change, The Standish Group International, Inc, 1997. 8. Kuhn, T.S. The Structure of Scientific Revolutions. The University of Chicago Press, Chicago, 1996.

(C) 2005 Václav Rajlich70 9. Rajlich, V. and Bennett, K. A Staged Model for the Software Lifecycle. Computer, 33 (7) (July 2000) 66-71. 10. Rajlich, V., Gosavi, P. Incremental Change in Object-Oriented Programming. IEEE Software, 21 (July/August 2004). 62 - 69. 11. Rajlich, V., Xu, S., Analogy of Incremental Program Development and Constructivist Learning. In Second IEEE Int. Conf. On Cognitive Informatics, (2003), IEEE Computer Society Press, 98 - 105. 12. Wilde, N., Buckellew, M., Page, H., Rajlich, V. and Pounds, L. A Comparison of Methods for Locating Features in Legacy Software. Journal of Systems and Software, 65 (2) (February 2003), 105-114. 13. A. Marcus, V. Rajlich, J. Buchta, M. Petrenko, A. Sergeyev, Static Techniques for Concept Location in Object-Oriented Code, IEEE International Workshop on Program Comprehension, 2005, 33 - 42 14. V. Rajlich, Changing Paradigm of Software Engineering, to be published in Communications of ACM

(C) 2005 Václav Rajlich 1 Software Evolution and Agile Development Václav Rajlich Department of Computer Science Wayne State University Detroit, Michigan.

Similar presentations

Presentation on theme: "(C) 2005 Václav Rajlich 1 Software Evolution and Agile Development Václav Rajlich Department of Computer Science Wayne State University Detroit, Michigan."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

(C) 2005 Václav Rajlich 1 Software Evolution and Agile Development Václav Rajlich Department of Computer Science Wayne State University Detroit, Michigan.

Similar presentations

Presentation on theme: "(C) 2005 Václav Rajlich 1 Software Evolution and Agile Development Václav Rajlich Department of Computer Science Wayne State University Detroit, Michigan."— Presentation transcript:

Similar presentations

About project

Feedback