Presentation on theme: "Overview of A Structured Experiment of Test-Driven Development Jim Kile."— Presentation transcript:
Overview of A Structured Experiment of Test-Driven Development Jim Kile
June 24, 2005DCS891C - Summer 2005 2 What is this paper about? Problem explored – To empirically analyze and quantify the effectiveness of Test-Driven Development (TDD) Why? – Unless objects are designed judiciously, dependency problems (e.g. tight coupling, fragile super classes, etc.) can creep into code affecting code quality – TDD proponents argue that reduced coupling occurs because the practice guides them to Build only those objects that are actually needed Improves code quality through continuous testing – There is no empirical evidence to support the claims made by proponents
June 24, 2005DCS891C - Summer 2005 3 What is this paper about? Hypotheses – The TDD practice will yield code with superior external code quality when compared with code developed in a waterfall-like practice – Programmers who practice TDD will develop code faster than programmers who develop code with a more traditional waterfall-like practice Results – TDD appears to yield code with superior external code quality Code produced in this manner passed 18% more functional black-box test cases – TDD programmers took more time (16%) than control group programmers – On average 80% of professional programmers thought TDD was effective and 78% believed productivity was improved – TDD facilitates simpler design and the lack of upfront design is not a hindrance
June 24, 2005DCS891C - Summer 2005 4 Technical background What is test-driven development? A practice of software development that involves the implementation of a system starting from the unit test cases of an object – If you cant write a test for what you are about to code, then you shouldnt even be thinking about coding  Perceived shortcomings – Lack of upfront design – Applicability of practice for code that is difficult to test using TDD (e.g. GUIs) – Reliance on refactoring to achieve code understanding and manage complexity – High skill level and experience requirements Perceived benefits – Program comprehension (especially useful for maintenance activities) – Efficiency through continuous feedback – Creation and availability of reusable test assets – Reducing defect injection during development
June 24, 2005DCS891C - Summer 2005 5 Context of this research Experiments were done with practitioners in their own working environment Sample size was relatively small Experiment was modified after the first trial All programmers worked in pairs – John Deere and RoleModel had used pair programming – Ericcson was introduced to pair programming The code generated was very small (200 LOC) Subjects had varying levels of experience
June 24, 2005DCS891C - Summer 2005 6 Research approach Experimentation (quantitative) – A set of three structured experiments conducted with 24 pair programmers One group developed using Test-Driven Development Another control group developed using a waterfall-like approach – Results monitored for black box test cases passed, time taken to complete, and percentage of code coverage Survey (qualitative) – A set of nine close-ended questions aimed at eliciting opinions about How productive the practice is for programmers? How effective is the practice? How difficult is the practice to adopt?
June 24, 2005DCS891C - Summer 2005 7 Related research problems (ideas) IDEA 1: Conduct a larger scale study along the same lines – Sample size was small in this study and though statistically there were correlations, one of the three experiments had to be thrown out IDEA 2: Further analyze if TDD can have a positive impact on productivity – This research proved there was a positive correlation between TDD and quality deliverables – Quality can yield future savings and productivity – Hypotheses: There is a learning curve when introducing TDD methods to a team. For small duration efforts, this learning curve will impact productivity. Longer efforts should see productivity equal to or superior to a waterfall baseline. – Note: This can be done in pair fashion as was done here or in non-pair fashion, though interestingly non-pair studies to date seem to have better results IDEA 3: Analyze enhancement projects where the original project followed TDD and there is a base of test cases for regression – Hypotheses: TDD provides a positive impact on development productivity when the initial project uses TDD and those regression cases are available for a subsequent enhancement project IDEA 4: Re-run the study using individual programmers rather than pairs – The study required people to be familiar with a programming technique which is not in wide usage in the industry – Significance: Is there any benefit to doing TDD in existing work environments without introducing pair programming?
June 24, 2005DCS891C - Summer 2005 8 Further analyze TDD productivity impact (idea 2) Other studies using students did find the practice to be neutral to productivity [P3]:P3 – This same study indicated that the vast majority of test subjects, on average, had only 4 months experience with Java or TDD Only one individual had set development practices – There was no productivity improvement or reduction using TDD in this environment (productivity-neutral) The conclusion expressed confusion at the results -- they were also expecting productivity improvement This could that the practice is productivity neutral once the learning curve is overcome, however Still other studies using professional programmers found the practice to be neutral or a slight impact on productivity [P2]P2 – This study did not use a control group during the actual experiment, but used ad-hoc testing of a similar prior project as a basis None of these studies used pair programming, but used traditional programming
June 24, 2005DCS891C - Summer 2005 9 Productivity for enhancement projects using TDD (idea 3) A couple of studies have indicated the potential for productivity improvements for future enhancement projects [P2, P4]P2P4 – Work to create a base of automated unit test cases would be complete – There would need to be time spent to update automated cases with the enhancements This doesnt seem to have a lot of existing studies – most do not go a second step to record enhancement productivity
June 24, 2005DCS891C - Summer 2005 10 Presentation references [P1] B. George and L. Williams, "An Initial Investigation of test Driven Development in Industry", Proceedings of the 2003 ACM Symposium on Applied Computing, pp. 1135-1139, Melbourne, Florida, UASA, ACM Press, 2003. [P2] E. M. Maximilien and L. Williams, "Assessing test-driven development at IBM ", Proceedings of the 25th International Conference on Software Engineering, pp. 564-569, Portland, Oregon, USA, IEEE Computer Society, 2003. [P3] D. Staff and M. G. Ernst, "An Experimental Evaluation of Continuous Testing During Development", Proceedings of the 2004 ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 76-85, Boston, Massachusetts, USA, ACM Press, 2004. [P4] L. Williams, E. M. Maximilien and M. Vouk, "Test-Driven Development as a Defect-Reduction Practice", 14th International Symposium on Software Reliability Engineering, p. 34, Denver, Colorodo, USA, IEEE Computer Society, 2003, November 2003.
June 24, 2005DCS891C - Summer 2005 11 Original references  K. Beck, Extreme Programming Explained: Embrace Change. Reading, Massachusetts: Addison-Wesley, 2000.  K. Beck, Test Driven Development: By Example: Addison Wesley, 2002.  K. Beck, "Aim, Fire," in IEEE Software, vol. 18, September/October 2001, pp. 87-89.  F. P. Brooks, The Mythical Man-Month: Addison-Wesley Publishing Company, 1995.  D. T. Campbell and J. C. Stanley, Experimental and Quasi-Experimental Design for Research. Boston: Houghton Mifflin Co., 1963.  D. Chaplin, "Test First Programming," TechZone, 2001.  T. A. Corbi, "Program Understanding challenge for the 1990s," IBM Systems Journal, vol. 28, pp. 294-306, 1989.  S. Cornett, "Code Coverage Analysis," Bullseye Testing Technology 2002.  B. Foote and J. Yoder, "Big Ball of Mud," presented at Fourth Conference on Patterns Languages of Programs, Monticello, Illinois, September 1997.  D. Gelperin and W. Hetzel, "Software Quality Engineering," presented at Fourth International Conference on Software Testing, Washington, DC, June 1987.  B. George, "Analysis and Quantification of Test Driven Development Approach MS Thesis," in Computer Science. Raleigh, NC: North Carolina State University, 2002.  D. Hamlet and J. Maybee, The Engineering of Software. Boston: Addison Wesley, 2001.
June 24, 2005DCS891C - Summer 2005 12 Original references Continued  W. S. Humphrey, Managing the Software Process. Reading, Massachusetts: Addison-Wesley, 1989.  C. Larman and V. Basili, "A History of Iterative and Incremental Development," IEEE Computer, vol. 36, pp. 47-56, June 2003.  M. M. Lehman and L. Belady, Program Evolution: Processes of Software Change. London: Academic Press, 1985.  C. R. Martin, Advanced Principles, Patterns and Process of Software Development: Prentice Hall, 2001, in press.  E. M. Maximilien and L. Williams, "Assessing Test-driven Development at IBM," presented at International Conference of Software Engineering, Portland, OR, 2003.  M. M. Muller and O. Hagner, "Experiment about Test-first programming," presented at Empirical Assessment In Software Engineering EASE '02, Keele, April 2002.  D. E. Perry and A. L. Wolf, "Foundations for the Study of Software Architecture," ACM SIGSOFT, vol. 17, pp. 40-52, October 1992.  W. W. Royce, "Managing the development of large software systems: concepts and techniques," presented at IEEE WESTCON, Los Angeles, CA, 1970.  A. vanDeursen, "Program Comprehension Risks and Opportunities in Extreme Programming," CWI, Amsterdam SEN-R0110, ISSN 1386-369X, 2001.  A. vanDeursen, L. Moonen, A. vandenBergh, and G. K. R. t. code, "Refactoring test code," presented at XP 2001, 2001.  L. Williams, E. M. Maximilien, and M. Vouk, "Test-Driven Development as a Defect-Reduction Practice," presented at IEEE International Symposium on Software Reliability Engineering, Denver, CO, 2003.  L. A. Williams, The Collaborative Software Process. Salt Lake City, UT: Department of Computer Science, 2000.