1. Center for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

2. “Multi-Objective Regression Test Suite Optimization with Fuzzy Logic”
Presented By:
Zeeshan Anwar
Ali Ahsan
University of Engineering and Technology (UET), Taxila
Centre for Advanced Studies in Engineering (CASE)

3. Contents Introduction Related Work Objectives Problem Definition
Simulation
Comparison & Suggested Solution
Conclusion & Future Work
References
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

4. Contents INTRODUCTION Related Work Objectives Problem Definition
Simulation
Comparison & Suggested Solution
Conclusion & Future Work
References
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

5. INTRODUCTION Regression Testing Regression Testing Techniques [2]
Testing of previously tested programs following modifications to ensure that defects have not been introduced into unchanged programs. [1]
Regression Testing Techniques [2]
Regression Testing
Retest all
Test Selection
Test Prioritization
Hybrid Approaches
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

6. INTRODUCTION Cont… RTO Optimization Regression Test Suite Optimization
Test Suite contains reusable, re-testable, obsolete, new structural and new specifications test cases. [30]
With the addition of test cases, regression test suite becomes so large and ineffective [4] we have to optimize it to save cost and resources. Regression Test Suite Optimization (RTO) is making the regression test suite efficient to save time, cost or resources.
RTO
Optimization
Minimization
Prioritization
Selection
Center for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

7. INTRODUCTION Cont… RTO Journey Hybrid Retest all Regression Techniques
SO Optimization
(Heuristic & CI)
Hybrid
MO Optimization
(Heuristic & CI)
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

8. Research Questions & Solution Methodology
INTRODUCTION Cont…
Research Questions & Solution Methodology
Regression Test Suite Optimization
Why Optimize?
Approaches?
Techniques?
Important Factors?
Multi-Objective Optimization?
Experiments
Case Study
Literature Review
Retest all
Validation
Fuzzy & RTO
CS1 & CS2
CI & Optimization
Computational Intelligence?
CI Techniques?
Advantages?
CI and RTO?
Fuzzy Logic and RTO
How Fuzzy Logic can be used for RTO?
Implementation?
Advantages?
Comparison?
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

9. Contents Introduction RELATED WORK Objectives Problem Definition
Simulation
Comparison & Suggested Solution
Conclusion & Future Work
References
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

10. RELATED WORK Heuristic Prioritization of Regression Test Suite
Researcher
Approach
Metrics
Results
W. Eric Wong [4]
Modification based prioritization
Size reduction, precision and recall
Minimization & Prioritization give good results.
Ruchika Malhotra [5]
Test cases that cover modified program has high priority
Prioritized test suite can find faults.
Gregg Rothermel [7]
Nine approaches for prioritization
Fault Detection Rate
Optimal Prioritization performs best
Jung-Min Kim [9]
Resource Constraints and Deadlines (MO)
1st New Version.
2nd Faults Detection
Safe techniques is better & equivalent to retest all
Nashat Mansour [14]
3 Phase RTS technique for C# programs
Inclusiveness and precision
Their technique is safe up to step 5
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

11. RELATED WORK Cont.. Heuristic Minimization of Regression Test Suite
Researcher
Approach
Metrics
Results
Machani SivaPrasad [6]
Execution tracking and coverage analysis
Execution tracking and dataflow analysis
Reduction in size and time can be achieved
Kunal Taneja [10]
Behavioral difference b/w two programs
Execution of changed statement and program state difference
12.9% fewer runs to cause execution of changed statements & 11.8% runs on average to cause program state difference.
Alessandro Orso [11]
Two level high level & in depth analysis to select test cases.
Time to execute original and selected suite.
Selected suite is save and consumes less time (62.5% on average).
Agastya Nanda [13]
Configuration and database files change may also effect RTS process.
Select test cases that cover modified external entities along with changed code.
Coverage based technique miss many test cases which must be executed for external entities.
Pavan Kumar Chittimalli [15]
Coverage data is outdated we have to update it.
Coverage data, RECOMPUTEMATRIX. Used Outdated, estimated and updated coverage data.
Applied RECOVER on 3 programs 1 to 80KLOC size
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

12. RELATED WORK Cont..
Regression Test Suite Optimization and Genetic Algorithms
Researcher
Approach
Metrics
Results
Zheng Li [12]
Studied 5 search based algorithms for prioritization
Block, decision & statement coverage
Genetic algorithm performs well, greedy approaches are surprisingly effective
Qian Zhongsheng [18]
User session based test suite optimization
Fitness value calculated using error coverage ratio and cost of test run
Does not increase the number of detected faults but executes in less time
S. Nachiyappan
[21]
Genetic algorithm for test suite reduction
Used coverage and run time of test case in fitness function
Applied on 3 projects and found that increased granularity of fitness function led to more effective test case reduction
Arvinder Kaur et al [23]
Genetic algo for regression test prioritization
Maximum faults covered in minimum execution time
Applied on 5 problems and efficiency of 50 to 100% was obtained.
Arvinder Kaur et al [25]
Average percentage of code covered
Applied on triangle problem and test suite was prioritized
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

13. RELATED WORK Cont.. RTO and Swarm Algorithm Researcher Approach
Metrics
Results
Luciano S. de Souza et al [26]
Multiobjective (2) particle swarm optimization for test suite selection.
Maximize requirement coverage & minimize cost
Find good Pareto front for objective function requirement coverage & cost
Arvinder Kaur et al [27]
Combination of Particle Swarm Optimization & GA
Percentage of faults detected. Best test suite is which cover maximum faults in less time.
75.6% faults coverage achieved by HPSO
Arvinder Kaur et al [29]
Faults & condition nodes covered. Complexity & correctness
Checked on 5 programs. Fault based experiment: APFD =90.3% in best case. Path based experiment: APCC 93.1% in best case.
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

14. RELATED WORK Cont.. RTO & Fuzzy Logic Researcher Approach Metrics
Results
Zhiwei Xu et al [38]
Mamdani Model of Fuzzy Logic
Schedule factor, defect impact factor and test coverage factor
Applied on 4 releases of GSM test database. 3 releases were used for history and fuzzy logic was applied on 4th release and found that fuzzy expert system can find the defects earlier and faster.
Ali M. Alakeel [39]
Fuzzy logic concept to measure the effectiveness of a test case to violate assertion
History of testing of original program. -
A. A. Haider [40]
Multi-objective fuzzy logic based expert system.
Performance, throughput and code coverage. Classify test cases based on P, T and C.
CI based multi-objective approaches will left may test cases where as fuzzy logic based approach is best suited for multi-objective optimization.
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

15. Contents Introduction Related Work OBJECTIVES Problem Definition
Simulation
Comparison & Suggested Solution
Conclusion & Future Work
References
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

16. OBJECTIVES 1 2 3 4 5 Critically review existing techniques for RTO.
Formulate Multi-Objective Optimization Problem for RTO 2
Implement Fuzzy based RTO 3 4
Compare Results
Propose Findings 5
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

17. Contents Introduction Related Work Objectives PROBLEM DEFINITION
Simulation
Comparison & Suggested Solution
Conclusion & Future Work
References
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

18. PROBLEM DEFINITION Fuzzy to Address Flaws
Expert Judgment for Selection
Multi-Objective
Continuous and more flexible
Easy to Implement
Flaws in existing approaches
Single Objective
Not Safe
Multi-Objective to make them safe
Discrete value based
Coverage based
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

19. PROBLEM DEFINITION cont…
Fault Detection Rate
FDR = Faults Detected / Total Faults
Requirement Coverage
RC = Requirements Covered / Total Requirements
Execution Time (ET )
Requirement Failure Impact (RFI)
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

20. PROBLEM DEFINITION cont…
Test Suite T is collection of n test cases where n is total number of test cases in test suite.
Suitability of each test case to become part of optimized test suite is defined as Best, Moderate and Low and test suite belongs to only one class at a time.
S = LS  MS  BS
Objective function for finding To is to save test suite execution time. It consists of multiple objective functions that are:
Max FDR (T)
Min ET (T)
Max Rc (T)
Max RFI (T)
Fitness =
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

21. Contents Introduction Related Work Objectives Problem Definition
SIMULATION
Comparison & Suggested Solution
Conclusion & Future Work
References
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

22. Experiments Strategy Zeeshan Anwar 22
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

23. Case Study Case Study 1: Previous Data Problem [62]
Case Study 2: Siemens Print Tokens [76]
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

24. Contents Introduction Related Work Objectives Problem Definition
Simulation
COMPARISON & SUGGESTED SOLUTION
Conclusion & Future Work
References
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

25. Comparison Zeeshan Anwar 25
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

26. Contents Introduction Related Work Objectives Problem Definition
Simulation
Comparison & Proposed Solution
Conclusion & Future Work
References
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

27. CONCLUSION 1
We can conclude from the experiments that Fuzzy Logic is effective to optimize the regression test suites and can save up to 50% time and is a safe technique and easy to implement. 2
Fuzzy Logic work well for Optimization of Regression Test Suites for Black Box and White Box Testing Methods. 3
Regression Test Suite Optimization Process can’t be fully automated using Fuzzy Logic.
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

28. FUTURE WORK 1
Regression Test Suite Optimization with MOGA, NSGA-II and MOPSO. 2
Propose Neuro-Fuzzy Modeling to make proposed system fully automated and introduce learning capabilities.
Regression Test Suite optimization with Neural Networks and Support Vector Machine. 3
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

29. Contents Introduction Related Work Objective Problem Definition
Simulation
Comparison & Proposed Solution
Conclusion & Future Work
REFERENCES
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

30. REFERENCES
M. Kumar, A. Sharma and R. Kumar, “Optimization of Test Cases using Soft Computing Techniques: A Critical Review”, WSEAS Transactions on Information Science and Applications, Issue 11, Volume 8, November 2011, pp 440 – 452.
A. A. Haider, S. Rafique and A. Nadeem, “Test Suite Optimization using Fuzzy Logic”, 8th International Conference of Emerging Techniques (ICET), 8th October 2012.
Mamdani, Ebrahim H., and Sedrak Assilian. "An experiment in linguistic synthesis with a fuzzy logic controller." International journal of man-machine studies 7.1 (1975): pp 1-13.
Takagi, Tomohiro, and M. Sugeno. "Fuzzy identification of systems and its applications to modeling and control." Systems, Man and Cybernetics, IEEE Transactions on 1 (1985): pp
“ISTQB® Glossary of Testing Terms Version:2.2”,
W. Eric Wong, J. R. Horgan, S. London and H. Agrawal, “A Study of Effective Regression Testing in Practice”, 8th IEEE International Symposium on Software Reliability Engineering (ISSRE’97), pp ,Albuquerque,NM, November 1997.
R. Malhotra, A. Kaur and Y. Singh, “A Regression Test Selection and Prioritization Technique”, Journal of Information Processing Systems, Vol.6, No.2, June 2010, pp
S. Prasad , M. Jain and S. Singh, “Regression Optimizer A Multi Coverage Criteria Test Suite Minimization Technique”, International Journal of Applied Information Systems (IJAIS) – ISSN : , Foundation of Computer Science FCS, New York, USA Volume 1– No.8, April 2012 – pp 5-11
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

31. REFERENCES cont…
L. Magdalena, “What is Soft Computing? Revisiting Possible Answers”, International Journal of Computational Intelligence Systems, Vol.3, No. 2 (June, 2010), pp
S. Sumathi and Surekha P., “Computational Intelligence Paradigms Theory and Applications using MATLAB”, CRC Press Taylor & Francis Group, Boca Raton, London, Newyork, ISBN: , 2010
Z. Li, M. Harman, and R. M. Hierons, “Search Algorithms for Regression Test Case Prioritization”, IEEE Transactions on Software Engineering, VOL. 33, NO. 4, APRIL 2007
Q. Zhongsheng, “Test Case Generation and Optimization for User Session-based Web Application Testing”, Journal of Computers, Vol. 5, NO. 11, November 2010, pp
S.Nachiyappan A.Vimaladevi and C.B. Selva Lakshmi, “An Evolutionary Algorithm for Regression Test Suite Reduction”, Proceedings of the International Conference on Communication and Computational Intelligence – 2010, pp
S. Yoo, and M. Harman. "Pareto efficient multi-objective test case selection." Proceedings of the 2007 international symposium on Software testing and analysis. ACM, 2007.
A. Kaur, “A Bee Colony Optimization Algorithm for Code Coverage Test Suite Prioritization”, International Journal of Engineering Science and Technology (IJEST), Vol. 3 No. 4 April 2011, pp
Luciano S. de Souza, Pericles B. C. de Miranda, Ricardo B. C. Prudencio and Flavia de A. Barros, “A Multi-Objective Particle Swarm Optimization for Test Case Selection Based on Functional Requirements Coverage and Execution Effort”, 23rd IEEE International Conference on Tools with Artificial Intelligence, 2011, pp
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

32. REFERENCES cont…
Xu, Zhiwei, Kehan Gao, and Taghi M. Khoshgoftaar. "Application of fuzzy expert system in test case selection for system regression test." Information Reuse and Integration, Conf, IRI-2005 IEEE International Conference on.. IEEE, 2005.
A. M. Alakeel, “A Fuzzy Test Cases Prioritization Technique for Regression Testing Programs with Assertions”, ADVCOMP 2012 : The Sixth International Conference on Advanced Engineering Computing and Applications in Sciences, pp
Nanda, Agastya, S. Mani, S. Sinha, M. J. Harrold, and A. Orso. "Regression testing in the presence of non-code changes." In Software Testing, Verification and Validation (ICST), 2011 IEEE Fourth International Conference on, pp IEEE, 2011.
S. Parsa and A. Khalilian, “On the Optimization Approach towards Test Suite Minimization”, International Journal of Software Engineering and Its Applications Vol. 4, No. 1, January 2010, pp
Whyte,G and Mulder, D ,L. “Mitigating the Impact of Software Test Constraints on Software Testing Effectiveness”, The Electronic Journal Information Systems Evaluation Volume 14 Issue , pp
Ashraf, E., A. Rauf, and K. Mahmood. "Value based Regression Test Case Prioritization." Proceedings of the World Congress on Engineering and Computer Science. Vol
K.K. Aggarwal, and Y. Singh, “A book on software engineering”, New Age International (P) Ltd.; Publishers, 4835/24, Ansari Road, Daryaganj, New Delhi, 2001.
Hutchins, Monica, et al. "Experiments of the effectiveness of dataflow-and control flow-based test adequacy criteria." Proceedings of the 16th international conference on Software engineering. IEEE Computer Society Press, 1994.
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

33. REFERENCES cont…
Z. Anwar, A. Ahsan, “Exploration and Analysis of Regression Test Suite Optimization”, ACM SIGSOFT Software Engineering Notes (2014). [Accepted for Publication]
Z. Anwar, “Neuro-Fuzzy Modeling based Regression Test Suite Optimization”, MSc Thesis, Center for Advance Studies in Engineering, Islamabad, Pakistan, 2013.
Z. Anwar, A. Ahsan, “Comparative Analysis of MOGA, NSGA-II and MOPSO for Regression Test Suite Optimization”, International Journal of Software Engineering (IJSE). [Submitted]
Do, Hyunsook, S. Elbaum, and G. Rothermel. "Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact." Empirical Software Engineering 10.4 (2005):
Centre for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

34. Questions?

35. Thank You !

36. Case Study 1: Sugeno Center for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

37. Case Study 1: Sugeno cont…
Surface Plot of FDR and ET with Suitability
Surface Plot of FDR and RC with Suitability
Surface Plot of FDR and RFI with Suitability
Surface Plot of RC and ET with Suitability
Surface Plot of RFI and ET with Suitability
Surface Plot of RFI and RC with Suitability
Center for Advanced Studies in Engineering (CASE)
Zeeshan Anwar

38. Case Study 2: Sugeno Center for Advanced Studies in Engineering (CASE)
Surface Plot of FDR and ET with Suitability
Surface Plot of FDR and RC with Suitability
Surface Plot of FDR and RFI with Suitability
Surface Plot of RC and ET with Suitability
Surface Plot of RFI and ET with Suitability
Surface Plot of RFI and RC with Suitability
Center for Advanced Studies in Engineering (CASE)
Zeeshan Anwar