1. Slide title In CAPITALS 50 pt Slide subtitle 32 pt Risk-based regression testing in a telecommunication system node Master’s thesis presentation 14.8.2007 Author: Mika Lehto Supervisor: Professor Heikki Hämmäinen Instructor: M.Sc. Pekka Hämäläinen

2. Top right corner for field-mark, customer or partner logotypes. See Best practice for example. Slide title 40 pt Slide subtitle 24 pt Text 24 pt Bullets level 2-5 20 pt Ericsson Confidential2007-07-202 Agenda  Background and objectives  Risk-based testing  Regression testing  Case Study –Ericsson’s Media Gateway for Mobile Networks (M-MGw) –M-MGw Node Functional Verification –Risk model –Risk analysis and prioritization –Scenarios for the case study –Results  Conclusions  Future research

3. Top right corner for field-mark, customer or partner logotypes. See Best practice for example. Slide title 40 pt Slide subtitle 24 pt Text 24 pt Bullets level 2-5 20 pt Ericsson Confidential2007-07-203 Background  The costs of regression testing comprise 20-30 % of the whole testing costs on average in software development → regression testing is often the target of cost reduction activities in order to achieve shorter time-to-market and lower cost structure of the product  The efficiency of regression testing is usually attempted to improve by using regression testing methods such as regression test selection, test suite reduction, and test case prioritization  One approach that has been studied is called risk-based regression testing but only a little information about the feasibility of risk-based regression testing is currently available

4. Top right corner for field-mark, customer or partner logotypes. See Best practice for example. Slide title 40 pt Slide subtitle 24 pt Text 24 pt Bullets level 2-5 20 pt Ericsson Confidential2007-07-204 Objectives  The primary objective of this thesis was to study whether it would have been beneficial to use risk-based regression testing in an old M-MGw Node Functional Verfication project  This thesis also analyzed the differences in test case selections between previously used and risk-based test case selections methods  The aim of this thesis was also to demonstrate how risk-based testing approach can be utilized in regression testing

5. Top right corner for field-mark, customer or partner logotypes. See Best practice for example. Slide title 40 pt Slide subtitle 24 pt Text 24 pt Bullets level 2-5 20 pt Ericsson Confidential2007-07-205 Risk-based testing  The fundamental difference between traditional testing and risk-based testing is that risk-based testing brings formal risk assessment methods (risk identification, risk analysis, and risk prioritization) in testing process  The aim of risk-based testing is to focus the test effort on the most critical areas of the system  Risk-based testing approach can be used for instance to: –Select the most important test objects and test cases –Evaluate the current level of system quality and risk –Provide input for the decision to continue or stop testing

6. Top right corner for field-mark, customer or partner logotypes. See Best practice for example. Slide title 40 pt Slide subtitle 24 pt Text 24 pt Bullets level 2-5 20 pt Ericsson Confidential2007-07-206 Regression testing  “Selective retesting of a system or component to verify that modifications have not caused unintended effects and that the system or component still complies with its specified requirements” (IEEE Software glossary)  Regression testing is typically used during iterative software development and to support application maintenance  The biggest challenge in regression testing is the test case selection  In risk-based regression testing, formal risk analysis methods are used to select the most risky test cases for execution

7. Top right corner for field-mark, customer or partner logotypes. See Best practice for example. Slide title 40 pt Slide subtitle 24 pt Text 24 pt Bullets level 2-5 20 pt Ericsson Confidential2007-07-207 Ericsson’s Media Gateway for Mobile Networks (M-MGw)  M-MGw operates in the connectivity layer of the core network and its fundamental purpose is to connect the core network to other networks such as UTRAN and PSTN  M-MGw also supports media conversion, bearer control and payload processing  M-MGw is remotely controlled by the MSC or SGSN Servers which operate in the network control layer.

8. Top right corner for field-mark, customer or partner logotypes. See Best practice for example. Slide title 40 pt Slide subtitle 24 pt Text 24 pt Bullets level 2-5 20 pt Ericsson Confidential2007-07-208 M-MGw Node Functional Verification  Testing in the NFV is based on requirements (e.g. Specifications and call flows)  NFV is a testing phase in which the M-MGw software is first time tested in a real node environment  Test environment is established of one M-MGw node and all the external interfaces to this node are simulated

9. Top right corner for field-mark, customer or partner logotypes. See Best practice for example. Slide title 40 pt Slide subtitle 24 pt Text 24 pt Bullets level 2-5 20 pt Ericsson Confidential2007-07-209 Risk model  The case study was based on a qualitative risk model and a risk was measured as a risk exposure: Risk Exposure = Probability x Consequence  Risk items were categorized into three classes according to the probability and consequence of a risk: tolerable, undesirable, and intolerable

10. Top right corner for field-mark, customer or partner logotypes. See Best practice for example. Slide title 40 pt Slide subtitle 24 pt Text 24 pt Bullets level 2-5 20 pt Ericsson Confidential2007-07-2010 Risk analysis and prioritization  Risk analysis was performed both for the test objects and test cases using systems experts –2 systems experts participated into the risk analysis of test objects and 8 systems experts analyzed the risks related to test cases  Test objects and test cases were prioritized according to risk exposure –In a case of test case prioritization, also other prioritization criteria (consequence, old test case priority) were needed because several test cases had the same risk exposure

11. Top right corner for field-mark, customer or partner logotypes. See Best practice for example. Slide title 40 pt Slide subtitle 24 pt Text 24 pt Bullets level 2-5 20 pt Ericsson Confidential2007-07-2011 Scenarios  The analysis was based on 5 scenarios in which the amount of executed test cases was changed  In scenario 1, as many test cases were selected for execution per test object as in the original case but the test cases were selected based on risk  In other scenarios, the test effort was focused on the most risky test objects by defining how many percent of the test cases per test object were executed –Scenario 2: Intolerable 25%, Undesirable 15%, Tolerable 10% –Scenario 3: Intolerable 20%, Undesirable 10%, Tolerable 5% –Scenario 4: Intolerable 15%, Undesirable 10%, Tolerable 0% –Scenario 5: Intolerable 10%, Undesirable 5%, Tolerable 0%

12. Top right corner for field-mark, customer or partner logotypes. See Best practice for example. Slide title 40 pt Slide subtitle 24 pt Text 24 pt Bullets level 2-5 20 pt Ericsson Confidential2007-07-2012 Results – used man hours  Significant savings in used man hours would have been achieved in scenarios 3-5  In the original case, the used man hours were distributed equally to all test objects  In risk-based approach (i.e. scenarios 2-5), approximately 55 % of man hours were consumed to 20 % of test objects

13. Top right corner for field-mark, customer or partner logotypes. See Best practice for example. Slide title 40 pt Slide subtitle 24 pt Text 24 pt Bullets level 2-5 20 pt Ericsson Confidential2007-07-2013 Results – risk level  Risk level defines how many percent of the total risk is still uncovered  Risk level would have been 10 percentage units lower in the original case if the test cases had been selected based on risk  In the scenarios 3-4, risk level is at the same level than in the original case even though much less man hours were consumed

14. Top right corner for field-mark, customer or partner logotypes. See Best practice for example. Slide title 40 pt Slide subtitle 24 pt Text 24 pt Bullets level 2-5 20 pt Ericsson Confidential2007-07-2014 Results – detected faults  Only 40 % of the faults were found in scenario 1 in which the test cases were selected based on risk but the test effort was not concentrated on the most risky test objects  By concentrating also the test effort to the most risky test objects, 40-60 % of the faults were found  Deeper analysis to these poor fault-detection rates revealed that the risk analysis of test cases of TO_04 had failed and caused faults to slip through testing

15. Top right corner for field-mark, customer or partner logotypes. See Best practice for example. Slide title 40 pt Slide subtitle 24 pt Text 24 pt Bullets level 2-5 20 pt Ericsson Confidential2007-07-2015 Results – test case selections  Differencies in test case selections between previously used and risk-based approach was analyzed by determining test case coverage which defines how many percent of test cases executed in the original case was also executed in designed scenarios  In addition, the average risk exposure per test case was determined to each scenario

16. Top right corner for field-mark, customer or partner logotypes. See Best practice for example. Slide title 40 pt Slide subtitle 24 pt Text 24 pt Bullets level 2-5 20 pt Ericsson Confidential2007-07-2016 Conclusions  Risk-based regression testing would not have brought cost savings in used man hours because the costs of the faults that slipped through testing would have eliminated all the gained cost savings  Testing effort was not concentrated on the most risky test object in the original case  It can not be said that testers would have utilized risk- based testing approach when selecting regression test cases in the original case

17. Top right corner for field-mark, customer or partner logotypes. See Best practice for example. Slide title 40 pt Slide subtitle 24 pt Text 24 pt Bullets level 2-5 20 pt Ericsson Confidential2007-07-2017 Future Research  A possible pilot project for risk-based testing in order to get more empirical experience how risk-based testing works in practice  The concept and usage of risk level should be investigated more because risk level could be possible used as an indicator to tell when it is time to stop testing  To develop a framework for assessing the costs of the faults quantitatively

18. Top right corner for field-mark, customer or partner logotypes. See Best practice for example. Slide title 40 pt Slide subtitle 24 pt Text 24 pt Bullets level 2-5 20 pt Ericsson Confidential2007-07-2018