1. Test Execution Effort and Capacity Estimation Eduardo Aranha and Paulo Borba Informatics Center Federal University of Pernambuco Recife, PE, Brazil {ehsa, phmb}@cin.ufpe.br Mobile Devices R&D Motorola Industrial Ltda Jaguariuna, SP – Brazil COCOMO FORUM 2006

2. Mobile Devices Software Group Page 1 Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc. 2004. Software Testing Companies that release products with poor quality may quickly lose its clients A usual activity performed to ensure quality is software testing Test team Developers teams Developer team

3. Mobile Devices Software Group Page 2 Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc. 2004. Research Context Software Requirements Test Cases Requirements and Code Coverage Test Execution Effort Estimates Test Generation Effort Estimation Coverage Analysis Test Case Selection Test Case Selection Test Team and Resource Planning Planning Quality Model This Research Others Motorola Brazil Test Center Researches Some Others Motorola Researches Motorola Brazil Test Center Research Project

4. Mobile Devices Software Group Page 3 Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc. 2004. Test Specification Language Controlled natural language (CNL) –Subset of English with restricted grammar and vocabulary –Domain specific language (e.g., mobile phone application) –Sentences written in a more concise and standard way –Make easer to understand and to have text processing Example of test procedure: StepDescriptionExpected Results 1Start the message center.The phone is in message center. 2Select the new message option.The phone is in message composer. 3 Insert a recipient address into the recipients field. The recipients field is filled. 4 Insert a SMS content into the message body. The message body is populated. 5Send the message. The send message transient is displayed. The message is sent.

5. Mobile Devices Software Group Page 4 Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc. 2004. Measuring Test Execution Complexity of a Test Case For each single test action (step) –Evaluate execution complexity levels –Rate execution complexity levels –Add points to the whole test execution complexity Example: StepDescriptionExpected Results 1Start the message center.The phone is in message center. 2Select the new message option.The phone is in message composer. 3 Insert a recipient address into the recipients field. The recipients field is filled. 4 Insert a SMS content into the message body. The message body is populated. 5Send the message. The send message transient is displayed. The message is sent. 100 Points Complexity LevelCharactistic 78Total... 20 LowNFC2 0 N/ANFC1... 10 HighFC2 30 LowFC1 2 Total: 630 execution points 31 Points Complexity LevelCharactistic Total... LowNFC2 N/ANFC1... HighFC2 LowFC1 Points Complexity LevelCharactistic Total... NFC2 NFC1... FC2 FC1

6. Mobile Devices Software Group Page 5 Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc. 2004. Step 1 – Evaluating execution complexity levels Knowledge base for the execution complexity levels of the test action –Verb and optionally its arguments mapped into complexity levels If it is a new action: –Evaluate it according to the exercised system functional and non functional characteristics –Record evaluation in the knowledge base FC2Average size of date inputs. Less than 3030 to 100More than 100 LowMediumHigh NFC2Use of Bluetooth. File transfer, print messages, use headset Application data synchronization LowMediumHigh

7. Mobile Devices Software Group Page 6 Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc. 2004. Step 2 and 3 Step 2: Rating execution complexity levels –Search the knowledge base for the values assigned for each execution complexity levels of each characteristic Step 3: Assigning the value for the test case –The execution complexity of a test case is calculated by summing the execution points of each of its test steps FC2Average size of date inputs. LowMediumHigh 1560150 NFC2Use of Bluetooth. LowMediumHigh 3050

8. Mobile Devices Software Group Page 7 Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc. 2004. Using Test Execution Complexity for Effort Estimation Test productivity –The proportion between execution points and time spent for executing each test –For example, a test with 600 execution points executed in 5 minutes gives a productivity of 120 execution points / minute –A new test case with 160 execution points can be estimated to be executed in approximately 1 minute and 20 seconds Risk factors –Environment conditions, team experience, the use of tools, the reuse of test setups and other factors must be considered –Effort multipliers

9. Mobile Devices Software Group Page 8 Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc. 2004. Method Validation Initial experiment –We observed the relation between execution times and the measured complexity of test cases We executed and analyzed 33 tests in a controlled environment –We extended the characteristics using a Delphi panel –Achieved improvements with respect to the current approach: TestMMREPRED(20)PRED(30) 1 36.75%100.00%37.50% 2 36.12%33.33%25.00% 3 17.19%50.00%0.00%

10. Mobile Devices Software Group Page 9 Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc. 2004. Test Execution Capacity Estimation Simulation of a high number of different test requests Average number of tests per test suite Average number of execution points per test Average execution effort per test suite Current manpower Current test execution capacity

11. Mobile Devices Software Group Page 10 Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc. 2004. Conclusions New method for measuring test execution complexity Applicable for new or existing tests CNL improves specifications and supports automation –Knowledge base reduces estimation effort Some drawbacks –Effort to create the knowledge base –CNL, functional and non functional characteristics are dependent of application domain More experiments to run –ANOVA, covariance –Estimation accuracy, regression analysis, Bayesian calibration, etc.

12. Mobile Devices Software Group Page 11 Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc. 2004. Questions?