2. Test Design and Management in context of IFDK reference product

3. About this course material -This material if for general training for Test Design and management -Material is more supportive in class room -Material will be updated during courses -FreeNest Portable Project Platform is used to demonstrate things only in practice. This is not limiting usage for material for other training environments (I hope ) About material

4. Example of IFDK product concept IFDK = Internal Flame Drum Kit Requirement Management

5. Yläotsikko Seller Customer Developer Different aspects to product QA Tester

6. Different requirement levels Customer/Business/Stake Holder Requirements System Requirements Design Requirements Component Requirements Implementation Scalability Stability Performance Security Performance Stress Usabilty

7. Customer Requirements Sub System Requirements Component RequirementsComponent/Unit Testing Integration Testing System Testing Acceptance Testing System Requirements Verification and Validation Verification = Are we building the product right? Validation = Are we building the right product? Yläotsikko

8. System Testing Integration Testing Unit Testing Customer/Business/Stake Holder Requirements System Requirements Design Requirements Component Requirements Acceptance Testing Why we need requirements from testing point of view? Implementation „Developer's Area“ „Test Engineers Area“ ”Traditional Testing Levels”

9. Customer/Business Requirements Customer/Business Requirements Sub System Requirements Sub System Requirements Component Requirements Component Requirements Component / Unit Testing Component / Unit Testing Integration Testing Integration Testing System Testing System Testing Acceptance Testing Acceptance Testing System Requirements System Requirements IFDK System Verification and Validation IFDK System Verification and Validation Use Cases User Storys Features Requirements Validation = Are we building the right product? Verification = Are we building the product right? Validation = Are we building the right product? Verification = Are we building the product right? Architecture& Design& Implementation Architecture& Design& Implementation Product VALIDATION VERIFICATION IFDK Product Ideas Yläotsikko VALIDATION vs VERIFICATION?

10. Features Test Case Use Cases User Storys Product Design & Implementation Requirements Test Case Ready to test Testing & Quality Assurance Can we deliver Product Ready to Deliver ? Customer Not ready to deliver Yläotsikko

11. Features Test Case Use Cases User Storys Product Design & Implementation Requirements Test Case Ready to test Testing & Quality Assurance Can we deliver Product Ready to Deliver ? Customer Not ready to deliver Verification Validation Validation = Are we building the right product? Verification = Are we building the product right?

12. SW Development Process (Waterfall) Requirement Gathering/Evaluatio n Requirement Gathering/Evaluatio n Design Implementation Verification & Validation Verification & Validation Maintenance Error Managment Process Error Managment Process Task1 Mile Stone 1 Mile Stone 2 Mile Stone 3 Task1

13. White Box Testing Grey Box Testing Black Box Testing “SYSTEM TESTING” Perspective “CODE LEVEL TESTING” Perspective

14. Black Box vs White Box Testing Unit Testing is White Box testing Black Box Testing for selected component ? ? ? ? Yläotsikko

15. TESTING LEVELS

16. Unit/Module/Compone nt Testing Customer/Business Requirements Customer/Business Requirements Sub System Requirements Sub System Requirements Component Requirements Component Requirements Component / Unit Testing Component / Unit Testing Integration Testing Integration Testing System Testing System Testing Acceptance Testing Acceptance Testing System Requirements System Requirements Architecture& Design& Implementatio n Architecture& Design& Implementatio n Product VALIDATION VERIFICATION Yläotsikko

17. What should be tested? How ?

18. What should be tested? How ?

19. How to verify component implementation -Unit Testing -Code Coverage -Branch Coverage -Complexity Analyse -Unit Testing -Code Coverage -Branch Coverage -Complexity Analyse Yläotsikko

20. Component /Unit Testing Class Attributes Class Methods Class Attributes Class Methods Attributes TestClass TestMethodCall Unit Test Frame Work Test Method Call Method Result A=1 B=2 C=Class.TestMethodCo untValues(A+B) C<>3 FAIL C=3 PASS Implemented Class MethodCountValues( int x, int y) z=x+y+1 Return z Developer

21. Code Coverage An analysis method that determines which parts of the software have been executed (covered) by the test suite and which parts have not been executed, e.g. statement coverage, decision coverage or condition coverage. http://en.wikipedia.org/wiki/Code_coverage http://www.atlassian.com/software/clover/ Yläotsikko

22. Branch coverage The percentage of branches that have been exercised by a test suite. 100% branch coverage implies both 100% decision coverage and 100% statement coverage. Yläotsikko

23. Code Complexity Example tool CCCC https://wiki.jenkins- ci.org/display/JENKINS/CCCC+Plugin http://sourceforge.net/projects/codeanalyze- gpl/?source=recommended Yläotsikko

24. Integration Testing Customer/Business Requirements Sub System Requirements Component Requirements Component / Unit Testing Integration Testing System Testing Acceptance Testing System Requirements Architecture& Design& Implementatio n Product VALIDATION VERIFICATION Yläotsikko

26. http://prosentti.vero.fi/veropros_tietojen_syotto2011.asp What should be tested? How ?

27. Why Integrate first? Avoid Big Bang! HW Component Data Base Component/Application 10% tested Component/Application 10% tested Web Service Tested Component/Application Yläotsikko

28. System Testing Customer/Business Requirements Sub System Requirements Component Requirements Component / Unit Testing Integration Testing System Testing Acceptance Testing System Requirements Architecture& Design& Implementatio n Product VALIDATION VERIFICATION Yläotsikko

29. What should be tested? How ?

30. What should be tested? How ? Sales DB Room Reserva tion DB DB ? CRM DB Application & Gateways Application & Gateways Application & Gateways Application & Gateways

31. System Acceptance Testing Customer/Business Requirements Sub System Requirements Component Requirements Component / Unit Testing Integration Testing System Testing Acceptance Testing System Requirements Architecture& Design& Implementatio n Product VALIDATION VERIFICATION Yläotsikko

32. What should be tested before so customer could be so happy ? How ? http://www.123rf.com

33. Feature Pack Project Manager Designer/Coder Integration Test Engineer Integration Test Engineer Test Manager System Testing IFDK System Testing Feature Unit/Integration Testing IFDK System Acceptance Testing System Test Engineer System Test Engineer Error Manager Acceptance Test Engineer Acceptance Test Engineer Validation Verification IFDK Verification/Validation (Testing Organization) Regression Testing Integration Testing System Testing Acceptance Testing Test Automation Unit Testing Product Release Error Database Error Database Test Management Database Test Management Database Report Staus Report Staus Regression Testing

34. Input ? Output?

35. IDEAL Implementation REQ-X REQ-Y REQ-Z REQ-O REQ Yläotsikko What I should check ?

36. Discuss about reasons for test design? Why we need to do design? Stupid work ! I wan’t to progress!??

37. Test Case Design

38. Feature Pack Project Manager Designer/Coder Integration Test Engineer Integration Test Engineer Test Manager System Testing IFDK System Testing Feature Unit/Integration Testing IFDK System Acceptance Testing System Test Engineer System Test Engineer Error Manager Acceptance Test Engineer Acceptance Test Engineer Validation Verification UNDERSTAND YOUR TEST LEVEL Regression Testing Integration Testing System Testing Acceptance Testing Test Automation Unit Testing Product Release Error Database Error Database Test Management Database Test Management Database Report Staus Report Staus Regression Testing

39. Verify what? Using configuration? With tools? Verify what? Using configuration? With tools? What is verdict? Verify drum track player pause mode functionality. Do this with IFDK software release X and playing song ”Show must go on by Freddy Mercury” Test should be done using android emulator environment and using your hands, ears and eyes” Verify drum track player pause mode functionality. Do this with IFDK software release X and playing song ”Show must go on by Freddy Mercury” Test should be done using android emulator environment and using your hands, ears and eyes” Add Information about case Pre State: Android emulator is running Release X is installed on emulator Test Case Steps: 1. Open drum kit player application 2. Select song ”Show must go on” 3. Start to play 4. Press Pause and check song is paused 5. Check memory usage from system application 6. Press Play 7. jump to 4 several time (<10) 8. Listen song to the end 9. Exit player using ”exit button” End State: IFDK Kit in main screen mode Pre State: Android emulator is running Release X is installed on emulator Test Case Steps: 1. Open drum kit player application 2. Select song ”Show must go on” 3. Start to play 4. Press Pause and check song is paused 5. Check memory usage from system application 6. Press Play 7. jump to 4 several time (<10) 8. Listen song to the end 9. Exit player using ”exit button” End State: IFDK Kit in main screen mode Test Case Id Test Case owner/writer Date comments Test Case Id Test Case owner/writer Date comments If Pause is working result is PASS. If Pause mode failed result is FAIL Define pre-state Define Steps Define end-state Define pre-state Define Steps Define end-state What Information Test Case should contain?

40. Stupid work! This takes ages! This Test Case documentation is old as soon I have changed some implementation? Why you need to do so hard documentation? Give me a one good reason!

41. Checklist? Check UI is workingWorking? Check color changeWorking? Check Counter value after 50 loginsWorking? Check disable mode for counterWorking? Check UI is workingWorking? Check color changeWorking? Check Counter value after 50 loginsWorking? Check disable mode for counterWorking? Checklist can be working great in small team! What happens if team is disbanded to other projects? And you are new maintainer for this project?

42. We have to automate all tests!! No sense to create documentation ? Who does automation without a design?

43. Test Design Method Negative Testing Requirement based test design Defect based test design Design based test desing Functional test design NonFunctional test design Specification based Testing design Test Case Step Test Case Step Customer's Idea Brainstorm Intitution Exploratory Customer's Idea Brainstorm Intitution Exploratory Test Engineer’s Daily Job? Test Engineer’s Daily Job? MixedMixed MixedMixed Write

44. What is Testing level What is Testing level Check different sources & strategy for Test Case design Check different sources & strategy for Test Case design Functional Test Case Non-Functional Test Cases How to create Test Case??? Acceptance Test Case Field Test Case Interoperability Test Case Conformance Test Case Regression Test Case Choose Case Type WRITE A Test Case! WRITE A Test Case!

45. Test Case Design in agile framework Definition for test cases Definition for Acceptance Criteria Definition for Acceptance Criteria Check list 1.Verify Test 2.Verify Test 3.Verify Test 4.Verify Test Check list 1.Verify Test 2.Verify Test 3.Verify Test 4.Verify Test User Story: Tested using test automation? Tested using test automation? As a user I would like to use my google account for login Implementation Done Implementation Done

46. Test Driven Development TDD in all levels! Test Driven Development TDD in all levels! Design Tests Implement Code Design draft Tests Case Define Architecture & Design?

47. Unit Testing

48. Test Driven Development and Unit Testing Class Attributes Class Methods Class Attributes Class Methods Attributes TestClass TestMethodCall Unit Test Frame Work Test Method Call Method Result A=1 B=2 C=Class.TestMethodCo untValues(A+B) C<>3 FAIL C=3 PASS Implemented Class MethodCountValues( int x, int y) z=x+y+1 Return z Developer DEFINE TEST CASES FIRST!!! DEFINE TEST CASES FIRST!!! IMPLEMENT CODE AGAINST TESTS IMPLEMENT CODE AGAINST TESTS

49. Implemented Software Component #1 Implemented Software Component #1 Implemented Software Component #3 Implemented Software Component #3 Implemented Software Component #4 Implemented Software Component #4 Implemented Software Component #2 Implemented Software Component #2 Software Product Integration Software Product Integration Developer 1 Developer 2 Developer 3 Developer 4 Test Sand Box TESTS Integration test engineer #2Integration test engineer #1

50. Integration Test with stubs/mocs Tested Component/Application Log STUB/MOCK Component Scripted STUB Interface Control Configure Control Configure Simulated Interface Simulated Interface Messages/Events STUB/MOCK Component Control Interface

51. Integration Testing with simulated Interfaces Fake Spotify Server Fake Spotify Server Fake Facebook Server Fake Facebook Server Simulated Bluetooth Interface Simulated Bluetooth Interface Simulated MIDI Interface Simulated MIDI Interface

52. Integration Test with stubs Tested Component/Application Log STUB/MOCK Component Scripted STUB Interface Control Configure Control Configure Simulated Interface Simulated Interface Messages/Events STUB/MOCK Component Control Interface Yläotsikko

53. in practice #1 IFDK android setup Tested Component Application Tested Component Application Activate/Control STUB/MOCK Component Scripted STUB Interface Control Configure Control Configure Simulated Interface Simulated Interface Messages/Events WEB SERVER simulating Service interface WEB SERVER simulating Service interface Control Interface Trace/Log Yläotsikko

54. in practice #2 server component testing Tested Component Application Tested Component Application Trace/Log Activate/Control Mock Server/Daemon Scripted STUB Interface Automated Test Interface Automated Test Interface Simulated Interface Simulated Interface Messages/Events WEB SERVER Control Interface Operating System Needed Fake Application Needed Fake Application Junit Scripted Interface Yläotsikko

55. Integration Testing Implementa tion Test Framework Error Database Error Database Test Plan Test Plan Implementa tion Test Framework Component 2

56. Regression Test Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It is performed when the software or its environment is Changed. (ISTQB Syllabus) Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It is performed when the software or its environment is Changed. (ISTQB Syllabus)

57. Regression testing reason for automation? Test Case #1 Test Case #1 Test Case #3 Test Case #3 Test Case #7 Test Case #7 Test Case #4 Test Case #4 Test Case #8 Test Case #8 REGRESSION TEST PLAN REGRESSION TEST PLAN Test Automation Solution Execute Regression Test when needed? ? ? Export Results as Report Export Results as Report

58. Create a Test Case! Functional? Your Sources For Test Case Functional Test Case: ● Verify functionality of XXXX Non-Functional Test Cases ● Verify Stability of XXXX ● Verify Performance of XXX ● Verify Security of XXXX ● Verify Usability of XXXX ● Verify Scalability of XXXX ● etc... Non-Functional? Which Type? ● Requirement ● Use Case ● Feature ● User Story ● Customer's Idea ● Brainstorm ● Intitution ● Exploratory Check also..... ● Correct functionality path ● Miss-usage of functionality ● Boundary Check Check also..... ● Check Possiblity to automated testing? How to create Test Case??? Regression Test Case?? Write a Case Yläotsikko

59. Feature Pack Project Manager Designer/Coder Integration Test Engineer Integration Test Engineer Test Manager System Testing IFDK System Testing Feature Unit/Integration Testing IFDK System Acceptance Testing System Test Engineer System Test Engineer Error Manager Acceptance Test Engineer Acceptance Test Engineer Validation Verification IFDK Verification/Validation (Testing Organization) Regression Testing Integration Testing System Testing Acceptance Testing Test Automation Unit Testing Product Release Error Database Error Database Test Management Database Test Management Database Report Staus Report Staus

60. What should be tested? How ? DB Appcliation

61. What should be tested? How ? Application

63. Using Testlink How to write test case? Creating a test plan

64. About Testlink TestLink is a web based Test Management tool. The application provides Test specification, Test plans and execution, Reporting, Requirements specification and collaborate with well-known bug trackers.

65. Test Management – Testlink Test Management Database Test Management Database Feature Implementation Feature Implementation TestCase Bug Report Error Database Error Database PASS FAIL TestCase Test Plan/Suite SourceCode Version Control SourceCode Version Control TESTLINK

66. Test Plan Life Cycle Test Plan 1.0 Test Plan 1.0 Week 0 Week 26 Week 52 REL 0.1 REL 0.2 REL 0.5 REL 1.0 Test Plan 2.x Test Plan 2.x Routine Test Needs Still Work Effort Needs Still Work Effort

67. Test Case 1 Test Case 2 V2.0 Test Case 3 V2.0 Test Case 1 Test Case 2 V2.0 Test Case 3 V2.0 Test Plan V 0.1

68. Test Case 1 Test Case 2 V1.0 Test Case 3 V1.0 Test Case 2 V2.0 Test Case 2 V2.0 Test Case 2 V3.0 Test Case 4 Draft

69. What we can do with Testlink Tool

70. Test Case 1 Test Case 2 V2.0 Test Case 2 V2.0 Test Case 2 V2.0 Test Case 2 V2.0 Test Plan V 0.1

71. What should be tested? Priority setting Test Case 1 Test Case 2 V2.0 Test Case 3 V2.0 P1 P3 P4

72. Test Case 1 Test Case 2 V2.0 Test Case 2 V2.0 Test Case 3 V2.0 Test Case 3 V2.0 Test Plan V 0.1 PASS / FAIL Executes Product RELEASE/BUILD VERSION 1.1 Product RELEASE/BUILD VERSION 1.1 Product RELEASE/BUILD VERSION 1.3 Product RELEASE/BUILD VERSION 1.3 Product RELEASE/BUILD VERSION 1.2 Product RELEASE/BUILD VERSION 1.2

73. Setting Configuration Version 0.4 Version 0.2.2.2 Version 0.2.2 Test Plan Test Cases For Features Test Plan Test Cases For Features Tested Release/configuration Error/De fect Report Error/De fect Report Error/De fect Report Error/De fect Report Error/De fect Report Error/De fect Report

74. Testlink use….

75. Needed Tools TESTLINK: http://sourceforge.net/projects/testlink/ BUGZILLA: http://www.bugzilla.org/

76. Execution of Test Case Test Engineer daily job ? Yläotsikko

78. Test Case SUT/DUT ”IFDK- unit” SUT/DUT ”IFDK- unit” What is generated as results from test case execution LOG FILE Indications Events NOTIFICATIONS SUT = System Under Test DUT = Device Under Test ENVIRONMENT/ EMULATOR TOOLS TEST CASE Yläotsikko

79. Regression Test Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It is performed when the software or its environment is Changed. (ISTQB Syllabus) Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It is performed when the software or its environment is Changed. (ISTQB Syllabus)

80. Test Plan & Regression Test Case #1 Test Case #1 Test Case #3 Test Case #3 Test Case #5 Test Case #5 Test Case #7 Test Case #7 Test Case #9 Test Case #9 Test Case #2 Test Case #2 Test Case #4 Test Case #4 Test Case #6 Test Case #6 Test Case #8 Test Case #8 Test Case #10 Test Case #10 TEST PLAN 1 Test Case #1 Test Case #1 Test Case #3 Test Case #3 Test Case #7 Test Case #7 Test Case #4 Test Case #4 Test Case #8 Test Case #8 REGRESSION TEST PLAN REGRESSION TEST PLAN Test Case #5 Test Case #5 Test Case #9 Test Case #9 Test Case #2 Test Case #2 Test Case #6 Test Case #6 Test Case #10 Test Case #10 Test Case #9 Test Case #9 Test Case #10 Test Case #10 Test Case #11 Test Case #11 TEST PLAN 1.1 Test Case #12 Test Case #12 Test Case #13 Test Case #13 New TCases Added! New TCases Added!

81. Regression testing reason for automation? Test Case #1 Test Case #1 Test Case #3 Test Case #3 Test Case #7 Test Case #7 Test Case #4 Test Case #4 Test Case #8 Test Case #8 REGRESSION TEST PLAN REGRESSION TEST PLAN Test Automation Solution Execute Regression Test when needed? ? ? Export Results as Report Export Results as Report

82. Feature Component Feature Component Project Manager Designer/Coder Integration Test Engineer Integration Test Engineer Test Manager System Testing Feature Unit/Integration Testing System Acceptance Testing System Test Engineer System Test Engineer Test Automation Engineer Test Automation Engineer Acceptance Test Engineer Acceptance Test Engineer Validation Verification Product verification/validation and test automation Regression Testing Integration Testing Functional System Testing Acceptance Testing Unit Testing Product Release Component Integration Testing A A A A A Load, Stress, performance Load, Stress, performance

83. Heading -Sed posuere interdum sem. -Quisque ligula eros ullamcorper quis, lacinia quis facilisis sed sapien. -Mauris varius diam vitae arcu. Sed arcu lectus auctor vitae, consectetuer et venenatis eget velit. -Sed augue orci, lacinia eu tincidunt et eleifend nec lacus. Yläotsikko

84. Component Design Class Attributes Class Methods Class Attributes Class Methods Yläotsikko

85. Feature X * n Calory Counter: Player can measure calories during training session. This can be seen as exercise result in web service eg. Facebook application Energy usage Yläotsikko

86. SW Development Process (Agile) User Story Y Sprint Task1 Task2 Task3 Sprint Task4 Task5 Task6 User Story X Design Implementation Verification Design Implementation Verification Product Backlog Design Implementation Verification Design Implementation Verification Design Implementation Verification Design Implementation Verification Design Implementation Verification Design Implementation Verification User Story Z Task7 Task8 Task9 User Story Z Yläotsikko