Error and Defect Management in context of IFDK reference product

About this course material -This material if for general training for Error and Defect Management -Material is more supportive in class room -Material will be updated during courses -FreeNest Portable Project Platform is used to demonstrate things only in practice. This is not limiting usage for material for other training environments (I hope ) About material

Customer/Business Requirements Sub System Requirements Component Requirements Component / Unit Testing Integration Testing System Testing Acceptance Testing System Requirements Architecture& Design& Implementatio n Product VALIDATION VERIFICATION Error Database Error Database MAINTENANCE Error Database & Management covers whole product life cycle PRODUCT END timelinetimeline

Test Plan Life Cycle Test Plan 1.0 Test Plan 1.0 Week 0 Week 26 Week 52 REL 0.1 REL 0.2 REL 0.5 REL 1.0 Test Plan 2.x Test Plan 2.x Routine Test Needs Still Work Effort Needs Still Work Effort

Testing Levels and Error Management UNIT TESTING – Developer's area Implementation Test Framework Error Database Error Database

System & Acceptace Testing Implem entati on Test Framewo rk Error Database Error Database Te st Pl an Te st Pl an Implem entati on Test Framewo rk Component 2 Test Plan Test Plan IFDK HARDWARE DESIGN IFDK HARDWARE DESIGN IFDK HARDWARE PROTOTYPE VERIFICATED IFDK HARDWARE PROTOTYPE VERIFICATED

Integration Testing Implementa tion Test Framework Error Database Error Database Test Plan Test Plan Implementa tion Test Framework Component 2

Error/Bug/Defect Report ● Defect/Burg/Error ID ● Reporter ● Time ● Founded where ● Which way? ● Test Case ● Test Setup/Configuration ● Describe scenario? ● Attachements? Picture/Log/etc.. ● Defect/Burg/Error ID ● Reporter ● Time ● Founded where ● Which way? ● Test Case ● Test Setup/Configuration ● Describe scenario? ● Attachements? Picture/Log/etc.. Error Database Error Database Id1 Id2 Id1 Id3 Id1

Hands On: Bugzilla Error Database ion-list/ ion-list/ What is Bugzilla? Bugzilla is a "Defect Tracking System" or "Bug-Tracking System". Defect Tracking Systems allow individual or groups of developers to keep track of outstanding bugs in their product effectively. Most commercial defect-tracking software vendors charge enormous licensing fees. Despite being "free", Bugzilla has many features its expensive counterparts lack. Consequently, Bugzilla has quickly become a favorite of thousands of organizations across the globe. What is Bugzilla? Bugzilla is a "Defect Tracking System" or "Bug-Tracking System". Defect Tracking Systems allow individual or groups of developers to keep track of outstanding bugs in their product effectively. Most commercial defect-tracking software vendors charge enormous licensing fees. Despite being "free", Bugzilla has many features its expensive counterparts lack. Consequently, Bugzilla has quickly become a favorite of thousands of organizations across the globe.

Error Management – Bugzilla Test Management Database Test Management Database Feature Implementation Feature Implementation TestCase Bug Report Error Database Error Database PASS FAIL TestCase Test Plan/Suite SourceCode Version Control SourceCode Version Control BUGZILLA TESTLINK

Other Defect Database Solutions JIRA – Commercial Requisite Pro – Commercial Rational Synergy - Commercial Mantis – Open Source

Bugzilla

Reporting, Metrics and daily usage

Feature or Bug? That's a question!

Definitions Failure - Fault, Defect, Bug - Incident, Failure, Error Example forum thread: ISTQB syllabus Yläotsikko

● TEST CASE ID XXXXX ● Step1 ● Step2 ● Step3. ● Step4. INCIDENT (Huomio) Bug/Defect (Vika) Defect Database IFDK System EXECUTE TEST ! Test Engineer Notes Reports What means error reporting? Yläotsikko

Error/Bug/Defect Report ● Defect/Burg/Error ID ● Reporter ● Time ● Founded where ● Which way? ● Test Case ● Test Setup/Configuration ● Describe scenario? ● Attachements? Picture/Log/etc.. ● Defect/Burg/Error ID ● Reporter ● Time ● Founded where ● Which way? ● Test Case ● Test Setup/Configuration ● Describe scenario? ● Attachements? Picture/Log/etc.. Yläotsikko

About Error Management Requirement Management Requirement Management Implementation Process Implementation Process Verificat ion& Validatio n Verificat ion& Validatio n Failure Report Failure Report Yläotsikko

Example Sources for error report CRM Field Testing Testing Process Customer Feedback / Customer Feedback / Error Report Change Request? Change Request? N x Incidents Yläotsikko

Change Management Sometimes founded defect can lead to change Bug? Change Request? Not Clear Requirements Not Clear Requirements Feature ? Yläotsikko

Error Reporting and Process Yläotsikko

Hands On: Bugzilla Error Database ion-list/ ion-list/ What is Bugzilla? Bugzilla is a "Defect Tracking System" or "Bug-Tracking System". Defect Tracking Systems allow individual or groups of developers to keep track of outstanding bugs in their product effectively. Most commercial defect-tracking software vendors charge enormous licensing fees. Despite being "free", Bugzilla has many features its expensive counterparts lack. Consequently, Bugzilla has quickly become a favorite of thousands of organizations across the globe. What is Bugzilla? Bugzilla is a "Defect Tracking System" or "Bug-Tracking System". Defect Tracking Systems allow individual or groups of developers to keep track of outstanding bugs in their product effectively. Most commercial defect-tracking software vendors charge enormous licensing fees. Despite being "free", Bugzilla has many features its expensive counterparts lack. Consequently, Bugzilla has quickly become a favorite of thousands of organizations across the globe. Yläotsikko

Other Defect Database Solutions JIRA – Commercial Requisite Pro – Commercial Rational Synergy - Commercial Mantis – Open Source Yläotsikko

Bugzilla Yläotsikko

Reporting, Metrics and daily usage Yläotsikko

CMMI Process framework CMMI – covers ”error management” in several process areas SP3.2 Analyze Verification results Typical Work Products: Trouble reports - Analyze the verificationd ata on defects - Record all results of the analysis in a report - Provide infromation on how defects can be resolved (including verfiocation methods, criteria, and verification environment) and initiate corrective action Project Monitoring and Control SG2 Manage Corrective Action to Closure SP2.1 Analyze Issues SP2.2 Take corrective Action SP2.3 Manage corrective Action SG2 Validate Product or Product Components SP2.2 Analyze Validation Results. Change request management & Configuration Management process SG2 Track and Control Changes SP2.1 Track Change Requests SP2.2 Control Configuration Items Yläotsikko

Traditional SW Project vs Open Source Project Open Source – Crowd Sourcing SW Relase tested without coordination by group of volunteers Release tested by customer Field Testing Test Group Yläotsikko

Example of IFDK product concept IFDK = Internal Flame Drum Kit Requirement Management

Example of product architecture Yläotsikko

Customer/Business Requirements Customer/Business Requirements Sub System Requirements Sub System Requirements Component Requirements Component Requirements Component / Unit Testing Component / Unit Testing Integration Testing Integration Testing System Testing System Testing Acceptance Testing Acceptance Testing System Requirements System Requirements IFDK System Verification and Validation IFDK System Verification and Validation Use Cases User Storys Features Requirements Validation = Are we building the right product? Verification = Are we building the product right? Validation = Are we building the right product? Verification = Are we building the product right? Architecture& Design& Implementation Architecture& Design& Implementation Product VALIDATION VERIFICATION IFDK Product Ideas Yläotsikko VALIDATION vs VERIFICATION?

UML OMT System Engineering Google: SysML, UML, Systems engineering, Software Design, Code Software Engineering - Architecture SysML UML OMT.NET, JAVA, C++ Software Engineering - Design Requirements

Yläotsikko Seller Customer Developer Different aspects to product QA Tester

Different requirement levels Customer/Business/Stake Holder Requirements System Requirements Design Requirements Component Requirements Implementation Scalability Stability Performance Security Performance Stress Usabilty

System Testing Integration Testing Unit Testing Customer/Business/Stake Holder Requirements System Requirements Design Requirements Component Requirements Acceptance Testing Why we need requirements from testing point of view? Implementation „Developer's Area“ „Test Engineers Area“ ”Traditional Testing Levels”

Customer Requirements Sub System Requirements Component RequirementsComponent/Unit Testing Integration Testing System Testing Acceptance Testing System Requirements Verification and Validation Verification = Are we building the product right? Validation = Are we building the right product? Yläotsikko

IDEAL Requirements help Design Requirements help Design REQ-X REQ-Y REQ-Z REQ-O REQ Yläotsikko

Defines Feature X * n Functional Requirements Functional Requirements Non-Functional Requirements Non-Functional Requirements Use Cases Vision of product Simple Requirement Management Process Problem DomainSolution Domain Solution Proposal Customer/Marketing/ business User Storys FEATURE VISION/NEED/PROPOSA L FEATURE VISION/NEED/PROPOSA L YOU! Design documents & implementation Test Case Yläotsikko

Features Test Case Use Cases User Storys Product Design & Implementation Requirements Test Case Ready to test Testing & Quality Assurance Can we deliver Product Ready to Deliver ? Customer Not ready to deliver Yläotsikko

SW Development Process (Waterfall) Requirement Gathering/Evaluatio n Requirement Gathering/Evaluatio n Design Implementation Verification & Validation Verification & Validation Maintenance Error Managment Process Error Managment Process Task1 Mile Stone 1 Mile Stone 2 Mile Stone 3 Task1

Black Box vs White Box Testing Unit Testing is White Box testing Black Box Testing for selected component ? ? ? ? Yläotsikko

Component /Unit Testing Class Attributes Class Methods Class Attributes Class Methods Class Attributes Class Methods Class Attributes TestClass Methods Unit Test Frame Work Result Yläotsikko

Component Testing Customer/Business Requirements Customer/Business Requirements Sub System Requirements Sub System Requirements Component Requirements Component Requirements Component / Unit Testing Component / Unit Testing Integration Testing Integration Testing System Testing System Testing Acceptance Testing Acceptance Testing System Requirements System Requirements Architecture& Design& Implementatio n Architecture& Design& Implementatio n Product VALIDATION VERIFICATION Yläotsikko

Create a Test Case! Functional? Your Sources For Test Case Functional Test Case: ● Verify functionality of XXXX Non-Functional Test Cases ● Verify Stability of XXXX ● Verify Performance of XXX ● Verify Security of XXXX ● Verify Usability of XXXX ● Verify Scalability of XXXX ● etc... Non-Functional? Which Type? ● Requirement ● Use Case ● Feature ● User Story ● Customer's Idea ● Brainstorm ● Intitution ● Exploratory Check also..... ● Correct functionality path ● Miss-usage of functionality ● Boundary Check Check also..... ● Check Possiblity to automated testing? How to create Test Case??? Regression Test Case?? Write a Case Yläotsikko

Heading -Sed posuere interdum sem. -Quisque ligula eros ullamcorper quis, lacinia quis facilisis sed sapien. -Mauris varius diam vitae arcu. Sed arcu lectus auctor vitae, consectetuer et venenatis eget velit. -Sed augue orci, lacinia eu tincidunt et eleifend nec lacus. Yläotsikko

Feature X * n Calory Counter: Player can measure calories during training session. This can be seen as exercise result in web service eg. Facebook application Energy usage Yläotsikko

SW Development Process (Agile) User Story Y Sprint Task1 Task2 Task3 Sprint Task4 Task5 Task6 User Story X Design Implementation Verification Design Implementation Verification Product Backlog Design Implementation Verification Design Implementation Verification Design Implementation Verification Design Implementation Verification Design Implementation Verification Design Implementation Verification User Story Z Task7 Task8 Task9 User Story Z Yläotsikko

Testing according ISEB standard Functional Testing Testing based on an analysis of the specification of the functionality of a component or system. See also black box testing.) Non-Functional Testing Testing the attributes of a component or system that do not relate to functionality, e.g. scalabilty, stability, reliability, efficiency, usability, maintainability and portability. Yläotsikko

We need to capsule all projects as sub projects, which have more independence Project Teams has to have own test engineer –> Integration Test Engineer this lead’s to better defect prevention Integration Test Engineer is part of project team, part of team at start of project Test Engineer provides valuable information for design and enables better testability for project product Define new roles for validation engineers Check next page Lower level for test automation Enables regression test (method to block new implementation defects) Support for executing more complex test scenarios Continuous test execution (stability, performance gain etc) Notes Yläotsikko

Test Level1 – Component/Unit testing Component/Class level unit testing (eg. xUnit framework) Test Level2 - Integration Testing (Feature, Application) Integration test for feature/application with “stub” interface components Functionality SW Testing Non-Functional SW Testing according needs (eg. feature/application specific performance, stability) Test Level3 – Regression Testing (Target HW+Android platform, tool, terminal etc.) Functional SW testing for all possible features Non-Functional SW testing according needs Non-Functional HW Testing according needs Test Level4 – System Integration Testing (Target Platform, Tool, Terminal etc) Functional SW testing for all new features Non-Functional SW Performance, Stability, Scalability Test Level5 – Acceptance Testing Functional/Non-Functional test according customer requirements HWT1 – Hardware Testing HW Integration test with limited environment. Conducted performance verified Testing Level descriptions Yläotsikko

VERIFICATION & VALIDATION Validation = Are we building the right product? Verification = Are we building the product right? Yläotsikko

Integration testing using emulator Yläotsikko

Test Case SUT/DUT IFDK What is generated as results from test case execution LOG FILE EVENTS NOTIFICATIONS SUT = System Under Test DUT = Device Under Test ENVIRONMENT ANDROID EMULATOR TOOLS scripts/grep TEST CASE Yläotsikko

How to verify component implementation -Unit Testing -Code Coverage -Branch Coverage -Complexity Analyse -Unit Testing -Code Coverage -Branch Coverage -Complexity Analyse Yläotsikko

Code Complexity Example tool CCCC Yläotsikko

Branch coverage The percentage of branches that have been exercised by a test suite. 100% branch coverage implies both 100% decision coverage and 100% statement coverage. Yläotsikko

Code Coverage An analysis method that determines which parts of the software have been executed (covered) by the test suite and which parts have not been executed, e.g. statement coverage, decision coverage or condition coverage. Yläotsikko

Release/Configuration Management & Integration Testing Day 6 Yläotsikko

Putting all tools together! Continous Integration Yläotsikko

Design verification - Unit Testing Yläotsikko

Unit Test example Re-run with Meego Example of xUnit in QT environment Yläotsikko

Integration Testing Customer/Business Requirements Sub System Requirements Component Requirements Component / Unit Testing Integration Testing System Testing Acceptance Testing System Requirements Architecture& Design& Implementatio n Product VALIDATION VERIFICATION Yläotsikko

Why Integrate first? Avoid Big Bang! HW Component Data Base Component/Application 10% tested Component/Application 10% tested Web Service Tested Component/Application Yläotsikko

Integration Test with stubs Tested Component/Application Log STUB/MOCK Component Scripted STUB Interface Control Configure Control Configure Simulated Interface Simulated Interface Messages/Events STUB/MOCK Component Control Interface Yläotsikko

in practice #1 IFDK android setup Tested Component Application Tested Component Application Activate/Control STUB/MOCK Component Scripted STUB Interface Control Configure Control Configure Simulated Interface Simulated Interface Messages/Events WEB SERVER simulating Service interface WEB SERVER simulating Service interface Control Interface Trace/Log Yläotsikko

in practice #2 server component testing Tested Component Application Tested Component Application Trace/Log Activate/Control Mock Server/Daemon Scripted STUB Interface Automated Test Interface Automated Test Interface Simulated Interface Simulated Interface Messages/Events WEB SERVER Control Interface Operating System Needed Fake Application Needed Fake Application Junit Scripted Interface Yläotsikko

Release Management and Integration Testing IFDK Application Facebook Web Service HW Component Calore Meter Enabled Drum Stick HW Component Calore Meter Enabled Drum Stick Calore Meter SW Component 10% tested Calore Meter SW Component 10% tested Data Base Schema Design Stubs neede d Stubs neede d Stubs neede d Stubs neede d Stubs neede d Stubs neede d Stubs neede d Stubs neede d First System Test With all components First System Test With all components Yläotsikko

Trace/Log as feedback Log contains important information. Log Simple tool for log analysing in Linux ”grep” command, TAIL -F /var/log/messages | grep error -Specific Inhouse log capturing and analyse tools Yläotsikko

Agenda Brief guidance for NEST Project Platform 1.3 Background story for reference project IFDK Some Definitions SW Development Process Release + Configuration Management Testing Levels and Error Management Hands On: Test Link + Bugzilla Error Reporting, Metrics and daily usage Closed Software Project vs Open Source Project Discussion Yläotsikko

Release Management Version 0.1 Version 0.2 Version 0.3 Version Version Version Version 0.4 Trunk Customer 1 Version Version Version Yläotsikko

Release & Configuration Managmement Version 0.1 Version 0.2 Version 0.3 Version Version Version Version 0.4 Trunk Customer 1 Version Version Version Feature s Release 1.0 Feature s Yläotsikko

Release Planning Yläotsikko

Validaton& Verificaton (Testing) Management Version 0.4 Version Version Test Plan Test Cases For Features Test Plan Test Cases For Features Tested Release/configuration Error/De fect Report Error/De fect Report Error/De fect Report Error/De fect Report Error/De fect Report Error/De fect Report Yläotsikko

Feature Pack Project Manager Designer/Coder Integration Test Engineer Integration Test Engineer Test Manager System Testing IFDK System Testing Feature Unit/Integration Testing IFDK System Acceptance Testing System Test Engineer System Test Engineer Test Automation Engineer Test Automation Engineer Acceptance Test Engineer Acceptance Test Engineer Validation Verification IFDK Verification/Validation (Testing Organization) Regression Testing Integration Testing System Testing Acceptance Testing Test Automation Unit Testing Product Release Yläotsikko