Presentation on theme: "COMP8130 4130Adrian Marshall Verification and Validation Processes Introduction Adrian Marshall Intro."— Presentation transcript:
COMP8130 4130Adrian Marshall Verification and Validation Processes Introduction Adrian Marshall Intro
COMP8130 4130Adrian Marshall Agenda Introduction Definitions V&V Test objectives Testing challenges Testing and the V model Testing Approaches Testing levels Classes of test Risk based testing Requirements based testing Test Methods Common testing types Testing tools
COMP8130 4130Adrian Marshall Intro Verification IEEE 1012: the process of determining whether or not the products of a given phase of the software development lifecycle fulfil the requirements established during the previous phase. ISO 12207: confirmation by examination and provision of objective evidence that specified requirements have been fulfilled
COMP8130 4130Adrian Marshall Intro Validation IEEE 1012: the process of evaluating software at the end of the software development process to ensure compliance with software requirements ISO 12207: confirmation by examination and provision of objective evidence that particular requirements for a specific intended use have been fulfilled
COMP8130 4130Adrian Marshall Intro Put more simply: We Verify that the output of each software phase meets its requirements, and We Validate that the software, at the end of the development effort, meets the overall intended use
COMP8130 4130Adrian Marshall Types of V & V Activities Requirements analysis and Traceability analysis Design analysis Interface analysis Implementation evaluation static - reviews, inspections, structure analysis dynamic - simulation, prototyping, execution time analysis formal - mathematical analysis of algorithms Testing Project & Management analysis
COMP8130 4130Adrian Marshall Sources of guidance on V & V V & V Standards IEEE 1012 - Software V & V Plans IEEE 1059 - Guide for Software V & V Plans IEEE 1028 - Software Reviews & Audits IEEE 829 - Software Test Documentation Related Standards ISO 12207 - Software Lifecycle Processes ISO 9126 - Software Quality Characteristics Text V & V of Modern Software -Intensive Systems - Schulmeyer & Mackenzie, Prentice Hall, 2000
COMP8130 4130Adrian Marshall Pros and Cons of V & V Positive early error detection better product quality better project planning better adherence to standards, methods and practices better decision support information Cost of detection & prevention < cost of corrective action Negative additional time and effort required for V&V activities additional cost (visible) Independence of V&V can be hard for small organisations
COMP8130 4130Adrian Marshall Defect Introduction by Phase What is known about the quality of software systems? Defect SourceMilitaryIndustry Requirements20.0%12.5% Design20.0%24.2% Code35.0%38.3% Documentation15.0%13.3% Bad Fixes10.0%11.7% Applied Software Measurement 2nd Edition, by Capers Jones. McGraw-Hill, 1997. ISBN: 0- 07-032826-9
COMP8130 4130Adrian Marshall Cost of Removing Defects What is known about the quality of software systems? Removal PhaseCost RatioExample Cost S/W Analysis1 : 1(say) $200 S/W Design2.5 : 1$500 Code & Unit Test5 : 1$1,000 Integration & Test36 : 1$7,200 Deployment150 : 1$30,000 Applied Software Measurement 2nd Edition, by Capers Jones. McGraw-Hill, 1997. ISBN: 0- 07-032826-9
COMP8130 4130Adrian Marshall Testing Definitions (1) Testing is the process of executing a program with the intent of finding errors Testing is an activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results Testing is the process by which we understand the status of the benefits and the risk associated with release of a software system Testing includes all activities associated with the planning, preparation, execution, and reporting of tests
COMP8130 4130Adrian Marshall Testing Definitions (2) “Testing cannot guarantee that errors are not present, rather it demonstrates that errors are present”….
COMP8130 4130Adrian Marshall Test Objectives Verifying the implementation of any or all products Requirements and solution validation Defect detection Provide assessment of deployment risks Provide performance & threshold benchmark data Establish testing processes, assets, data and skills for on-going testing activities
COMP8130 4130Adrian Marshall Testing Challenges Complete testing is not possible Testing work is creative and difficult Testing is costly Testing is often not seen as a core activity Testers aim to find and report problems
COMP8130 4130Adrian Marshall Testing Challenges Technical personnel often do not want to become testers, leaving testing to non-technical system users Testing requires independence Testing is often a critical path activity Testing is often trimmed to solve schedule or budget problems.…
COMP8130 4130Adrian Marshall Testing and the V Model TESTING ACTIVITIES BUILD ACTIVITIES Review requirements Analyse test requirements Install system Integrate components Test against system requirements Accept system Design component solution Test components Buy / build components Design solution Determine system requirements Determine business requirements Test against business requirements Review solution Develop Master Test Plan Test integration of components COMPONENT LEVEL BUSINESS REQUIREMENTS LEVEL SYSTEMS REQUIREMENTS LEVEL INTEGRATION LEVEL DESIGN LEVEL SYSTEM LEVEL ACCEPTANCE LEVEL Develop Detailed Test Plan/s DESIGN ACTIVITIES COMPONENT LEVEL
COMP8130 4130Adrian Marshall Testing Approaches Bottom up tests smallest components / sub functions first test drivers are required Top down tests major functional areas from the top down stubs are used where lower levels are incomplete Functional thread process / path-oriented approach which crosses unit boundaries Combined….
COMP8130 4130Adrian Marshall Unit / Component testing conducted to verify the implementation of the design for one software element (for example, unit, module, function, class instance, method) or a collection of software elements Integration an orderly progression of testing in which software elements, hardware elements, or both are incrementally combined and tested until the entire system has been integrated System the process of testing an integrated hardware and software system to verify that the system meets its specified requirements…. Testing Levels
COMP8130 4130Adrian Marshall Classes of Test White (or glass) box testing designed with knowledge of how the system is constructed aims to exercise the internal logical structure statements, decisions, paths & exception handling evaluated Black box testing designed without knowledge of how the system is constructed verifies that functional & performance requirements have been satisfied focuses on the external behaviour of the system…. Grey Box designed with some knowledge of how the system is constructed
COMP8130 4130Adrian Marshall White Box Testing White box testing techniques Control flow based testing (e.g. decision & statement coverage testing) Statement coverage – each statement is executed at least once Decision coverage – each conditional statement is executed at least once each way Complexity based testing – (eg McCabe cyclomatic complexity measure) – higher concentration of tests for more complex software Boundary case and exception handling
COMP8130 4130Adrian Marshall Black box testing techniques Equivalence partitioning Boundary value analysis Decision table Testing from formal specifications Error guessing Exploratory testing Black Box Testing
COMP8130 4130Adrian Marshall Test areas of risk with more rigor (greater coverage of functionality, and/or code) Product risks may include: Performance (capacity, throughput, accuracy, etc) Safety Security (authentication…) Complexity Test areas of higher risk first. Focus on consequences and likelihood. Risk-Based Testing
COMP8130 4130Adrian Marshall Systematic requirements based testing ensures complete testing scope is analysed. Focus Areas (examples) Functionality - Security, Accuracy, Regulatory Compliance, Technical Compliance, Reliability - Data Integrity, Error Handling, Fault Tolerance, Recoverability Useability - User Friendliness, User Guidance, Adaptability, Clarity of Control, Error Handling, Conciseness, Ease of Learning, Documentation Quality, Ease of Installation, Performance - Throughput, Acceptable Response Time, Data Storage Requirements, Acceptable Memory Capacity, Acceptable Processing Speed Portability - Portability to Different Hardware Platforms, Compatibility With Different Operating Systems, Conformance, Replaceability, Languages Supported…. Requirements-Based Testing
COMP8130 4130Adrian Marshall Test Methods MethodDescription Demonstration The actual operation of an item to provide evidence that it accomplishes the required functions under specific scenarios. This requires human observation against definable objectives. Inspection Used to verify conformance with specified standards, by examination of, and comparison with, drawings, code, unit test results or other documentation. Analysis Analysis is the use of established technical or mathematical models or simulations, algorithms, or other scientific principles and procedures to provide evidence that the item meets its stated requirements Test Test is the application of scientific principles and procedures to determine the properties or functional capabilities of items. The process will generate data that is presented by calculated, precision measurement equipment for subsequent evaluation.
COMP8130 4130Adrian Marshall Inspections and reviews require visual examination. They can be conducted at the early definitions phase and hence provide efficient defect rectification. Can be influenced by the ability of the inspector/reviewer (use checklists to standardise). Inspections and Reviews
COMP8130 4130Adrian Marshall Acceptance testing / User Acceptance Testing (UAT) Testing a system’s behaviour against the customer’s requirements Alpha & beta testing Testing by a representative sample of users (internal = alpha, external = beta) Installation testing Testing a system after installation in the target environment Performance testing Testing against specified performance requirements (eg. response time) Reliability testing Testing of stability, endurance, robustness, and recoverability…. Common Testing Types (1)
COMP8130 4130Adrian Marshall Regression testing Re-testing previously run tests to evaluate the impact that a software change may have on unaltered software components Security testing Testing a system’s ability to prevent unauthorised use or misuse, authentication Compatibility / Interoperability testing Testing the ability of software to operate and coexist with other (application and system) software and hardware Stress testing Exercising a system at the maximum design load and beyond Usability testing Testing a system’s user friendliness, ease of learning, and ease of use…. Common Testing Types (2)
COMP8130 4130Adrian Marshall Test management tools information repositories document generators defect management tools requirements traceability and test coverage tools Test execution tools compliers, debuggers, link loaders source code analysers (coverage & complexity) GUI testers functional record and replay tools / robots performance / load and stress testing tools security vulnerability analysis tools…. Testing Tools
COMP8130 4130Adrian Marshall Review V & V can provide continuous information about the quality of the system and the development effort cost of detection & prevention < cost of corrective action Testing is a process by which we understand the status of the benefits and the risk associated with release of a software system. There are many testing techniques available for developers and testers. Risk based testing is used to focus scarce testing resources. Systematic requirements based testing ensure complete testing scope is analysed. Automated testing tools may by used to assist test management and execution.