Week 2 Quality Assurance Quality Assurance in Context Software Qualit Engineering

What is SQA SQA is a well defined, repeatable process that is integrated with project management and the software development lifecycles to review internal control mechanisms and assure adherence to software standards and procedures. The objective of the process is to assure conformance to requirements, reduce risk, assess internal controls and improve quality while conforming to the stated schedule and budget constraints. (Owens and Khazanchi)

More definitions (Owens and Khazanchi)

What is QA? QA as Dealing with Defect Defect Prevention Error blocking Error source removal Defect Detection and Removal Inspection Testing etc. Defect Containment Failure prevention Failure containment. Tian defines QA in 3 contexts.. i.e the process to deal with defects, prevent, eliminate, reduce or contain various specific problems associated with different aspects of defects.

QA: quality assurance Focus on correctness aspect of Q QA as dealing with defects post-release: impact on consumers pre-release: what producer can do what: testing & many others when: earlier ones desirable (lower cost) how =>classification as follows Defect prevention and reduction activities directly deal with the competing processes of defect injection and removal. After release the failure observed and problems reported by customers and users also need to be fixed, which in turn could lead to reduced defect and improved product quality. However, one shouldn’t rely on these post release activities and give up pre release activities. Coz error correction after release can be many time more expensive. Can also leads to damage to the reputation of SW development organisation. Pre release activities may be complimented with ‘Controlled field testing’ commonly referred to as ‘beta testing’, and similar techniques. Defect containment activities aim at minimizing the –ve impact of these remaining faults during operational use after product release. Defect containment activities involve a lot of duplication and redundancy and requires more effort to design and implement related features. Therefore, they are limited to situation where in field failures are associated with substantial damage. .

Error/Fault/Failure & QA Removal of faults (pre: detection) inspection: faults discovered/removed testing: failures trace back to faults Failure prevention and containment: local failure !=> global failure via dynamic measures to tolerate faults failure impact =>safety assurance

QA Classification

Defect Prevention Overview Error blocking error: missing/incorrect actions direct intervention to block errors fault injections prevented rely on technology/tools/etc. Error source removal root cause analysis identify error sources removal through education/training/etc. Focus on Product and domain specific knowledge, SW dev knowledge and expertise, developmental methodologies, development process knowledge Systematic defect prevention via process improvement. Education and training to remove human misconceptions. For error source elimination one can focus on

Formal Methods Motivation (formal methods =>fault absent) fault present: revealed through testing/inspection/etc. fault absent: formally verify. (formal methods =>fault absent) Basic ideas behavior formally specified: pre/post conditions, or as mathematical functions. verify “correctness": intermediate states/steps, axioms and compositional rules. Approaches: axiomatic/functional/etc.

Defect Reduction:Inspection Overview Artifacts (code/design/test-cases/etc.) from (req./design/coding/testing/etc) phases. Informal reviews: self conducted reviews. independent reviews. orthogonality of views desirable. Formal inspections: Fagan inspection and variations. process and structure. individual vs. group inspections. what/how to check: techniques .

Testing Overview Product/Process characteristics: scale/order: unit, component, system, : : : who: self, independent, 3rd party What to check: external specifications (black-box) internal implementation (white/clear-box) Criteria: when to stop? coverage of specs/structures. Reliability: usage-based testing

Defect Containment: Fault Tolerance Overview Idea originated from hardware systems. Focus is on reliability, availability and dependability. Motivation fault present but removal infeasible/impractical (fault tolerance ) contain defects. FT techniques: break fault-failure link Recovery blocks: rollback and redo NVP: N-version programming fault blocked/out-voted RECOVERY blocks: like databases. N-version programming: Many parallel programs implementing same functionality but using different logic, algorithm..

Safety Assurance Overview Extending FT idea for safety: fault tolerance to failure “tolerance" Safety related concepts: safety: accident free accident: failure with severe consequences hazard: precondition to accident Safety assurance: hazard analysis hazard elimination/reduction/control damage control Hazard elimination through substitution, simplification, decoupling etc. Hazard reduction through design for controllability.. For example pressure release valve.. Hazard control through reducing exposure, isolation and containment Damage control: through escape routes, safe abandonment of product and materials, and devices for limiting physical damage to equipments

QA in Context QA and the overall development context defect handling/resolution activities in process alternative perspectives verification/validation (V&V) view Now defects are found through various activities outlined earlier. Now the question is what we need do to handle these defect. Handling defect is called defect resolution

Defect handling/resolution status and tracking Logging Monitoring causal (root-cause) analysis resolution: defect removal/etc. Many stakeholders: testers, developer, managers, client (except unit testing) Part of project management activity improvement: break causal chain

Defect Measurement and Analysis parallel to defect handling where injected/found? type/severity/impact? more detailed classification possible? consistent interpretation timely defect reporting Defect analyses as follow up to defect handling. data and historical baselines goal: assessment/prediction/improvement causal/risk/reliability/etc. analyses

Defect Handling in Different QA Activities. Among 3 classes of QA activities: defect detection and removal activities are more closely associated with defect handling. Various defect prevention activities do not directly deal with defect or discovered faults. On defect containment the focus is not on fault finding or removal. So defect handling is not closely associated with prevention and containment activities

QA in Software Processes Development process components: requirement, specification, design, coding, testing, release. Process variations: waterfall development process iterative development process spiral development process lightweight/agile development processes such as XP (extreme programming) maintenance process too mixed/synthesized/customized processes QA important in all processes

QA in Waterfall Process

QA in Waterfall Process: QA throughout process defect prevention in early phases focused defect removal in testing phase defect containment in late phases phase transitions: inspection/review/etc.

QA in Software Processes Process variations (: waterfall) and QA: iterative: QA in iterations/increments spiral: QA and risk management How we can do it in agile processes?? mixed/synthesized: case specific more evenly distributed QA activities QA in maintenance processes: focus on defect handling; some defect containment activities for critical or highly-dependable systems; data for future QA activities QA scattered throughout all processes test-driven development, pair programming, pair inspection, involvement of customer.

V&V Core QA activities grouped into V&V. Validation: w.r.t. requirement (what?) appropriate-for-use “ doing right thing"? scenario and usage inspection/testing; system/acceptance testing; beta testing and operational support. Verification: w.r.t. specification/design (how?) Correct “doing things right" design as specification for components; structural and functional testing; inspections and formal verification.

V&V in Software Process

V&V vs DC View Two views of QA: Mapping between V&V and DC view: V&V view DC (defect-centered) view Interconnected: mapping possible? Mapping between V&V and DC view: V&V after commitment (defect injected already) defect removal & containment focus Verification: more internal focus Validation: more external focus In V-model: closer to user (near top) or

DC-V&V Mapping

Time for a break

Software Quality Engineering SQE: Software Quality Engineering Key SQE Activities SQE in Software Process Since there are diferent expectations by customer and developer, we need to move beyond just focusing on QA activities to SQE by managing these quality expectation as engineering problem. Goal is to meet or exceed these quality expectations through the selection and execution of selected appropriate QA activities while minimizing the cost and risk under the project constraints.

QA to SQE QA activities need additional support: Planning and goal setting Management: when to stop? adjustment and improvement, etc. all based on assessments/predictions Assessment of quality/reliability/etc.: Data collection needed Analysis and modeling Providing feedback for management QA + above => software quality engineering (SQE)

SQE Process

SQE and QIP QIP (quality improvement paradigm): Step 1: understand baseline Step 2: change then assess impact Step 3: package for improvement SQE as expanding QA to include QIP ideas.

Pre-QA Planning Pre-QA planning: Setting quality goals: Quality goal Overall QA strategy: QA activities to perform? measurement/feedback planning Setting quality goals: Identify quality views/attributes Select direct quality measurements Assess quality expectations vs. cost

Setting Quality Goals Identify quality views/attributes customer/user expectations, market condition, product type, etc. Select direct quality measurements direct: reliability defect-based measurement other measurements Assess quality expectations vs. cost cost-of-quality/defect studies

Forming QA Strategy QA activity planning evaluate individual QA alternatives strength/weakness/cost/applicability/etc. match against goals integration/cost considerations Measurement/feedback planning: define measurements (defect & others) planning to collect data preliminary choices of models/analyses feedback & followup mechanisms, etc.

Analysis and Feedback Measurement: handling process defect measurement as part of defect handling process other data and historical baselines Analyses: quality/other models input: above data output/goal: feedback and follow up focus on defect/risk/reliability analyses Feedback and followup: frequent feedback: assessments/predictions possible improvement areas project management and improvement

SQE in Software Processes SQE activities development activities: quality planning product planning QA activities development activities analysis/feedback project management Fitting SQE in software processes: different start/end time different sets of activities, sub-activities, and focuses in waterfall process: more staged (planning, execution, analysis/feedback) in other processes: more iterative or other variations

SQE Effort Profile QE activity/effort distribution/dynamics: different focus in different phases different levels (qualitatively) different build-up/wind-down patterns impact of product release deadline (deadline-driven activities) planning: front heavy QA: activity mix (early vs. late; peak variability? deadline?) analysis/feedback: tail heavy (often deadline-driven or decision-driven)

SQE Effort in Waterfall Process

Enough for this week Reading Tian And lets discuss your past papers.. Chapter 3, 4 and 5. And lets discuss your past papers..