Presentation on theme: "Software Quality Assurance"— Presentation transcript:
1 Software Quality Assurance Industry statistics for 1994 (WW. Gibb):For every 6 systems that work, 2 are cancelledIt is worse for large systems: 3 out of every 4 large systems were reduced in scope or did not work at all (e.g. Denver airport)Still bad today for low maturity software developmentGood quality software doesn’t just happen, however good the team producing the software and however good the development techniques they use.Good software development needs procedures and management.That is what QA is all aboutWhy does it need procedures and management?* Because people can’t hold everything they need to know in their head at once* Because many techniques are not ‘turn the handle’ - they are creative - and hence there is opportunity for human error
2 Why are procedures and management necessary for quality? Software development is complex: much to rememberHence standards, procedures and guidelines which embody best practiceThese also help training of new staffWhen the pressure’s on… something has to give...Hence defined points at which quality of work so far is assuredHence being explicit with the customer about cost-time-scope-quality trade-off: ‘if you want software of this quality...’Some defects may go unnoticed by the authorHence reviews… which wouldn’t happen without management!Helps assure the customer that the software is of sufficient quality
3 This lecture What is quality? What is quality assurance? Quality factorsWhat is quality assurance?Quality manuals and plansStandards for quality manuals: the ISO 9000 familyGroup Project StandardsReviewsKinds of review: particularly the Formal Technical ReviewHow to do a formal review
4 What is quality? ISO definition (ISO 8402:1994) Or… “The totality of features and characteristics of a product, process or service that bears on its ability to satisfy stated and implied needs.”Or…Software quality is meeting the requirements of the userWith the caveats that…The user requirements may not be precisely statedSome user requirements may not be stated at all, because of ‘shared assumptions’ between user and developerThe developer may also have requirements which may not be part of the user requirements (e.g. testability, maintainability)
5 Quality factors (ISO 9126:1992) FunctionalitySatisfying stated or implied needs (regarding the software’s functions)ReliabilityMaintaining performance under stated conditions for stated period of timeUsabilityThe effort needed for use, and the individual assessment of such use by a stated or implied set of usersEfficiencyThe relationship between performance and resources requiredMaintainabilityThe effort needed to make specified modificationsPortabilityThe ability of software to be transferred from one environment to anotherSuitability: Right feature set: suitable functions for tasks to be performedAccuracy: Same as correctness: Provision of right/agreed results or effectsInteroperability: interaction with particular systemsSecurity: Ability to prevent unauthorised accessMaturity: Frequency of failureFault tolerance: Ability to maintain some level of performance in spite of failureRecoverability: Ability to recover original level of performance after a failure (e.g. time to recover)Operability: Effort required to operate the software once it’s learnt
6 What is Quality Assurance? Quality Assurance is the overall management of quality.Within a company, it includes:Specifying a quality policy: General quality objectives and commitmentsMaintaining a quality manual, which contains details regarding:What constitutes good quality: Standards, procedures, guidelinesQuality controls: methods of assuring that quality is goodProducing quality plans for each projectThe parts of the quality manual which apply to this projectAssigning specific people to quality assurance, often including a project-independent quality assurance groupPlanning (schedule/budget), performing, evaluating quality assurance activitiesThis definition of quality control, something that assures quality rather than the general activity of ensuring quality, is consistent with ISO 8402.
7 Recognised standards of Quality Assurance Increases customer’s confidence that acompany produces quality productsGeneral QualityStandardse.g. ISO 9000QualityManualFrom Sommerville.Project 1Quality PlanProject 2Quality PlanProject nQuality Plan
8 The ISO 9000 familyISO 9000: Guidelines for selection and use of the other documents in the family.ISO 9001: Model for QA in design, development, production, installation and servicing.ISO 9002: Model for QA in production, installation and servicing.ISO 9003: Model for QA in final inspection and test.ISO 9004: Guidelines on quality management and quality system elements.Also ISO : Application of ISO 9001 to the development, supply and maintenance of software. Direct correspondence between clauses of ISO 9001 and ISO
9 Some example sections from ISO 9000-3 Management Responsibility.Quality System.Internal quality system audits.Corrective action.Contract review.Purchaser’s requirements spec.Development planning.Quality planning.Design and implementation.Testing & Validation.Acceptance.Maintenance.Configuration Management.Document Control.Measurement.Rules, practices, conventions.Training.Purchasing.ISO 9001/ are short (10-15 pages): open to interpretation
10 ISO 9001 versus ISO 9000-3: ‘quality planning’ section … Quality planning shall be consistent with all other requirements of a supplier’s quality system and shall be documented in a format to suit the supplier's method of operation. The supplier shall give consideration to...ISOQuality planning should address…quality requirements, expressed in measurable terms, where appropriate;the life cycle model to be used for software developmentdefined criteria for starting and ending each project phaseidentification of types of reviews, tests and other verification and validation activities to be carried outidentification of configuration management procs to be carried out
11 ISO 9000: A standard for quality manuals Companies can, and do, get certification that:Their quality manual meets the ISO 9000 standardsThey run their projects with adherence to their quality manualA company is ISO certified by an independent bodyThe company first receives training in ISO 9000 etc.The quality manual is sent to the body, which assesses it and reports backAn assessment team visits the company to ensure that it follows its manualThe body then awards the company ISO 9000 status unconditionally, conditionally, or not at allAfter certification, an assessment team visits unannounced 2-4 times per year - and conducts a full reassessment every 3 yearsFrom Darrel Ince, ISO 9001 and Software Quality Assurance, (National Library)
12 Software Engineering Institute’s Capability-Maturity Model (SEI CMM) Level 5OptimizingOrganisational commitmentto continuous processimprovement; defect preventionLevel 4ManagedManagement by measurement: quantitativetargets for each project; productivity andquality measures; measurementdatabaseLevel 3DefinedStable process model includingentry/exit criteria; reviews, qualitytracking, full process training programLevel 2RepeatableStable project planning, tracking of cost/schedule/functionality, problem identification, project standards;can ‘repeat the success of previous projects’From Paulk, Mark C. et al. “Capability Maturity Model, Version 1.1”, IEEE Software, July 1993,Level 1InitialUnstable, ad-hoc processes;project success depends on staff skills/experience (‘heroics’)
13 Comments on the CMM CMM versus ISO 9000 Disadvantages of the CMM CMM much more prescriptive (500 pages versus pages)Companies rated ISO 9000 are probably partly level 2, partly level 3; but they may only be at level 1 (because ISO open to interpretation)CMM is improvement-oriented, ISO 9000 is ‘minimum quality’Disadvantages of the CMMNothing about technology or risk analysisOriginally to help DoD select contractors: large, slow-moving systemsKey areas a bit arbitrary: 80% level % level 3 = level 1!Too bureaucratic for small companies?At least it collects together some best practicesCMM versus ISO 9000 is from a paper by Mark Paulk, IEEE Software January 1995.Disadvantages from Sommerville.‘Too bureaucratic’ has a question mark because it seems that plenty of small companies have been assessed with respect to the CMM (although they may actually be small suborganisations within large organisations).
14 A project’s quality plan Should be defined at an early stage of the projectCan use quality factors to determine which parts of the quality manual are particularly appropriate to this projectIs portability important?How long will this product live?Define activities to be performed. Answer questions for each activity:What is checked and against what?Who performs the check and who approves the checks?Who takes the final decision if there is controversy?How are results recorded?How are corrective actions controlled?
15 Quality controls: The general process 1 Define things which are considered ‘good quality’.2 Define the procedure for checking quality.3 Carry out the procedure on a specified item.4 Record the result.5 Take and record any corrective actionRegarding both the item and the quality systemFeedback (step 5) is important - if there are any deficiencies in the QA system, then they should be improved.
16 Group Project Standards A cut down version of what you might see in industryThey cover the following areas:QA.01 Quality plan - see previous slide on ISO-9001 Quality planningQA.02 Project management - how we will run the projectQA.03 Documentation - what they will cover/look likeQA.05 Design Spec standardsQA.06. Test Spec standardsQA.07 Review standards - will cover now...QA.08 Configuration managementQA.09/10 Coding standards for Java/C#QA.11 Final report / maintenance documentation
17 An important quality control: the review Bugs are caught more efficiently by review than by testingA review is…A meeting at which some aspect(s) of the project are inspected and assessed by a group of peopleAll reviews have an element of formality, and their outcome is minutedFormal Technical ReviewTo identify defects in a workproduct (design document, code listing, …)WalkthroughTo identify defects in a workproduct, but with the emphasis on learning and evaluating alternatives - less formal
18 Formal Technical Review: process Author submits document or code to the review convenorBriefly examined before accepting it for reviewReview preparationReview team formedCriteria formed by which the document/code will be examinedAll necessary documents sent to reviewers, who read themThe review meeting (2 hours maximum)Issues (possible defects) are detected, but not correctedFollow-up: document changed in response to issues highlighted in review meeting, and resubmitted if necessary
19 Documents needed for a review The item under review!The document(s) from which the item is derivedFor test spec: requirements, user interface design, design?For code: detailed design (or for us, list of classes/attributes/methods)The relevant quality plan documentsA review checklist: questions to answerSummarises what to look for when reviewingDerive from the quality plan documents and (to an extent) from the documents from which the item is derivedTry to fit on one sheet of paper, to concentrate on major issues
20 Example review checklist: design Do the reviewers understand the design?Will the design meet the requirements?Does the design satisfy the relevant standards?Can the design be simplified?Will the design lead to a maintainable system?Could the design be modified so as to offer more opportunity for reusing existing software?The more specific the quality plan, the easier to create checklistsSE.QA.07 has a more specific design checklist for group project.
21 Example review checklist: code Are all variables initialised before their values are used?Have all constants been named?Is each loop certain to terminate?Are upper and lower bounds of arrays sensible?Have pointers been correctly reassigned after modification?Does code conform to QA code standards for the language?
22 The review meetingThe item being reviewed is ‘walked through’ from start to endIssues are highlighted as they occurIncludes issues found before, as well as during, the meetingThe author evaluates each issue, and…Accepts as a defect, or explains why not a defect, or answers a question which a reviewer hasMinutes are taken by the scribe (QA Manager for us!)‘Issue’ is not a euphemism for ‘problem’, it means ‘possible problem’(Note that the item is ‘walked through’ even though this isn’t a walkthrough!)
23 ConclusionsQA is the management of doing things in such a way to ensure that end results meet the users requirements.Steps must be appropriate & economically realistic.Aim is not highest possible quality, but is a compromise betweenTimeCostQualityScope