Presentation on theme: "Software Development QA Best Practices"— Presentation transcript:
1 Software Development QA Best Practices Suzette Hackl, CSMSenior Project ManagerSkyline Technologies, Inc.May 20, 2010
2 Agenda Definition of “Best Practices” Software Quality Assurance as it relates to:ProcessPeopleTechnologyApplication to SCRUMQuestions?
3 “Best Practices”A collection of Software Quality “What Works Now” practices.Suzette’s collection of practices come from many sources and span across industries, sectors, and experiences.Recognizable, Obvious, Debatable…..The list of practices in the QA space is large. A large laundry list is hard to conceptualize and translate into implementation. I have grouped my culled down list by People, Process, and Technology and categorized them by:Basic: The basic are exactly that. They are the training wheels you need to get started and when you take them off, it is evident that you know how to ride. When you eventually take off your training wheels and ride that does not mean that you always remember how to ride. Each one of these basic practices is “needed” for successful quality assurance. Removing or discontinuing one of these basic practices will degrade quality of your systems and potentially put business at risk.Foundational: The foundational practices are the rock and mortar that will hold your “Quality” program together. They have significant value adds to an organization. Unlike the basic, they are probably not as well known and therefore need implementation help.Incremental: The incremental practices are to be judiciously applied in special conditions and bring many returns in terms of value. Individually they do not provide broad gains across the board of testing because of their specialized nature.Not a new approach. Evidence of IID use in mid-late 50’s. IBM Federal Systems Division used an approach that is almost indistinguishable from XP.IID development has proven scalable. Used on very large -- critical life and death projects. Canadian ATC started as waterfall in 1989 and failed by 1992 but restarted as IID project.DoD MIL-STD-498 replaced previous waterfall standard.
4 Agenda Definition of “Best Practices” Software Quality Assurance as it relates to:ProcessPeopleTechnologyApplication to SCRUMQuestions?
5 QA ProcessesStandard & Consistent Requirements for Test PlanningOne of the roles of software testing is to ensure that the product meets the requirements of the client. Capturing the requirements therefore becomes an essential part not only to help develop but to create test plans that can be used to gauge if the developed application is likely to meet the customers’ needs. Therefore, requirement management and its translation to produce test plans is an important step. The premise of this practice is that requirements are consistently and clearly documented so as to be interpreted correctly by the development and QA team members.Standard & Consistent Functional Specifications or Solution Design DocumentationFunctional Specifications are a key part of many development processes. While it is a development process aspect, it is critically necessary for software functional testing. The advantage of having a functional specification is the test generation activities could happen in parallel with the development of the code. This is ideal from several dimensions. Firstly, it gains parallelism in execution, removing a serious serialization bottleneck in the development process. By the time the software code is ready, the test cases are also ready to be ran against the code. Secondly, if forces a degree of clarity from the perspective of a designer and an architect so essential for the overall efficiencies of development. Thirdly, the functional specifications become documentation that can be shared with the business to gain an additional perspective on what is being developed.Common Glossary of QA TerminologyThis may seem to be the most basic of all concepts but it is one of the most important. Having clearly defined terms communicated across the organization will go a long way in the acceptance and engagement of Quality AssuranceA functional specification often describes the external view of an object or a procedure indication the options by which a service could be involved.
6 QA Processes - continued Reviews & Inspection This best practice includes code reviews, Internal and external QA Audits, and Project Audits.Test Monitoring & GovernanceThe premise of test monitoring is to ensure quality key performance indicators are being met. This is done thru verification across project phases and deliverables. By test monitoring, QA assures software completeness and readiness for delivery.Verification: The process of determining whether or not the products of a given stage of the software development life cycle fulfill the requirements established during the previous stageValidation: The process of evaluating software at the end of the software development process to ensure compliance with software requirementsGovernance: Formal software reviews should be conducted at the end of each phase of the life cycle to identify problems and determine whether the interim product meets all applicable requirements. An example of these formal reviews is Gate Readiness Reviews. Reviews are part of the software development lifecycle, designed to provide a ready/not-ready decision to begin the next phase. In formal reviews, actual work done is compared with established standards. QA's main objective in reviews is to assure that the Management and Development Plans have been followed, and that the product is ready to proceed with the next phase of development. Although the decision to proceed is a management decision, QA is responsible for advising management and participating in the decision.
7 QA Processes - continued Formal Entry and Exit CriteriaThe idea is that every process step has a precise entry and precise exit criteria. These are defined by the development process and are watched by management to gate the movement from one stage to another. The preciseness of these criteria is determined by each individual project leadership. However, this practice allows much more careful management of the software development process.Functional Test - Variations Most functional tests are written as black box tests working off a functional specification. The numbers of test cases that are generated usually are variations on the input space coupled with visiting the output conditions. Functional testing involves writing down different variations to cover as much of the state space as one deems necessary for a program. The best practice involves understanding how to write variations and gain coverage which is adequate enough to thoroughly test the function. Given that there is no measure of coverage for functional tests, the practice of writing variations does involve an element of artUser Validation and Acceptance The idea is to release an application to a limited number of users and get feedback to fix problems before implemented to the entire user community. This best practice has everything to do with getting functionality in the hands of users as early in the process so as to reduce cost / business interruptions and post implementation issues(i.e. User Acceptance Testing, Prototyping, Betas, Pilots, Agile/SCRUM)
8 QA Processes - continued Automated Test Execution The use of automated testing to support rapid testing, capturing issues earlier in the development lifecycle, and providing detailed logs of frequent test results (sometimes through nightly builds). The automated test environment must be carefully setup to ensure accurate and consistent testing outcomes. Automatic testing should be run at code check-in to minimize changes of disrupting a working build. The goal of automated test execution is to minimize the amount of manual work involved in test execution and gain higher coverage with a larger number of test cases. The automated test execution has a significant impact on both the tool sets for test execution and also the way tests are designed. Integral to automated testing is the verification of current operations and logging of test failures with diagnosis information“Nightly” Builds & Continuous Testing The concept captures frequent builds from changes that are being promoted into the change control system. The advantage is firstly, that if a major regression occurs because of errors recently generated, they are captured quickly. Secondly, regression tests can be run in the background and cycled when it makes sense. Thirdly, the newer releases of software are available to developers and testers sooner. Continuous testing uses excess cycles on a developer's workstation to continuously run regression tests in the background, providing rapid feedback about test failures as source code is edited. It reduces the time and energy required to keep code well-tested, and prevents regression errors from persisting uncaught for long periods of time.
9 QA Processes - continued Integration Testing with User ScenariosAs we integrate multiple software products and create end user applications that invoke one or multiple systems, the task of testing the end user features gets complicated. One of the viable methods of testing is to develop user scenarios that exercise the functionality of the applications. These are called user scenarios. The advantage of the user scenario is that it tests systems in the ways that most likely reflect a user’s everyday usageSoftware Development Lifecycle & Incorporating QA Quality Assurance encompasses the entire software development process, which includes processes such as software design, coding, source code control, code reviews, change Management, configuration management, and release management. There are specific phase activities that should be conducted during the software life cycle. At the conclusion of each phase there is a key gate check and management decision that the existing phase has been completed and approval to proceed to the next phase. Below is an example of how this applies to SCRUM.
10 QA Processes - continued Quality Assurance Benchmarks and Performance IndicatorsEstablish and collect useful metrics that will facilitate decision making. Benchmarks and Performance Indicators (PI´s) cannot be meaningfully formulated unless there is a direct connection between them and the related policies and goals.Usability TestingUsability testing means a way to measure how people (intended/end user) find it (easy, moderate or hard) to interact with and use the system keeping its purpose in mind. It is a standard statement that "Usability testing measures the usability of the system". Some would argue that application usability perceived by the user community is the utmost factor that determines success. Usability testing needs to not only assess how usable a system is but also provide feedback on methods to improve the user experience and thereby gain a positive quality image.Test Case Quality and Re-usabilityThis practice is two-fold. Firstly, the valuation of a test case’s quality. Secondly, the value of re-using test cases where applicable. Secondly, the re-usability of test cases.PairingThere are various methods of pairing skilled individuals of a project team to produce higher quality of work more efficiently; pairing developers, teaming testers with developers, etc. It’s been recognized for a long time that the close coupling of testers with developers improves both the test cases and the code that is developedVisionMissionPolicyGoals (Objectives)BenchmarksPIs
11 QA Processes - continued Risk Management and Confidence The notion of this practice is to focus testing on risky areas and to find the important bugs; you have to practice some kind of Risk Management. There is a factor of confidence used here. The practice of accessing a risk factor to tests can also help with resource modeling. For example; testing of high risk, low confidence functionalities make more sense to be staffed by those QA Team members with high domain knowledge while low risk, high confidence functionalities could be staff by lower priced resources.
12 QA Processes - continued Bug-Tracking and Resolution with Root Cause AnalysisEstimatingTraceabilityThe linkages between test cases, their parent design elements, and their parent requirements are maintained through a technique termed Requirements Traceability. In essence, it is necessary to be able to trace the linkage from a test case all the way through to a grandparent goal.
13 Agenda Definition of “Best Practices” Software Quality Assurance as it relates to:ProcessPeopleTechnologyApplication to SCRUMQuestions?
14 People Software Quality Ownership The premise of this practice is that business owns the task of determining the quality expectations and KPIs of an application. Quality Control owns the task of assuring that software meets those quality guidelines.Clearly Defined QA Roles & ExpectationsQA Professional/Senior Lead (QA Analyst III)QA Lead (QA Analyst II )QA Analyst (QA Analyst I)SCRUM Team MemberQA Analyst Domain KnowledgeTesters require domain knowledge and have to be at least as skilled as an expert user/power user.Good Writing Skills are MandatoryWriting understandable and repeatable test scripts, which also contains information about the idea and the intention of the test, the expected results and technical background, isn’t easy. There's also a fine line between too detailed test scripts, which are difficult to maintain, and too superficial ones. This is truly an “art” and not a science and takes experience. A good test case is one which brings you closer to your goal. There’s no simple formula or prescription for generating “good” test cases.
15 People - continuedSoftware Delivery & Capacity Planning A unit of work represents both the functionality being delivered and the quality of the functionality. We see that each work unit has both functionality and a quality component. We don’t want to intentionally plan to deliver functionality without qualityQA Members have broad and comprehensive knowledge in ITThe premise of this best practice is that QA folks mustn’t be experts or gurus, but they should be able to warm up fast to many technologies. For example different operating systems, networks, Internet, databases, program languages, etc. Quality Assurance is not only closely related to development but also support, technical writing and field service. Therefore QA engineers should have some knowledge and experience in at least one of these areas. There shouldn't also be any borders or constraints that prevent or discourage QA people to switch to development, technical writing, field service or support - and vice versa.
16 People - continuedQA people need to be good communicators and team players They must be diplomatic (but sometimes tough), convincing and able to present.QA Job Satisfaction - Paradigm shiftAwareness of the value of Quality Assurance to an organization greatly impacts job satisfaction for a QA professional. This includes the understanding and acceptance of testing limitations.
17 Agenda Definition of “Best Practices” Software Quality Assurance as it relates to:ProcessPeopleTechnologyApplication to SCRUMQuestions?
18 Technology System Intellectual Intelligence Application Ownership This practice is related to the system knowledge held internally by an organization and its owning group. This knowledge ranges from architectural design to code practices and standards used in the application. This is a control measure for protection. Augmenting development teams with consultants is a common practice as a means of meeting surges and recessions in business demands. Nevertheless, retaining an underlying knowledge level is important for continuity and risk mitigation. The level of intellectual intelligence retained is unique to the specific software application, what business functions, and the value of these functions.Application OwnershipWe have already discussed the concept of quality ownership. The practice is the answer to the question “Who is the person that has the ultimate ownership of the application?” It is the one single ring able neck and product owner as described in SCRUM. IT has the responsibility for ensuring the system is maintained, sustained, and operational.Automation Testing Tools Unit test automation, System or Functional test automation, Performance test automation, Regression test automationTechnical Infrastructure and SupportTest Environments & Supporting applications and job scheduling tools
19 Technology Utilization of single platform to support QA Processes The concept of a single platform to support multiple QA processes is not new. There are studies that show how quality attributes can be lost between systems resulting in delivery of a product which is not what the client expected. The use of a single platform for work/project requests, requirements, test plans, test cases, design, development, test execution is very useful. It reduces the potential for loss of data and redundancy as well as provides one source for metric reporting.RefactoringRefactoring is the concept of cleaning up, improving the inner working of a system. All system functionality and outcomes are to remain unaffected given the same inputs. The amount of refactoring is dependent on the “swamp” or “ball of spaghetti” inside the applications. Lack of architecture direction and strategic planning of application growth can often result in this situation. Refactoring neither fixes bugs nor adds new functionality, though it might precede either activity. Rather, it improves the understandability of the code, changes its internal structure and design, and removes dead code. The amount of refactoring is based on the technical team’s willingness and speed to technical debt repayment plan.Code Coverage Analysis/ToolThe process of finding areas of a program not exercised by a set of test cases, creating additional test cases to increase coverage, and determining a quantitative measure of code coverage, which is an indirect measure of quality. An optional aspect of code coverage analysis is identifying redundant test cases that do not increase coverage. Analysis of code coverage (or test coverage analysis) is a technique whereby test cases are created to exercise all parts of the software.
20 Agenda Definition of “Best Practices” Software Quality Assurance as it relates to:ProcessPeopleTechnologyApplication to SCRUMQuestions?
21 Product & Team Backlog Formation The Scrum FrameworkScrum PlanningPotential DeploymentProduct & Team Backlog FormationSprint 2-4 WeeksTeam RetrospectiveSprint Planning 2 Parts: Selection and DecompSprint ReviewDaily Scrum
22 Agenda Definition of “Best Practices” Software Quality Assurance as it relates to:ProcessPeopleTechnologyApplication to SCRUMQuestions?