Just-In-Time Testing Workshop Robert Sabourin AmiBug.Com, Inc. Montreal, Canada rsabourin@amibug.com March 25, 2017 © Robert Sabourin, 2012

Overview Welcome Some Philosophy Be Pre[pared Testing Ideas Test Triage March 25, 2017 © Robert Sabourin, 2012

Just In Time Testing Robert Sabourin , Software Evangelist President AmiBug.Com Inc. Montreal, Quebec, Canada rsabourin@amibug.com www.amibugshare.com March 25, 2017 © Robert Sabourin, 2012

Just-In-Time Testing Welcome March 25, 2017 © Robert Sabourin, 2012

Just-In-Time Testing Pain points? What hurts? How Much? March 25, 2017 © Robert Sabourin, 2012

Just-In-Time Testing Just In Time For ________? March 25, 2017 © Robert Sabourin, 2012

Just-In-Time Testing Some Philosophy March 25, 2017 © Robert Sabourin, 2012

Fundamental Question How do you know when you are finished? March 25, 2017 © Robert Sabourin, 2012

Crosby on Quality “Quality is defined as conformance to requirements” “Quality is not a measure of GOODNESS” Phil B. Crosby, Quality is Free March 25, 2017 © Robert Sabourin, 2012

Joseph Juran “Quality is fitness for use” Quality Control Handbook March 25, 2017 © Robert Sabourin, 2012

Gerald M. Weinberg “Quality is value to some person” Exploring Requirements Quality Before Design Dorset House March 25, 2017 © Robert Sabourin, 2012

Edsger W. Dijkstra “Program testing can be used to show the presence of bugs, but never to show their absence” March 25, 2017 © Robert Sabourin, 2012

Quiz Application screens are selected with three controls: (a) has 5 options (b) has 6 options (c) has 2 options How many screens can a user choose? March 25, 2017 © Robert Sabourin, 2012

Quiz Total Combinations = 6 x 5 x 2 = 60 To exercise each combination once a total of 60 tests would be required. March 25, 2017 © Robert Sabourin, 2012

Quiz How many tests would be required to exercise all possible screens in every possible order?. March 25, 2017 © Robert Sabourin, 2012

Pop Quiz To exercise all screens in every possible order would require 60! Test cases 60! = 60 x 59 x 58 x ... 3 x 2 x 1 60! ≈ 8.32 x 10**81 March 25, 2017 © Robert Sabourin, 2012

Pop Quiz How many atoms are in the observable universe? From 7.0 × 10**79 To 1.5 × 10**82 March 25, 2017 © Robert Sabourin, 2012

Control Flow Testing Model flow Create control flow diagram Find basis paths Minimal set of transactions Exercise at least once Each step Each decision March 25, 2017 © Robert Sabourin, 2012

Control Flow Testing Minimal basis paths N=9 E=10 P=10-9+2 P=3 N – number of nodes E – number of edges P – number of basis paths P = E – N + 2 McCabe Cyclomatic Complexity C D E F G H I March 25, 2017 © Robert Sabourin, 2012

Control Flow Testing Finding basis paths: N=9 E=10 P=10-9+2 P=3 Start with a typical baseline Flip first decision keep rest as similar as possible Continue flipping decisions on baseline After all decisions on baseline have been flipped continue on next path Stop when all paths have been exhausted C D E F G H I March 25, 2017 © Robert Sabourin, 2012

Control Flow Testing March 25, 2017 © Robert Sabourin, 2012

March 25, 2017 © Robert Sabourin, 2012

ST Di AD AC X L K I A Y Z AE M O J B AF AA AG AH AJ AB N C AM AL P H D AI AN AK Q S E AO AP AQ R T G F U V W March 25, 2017 © Robert Sabourin, 2012

March 25, 2017 © Robert Sabourin, 2012

Purpose of Testing Common definition: Broader definition: To find bugs before our customers do! Broader definition: The role of testing is to provide objective input to facilitate business decisions! Keeps stakeholders aware of all issues or concerns that relate to shipping a product! March 25, 2017 © Robert Sabourin, 2012

Bug Defined To make our job more fun, whenever we have a concern with software, we call it a “bug”. March 25, 2017 © Robert Sabourin, 2012

Just-In-Time Testing It’s all about people! (and the occasional bug too) March 25, 2017 © Robert Sabourin, 2012

Just-In-Time Testing Context Drivers March 25, 2017 © Robert Sabourin, 2012

Context Drivers - BTO Business Technology Organization Value To whom? Why? Technology Solutions Organization Corporate Structure Team Structure Roles and Responsibilities March 25, 2017 © Robert Sabourin, 2012

March 25, 2017 © Robert Sabourin, 2012

Context Listeners Find Sources Monitor Drivers Anticipate Change React March 25, 2017 © Robert Sabourin, 2012

Get Ready, Get Set, Cause here it comes Just In Time Testing Get Ready, Get Set, Cause here it comes March 25, 2017 © Robert Sabourin, 2012

Just-In-Time Testing Turbulence March 25, 2017 © Robert Sabourin, 2012

Just-In-Time Testing Unprepared March 25, 2017 © Robert Sabourin, 2012

Just-In-Time Testing Sharpen Testing Skills Thinker Detective Reporter Diplomat Negotiator Cheer Leader Pragmatist March 25, 2017 © Robert Sabourin, 2012

Philosophy We have precious little time to run tests! We must always be prepared! March 25, 2017 © Robert Sabourin, 2012

Time March 25, 2017 © Robert Sabourin, 2012

Just In Time Testing Test Triage March 25, 2017 © Robert Sabourin, 2012

Yoda Plan to support change "No! Try not, Do. Or do not. There is no try." March 25, 2017 © Robert Sabourin, 2012

Testing Ideas Collect all testing ideas you can find! List Sort Organize Shuffle Plan to support change March 25, 2017 © Robert Sabourin, 2012

Testing Ideas How to find them? Does system do what it is suppose to do? Does the system do things it is not supposed to? How can the system break? How does the system react to it’s environment? What characteristics must the system have? Why have similar systems failed? How have previous projects failed? Plan to support change March 25, 2017 © Robert Sabourin, 2012

Testing Ideas Collect testing ideas From testing ideas build a series of testing objectives Each can be assigned as work to testers Each can include all, part of, or multiple testing ideas Capture testing ideas March 25, 2017 © Robert Sabourin, 2012

Testing Ideas I often use Index Cards Unique id One testing idea per card Colour indicates source Shuffled and reviewed Organized and reorganized Sorted, grouped, prioritized and collected Capture testing ideas March 25, 2017 © Robert Sabourin, 2012

Test Idea Sources Capabilities Failure Modes Quality Factors Usage Scenarios Creative Ideas States Data Environments White Box Taxonomies Capture testing ideas March 25, 2017 © Robert Sabourin, 2012

Testing Ideas Investigative approaches We become truffle snorting pigs and try to find useful information in all evidence we discover We can even get good ideas from out of date sources Capture testing ideas March 25, 2017 © Robert Sabourin, 2012

Testing Ideas Capabilities Use cases Functional requirements Documented requirements Implicit requirements Capture testing ideas March 25, 2017 © Robert Sabourin, 2012

Testing Ideas Failure Modes Capture testing ideas What can break? Reaction to invalid input? How does software behave in constrained environment? Memory Disk Space Network Bandwidth CPU capacity Shared resources Stress, Load, Volume Capture testing ideas March 25, 2017 © Robert Sabourin, 2012

Quality Factors Importance Different Application Types Capture testing ideas Quality Factors Importance Different Application Types March 25, 2017 © Robert Sabourin, 2012

Testing Ideas Usage Scenarios Identify classes of users Identify how users will use system Describe scenarios Use Story board or similar approaches Identify variations Capture testing ideas March 25, 2017 © Robert Sabourin, 2012

Testing Ideas Creative approaches Action verbs Mind Maps Soap Operas Lateral Thinking Capture testing ideas March 25, 2017 © Robert Sabourin, 2012

Testing Ideas State Models Capture testing ideas power up service needed reset button idle coin inserted inserting coins coin return cup removed no cups OR no coffee OR sensor jam right amount entered coin return make coffee user choose Capture testing ideas button pushed State Models March 25, 2017 © Robert Sabourin, 2012

Testing Ideas Data Flow Structure Create Update Change Capture testing ideas March 25, 2017 © Robert Sabourin, 2012

Testing Ideas Environment Capture testing ideas Hardware Software Operating systems Locales Browsers Plug-ins Co-dependent software Capture testing ideas March 25, 2017 © Robert Sabourin, 2012

Testing Ideas White Box Capture testing ideas Design Internal structure Code Capture testing ideas March 25, 2017 © Robert Sabourin, 2012

Testing Ideas Bug taxonomies Collections of possible bugs Appendix A of Testing Computer Software, Kaner, Falk, Nguyen Boris Biezer Taxonomy Otto Vinter manages Shopping cart taxonomy Giri Vijayaraghavan Capture testing ideas March 25, 2017 © Robert Sabourin, 2012

March 25, 2017 © Robert Sabourin, 2012

Which test? Impact estimation For each test idea guesstimate: benefit of implementation consequence of implementation benefit for not implementing consequence of not implementing How credible is the information? Triage testing ideas March 25, 2017 © Robert Sabourin, 2012

Which test? Test Idea Rejection – What If? If the cost/benefit does not make business sense then consider implementing: part of the test, could that lead to part of the benefit at a more reasonable cost? more than the stated test, would that generate more benefit? a different test than the stated idea, could that generate more benefit for less cost? Triage testing ideas March 25, 2017 © Robert Sabourin, 2012

Test Triage Test Triage Meeting Review Context New Information Business Technical Organizational New Information Test results Bug results New testing ideas Triage testing ideas March 25, 2017 © Robert Sabourin, 2012

Test Triage Allocate Testing Assignments to Testers Make sure testers know context Best thing to test Best person to test it Best people to explore it Best lead Are subject matter experts required Triage testing ideas March 25, 2017 © Robert Sabourin, 2012

Test Triage Life of a test idea Triage testing ideas Comes into existence Clarified Prioritized Test Now (before further testing) Test before shipping Nice to have May be of interest in some future release Not of interest in current form Will never be of interest Integrate into a testing objective Triage testing ideas March 25, 2017 © Robert Sabourin, 2012

Which test is next? Questions Given state of project, state of business, state of technology, our abilities, our experience and our history, what we know and what we do not know, what should we test next? How much effort are we willing to spend continuing to test this project? Can we ship yet? Triage testing ideas March 25, 2017 © Robert Sabourin, 2012

Deciding what not to test? Time pressure Should we skip a test? If test failed could system still be of value to some stakeholder? If test was skipped could important bugs have been otherwise found? Triage testing ideas March 25, 2017 © Robert Sabourin, 2012

Guidelines and Decisions To each stakeholder risk of failure consequence of failure value of success how much certainty do we have is it a wild guess or an absolute truth? Get Started Right March 25, 2017 © Robert Sabourin, 2012

Bottom Line My experience is that it is better to omit a test on purpose than to skip it because you ran out of time or forgot about it! Systematically collecting, evaluating and triaging testing ideas helps me decide what not to test - at least for now? Get Started Right March 25, 2017 © Robert Sabourin, 2012

Thank You Questions? March 25, 2017 © Robert Sabourin, 2012