Presentation is loading. Please wait.

Presentation is loading. Please wait.

AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 1 Just-In-Time Testing Workshop Robert Sabourin AmiBug.Com, Inc. Montreal, Canada

Similar presentations


Presentation on theme: "AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 1 Just-In-Time Testing Workshop Robert Sabourin AmiBug.Com, Inc. Montreal, Canada"— Presentation transcript:

1 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 1 Just-In-Time Testing Workshop Robert Sabourin AmiBug.Com, Inc. Montreal, Canada

2 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 2 Overview Welcome Some Philosophy Be Pre[pared Testing Ideas Test Triage

3 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 3 Just In Time Testing Robert Sabourin, Software Evangelist President AmiBug.Com Inc. Montreal, Quebec, Canada

4 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 4 Just-In-Time Testing Welcome

5 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 5 Just-In-Time Testing Pain points? –What hurts? –How Much?

6 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 6 Just-In-Time Testing Just In Time For ________?

7 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 7 Just-In-Time Testing Some Philosophy

8 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 8 Fundamental Question How do you know when you are finished?

9 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 9 Crosby on Quality Quality is defined as conformance to requirements Quality is not a measure of GOODNESS –Phil B. Crosby, Quality is Free

10 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 10 Quality is fitness for use Joseph Juran Quality Control Handbook

11 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 11 Gerald M. Weinberg Quality is value to some person Exploring Requirements Quality Before Design Dorset House

12 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 12 Edsger W. Dijkstra Program testing can be used to show the presence of bugs, but never to show their absence

13 AmiBug.Com, Inc. Quiz Application screens are selected with three controls: (a) has 5 options (b) has 6 options (c) has 2 options How many screens can a user choose? November 10, 2013© Robert Sabourin, 2012Slide 13

14 AmiBug.Com, Inc. Quiz Total Combinations = 6 x 5 x 2 = 60 To exercise each combination once a total of 60 tests would be required. November 10, 2013© Robert Sabourin, 2012Slide 14

15 AmiBug.Com, Inc. How many tests would be required to exercise all possible screens in every possible order?. November 10, 2013© Robert Sabourin, 2012 Slide 15 Quiz

16 AmiBug.Com, Inc. To exercise all screens in every possible order would require 60! Test cases 60! = 60 x 59 x 58 x... 3 x 2 x 1 60! 8.32 x 10**81 November 10, 2013© Robert Sabourin, 2012 Pop Quiz Slide 16

17 AmiBug.Com, Inc. Pop Quiz From 7.0 × 10**79 To 1.5 × 10**82 November 10, 2013© Robert Sabourin, 2012 How many atoms are in the observable universe? Slide 17

18 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 18 Control Flow Testing Model flow –Create control flow diagram –Find basis paths Minimal set of transactions Exercise at least once –Each step –Each decision

19 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 19 Control Flow Testing –Minimal basis paths N – number of nodes E – number of edges P – number of basis paths P = E – N + 2 McCabe Cyclomatic Complexity B A CD FG E H I N=9 E=10 P= P=3

20 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 20 Control Flow Testing –Finding basis paths: 1.Start with a typical baseline 2.Flip first decision keep rest as similar as possible 3.Continue flipping decisions on baseline 4.After all decisions on baseline have been flipped continue on next path 5.Stop when all paths have been exhausted B A CD FG E H I N=9 E=10 P= P=3

21 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 21 Control Flow Testing

22 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 22

23 AmiBug.Com, Inc. ST Di M R N Q P L A B C D E FG I J K O H S T U VW X ZY ACAD AE AF AA AGAHAJAB AM AL AI AK AN AOAP AQ November 10, 2013© Robert Sabourin, 2012Slide 23

24 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 24

25 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 25 Purpose of Testing Common definition: –To find bugs before our customers do! Broader definition: –The role of testing is to provide objective input to facilitate business decisions! –Keeps stakeholders aware of all issues or concerns that relate to shipping a product!

26 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 26 Bug Defined To make our job more fun, whenever we have a concern with software, we call it a bug.

27 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 27 Its all about people! (and the occasional bug too) Just-In-Time Testing

28 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 28 Just-In-Time Testing Context Drivers

29 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 29 Context Drivers - BTO Business –Value –To whom? –Why? Technology –Solutions Organization –Corporate Structure –Team Structure –Roles and Responsibilities

30 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 30

31 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 31 Context Listeners Find Sources Monitor Drivers Anticipate Change React

32 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 32 Just In Time Testing Get Ready, Get Set, Cause here it comes

33 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 33 Turbulence Just-In-Time Testing

34 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 34 Unprepared Just-In-Time Testing

35 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 35 Just-In-Time Testing Sharpen Testing Skills Thinker Detective Reporter Diplomat Negotiator Cheer Leader Pragmatist

36 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 36 Philosophy We have precious little time to run tests! We must always be prepared!

37 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 37 Time

38 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 38 Just In Time Testing Test Triage

39 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 39 "No! Try not, Do. Or do not. There is no try." Yoda Plan to support change

40 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 40 Testing Ideas Collect all testing ideas you can find! –List –Sort –Organize –Shuffle Plan to support change

41 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 41 Testing Ideas How to find them? –Does system do what it is suppose to do? –Does the system do things it is not supposed to? –How can the system break? –How does the system react to its environment? –What characteristics must the system have? –Why have similar systems failed? –How have previous projects failed? Plan to support change

42 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 42 Testing Ideas Collect testing ideas From testing ideas build a series of testing objectives –Each can be assigned as work to testers –Each can include all, part of, or multiple testing ideas Capture testing ideas

43 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 43 Testing Ideas I often use Index Cards –Unique id –One testing idea per card –Colour indicates source –Shuffled and reviewed –Organized and reorganized –Sorted, grouped, prioritized and collected Capture testing ideas

44 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 44 Test Idea Sources Capabilities Failure Modes Quality Factors Usage Scenarios Creative Ideas States Data Environments White Box Taxonomies Capture testing ideas

45 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 45 Testing Ideas Investigative approaches –We become truffle snorting pigs and try to find useful information in all evidence we discover –We can even get good ideas from out of date sources Capture testing ideas

46 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 46 Testing Ideas Capabilities –Use cases –Functional requirements –Documented requirements –Implicit requirements Capture testing ideas

47 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 47 Testing Ideas Failure Modes –What can break? –Reaction to invalid input? –How does software behave in constrained environment? Memory Disk Space Network Bandwidth CPU capacity Shared resources –Stress, Load, Volume Capture testing ideas

48 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 48 Quality Factors Importance Different Application Types Capture testing ideas

49 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 49 Testing Ideas Usage Scenarios –Identify classes of users –Identify how users will use system –Describe scenarios –Use Story board or similar approaches –Identify variations Capture testing ideas

50 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 50 Testing Ideas Creative approaches –Action verbs –Mind Maps –Soap Operas –Lateral Thinking Capture testing ideas

51 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 51 State Models power up idle inserting coins user choose make coffee service needed coin insertedreset button coin return right amount entered coin return button pushed no cups OR no coffee OR sensor jam cup removed Testing Ideas Capture testing ideas

52 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 52 Testing Ideas Data –Flow –Structure –Create –Update –Change Capture testing ideas

53 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 53 Testing Ideas Environment –Hardware –Software –Operating systems –Locales –Browsers –Plug-ins –Co-dependent software Capture testing ideas

54 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 54 Testing Ideas White Box –Design –Internal structure –Code Capture testing ideas

55 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 55 Testing Ideas Bug taxonomies –Collections of possible bugs –Appendix A of Testing Computer Software, Kaner, Falk, Nguyen –Boris Biezer Taxonomy Otto Vinter manages –Shopping cart taxonomy Giri Vijayaraghavan Capture testing ideas

56 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 56

57 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 57 Which test? Impact estimation –For each test idea guesstimate: benefit of implementation consequence of implementation benefit for not implementing consequence of not implementing –How credible is the information? Triage testing ideas

58 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 58 Which test? Test Idea Rejection – What If? –If the cost/benefit does not make business sense then consider implementing: part of the test, could that lead to part of the benefit at a more reasonable cost? more than the stated test, would that generate more benefit? a different test than the stated idea, could that generate more benefit for less cost? Triage testing ideas

59 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 59 Test Triage Test Triage Meeting –Review Context Business Technical Organizational –New Information Test results Bug results New testing ideas Triage testing ideas

60 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 60 Test Triage Allocate Testing Assignments to Testers –Make sure testers know context –Best thing to test –Best person to test it –Best people to explore it –Best lead –Are subject matter experts required Triage testing ideas

61 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 61 Test Triage Life of a test idea a.Comes into existence b.Clarified c.Prioritized a.Test Now (before further testing) b.Test before shipping c.Nice to have d.May be of interest in some future release e.Not of interest in current form f.Will never be of interest d.Integrate into a testing objective Triage testing ideas

62 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 62 Which test is next? Questions –Given state of project, state of business, state of technology, our abilities, our experience and our history, what we know and what we do not know, what should we test next? –How much effort are we willing to spend continuing to test this project? –Can we ship yet? Triage testing ideas

63 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 63 Deciding what not to test? Time pressure –Should we skip a test? –If test failed could system still be of value to some stakeholder? –If test was skipped could important bugs have been otherwise found? Triage testing ideas

64 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 64 Guidelines and Decisions To each stakeholder –risk of failure –consequence of failure –value of success –how much certainty do we have –is it a wild guess or an absolute truth? Get Started Right

65 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 65 Bottom Line My experience is that it is better to omit a test on purpose than to skip it because you ran out of time or forgot about it! Get Started Right Systematically collecting, evaluating and triaging testing ideas helps me decide what not to test - at least for now?

66 AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 66 Thank You Questions?


Download ppt "AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 1 Just-In-Time Testing Workshop Robert Sabourin AmiBug.Com, Inc. Montreal, Canada"

Similar presentations


Ads by Google