3 Introduction What is Exploratory Testing ? How is it different from Ad hoc Testing ?Where does ET fit ?Characteristics of Exploratory tester.Learning styles and ET.Styles of Exploration.Pros and Cons of ET.
4 What is ET ? "Exploratory testing involves simultaneously learning, planning, running tests, and reporting / troubleshootingresults."- Dr. Cem Kaner."Exploratory testing is an interactive process of concurrentproduct exploration, test design and test execution. To theextent that the next test we do is influenced by the resultof the last test we did, we are doing exploratory testing.”- James Bach.
5 How is it different from Ad hoc Testing ? “It is an impromptu and unrehearsed testing to uncoverthe bugs with no mission or goal.”
6 Ad Hoc Vs ET Increasing degree of ET Ad hoc testing Pure Scripted testingChartersVague ScriptsDocumentation(%)
7 Where does ET fit ?No specifications or little documentation available.No time to specify,script and test.Thread Pulling.When the application does not have any future releases.ET can be associated with scripted testing to improvethe coverage.To help assess risks.When there is open communication betweendevelopment and testing teams.
8 Characteristics of Exploratory Tester : Design and Excel :An exploratory tester is first and foremost a TestDesigner.The excellent exploratory tester is able to crafttests that systematically explore the product. One wayto approach it is to consider it a questioning process.To design a test is to craft a question for a productthat will reveal vital information.
9 Characteristics of Exploratory Tester : …contdHawk eyes :Excellent exploratory testers are excellent observers .The exploratory tester must watch for anything unusualmysterious, or otherwise relevant to the testing.Exploratory testers must be careful to distinguishobservation from inference(even under pressure) andavoid to allow preconceived assumptions to blind them.
10 Characteristics of Exploratory Tester : …contdCritical Thinking :Excellent exploratory testers are able to review andexplain their logic, looking for errors in their ownthinking.Break the Rules :Exploratory tester clears their mind from the shackles ofthe past, thinks creatively and laterally, to produce betterideas.Use heuristics to accomplish this.
11 Characteristics of Exploratory Tester : …contdRace up the learning curve :Excellent exploratory testers climb learning curves morequickly than most. Intelligence helps but its more a matterof skill and practice that is build over a period of time.Rich Resources :Excellent exploratory testers build a deep inventory oftools,information sources, test data, and friends to drawupon. While testing, they remain alert for opportunities toapply those resources to the testing at hand.
12 Characteristics of Exploratory Tester : …contdManage and Succeed :Excellent exploratory testers know the value of theirprecious time.They differentiate between a dead end and apromising lead. They follow their main mission whiletaking deviations to ensure the coverage.Keep Track :Excellent exploratory tester keeps track of what wastested, techniques used and risks the tests were intendedto explore, and how they relate to the mission of testing.
13 Learning Styles and ET: A learning style is a characteristic strengths and preferencesin the ways they take in and process information.Sensory/Intuitive : Preferred type of information.Visual/Verbal : Preferred mode of information received.Active/Reflective : Preferred organization of information.Inductive/Deductive : Information processing approach.Sequential/Global : Preferred way of understanding.This is based on an informal survey of a small number of testers by Andy Tinkham and Cem Kaner.Reference “Learning Styles and Exploratory Testing”.
14 Learning Styles and ET : …contdSensory/IntuitiveA sensory learner preferentially perceives information by the external senses.Examples :Sights , Sounds , Physical sensations.etc. The sensory-based person focuses on the details gained on actual observations. Inclined to develop a standard pattern for approaching exploratory testing.An intuitive learner perceives information relies on the internal information (generated from memory, conjecture, and interpretation) and intuition. The intuitor focuses on the internal model , applies risk-based approach and different theories of error to the software.
15 Learning Styles and ET : …contdVisual/VerbalVisual learners perceive information from visual images such as pictures, movies, diagrams or demonstrations.They tend to work off an internal model that is picture-based.They make diagrams and pictures for their notes as they explore.Verbal learners perceive information hearing (or reading) to lectures, written words, and mathematical formulas.They use a textual model like descriptions for their testing.
16 Learning Styles and ET : …contdInductive/DeductiveAn inductive learner prefers to work from specifics and derive the generalities.They gather as many specifics as possible along with the historical data and generalize them to the application.A deductive learner starts with the generalities and applies them to the specific situations they encounter.They apply guidelines and heuristics immensely in their approach.
17 Learning Styles and ET : …contdActive/Reflective :Active learners experiment with the information they receive and like to work in groups and find solutions. They do very rapid hands-on testing.They solicit the feedback from the teams.Reflective learners prefer to think about information before they use it and tend to work alone.They have a thought process that precedes each test.They make up for their lack of speed by executing the “good” tests that are most likely to find bugs.
18 Learning Styles and ET : …contdSequential/GlobalSequential learners learn in a logically structured incremental manner. They follow “linear reasoning processes”. The sequential tester will seem to get off to a faster start and test plans will grow in complexity over time as the understanding deepens.Global learners learn in chunks and have a holistic view of the information. A global tester will get off to a slower start. They need demonstrations on the application.Once the understanding is complete , he creates complex tests and can catch the bugs which are normally not seen.
19 Styles of Exploration : HunchesError handlingExamplesTroubleshootingModelsGroup InsightSpecificationsInvariancesInterference
23 Styles of Exploration : …contdInvariances :Tests that change things that should have no impactExample:Printing a word document or a power point.
24 Styles of Exploration : …contdInterferences :InterruptAlterTerminateSuspendSwitchContend.
25 Styles of Exploration : …contdError Handling :Check if errors and exceptions are handled correctly.Specifications :HeuristicsUser ManualActive reading
26 Styles of Exploration : …contdGroup Insights :Brainstormed test listsGroup discussion of related componentsFishbone analysisPaired Exploratory TestingTroubleshooting :Bug Analysis.Bug regression.
27 Pros and Cons of ET: Pros :- Does not require extensive documentation. Responsive to changing scenarios.Under tight schedules, testing can be more focuseddepending on the bug rate or risks.Improved coverage.
28 Pros and Cons of ET: Cons :- Dependent on the tester’s skills. Test tracking not concrete.More prone to human error.No contingency plan if the tester is unavailable.
29 Formalised Methods of ET What is Formalised ET ?Scripted Testing vs Formalised ETWhat does ET achieve ?Approaches used for Formalised ET
30 What is Formalised ET ?A structured and reasoned approach to exploratorytesting is termed as Formalised ET. This approachconsists of specific tasks, objectives, and deliverablesthat makes it a systematic process.Using the systematic approach, arrive at an outline ofwhat to achievewhat to attackwhat would be the scopewhat effort to spendApplying planning to Exploratory Testing, we haveExploratory Planning.
31 Scripted Testing Vs ET Increasing degree of formalised ET Ad hoc testingPure Scripted testingChartersVague ScriptsDocumentation(%)
32 What does Formalised ET achieve? How much planning is done with ET?How do you execute the tests with ET?What kind of documentation is produced with ET?How to use Pair Testing with ET?
33 Approaches used for Formalised ET Identify the application domainIdentify the purposeIdentify the primary and secondary functionsIdentify the workflowsIdentify the break pointsIdentify the UI standards
34 Approaches used for Formalised ET …contdIdentify the expected resultsIdentify the the interfaces with other interfaces/externalapplicationsRecord failuresDocument the issues and questionsDecomposing the activites to smaller tasksCharter – states the goal and the tactics to be used
35 Approaches used for Formalised ET …contdSession Based Test Management (SBTM)It’s the concept of Testing in SessionsA session, is not a test case or bug report, is the basictesting work uninterrupted unit of reviewable,chartered test effort. It operates within a documentedprocess structure and produces records that complementverification documentation.Reference :“Session-Based Test Management” by Jonathan Bach (first published in Software Testing and Quality Engineering magazine, 11/00)
36 Approaches used for Formalised ET …contdUnInterrupted : No significant disturbances during the testing( e.g mails, calls, meetings etc.).Reviewable : A report , typically called a Session Sheet is produced on the testing performed which can be comprehended by competent authority and client.Chartered : States the ‘What’ and ‘How’ of testing and the issues to look for.It is not suggested to make it a detailed plan.
37 Approaches used for Formalised ET …contdWe can separate test sessions into broadly 3 tasks(called the TBS metrics) :Test design and Execution: Scanning the product and looking for problems.Bug investigation and Reporting : Begins when the tester stumbles on something what typically looks to be a fault.Session Setup : Setting up the test environment and improving the understanding of the product.
38 Approaches used for Formalised ET …contdSBTM uses a very generalized light weight report called the Session Sheet.A Session sheet consists of:Session charterData filesTester name(s)Test notesDate and time startedIssuesTask breakdown (the TBS metrics)Bugs
39 Approaches used for Formalised ET …contdEach session is debriefed.During debriefing the manager reviews the session sheets, makes improvements to the charters, gets feedback from the testers and estimates and plans made for further sessions.Agenda for debriefing session is :PROOFPast : What happened during the session?Results : What was achieved during the session?Outlook : What still needs to be done?Obstacles : What got in the way of good testing?Feelings : How does the tester feel about all this?
40 Approaches used for Formalised ET …contdDefect Driven Exploratory Testing:Defect Driven Exploratory Testing (DDET) is a goal orientedapproach focused on the critical areas identified on the Defectanalysis study based on Procedural Testing results.Scripted TestsBug ReportsDDET*Reference: Defect Driven Exploratory Testing, Ananthalakshmi H.V,SISL
41 Approaches used for Formalised ET …contdDDET:The Pre-Requisites for productive DDET are :In-depth knowledge of the product.Procedural Testing has to be carried out.Defect Analysis based on Scripted Tests.Key advantages of DDET are :Tester has clear clues on the areas to be explored.No wastage of time.Goal oriented approach , hence better results.
42 Approaches used for Formalised ET …contdDDET:Scripted Testing
43 Approaches used for Formalised ET …contdDDET:Applying DDET
44 Approaches used for Formalised ET …contdDDET:Normal ETTimeDefectsFormalized ETDDET
45 Improvements Improvements Test Session SheetsMetricsImproved Approaches using ETUsing DDET / RDET.
46 Improvements Test Session Sheets Charter \ Mission Areas Covered …contdTest Session SheetsCharter \ MissionAreas CoveredStrategy UsedSession SheetSession and Tester detailsTBS metricsTest NotesAnomaliesQuestionsIssues
47 Improvements Why “Test Session Sheets” ? …contdWhy “Test Session Sheets” ?Documented record for regulatory purposes.Managers can have track of quantum of testing.Client can have track of billable hours.Standardized reporting mechanism across the organization.Prevents overlapping of the areas covered.Acts as a feedback medium.Provides the structure for organization and management.
48 Improvements Metrics Helps determine Test Coverage details. …contdMetricsHelps determine Test Coverage details.Helps in debriefing sessions.Helps the manager to plan resource and work allocation.Provides a simple medium for traceability.Provides an insight into the quality of testing.Helps collect important data for organizational purposes.Test Metrics
49 Improvements Coverage Metrics Session Division : Session Name …contdCoverage MetricsSession Division :Session NameDate of sessionTimeDurationS107/01/200409:00 Hrs90 mins S2S3S4
50 Description of the Role Improvements…contdGroup Division:GroupTestersG1Joe, MaryG2G3G4User groups :User Group AcronymApplication RolesDescription of the RoleU1AdminAdministratorU2UserEnd user
51 Module/Element Tested Improvements…contdModule/Element – Session Tracker:Session id/nameModule/Element TestedM1M2M3M4S1 G1 G3 G4 G2S2G2/G3S3S4(This table is used to track modules covered by different groups in the different sessions).
52 Improvements Group – Role Tracker : X X For Session S1: User Group …contdGroup – Role Tracker :For Session S1:UserGroupGroupsG1G2G3G4U1 XU2 X
53 Improvements Test Summary Metrics: …contdTest Summary Metrics:Can be used for designing better sessions and estimation in future .Top management can track the progress of the project.Helps in knowing the maturity of the existing process and team.Test Summary Metrics
54 Improvements What are Q-patterns ? …contdWhat are Q-patterns ?A set of related questions on a software requirement whichprovides you with various alternatives that can beenhanced and reused.*Reference : by Vipul Kocher.
55 Improvements Structure of a Q-Pattern Associated patterns …contdStructure of a Q-PatternAssociated patternsName of the patternExamplesIntent/Explanation/definitionSpecializationQuestions related to:AdministrationUsage/UserUISecurityPerformance
56 Improvements Using DDET / RDET: Review ET Scripted Test cases Scripted …contdUsing DDET / RDET:ReviewETScriptedTest casesScriptedTestingDDET /RDETQ patterns
57 Using Orthogonal Arrays, Decision tables and State Diagrams
58 Orthogonal Arrays Orthogonal Arrays Testing Strategy: The Orthogonal Array Testing Strategy (OATS) is asystematic, statistical way of testing pair-wise interactionsand deriving a suitable small set of test cases(from alarge number of possibilities).It provides representative (uniformly distributed) coverageof all variable pair combinations.
59 Orthogonal Arrays…contdOrthogonal arrays are two dimensional arrays of numbers which possess the interesting quality that by choosing any two columns in the array an even distribution of all the pair-wise combinations of values in the array can be tested.Terminology for working with orthogonal arrays:Runs: The number of rows in the array which translates to the number of test cases that will be generated by the OATS technique.
60 Orthogonal Arrays…contdFactors : The number of columns in an array. This directly translates to the maximum number of variables that can be handled by this array.Levels : The maximum number of values that can be taken on by any single factor. An orthogonal array will contain values from 0 to Levels-1.Strength : The number of columns it takes to see each of the Levels Strength possibilities equally often.Orthogonal arrays are represented by the pattern LRuns(LevelsFactors).
61 Orthogonal Arrays Steps to create Orthogonal Arrays: …contdSteps to create Orthogonal Arrays:Identify the independent variables that will be tested for interaction. This will map to the Factors of the array.Identify the maximum number of values that each independent variable will take on. This will map to the Levels of the array.
62 Orthogonal Arrays…contdIdentify a suitable orthogonal array with the smallest number of Runs.(A suitable array is one that has at least as many Factors as needed from Step 1 and has at least as many levels for each of those factors as decided in Step 2.)Map the Factors and values onto the array.Choose values for any "left over" Levels.Transcribe the Runs into test cases, adding any particularly suspicious combinations that aren't generated.
63 Orthogonal Arrays…contdExample : Consider a screen with three drop downs list with three values.OA for the case is shown belowFACTORSRUNSFactor1Factor2Factor3123
64 Decision tables What are decision tables ? Decision table is a tabular representation ofcombinational logic used to derive test cases andprovide a means of tracking complex combinationsof conditions and resulting actions.Ideal for describing situations where varying conditionsproduce a combination of different actions.
65 Decision tables Why use a Decision Table? …contdWhy use a Decision Table?Provides a representation of requirements where aresponse or subset of the response is selected by evaluating many related conditions.Effective at finding faults in both implementation andspecification.Can support test design at all scopes from method tosystem.Supports manual or automated generation of test cases.
66 Decision tables Decision tables …contdSteps to create Decision Table:List all causes in the decision tableCalculate the number of possible combinationsFill columns with all possible combinationsReduce test combinationsCheck covered combinationsAdd effects to the table
67 Decision tables…contdExample : Decision Table of a credit card billing system
68 State Diagrams What are State Diagrams ? State diagrams show the dynamic behaviour of a system.The diagram shows the various states that an object canget into and the transitions that occur between the states.Terminology for State Diagrams:State: The state object is a snapshot of an object at a particular point in its life. A state may have an activity describing the function being performed.
69 State Diagrams…contdInitial State : The initial state is the starting state of the object with reference to the behaviour that the diagram explains. Each state diagram should have only one initial state.Final State : Each final state is the ending state of the object with reference to the behaviour that the diagram explains. There may be multiple final states for an object.
70 State Diagrams…contdTransition : The transition link represents the relationship between different states of an object. The transition guard is a condition which limits the cases in which a transition can occur. The transition action is performed during the transition and cannot be interrupted.
72 Team Management Building a ET team Role of QA manager/Lead Pair TestingMaintaining Morale & Accountability
73 Building an ET team Look for diverse skilled resources. Convincing the experienced testers to join the ET team.Find the right mixTechnical and Domain knowledge.Experience.Learning Styles.
74 Role of QA manager/Lead Decide the charters and sessions.Decide the risk factors involved.Hold debriefing sessions.Deciding the strategies for ET.Prepare the metrics.Maintaining the morale of the team.Updating the Q-Patterns.Resource allocation.
75 Pair TestingPair testing is a way of approaching a test design processtwo people test the same thing at the same time and placeone works the PC, the other makes creative suggestionsand writes deviation reportscontinuously exchanging ideasgeneration of more and different ideas than either tester islikely to produce on his ownhigh productivity, high creativityan effective complement to individual testing
76 Pair Testing…contdSuccessful pair testing requires three specific conditions:At least one tester is available who can be trusted to testwithout supervision.Another tester is available who can join the first tester fora session of test design.The two testers are otherwise capable of workingtogether.
78 Maintaining Morale & Accountability Informal QC groups meetings to discuss the learnings,experience and suggestions for improvement.Assigning challenging and varied tasks to avoidmonotony.Develop an whale done attitude towards the tester.Change the pairs regularly to avoid dependency.QA lead or the manager involved in testing/Coachingthe novice testers.
79 Exploratory testing in corporate world TimeResourcePersonnelInfrastructureExperience,Skill and knowledgeAvailable DocumentationFrequent Requirement changesCost and QualityMorale