© Workroom Productions Slide 1 Further Adventures in Session-based Testing James Lyndsay Workroom Productions

© Workroom Productions Slide 2 ? What?

© Workroom Productions Slide 3 Session-Based Testing? Manages and controls unscripted testing Limited Duration Directed Exploration Recorded Activities Supports ‘Agile’ Process Allows quick metrics Has wider uses

© Workroom Productions Slide 4 Adventures? Two case studies Permanent team - 20 coders, 4 testers Inexperienced and under-motivated Internet-based product, ASP model Part-time team 2-6 coders, 0-3 testers bought in from pool of 6-8 No permanent test team Windows client / server product around known application Shared business context Rapid change Commercial constraints; budget, time-to-market

© Workroom Productions Slide 5 Further? Paper - Adventures in Session-based Testing Contains detailed methods Contains problems and (some) solutions Now a year old STARWest Further Adventures... Methods Tools Training, Coaching

© Workroom Productions Slide 6 Strategy Stay within business constraints Deal with change rather than enforce stasis Rapidly transfer knowledge to allow independence Four goals Control the scope Control the work Measure risk and coverage Promote ongoing improvement

© Workroom Productions Slide 7 1 Methods

© Workroom Productions Slide 8 Methods: Overview Control and recording Control Scope Control Work Recording a Session Assessment and Measurement Assessing Risk Assessing Coverage Infrastructure Real-time Metrics Social techniques

© Workroom Productions Slide 9 Example: Test Points (Excel-based tool)

© Workroom Productions Slide 10 Methods: Control Scope - made of Test Points Test Point - piece of work Risk, Estimated cost Four groups New tests Regression tests Retests for fixes Investigatory tests Work done - made of Test Sessions Test Session - piece of time Actual cost, Coverage

© Workroom Productions Slide 11 Example: A Test Session

© Workroom Productions Slide 12 Methods: Control Recording a Test Session Plan: Approach, Scope, Expectations Record: Conditions, Actions, Observations Bug details Actual Cost How done are we?  Coverage Allows Review, coaching Forgetting Retrospectives

© Workroom Productions Slide 13 Methods: Assessment Risk Coverage Ask the Experts - the testers! How done are you? How much is left? Collect many subjective  few objective assessments Overall figures can go down as well as up Ongoing learning - estimates improved

© Workroom Productions Slide 14 Example: Metrics (Excel-based tool)

© Workroom Productions Slide 15 Methods: Support Real-time metrics Immediate feedback State of testing, not of system Fast, frequent, informed re-planning Soft skills/Social Techniques Experimental Approach Ongoing learning Communication Empowerment

© Workroom Productions Slide 16 2 Tools

© Workroom Productions Slide 17 Tools Freely available - under GPL Excel spreadsheet Single-user, single location Download from: Web: PHP/MySQL Multiple user Download from:

© Workroom Productions Slide 18 3 Training and coaching

© Workroom Productions Slide 19 Ongoing learning and development Training - from unskilled to skilled Skilled involvement - pair expert / novice Show - practice - challenge Careful choice of sessions Coaching - from good to great! Peer involvement, neutral assessment Challenge - innovate Discussion, metrics, personal investigation

© Workroom Productions Slide 20 Supporting the learning process Session-based testing Introduces discipline and framework Helps overlap training and testing Helps match challenge to individual Experts, managers - get involved! Sometimes compromise: Good bugs found quickly, or Better testers long-term Feedback is integral Before testing - techniques, planning, charter After testing - session log, bug log Stats and metrics - not just for management

© Workroom Productions Slide 21 Encouraging the team Involve the test team Allocate responsibility / ownership for tests Allow direct influence on clear, open metrics Relate metrics to business decisions Share successes, failures Reward discovery Discourage blind faith, boredom Step away - look at the wider picture

© Workroom Productions Slide 22 Changes: Case Study 1 Active learning process is encouraged Feedback has improved estimating skills Session review has improved test techniques Test Point ownership has improved planning and scoping skills Weekly 2-hour self-training session as a team Greater interest in testing, greater motivation Useful, visible, up-to-date metrics Increase openness, reduce surprise, blame Joint ownership of decisions, successes, failures

© Workroom Productions Slide 23 ! Conclusion

© Workroom Productions Slide 24 Conclusion Communication Sessions gave us the tools to communicate Better communication improved trust and effectiveness Ongoing improvement Sessions allow directed and disciplined approach Feedback improved techniques and estimation Empowerment increased morale and confidence Openness All figures available, all of the time, always current Encouraged trust and interest Allowed neutral feedback

© Workroom Productions Slide 25 Conclusion Test sessions provide: Control of unscripted testing Framework for ongoing improvement Overall lessons Simple measures are the best Favour effective communication and involvement over knee-jerk documentation Unobtrusive, immediate metrics allow real- time control and feedback

© Workroom Productions Slide 26 More Further details: Paper, updates, references etc. Contact: James Lyndsay