1. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare By: Scott Barber Chief Technologist PerfTestPlus, Inc. Easy & Valuable, yet Rarely Implemented Aspects of Performance Testing Created for:

2. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare Chief Technologist, PerfTestPlus, Inc. sbarber@perftestplus.com www.perftestplus.com sbarber@perftestplus.com www.perftestplus.com @sbarber Co-Founder: Workshop On Performance and Reliability www.performance-workshop.org www.performance-workshop.org Scott Barber Co-Author: Author: Contributing Author: Books: www.perftestplus.com/pubs About me: about.me/scott.barber www.perftestplus.com/pubs about.me/scott.barber Easy & Valuable, yet Rarely Implemented Aspects of Performance Testing

3. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare Belief it is done at the “end” & only under load Low priority/visibility (until prod has issues) Dependency on pre-defined requirements No explicit goals for the testing effort Ignoring the simple or obvious “Unionizing” the testing Tool dependency Dismissal of “Common Sense” Top Performance Testing Pathologies Belief that only “Experts” can add value

4. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare “Let’s face the truth, performance testing *IS* rocket science.” --Dawn Haynes … but even rocket science involves *SOME* easy stuff. -- Addendum added by: Scott Barber

5. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare System or application characteristics related to: Speed: - responsiveness - user experience Scalability: - capacity - load - volume Stability - consistency - reliability - stress What is Performance?

6. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare What mom tells people: I help people make websites go fast. What is Performance Testing? What I tell people: I help and/or teach individuals and organizations to optimize software systems by balancing: Cost Time to market Capacity while remaining focused on the quality of service to system users.

7. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare Conception to Headstone Not Cradle to Grave The Performance Lifecycle is:

8. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare Who is Responsible for Performance?

9. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare “With an order of magnitude fewer variables performance testing could be a science, but for now, performance testing is at best a scientific art.” --Scott Barber

10. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare So then, its hopeless?

11. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare Fact: One does not need to be a performance testing rock star to have a significant positive impact on performance… …and thus add significant business-value… …quickly and simply.

12. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare What we actually hope to gain by testing performance Are sometimes completely unrelated to stated requirements, goals, thresholds, or constraints Should be the main drivers behind performance test design and planning Usually indicate the performance-related priorities of project stakeholders Will frequently override good/bad in “go-live” decisions Value Begins with Clear Objectives

13. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare Start with these questions: In Other Words… “What’ve we got? What do we want? How do we get there…?” --Bob Barber (Scott’s dad) … as quickly, simply, and cheaply as possible? -- Addendum added by: Scott Barber

14. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare Talk about how focusing on performance adds value and mitigates risk from “bar napkin to delete key”. Get performance in the dev, test, & delivery plans. Don’t let performance fall off the plate. Be the advocate, even if it makes you “annoying”. Some Ways to Get Started How is the performance today? How will this [change] effect performance? Ask during dev, test, & management meetings – even if you weren’t invited. How is the performance today? How will this [change] effect performance? Ask during dev, test, & management meetings – even if you weren’t invited. Break out the puzzle!

15. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare Development / Sprint Build / Minor Release Major Release Candidate Production Goals/Budgets Unit Tests/Trends Story Acceptance Build Validation/Regression By Feature Load (Comprehensive) Peak Load (Comprehensive) Weakness Identification External Integrations Infrastructure/Tuning Break Point Identification Future Planning Deployments Maintenance Rapid Response Data Forward

16. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare Super Simple Stuff Market Research Unit-Level Tests Acceptance Criteria Integration-Level Tests Field Studies Snapshots T 4 APM

17. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare Market Research: http://www.keynote.com/keynote_competitive_research/ http://gtmetrix.com/top1000.html http://www.vertain.com/?peti http://www.gomez.com/benchmarks/ Conduct Market Research

18. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare Tools: FireBenchmarks; Performance testing addin for NUnit JUnitPerf; a collection of JUnit test decorators for performance Firefox Performance Tester's Pack HTTPerf Unit-Level Testing

19. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare Field Studies & Acceptance Tests Ask

20. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare Why am I annoyed? How annoyed am I? Does this annoy me all the time, or just sometimes? What impact is this likely to have on product value? Advocate something better. Thoughts on Annoyance

21. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare Concepts: Stick the pain chart in your Manual Test Scripts/Charters Jot down response times from browser plugin Build a “relationship table” associating times to annoyance, page/interaction by page/interaction Put timers in BVTs & Automated Regression Suites: Keep results in a “trend” file “Flag” results that are “worse” than goal or “best” previous result Don’t be afraid of stubs, mocks & virtual services… … but don’t confuse them with the real deal! Integration-Level Testing

22. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare Field Study: https://docs.google.com/spreadsheet/viewform?formkey =dDMzTUFQWnRsSW5sVDJsU0N1Z0tqMmc6MA Field Studies

23. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare Tools: http://www.webpagetest.org/ https://developers.google.com/speed/pagespeed/insights http://webwait.com/ http://www.websitepulse.com/help/tools.php http://www.websiteoptimization.com/services/analyze/ http://mobitest.akamai.com/m/index.cgi http://www.gtmetrics.com Snapshots

24. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare Units Stories Tiers Resources Goals Units Stories Tiers Resources Goals Assess Compare Investigate Accept Answer Assess Compare Investigate Accept Answer Dev & Prod Times Resources Sizes Frequencies Dashboard! Dev & Prod Times Resources Sizes Frequencies Dashboard! Proactive Micro & Macro Establish Goals Update Targets Proactive Micro & Macro Establish Goals Update Targets

25. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare If you don’t have speed targets, don’t fret... Trends are trendy! This particular data comes from automated BVTs!

26. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare So, what is Performance Testing? In effect: Performance testing helps stakeholders make decisions regarding product value and project risk; Specifically value and risk related to speed, scalability, and stability attributes of a system and it’s components throughout the product life-cycle.

27. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare The Bottom Line For each goal, determine what information will answer: Has this goal been achieved? To what degree? What needs to be done to achieve this goal? Decide what data must be collected to provide that information Figure out how to collect that data For each goal, determine what information will answer: Has this goal been achieved? To what degree? What needs to be done to achieve this goal? Decide what data must be collected to provide that information Figure out how to collect that data

28. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare Did we learn anything? Review & Questions

29. © 2012 PerfTestPlus, Inc. All rights reserved. Easy, Valuable & Rare Contact Info Scott Barber Chief Technologist PerfTestPlus, Inc E-mail: sbarber@perftestplus.com Blog: scott-barber.blogspot.com Web Site: www.PerfTestPlus.com Twitter: @sbarber