1. www.PerfTestPlus.com © 2008 PerfTestPlus, Inc. All rights reserved. Performance Tuning in a Virtual Environment Page 1 StickyMinds.com and Better Software magazine presents… Whose Environment is It Anyway? Performance Tuning in a Virtual Environment. Sponsored by Empirix Non-streaming participants should call 1-866-761-8643 International Non-streaming participants should call 1-904-596-2362

2. www.PerfTestPlus.com © 2008 PerfTestPlus, Inc. All rights reserved. Performance Tuning in a Virtual Environment Page 2 Scott Barber, (A.K.A. “The Perf Guy”) sbarber@perftestplus.com sbarber@perftestplus.com Chief Technologist, PerfTestPlus, Inc. www.perftestplus.com www.perftestplus.com Vice President & Executive Director, Association for Software Testing www.associationforsoftwaretesting.org www.associationforsoftwaretesting.org Co-Author, Performance Testing Guidance for Web Applications www.amazon.com/gp/product/0735625700 www.amazon.com/gp/product/0735625700 A Heuristic Approach

3. www.PerfTestPlus.com © 2008 PerfTestPlus, Inc. All rights reserved. Performance Tuning in a Virtual Environment Page 3 Developers write code Developers release code for test Testers test code, often in various environments Testers assess system performance in not very production-like environments Managers make decisions based on test results Some defects get fixed before release The team gets surprised by defects in production The team puts in lots of overtime to fix the defects Testing in a Non-Virtual World

4. www.PerfTestPlus.com © 2008 PerfTestPlus, Inc. All rights reserved. Performance Tuning in a Virtual Environment Page 4 Wikipedia lists 8 categories of platform virtualization and 4 categories of resource virtualization Platform Virtualization Software simulates a computer environment on a host machine, usually called “virtual machines” Applications run on “virtual machines” as if on a stand- alone hardware platform As if stand-alone ≠ Isolated Platform virtualization software has overhead (http://en.wikipedia.org/wiki/Virtualization)http://en.wikipedia.org/wiki/Virtualization What is Virtualization?

5. www.PerfTestPlus.com © 2008 PerfTestPlus, Inc. All rights reserved. Performance Tuning in a Virtual Environment Page 5 Resource Virtualization Identify a resource and exclusively assign it’s use to a specific application Sometimes partitions single resources, sometimes spans multiple resources Resources may or may not be co-located on a physical machine Exclusively assigned ≠ Isolated Resource virtualization has overhead (http://en.wikipedia.org/wiki/Virtualization)http://en.wikipedia.org/wiki/Virtualization What is Virtualization?

6. www.PerfTestPlus.com © 2008 PerfTestPlus, Inc. All rights reserved. Performance Tuning in a Virtual Environment Page 6 Developers write code Developers release code for test Testers test code, often in various configurations Testers assess system performance in not very production-like environments Managers make decisions based on test results Some defects get fixed before release The team gets surprised by defects in production The team puts in lots of overtime to fix the defects Testing in a Virtual World

7. www.PerfTestPlus.com © 2008 PerfTestPlus, Inc. All rights reserved. Performance Tuning in a Virtual Environment Page 7 Virtualization, if done well, can: Dramatically reduce the overhead involved in managing test environments Create opportunities for unique, targeted, ad-hoc environments without additional hardware costs Simplify configuration and release management Put the test team in control of their test environments But what about performance testing? What’s the Difference?

8. www.PerfTestPlus.com © 2008 PerfTestPlus, Inc. All rights reserved. Performance Tuning in a Virtual Environment Page 8 “With an order of magnitude fewer variables performance testing could be a science, but for now, performance testing is at best a scientific art.” --Scott Barber

9. www.PerfTestPlus.com © 2008 PerfTestPlus, Inc. All rights reserved. Performance Tuning in a Virtual Environment Page 9 Test environments are still test environments Virtual test environments are less likely to match virtual production environments There are more places for performance issues to hide in virtual environments Exclusively assigned ≠ Isolated As if stand-alone ≠ Isolated Virtualization adds overhead Study: “virtualized servers can handle 14% lower [load] than traditional server configurations” (http://www.sys-con.com/read/503412.htm)http://www.sys-con.com/read/503412.htm Testing Virtual Performance?!?

10. www.PerfTestPlus.com © 2008 PerfTestPlus, Inc. All rights reserved. Performance Tuning in a Virtual Environment Page 10 “Linear extrapolation of performance test results is, at best, black magic. Don’t do it. (Unless you learned from Connie Smith, PhD or Daniel Menasce, PhD)” --Scott Barber

11. www.PerfTestPlus.com © 2008 PerfTestPlus, Inc. All rights reserved. Performance Tuning in a Virtual Environment Page 11 If done well, there can be benefits: Dramatically reduce the overhead involved in managing test environments Easier for performance testing to have it’s own test environment Enable collaborative tuning without having to “borrow” hardware resources Put the test team in control of their test environments Virtualization does not change the core principles of successful performance testing Testing Virtual Performance?!?

12. www.PerfTestPlus.com © 2008 PerfTestPlus, Inc. All rights reserved. Performance Tuning in a Virtual Environment Page 12 “Only performance testing at the conclusion of system or functional testing is like ordering a diagnostic blood test after the patient is dead.” --Scott Barber

13. www.PerfTestPlus.com © 2008 PerfTestPlus, Inc. All rights reserved. Performance Tuning in a Virtual Environment Page 13 Core Principles Project context is central to successful performance testing. Business, project, system, & user success criteria. Identify system usage, and key metrics; plan & design tests. Install & prepare environments, tools, & resource monitors. Script the performance tests as designed. Run and monitor tests. Validate tests, test data, and results. Analyze the data individually and as a cross-functional team. Consolidate and share results, customized by audience. "Lather, rinse, repeat" as necessary.

14. www.PerfTestPlus.com © 2008 PerfTestPlus, Inc. All rights reserved. Performance Tuning in a Virtual Environment Page 14 Contact Info Scott Barber Chief Technologist PerfTestPlus, Inc E-mail: sbarber@perftestplus.com Web Site: www.PerfTestPlus.com

15. Whose Environment Is It Anyway? Dan Koloski Web BU CTO and Director of Strategy Empirix IMAGE

16. Empirix – At a Glance Testing & Monitoring Solutions for: Privately held, founded 2000 380+ employees Worldwide presence; HQ in Boston & offices in Japan, UK, Germany and South Korea Patents in test, monitoring and VoIP diagnostics methodology Web Applications & SOA Contact Center & CRM Carrier-Class VoIP & Telco

17. Empirix in the Quality Ecosystem Empirix Offerings Delivered Outcome Empirix Offerings Delivered Outcome PRODUCTS MANAGED SERVICES EXPERT SERVICES Onsite Consulting, Training, Mentoring e-TEST suite ™ OneSight™ e-LoadExpert™ Do-it-yourself Remote Validation Performance Tuning Supplement Your Team We Do it for You

18. A few points to add to Scott’s presentation… Scott said… –Virtualization does not change the core principles of successful performance testing Why? –Hardware is not virtual –Whose environment is it, anyway? –Rapid Bottleneck Identification Tuning a Virtualized Context

19. Hardware is not virtual Virtualization is app/OS or hardware emulation specific Underneath it all is a fixed pool of HW resources! Performance Testing is designed to understand the impact on the hardware – as seen from the perspective of the user Common Virtualization Types. Source: QAZone Whitepaper, Virtualization from the perspective of a QA/IT Operation Professional, http://qazone,empirix.com.

20. The Best Performance Data in a Virtual Environment Understand the impact on the hardware… –Requires accurate workload stimulus representing activity on and through all VMs associated with that device –Requires measurement of hardware characteristics at the base layer as well as OS and app-level characteristics in each VM –Requires real-time collaboration during load testing to differentiate intra-VM bottlenecks (business logic) vs Hardware bottlenecks (CPU, Memory, NIC) …as seen from the perspective of the user! –Your users don’t care about Virtualization –Metrics that matter are end-user perceived performance –Performance testing from inside the VM helps create stimulus but doesn’t report on end-user-perceived performance

21. Rapid Bottleneck Identification Load Testing in a Virtual Environment What is Load testing? “Testing conducted to isolate and identify the system and application issues (bottlenecks) that will keep the application from scaling to meet its performance requirements”

22. What is a Bottleneck? Any resource (hardware, software or bandwidth) that places defining limits on data flow or processing speed: your application is only as efficient as its least efficient element On the Web, bottlenecks directly affect performance and scalability Most untested systems have more than one bottleneck, but they can only be identified and resolved one at a time.

23. Where are the Bottlenecks? The vast majority of bottlenecks are caused by throughput limitations – not concurrency. Source: Empirix

24. Sample throughput limitations in virtual environments Disk I/O –Multiple applications accessing shared disk resources –Virtual Memory Paging on shared disk resources NIC –Multiple VMs shipping data through the same physical NIC CPU –Processor-intensive (i.e., database) applications will compete for shared CPU time. Memory –Depends on type of VM configuration how much of an issue this can be

25. Whose Environment Is It Anyway? Virtualization blurs the line between Dev/Test/Ops environments and ownership –Can exacerbate “silo” problems between Dev/Test/Ops However, there is an opportunity… …A common set of measurements can un-blur the line –Test data should inform production SLAs –Common measurement/metrics can bridge the gap –Measurement consistency is the cornerstone of ALM

26. Load Testing Tools DON’T Collaborate Load Testing Tools Apply Load and Make Measurements People (and their expertise & experience) Find Bottlenecks & Resolve Them Collaboration With Experts Resolves Issues Faster & Cheaper Key to Successful Collaboration = Access to People & Access to Measurements Tools Should Support Collaboration

27. e-TEST suite is designed for collaboration Distributed, high-scale load control Multi-user Web UI for real-time individual analysis Each user can view results from the same load test as it runs

28. Summary Hardware is not virtual – virtualization does NOT change core performance test best practice However… –Virtualization introduces new test planning challenges to cover –Workload/stimulus –Measurement of impact on end-user In a world of scarce time, focus efforts on throughput- related workload to identify most common bottlenecks

29. Visit QAZone for the slides http://qazone.empirix.com Free resource, open to all (Empirix & non-Empirix users) –Forums (both QA-centric and Product related discussions) –Knowledge Base –Resource Center –Events, Announcements, Community & Industry News Allows members to interact with peers, industry experts and a larger Empirix audience

30. www.PerfTestPlus.com © 2008 PerfTestPlus, Inc. All rights reserved. Performance Tuning in a Virtual Environment Page 30 Have a question for the speakers? Ask now. Q & A