Whose Environment Is It Anyway? Dan Koloski Web BU CTO and Director of Strategy Empirix IMAGE
Empirix – At a Glance Testing & Monitoring Solutions for: Privately held, founded 2000 380+ employees Worldwide presence; HQ in Boston & offices in Japan, UK, Germany and South Korea Patents in test, monitoring and VoIP diagnostics methodology Web Applications & SOA Contact Center & CRM Carrier-Class VoIP & Telco
Empirix in the Quality Ecosystem Empirix Offerings Delivered Outcome Empirix Offerings Delivered Outcome PRODUCTS MANAGED SERVICES EXPERT SERVICES Onsite Consulting, Training, Mentoring e-TEST suite ™ OneSight™ e-LoadExpert™ Do-it-yourself Remote Validation Performance Tuning Supplement Your Team We Do it for You
A few points to add to Scott’s presentation… Scott said… –Virtualization does not change the core principles of successful performance testing Why? –Hardware is not virtual –Whose environment is it, anyway? –Rapid Bottleneck Identification Tuning a Virtualized Context
Hardware is not virtual Virtualization is app/OS or hardware emulation specific Underneath it all is a fixed pool of HW resources! Performance Testing is designed to understand the impact on the hardware – as seen from the perspective of the user Common Virtualization Types. Source: QAZone Whitepaper, Virtualization from the perspective of a QA/IT Operation Professional, http://qazone,empirix.com.
The Best Performance Data in a Virtual Environment Understand the impact on the hardware… –Requires accurate workload stimulus representing activity on and through all VMs associated with that device –Requires measurement of hardware characteristics at the base layer as well as OS and app-level characteristics in each VM –Requires real-time collaboration during load testing to differentiate intra-VM bottlenecks (business logic) vs Hardware bottlenecks (CPU, Memory, NIC) …as seen from the perspective of the user! –Your users don’t care about Virtualization –Metrics that matter are end-user perceived performance –Performance testing from inside the VM helps create stimulus but doesn’t report on end-user-perceived performance
Rapid Bottleneck Identification Load Testing in a Virtual Environment What is Load testing? “Testing conducted to isolate and identify the system and application issues (bottlenecks) that will keep the application from scaling to meet its performance requirements”
What is a Bottleneck? Any resource (hardware, software or bandwidth) that places defining limits on data flow or processing speed: your application is only as efficient as its least efficient element On the Web, bottlenecks directly affect performance and scalability Most untested systems have more than one bottleneck, but they can only be identified and resolved one at a time.
Where are the Bottlenecks? The vast majority of bottlenecks are caused by throughput limitations – not concurrency. Source: Empirix
Sample throughput limitations in virtual environments Disk I/O –Multiple applications accessing shared disk resources –Virtual Memory Paging on shared disk resources NIC –Multiple VMs shipping data through the same physical NIC CPU –Processor-intensive (i.e., database) applications will compete for shared CPU time. Memory –Depends on type of VM configuration how much of an issue this can be
Whose Environment Is It Anyway? Virtualization blurs the line between Dev/Test/Ops environments and ownership –Can exacerbate “silo” problems between Dev/Test/Ops However, there is an opportunity… …A common set of measurements can un-blur the line –Test data should inform production SLAs –Common measurement/metrics can bridge the gap –Measurement consistency is the cornerstone of ALM
Load Testing Tools DON’T Collaborate Load Testing Tools Apply Load and Make Measurements People (and their expertise & experience) Find Bottlenecks & Resolve Them Collaboration With Experts Resolves Issues Faster & Cheaper Key to Successful Collaboration = Access to People & Access to Measurements Tools Should Support Collaboration
e-TEST suite is designed for collaboration Distributed, high-scale load control Multi-user Web UI for real-time individual analysis Each user can view results from the same load test as it runs
Summary Hardware is not virtual – virtualization does NOT change core performance test best practice However… –Virtualization introduces new test planning challenges to cover –Workload/stimulus –Measurement of impact on end-user In a world of scarce time, focus efforts on throughput- related workload to identify most common bottlenecks
Visit QAZone for the slides http://qazone.empirix.com Free resource, open to all (Empirix & non-Empirix users) –Forums (both QA-centric and Product related discussions) –Knowledge Base –Resource Center –Events, Announcements, Community & Industry News Allows members to interact with peers, industry experts and a larger Empirix audience