Presentation is loading. Please wait.

Presentation is loading. Please wait.

Presented by David Davis Director of Infrastructure

Similar presentations

Presentation on theme: "Presented by David Davis Director of Infrastructure"— Presentation transcript:

1 Presented by David Davis Director of Infrastructure
Performance Tuning, Management and Optimization in a Virtual Infrastructure Presented by David Davis Director of Infrastructure

2 Who Is David Davis? Director of Infrastructure Train Signal, Inc -- the leader in Professional IT video training Over 15 years in enterprise infrastructure management and years of real-world virtualization experience Have obtained the following certifications: CCIE#9369, VCP, MCSE and CISSP

3 Who Is David Davis? Author of six video training courses and hundreds of articles for well-known websites such as: and Best known for my Train Signal VMware ESX Server video training course Best of VMworld 2008 Awards Judge Company website: Personal website:

4 Abstract Virtualized Infrastructures can perform as well or better than physical infrastructures, if performance tuning and management are done correctly. Today’s Applications are complex: Virtualized Distributed Intensive Tied to SLAs

5 Abstract We will cover…
Performance Tuning and Management in the virtual infrastructure Best practices for virtualization performance How to troubleshoot bottlenecks on existing systems Tools that allow IT Pros to manage virtual infrastructure properly

6 Abstract What are common mistakes that can hurt performance and how can you prevent them? What tools are available for managing the performance of your virtual infrastructure? What are the best practices for configuring your virtualization infrastructure to ensure ideal performance?

7 What I Assume You Already Know
Assumptions… Good understanding of server virtualization concepts May or may not already be using virtualization Have, or will have, performance concerns (that’s everyone, right?)

8 By The End Of The Session, You’ll Know The Following:
How to manage performances How to optimize performance How to troubleshoot performance issues How to design your VI so that you prevent performance issues

9 Virtualization Basics: Virtualization Guest Encapsulation

10 Virtualization Performance Basics
Hardware Device Drivers Storage Stack Network Stack VMM VMkernel Resource Management UserWorlds POSIX API VM Other Peripheral I/O Hosted Management Agents and Interfaces Service Console

11 Managing Performance

12 Managing Performance

13 Managing Performance

14 ESX 3.5 And Update 1 Up to 32 logical processors per host (64 LP experimental) Large memory support – 250 GB physical and 64Gb per VM Large page size – 2 MB VMKernel pages can be allocated to guest OS 192 vCPUs per Host -- Update 1

15 ESX “4.0” CPU and RAM “hot add”
Historical performance tracking and performance alerts Clustered VirtualCenter Servers ESX hosts profile management Cross-hosts virtual networking 8-way virtual SMP Virtual machines fault tolerance across multiple hosts dubbed “vlockstep”

16 Virtualization Overhead
CPU -- special handling of instructions Memory -- additional management tasks Devices and resource management -- not direct access to hardware Typically, difficult to notice -- in my experience

17 Memory Overhead Service console 272 MB (not in ESXi) VMKernel 100 MB+
Per-VM memory overhead increases with: Number of VCPUs Size of guest memory 64-bit guest OS

18 Virtual CPU Recommendations
Single threaded app = uni proc VM Multi threaded app = SMP VM But only as many as required Unused VCPU in SMP VM = scheduling overhead -- see KB 1077 and 1730

19 Virtual CPU Recommendations
Make sure OS HAL matches number of CPUs -- MP vs. UP HAL Use 64-bit guests, if possible -- more registers, larger kernel tables Still, remember 64-bit OS vs. app compatibilities

20 Performance Issues Due To Interrupts
Any controller but usually USB Disable USB See KB 1290

21 Overall Device Recommendations
Disable / remove all unused devices USB, CDROM, Floppy Can consume CPU when idle

22 Large Guest Pages Backed By Host
New in ESX 3.5 Significant performance improvement for memory intensive apps Best to allocate large pages immediately after VM boot Page sharing not supported for large pages

23 Network Performance Check NICs for proper speed and duplex, hardcoded
NIC teaming distributes load and offers passive failover Separate NICs avoid contention -- console, VMKernel, and VM Tune VM to VM networking and rx/tx buffers (KB 1428)

24 Network Performance Use 32-bit vmxnet driver instead of vlance
To use vmxnet, install tools E1000 is for 64-bit guests Enhanced vmxnet is offered for several guests

25 Network Performance Use a network adapter that supports the following:
Checksum offload TCP segmentation offload (TSO) Jumbo frames (JF), available in enhanced vmxnet vNIC Capability to handle high memory DMA Capability to handle multiple scatter/gather elements per Tx frame 10 G

26 Install VMware Tools Vmxnet -- high speed net driver
Memory balloon driver Improved graphics Timer sponge for correct accounting of time Timesync -- syncs time with host every minute

27 ESX Server Storage Performance
FC Switch ESX Server HBA1 HBA2 HBA3 HBA4 Storage array SP2 SP1 1 2 3 4 Hardware configuration affects storage performance Consult SAN Configuration Guides Ensure caching is enabled Consider tuning layout of LUNs across RAID sets Spread I/O requests across available paths

28 Storage Performance Fibre Channel SAN storage best practices
Set LUN queue depth appropriately (KB 1267) Networked storage best practices (NFS, iSCSI) Ensure sufficient CPU for software-initiated iSCSI and NFS Avoid link oversubscription Ensure consistent configuration across the full network path Use multiple mount points with multiple VMs

29 Benchmarking With VMark
VMmark: A scalable benchmark for virtualized enterprise systems Provides meaningful measurement of virtualization performance Generates metric that scales with underlying system capacity Used to compare the performance of different hardware and virtualization platforms Employs realistic, diverse workloads running on multiple operating systems

30 Storage Performance VM Configuration Esxtop enhancements
Choose placement of data disks and swap files on LUNs appropriately RAID type, spindles available, concurrent access of LUNs etc. Increase VM’s max outstanding disk requests if needed (KB 1268) Esxtop enhancements Per device and path stats Per VM device stats

31 DRS Performance Ensure hosts in a cluster are VMotion compatible
Minimize reservations if possible Use VM affinity and anti-affinity rules only when needed Migration threshold should be set less aggressively when Hosts in the cluster are inhomogeneous VM resource utilization is highly variable in time More affinity and anti-affinity rules Use DRS to achieve max performance

32 Troubleshooting Performance
Know your applications Have a baseline Esxtop Decent tools in VI Client Find Bottleneck, CPU, Disk, RAM or Net Host or Guest? Some components may be out of your domain -- both SAN and Net are critical

33 Designing VI To Prevent Performance Issues
Capacity Planning is key Know your apps Understand the SAN Use DRS / Resource Pools Don’t skimp on hardware

34 Prevent Common Performance Mistakes
P2V Poor Improper Sizing Poor Hardware Selection Alerting not configured Not using DRS

35 Performance Tools Esxtop vKernel Capacity Bottleneck Analyzer
vKernel Modeler Solarwinds Orion VMware Edition and free VM Monitor Veeam Monitor Nagios Vizioncore vCharter / vFoglight eG VM Monitor

36 INSERT GRAPHICS – estimated to be about 5 slides
Performance Tool Demo INSERT GRAPHICS – estimated to be about 5 slides

37 David’s Five Performance Tips
Know how to use esxtop -- quick and simple Know your applications and environment Have a baseline Don’t skimp on hardware Use third-party performance tools -- historical performance monitoring is required

38 Conclusion Virtualization environments continue to grow in complexity
Managing performance doesn’t have to be difficult Follow best practices, know your environment and use third-party performance tools With that, performance can be improved and troubleshooting can be simplified

39 References Related papers for best practices and benchmarking:
ESX Server 3 performance tuning best practices VMmark SAN Configuration Guide:

40 Questions?

Download ppt "Presented by David Davis Director of Infrastructure"

Similar presentations

Ads by Google