2. Improving Disk Latency and Throughput with VMware
Presented by
Raxco Software, Inc.
March 11, 2011

3. Today’s Agenda
Provide technical information on how NTFS impacts VMware I/O performance
Examine ESX I/O test results
Economic impact of Windows guests
Solutions

4. Virtualization Benefits
Server consolidation
Less physical space for data centers
Lower energy costs
Easier management
Eco-friendly alternative
These benefits can only be realized if everything works as it should. The one thing the virtualization layer never touches is the guest system. If the guest systems overload the hypervisor, I/O contention occurs and performance suffers.

5. Identifying and Correcting Problems
Latency is your best indicator of a performance problem
Device latency is vSphere’s report of the physical storage response time
Kernel latency is vSphere’s report of ESC’s ability to manage IO
Experts disagree on specifics, but most agree that…
Device latency in excess of 15ms is worth inspection
Device latency in excess of 30ms is likely a problem
Kernel latency in excess of 2ms means ESX queues are overflowing
High device latency can result in ESX queuing
So, correct slow hardware first!
Then, consider reducing VMDKs on a VMFS volume
Only then consider changing queue depths
In a presentation at EMC World in 2010, VMware’s Scott Drummonds, discussed the importance of disk latency. When latency is > 15 ms you should be concerned, when it exceeds 30ms, you have a real problem. File and free space fragmentation increases disk latency because they increase the time it takes to read/write a file.
© Copyright 2010 EMC Corporation. All rights reserved.

6. Storage Contention Solution: Storage IO Control
SIOC calculates data store latency to identify storage contention
Latency is a normalized, average across virtual machines
IO size and IOPS included
SIOC enforces fairness when data store latency crosses threshold
Default of 30ms
Fairness enforced by limiting VMs access to queue slots
Net effect: trade throughput for latency
With Storage IO Control
Actual Disk Resources utilized by each VM are in the correct ratio even across ESX Hosts
VMware’s Storage I/O Control monitors latency and trades off throughput to improve latency. While this treats the symptom (slow latency), the solution (reduced throughput) is still an issue.
The real question is why are the disks slow in the first place?
© Copyright 2010 EMC Corporation. All rights reserved.

7. NTFS I/O Storms
All of your work is done in the guest systems. With Windows guests, the NTFS file system can work against you by creating I/O storms resulting from file and free space fragmentation. These I/O storms lead poor disk latency, less throughput, and poor performance.

8. NTFS Behavior NTFS fragments files and free space
Increases logical I/O to storage controller
More logical I/O = More physical I/O
Multiple instances of Windows on host can lead to I/O contention
The Windows NTFS file system is notorious for fragmenting files and free space. This fragmentation is related to how NTFS sees the disk. Each fragment results in a scsi command going to the disk controller. A file in one piece issues one logical I/O; a file in 100 pieces issues 100 logical I/O. Each logical I/O maps to the disk in one or more physical I/O. In a virtual system each logical I/O is overhead for the hypervisor.

9. What is Fragmentation? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
Fragmentation happens in the NTFS file system and on the physical disk, but the two are different. NTFS fragmentation is a result of how the file system allocates space logically, it’s just how the file system works. NTFS fragments files before anything is written to the disk.
Once the data is passed to the disk controller the firmware in the controller maps the logical data to physical blocks on the disk. If a file is in a single contiguous string of logical clusters, and it is mapped to a stripe set across three platters, you could have the best possible solution. The file system sends a single logical I/O and the controller mapped it for efficient retrieval. This breaks down when the file system sends the controller 100 logical I/O for a s fragmented file requiring 350 physical I/O to the disk to accommodate the file. This is how logical and physical fragmentation combine to hammer performance. ?

10. Logical v Physical Logical Level
NTFS needs disk and cluster size, enumerates LCNs
Creates $MFT and $Bitmap metadata
$Bitmap is how NTFS “sees” the disk
Has no idea about physical/virtual disk types
When a disk is formatted NTFS needs to know its size and the cluster size. It does not know what kind of disk it is or if it is physical of virtual.
When a file is created a record for the file is created in the $MFT, the volume index. NTFS then asks for space from the $Bitmap file. $Bitmap offers space in one chunk if it can locate it (a contiguous file). If contiguous space is not available $Bitmap offers space wherever it can find it (a fragmented file).

11. Anatomy of an MFT Record
(vcn, lcn, run length): (8a85, 9189a, 7)
This is a look inside a MFT record. The Extent List is where information about the logical location of a file is kept.
The VCN is the virtual cluster number and indicates which part of the file this piece is 1st, 2nd, 3rd…
The LCN is the starting logical cluster number for the fragment and it is followed by the run length which is the number of clusters from the LCN. The MFT record is 1KB in size. Severely fragmented files will have multiple linked MFT records which are needed to accommodate lengthy Extent Lists. This also has an adverse effect on performance.

12. File Allocation Create $MFT record (one or more)
$Bitmap accessed to locate free space
$MFT record is updated with content
NTFS creates a file record in the $MFT for a new file. It then goes to $Bitmap to find space. The logical clusters allocated are recorded in the MFT Extent List.
Bitmap
Access
MFT
Update
Create

13. File Access Load portion of MFT with correct record via directory
Locate file in the MFT
Pass starting LCN’s and run lengths to disk controller
Number of logical fragments influences number of physical seeks
A file that is badly fragmented at the file system level often results in excess physical I/O to the disk.
Locate
File
# Physical
Seeks
Load
# LCN’s

14. Logical v. Physical Physical Level Disk controller Maps LCNs to PCNs
Writes data to disk
The logical I/O is sent to the disk controller and mapped to physical blocks on the disk.

15. Wasted Seeks 58.30% 12.72% Fragmented After PerfectDisk
Partition State
Total Number of I/O Requests Sent to the File System
Total Number of Resulting Disk Accesses/Seeks
Net Wasted Seeks When Running SYSmark
Percent Net Wasted Seeks When Running SYSmark
Fragmented
1,320,686
2,090,649
769,963
58.30%
After PerfectDisk
1,434,454
1,616,847
182,393
12.72%
After Built-In
1,411,613
1,931,395
519,782
36.82%
David Goebel, a Microsoft engineer and one of the original 4-man team that wrote the NTFS, did an analysis to measure the importance of consolidated files and free space. Goebel ran a benchmark against two identical disks, one with the free space fragmented and the other with the free space consolidated. He wrote software that counted every request from the file system and then counted how many physical I/O were required to write the data. Ideally, each file system request should result in a single disk I/O.
The results show that the fragmented disk performed 58% more seeks than there were file system requests. When the disk was defragmented and the free space consolidated only 13% of file system requests required additional disk I/O. The extra logical I/O create extra physical I/O with its attendant impact on performance.
In this benchmark the I/O count difference between the fragmented disk and the PerfectDisk disk was 587,570. At 15ms per I/O this is an additional 2.4 hours of I/O processing time.

16. How This Affects A Virtual Environment
P2V Conversion
Extra Hypervisor Overhead
Disk Latency Degradation
Overall Performance
System Throughput
Wasted Space
Costly
File and free space fragmentation can bloat the size of going P2V.
Each extra command (one per fragment) across the virtual storage stack increases hypervisor overhead
Excess disk I/O increases disk latency time with an impact on overall performance
Disk latency affects system throughput
Fragmentation wastes valuable disk space
Problems associated with fragmentation cost more in troubleshooting, downtime and potentially unnecessary hardware upgrades.

17. P2V Conversion Physical Drive No Optimization Optimization 24GB 22GB
2GB Smaller
If you go P2V on a fragmented volume the virtual disk is almost the same as the physical disk. When you go P2V with defragmentation, free space consolidation and then zero-fill the free space you get a much smaller image. This saves P2V storage costs.

18. ESX Cluster Testing Identical disks - 40% free space
Optimized one set, the other “as is”
Installed MS Office and MS SQL
Captured metrics with VMware’s vscsiStats utility
Using VMware’s vscsiStats utility we measured the impact of defragmentation on two identical sets of disks. The first set was fragmented the second was optimized with Raxco’s PerfectDisk. The test involved installing MS Office and MS SQL to both sets of disk and collecting the I/O data on both.

19. Total I/O Count 37191 29238 21.3 Total IO Count Fragmented PerfectDisk
% Improvement
Total IO Count
37191
29238
21.3
Read IO Count
3066
2799
8.7
Write IO Count
34125
26439
22.5
The PerfectDisk disk reduced the total I/O crossing the stack by 21.3%. VMware has a mantra that says “More IOPS are bad, less IOPS are good.”
Optimizing the Windows guest reduced the total I/O by

20. 49% Reduction in Latency! 40,442 20,764 30ms 50ms 100ms >100ms
Total
Fragmented
I/O
12749
9877
8700
9116
40,442
PerfectDisk I/O
6707
4923
4081
5053
20,764
Disk latency is the time it takes an I/O to complete. As noted in slide 5, VMware and EMC state that disk latency in excess of 30ms is a problem. In our test the PerfectDisk disk reduced the number of I/O taking more than 30ms by 49%.
VMware’s Storage I/O Control monitors latency and limits access to HBA-LUN queues when latency exceeds 30ms. In effect, trading throughput for latency. By contrast, optimizing the Windows guest delivered a huge reduction in latency with no adverse effect on throughput.

21. Disk Latency
Actual vscsiStat histogram. Note that the red lines (PerfectDisk) are about 50% of the blue for all I/O greater than 1ms. This means PerfectDisk reduced the total number of I/O in each time interval by half.

22. 12X More Large I/O 247 2959 Total IO > 524K Fragmented Disk
PerfectDisk Disk
Total IO Equal to 524K
2512
848
Total IO > 524K
247
2959
Read IO Equal to 524K 33 7
Read IO >524K
125 65
Write IO Equal to 524K
2480
841
Write IO >524K
122
2894
One of the benefits of disk optimization is you can do larger I/O and thereby improve throughput. Our testing showed that PerfectDisk performed 12 times as many of the largest I/O as measured by vscsiStats.

23. 12 times more of the largest IO
Large I/O
Actual vscsiStats histogram. Note right-most red line for I/O >524K.
12 times more of the largest IO

24. Improved Sequential I/O
Fragmented
PerfectDisk
Improvement
Percent Sequential
17%
27%
58%
Total IO
127703
90526
25%
Sequential IO
22126
24340
33%
Not surprisingly, disk optimization improves sequential data access since the files are contiguous. Testing showed PerfectDisk improved sequential I/O by 58%.

25. Improved Sequential I/O
Center column shows that PerfectDisk exceed the fragmented disk in sequential I/O. Conversely, the left and right-most columns show the fragmented disk outdid PerfectDisk in the worst category where the I/O were the farthest apart (500,000 LBNs).

26. Installation Time Comparison
Fragmented
PerfectDisk
% Improvement
MS Office Install
20 min
15 min 25
MS SQL Install
76 min
51 min 33
The software installation times in the benchmark improved dramatically with the PerfectDisk disk.

27. The Cost of Fragmentation
EXAMPLE:
20 files x 6 seconds = 2 minutes
300 users x 2 min = 10 hours/day
10 hrs x $25/hr = $250/day
Annual cost = $62,500
The primary cost of the fragmentation is the cost of waiting. Whether it is on workstations or servers, as files and free space fragments the time is takes to read/write files increases. The figures above were taken from two separate customer ROI analyses. Neither company had ever defragmented their machines. An analysis of several disks showed files and free space were fragmented and performance was degrading.
An assumption was made that each user had al least 20 frequently used files that took at least 6 seconds too long to open. This was a loss of 2 minutes per day per user.
There were 300 users so lost time equaled 10 hours per day with a loaded cost of $25/hour.
The annual cost of fragmentation was $62,500. Management also recognized that if they did nothing the read/write times would continue to increase, performance-related problems would increase and they might buy hardware they didn’t really need.
The cost of PerfectDisk for the workstations and servers here was about $10,000. The ROI based on lost productivity alone was 40 workdays.

28. Virtual Guest Fragmentation
Windows guests have all the same NTFS behavior
Fragmentation produces more IOPS
Fragmentation reduces ESX throughput
Fragmentation increases ESX disk latency
Fragmentation creates resource contention between host & guests
Everything described thus far about the NTFS file system applies to virtual Windows servers. NTFS behavior will increase IOPS, increase latency, degrade throughput and potentially create I/O contention issues.

29. Solutions Expensive Inexpensive More disks and faster disks
Upgrade Fibre Channel
Troubleshooting
Inexpensive
Optimize the Windows guest systems
The expensive options are more and better hardware. This means fragmentation, a natural byproduct of using Windows, can happen faster.
The inexpensive solution is to optimize the Windows guests with PerfectDisk.

30. PerfectDisk 12 vSphere Virtualization Awareness/host & client
NEW
Virtualization Awareness/host & client
OptiWrite Fragmentation Avoidance
“Zero-fill” free space
NEW
NEW
PerfectDisk offers:
Optimization awareness between host and guests to avoid contention issues
OptiWrite fragmentation avoidance that prevents 95% of the fragmentation that would occur
A “zero-fill” free space option that aids in P2V conversion and offers a better VM compaction option. If you have a zero-detect SAN you can recover white space after the zero-fill completes

31. PerfectDisk 12 vSphere “Short stroking” for thin provisioned disks
Schedule guest compaction
Snapshot & Linked Clone recognition
NEW
NEW
NEW
Short-stroking makes it possible to defragment a thin-provisioned volume without blowing out all the storage.
New algorithms minimize file movement so defragmentation and zero-fill can take place
Snapshot and linked clone recognition. These are disks you generally don’t want to defrag so PerfectDisk flags them for you. PerfectDisk also differentiates between dependent/independent disks and persistent/non-persistent disks for use in virtual desktop environment.

32. PerfectDisk Benefits on ESX
Saves $$$ in productivity and admin
Reduces resource contention for VM’s
Reduces total IO workload
Improves throughput
Reduces disk latency
Delivers optimal performance
Optimizing Windows guest systems eliminates a well –documented problem with Windows. Keeping guest systems in good shape reduces the potential for I/O issues related to latency and throughput. It is always better to be proactive than reactive.

33. Contact Raxco Free Evaluation Software
Excellent Support to Get You Started
White Papers
Great ROI
Toll Free: