Vital Signs: Performance Monitoring Windows Server Module 2: Disk

Vital Signs: Performance Monitoring Windows Server Module 2: Disk
Microsoft Confidential

Conditions and Terms of Use
Microsoft Confidential This training package is proprietary and confidential, and is intended only for uses described in the training materials. Content and software is provided to you under a Non-Disclosure Agreement and cannot be distributed. Copying or disclosing all or any portion of the content and/or software included in such packages is strictly prohibited. The contents of this package are for informational and training purposes only and are provided "as is" without warranty of any kind, whether express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, and non- infringement. Training package content, including URLs and other Internet Web site references, is subject to change without notice. Because Microsoft must respond to changing market conditions, the content should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication. Unless otherwise noted, the companies, organizations, products, domain names, addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, address, logo, person, place, or event is intended or should be inferred. Copyright and Trademarks © 2011 Microsoft Corporation. All rights reserved. Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation. For more information, see Use of Microsoft Copyrighted Content at Microsoft®, Internet Explorer®, and Windows® are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. Other Microsoft products mentioned herein may be either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. All other trademarks are property of their respective owners. © 2011 Microsoft Corporation Microsoft Confidential

Students: How to View this Presentation
Switch to Notes Page view Click View on the ribbon and select Notes Page Use page up or page down to navigate Zoom in or out as needed Most slides will have supporting text that you can view now or after the delivery Add notes to your copy of the presentation if you want to. You take the presentation files home with you. Microsoft Confidential

Microsoft Confidential
Module 2: Disk Section 1: Disk hardware Section 2: Concepts Section 3: Disk counters Section 4: Disk analysis tools Microsoft Confidential

Section 1: Disk hardware
Hard disk magnetic media Solid state drive (SSD) Direct attached storage (DAS) Just a Bunch of Disks (JBOD) Network attached storage (NAS) Internet Small Computer System Interface (iSCSI) Storage area network (SAN) I/O operations per second (IOPS) Dedicated spindles Shared spindles Microsoft Confidential

Hard disk magnetic media
A magnetic disk (spindle) rotates at high speed so data can be read or written This basic design has not changed much in many years, although capacity and speed have increased greatly Hard disk drives (magnetic media) have not changed in many years and are still widely in use today. Microsoft Confidential

Solid state drive (SSD)
A data storage device that uses solid-state memory to persistently store data, in order to provide access in the same manner as a traditional block I/O hard disk drive Uses NAND-based flash memory, with no moving parts Generally uses less power and has significantly faster access times than spinning magnetic media A solid state drive (SSD) is a data storage device that uses solid-state memory to store persistent data. It provides access in the same manner as a traditional block I/O hard disk drive. An SSD uses NAND-based flash memory, with no moving parts. It generally uses less power and has significantly faster access times than spinning magnetic media. Microsoft Confidential

I/O operations per second (IOPS)
An industry measurement of the hardware throughput of a disk or a collection of disks For example, a single 7200 RPM disk drive has an uncached, full stroke access time of 13 ms, which calculates to at least 77 IOPS Average response time longer than 15 ms is generally considered slow The following chart shows worst-case IOPS and access times: Storage IOPS* Access time (ms)* 3.5“ floppy disk USB drive 1.54 600 5400 RPM hard disk 59 17 7200 RPM hard disk 71 13 10K RPM hard disk 122 8 15K RPM hard disk 143 7 Solid state drive (SSD) 5000 0.2 IOPS (I/O operations per second) is an industry standard for measuring the hardware throughput of a disk subsystem. You can calculate the IOPS of your disk hardware if you read its specifications and determine what its access time is. All of the metrics mentioned in the slide above can be compared to performance counters. The chart on this slide shows the worst-case (no cache; full stroke) access times for common hard drives. Hardware specifications and performance vary greatly between manufacturers. 15 ms is a generally accepted threshold * General approximations of full stroke (worst case) without cache. Does not reflect actual products. Microsoft Confidential

I/O operations per second (IOPS) continued…
Disk speeds are determined by moving parts. Rotational Latency is the rate at which the platter rotates. [speed] / 60 for rotations / sec and [speed] / 60,000 for rotations / ms A 15k disk rotates 250 / sec, which means one full rotation takes 4 ms A 10k disk rotates 167 / sec, which means one full rotation takes 6 ms Average Seek Time is the time it takes the head assembly on the actuator arm to travel to the track of the disk where the data will be read or written. Average Seek Time can vary greatly; not all disks are created equal. If the Average Seek Time is not listed on the HDD label, you may need to contact the vendor to attain this value. A 15k disk may have an Average Seek Time of 3 ms or more. A 10k disk may have an Average Seek Time of at least 5.5 ms A 7200RPM disk Average Seek Time may be closer to 9 ms IOPS (I/O operations per second) is an industry standard for measuring the hardware throughput of a disk subsystem. You can calculate the IOPS of your disk hardware if you read its specifications and determine what its access time is. All of the metrics mentioned in the slide above can be compared to performance counters. The chart on this slide shows the worst-case (no cache; full stroke) access times for common hard drives. Hardware specifications and performance vary greatly between manufacturers. Microsoft Confidential

I/O operations per second (IOPS) continued…
To determine the IOPS for a disk use the following formula: (1/[Rotational Latency (half rotation) + Average Seek Time])*1000 A 15k disk may have an IO time from 5 to 6 ms and be capable of 167 to 200 IOPS A 10k disk may have an IO time of around 8.15 ms and be capable of 122 IOPS A 7200 RPM disk may have an IO time of 14.1 ms and be capable of only 71 IOPS Consult your HDD manufacturer for the Average Seek Time values. These numbers do not account for any type of cache and are not for Solid State Disks. Why is an access time of 15 ms or more generally considered slow? Because 15ms is slower than a 7200 RPM disk. IOPS (I/O operations per second) is an industry standard for measuring the hardware throughput of a disk subsystem. You can calculate the IOPS of your disk hardware if you read its specifications and determine what its access time is. All of the metrics mentioned in the slide above can be compared to performance counters. The chart on this slide shows the worst-case (no cache; full stroke) access times for common hard drives. Hardware specifications and performance vary greatly between manufacturers. Microsoft Confidential

Challenge Question: If your application requires 2000 IOPS, how many disks are needed (not accounting for RAID overhead)? Answer: It depends on the disk speed. You might need 10 15k disks or you may need 17 10k disks or you may only need 1 solid state disk. Required IOPS / IOPS per disk = Amount of disks needed 2000 (IOPS needed) / (1 15k disk) = 10 2000 (IOPS needed) / (1 10k disk) = 17 2000 (IOPS needed) / 5000 (1 SSD) = 1 The amount of disks needed to perform 2000 IOPS depends on the type of disk used. Faster disks allow for more IOPS. If fast 15k disks are used, there would need to be at least 10 disks to cover 2000 IOPS. Each 15k disk should be capable of IOPS. Ensure that the Average Seek Time on the disks is at least 3ms. If the Average Seek Time is slower you would need more disks. Microsoft Confidential

Direct attached storage (DAS)
Presented as a Logical Unit Number (LUN) A hard drive that is “directly” attached to a server or workstation, without a storage network in between Called a “local” disk Generally implemented as IDE, SCSI, SATA, etc. Direct Attached Storage (DAS) are disks that are represented to Windows and Windows Server as a single physical disk or LUN (Logical Unit Number). ID 0 LUN 0 Disk controller IDE, SATA, SCSI, etc. LUN 0 & 1 ID 1 LUN 1 Microsoft Confidential

Just a Bunch of Disks (JBOD)
JBOD is direct attached storage (DAS) that is configured, typically, in a Redundant Array of Independent Disks (RAID) set Disks might be independently addressed or concatenated by hardware LUN 1 Server A JBOD (Just a Bunch Of Disks) is direct attached storage that is very close in physical proximity to the physical server. The server might see each spindle as an individual disk, or the hardware controller might consolidate the spindles into one or more LUNs. J B O D J B O D Microsoft Confidential

Network attached storage (NAS)
File-level computer data storage that is connected to a TCP/IP Ethernet network Uses Server Message Block (SMB), Network File System (NFS), or other network protocols Available via UNC network paths (\\svr\share) or in some cases as direct attached storage (DAS) NAS Controllers NIC \\svr\share1 Network Attached Storage (NAS) is file-level computer data storage that is connected via a TCP/IP Ethernet network. To access data on a NAS, a UNC network path is needed. A NAS uses the Server Message Block (SMB) network communication protocol. A NAS hardware device or appliance might or might not be running a Windows Server operating system, and it is effectively a Windows file server on the network. Computers access a NAS device as if it was a Windows file share by using the UNC naming convention, such as \\srv\share. TCP/IP NIC \\svr\share2 Other NAS Microsoft Confidential

Internet Small Computer System Interface (iSCSI)
A gigabit, IP-based storage networking standard for linking data storage facilities A protocol suite that makes it possible for SCSI commands to be encapsulated and transmitted over the TCP/IP stack To access the iSCSI network, computers use an Ethernet network interface card (NIC) SAN Controllers iSCSI NIC LUN 1 & 2 iSCSI and encapsulates and transmits SCSI commands using TCP/IP over Ethernet. Unlike a NAS device, which accesses data via a UNC path, iSCSI can present one or more LUNs to computers that are bound to it. The LUNs appear as local disk drives to the computers. TCP/IP NIC LUN 1 & 2 Other SANs Microsoft Confidential

Storage area network (SAN) and Fibre Channel (FC)
A SAN is a network of storage resources A SAN consolidates and shares storage media, using hardware efficiently Computers connect to one or more SANs by using a host bus adapter (HBA) to connect to a Fibre Channel network Fibre Channel (FC) is a gigabit-speed network technology over fiber-optic cables that is primarily used for network storage SAN Fibre Channel or SAN fabric SAN Controllers HBA LUN 1 & 2 A storage area network (SAN) is a network of physical spindles that is managed by one or more controllers. Host Bus Adapters (HBA) are physical cards that are installed on computers that need to access a SAN. HBAs access one or more SANs by using a Fibre Channel (FC) network, which uses fiber optic cables. Zero or more Fibre Channel switches can be used to manage the network traffic between HBAs and SANs. A SAN presents direct attached storage LUNs to the HBAs, giving the computers the appearance of local physical disks. Fibre Channel switch HBA LUN 1 & 2 Other SANs Microsoft Confidential

Dedicated spindles Physical spindles that are not used by or shared with any other computer Generally reliable and consistent IOPS and response times Each server has its own disk usage Some servers might have idle spindles Poor performance affects only one server In this case, the Exchange server needs more spindles. The SQL server and the file server have idle spindles 0 idle spindles 6 idle spindles 9 idle spindles Before the arrival of SANs, servers running Windows Server had JBOD disk arrays that were dedicated to the server. This gave the servers consistent and reliable amounts of IOPS and response times, but it also caused spindles to frequently be idle. Exchange server SQL server File server over utilization moderate utilization under utilization Microsoft Confidential

Shared spindles Physical spindles that are used by or shared with other computers Higher potential for unreliable or inconsistent IOPS and response times than dedicated spindles Spindles (and potentially controllers) are pooled together and share the load Less likely to have idle spindles compared to a dedicated spindle model Some servers might have better or worse disk performance Good or poor performance can affect multiple servers In this example, disk performance of the Exchange server is better, at the cost of using the spindles for the SQL server and the file server ? idle spindles Now that SAN technology has arrived, Windows Server administrators can share spindles across multiple servers, allowing all of the spindles to be used (so they are less likely to be idle) and maximizing the monetary investment in hardware. If the shared spindles become overwhelmed, this can affect all of the servers with a LUN attached to the same spindles. SAN Controllers Exchange server SQL server File server variable? variable? variable? Microsoft Confidential

Section 1 review What is the difference between NAS and a SAN? Why is an access time of 15 ms or more generally considered slow? Microsoft Confidential

Section 1 review (answers)
A SAN provides direct attached storage (C:, D:, etc.), and a NAS provides network shares (\\svr\share). Because 15 ms is the uncached access time median between 5400 and 7200 RPM hard disks, response time averages above it are generally considered slow. Microsoft Confidential

Section 2: Concepts Physical disks and logical disks Software RAID Hardware RAID I/O request packets (IRPs) Microsoft Confidential

Physical disks and logical disks
A LUN that is presented to the operating system Unknown hardware behind the LUN—unknown spindles Direct attached storage (DAS) Logical disk Disk volume with a drive letter Multiple logical disks can exist on a physical disk, and vice versa Disk Manager A physical disk is represented to Windows and Windows Server as a LUN. Physical disks are shown in the left pane of Disk Manager (see the screenshot in the slide). Multiple logical disks can be on a single physical disk, and a logical disk can be stripped across multiple physical disks. Logical disks (drive letters and mount points) Physical disks (LUNs) Microsoft Confidential

Drive G is a software RAID 0 set Microsoft Confidential
Software RAID combines Windows and Windows Server physical disks into a logical RAID set Windows Server 2003 and later support software RAID 0, 1, and 5 In this example, drive G is a logical software RAID 0 set, or a “spanned volume” Disk Manager Software RAID combines physical disks in Windows and Windows Server into a logical RAID set. Windows Server 2003 and later support software RAID 0, 1, and 5. In this example, the G drive is a logical software RAID 0 set, which is also called a spanned volume. Drive G is a software RAID 0 set Microsoft Confidential

Hardware RAID Presented to Windows and Windows Server as a single physical disk (LUN) Physical architecture is unknown to Windows and Windows Server Disk 2 might have one or more physical spindles behind it Hardware RAID presents Windows or Windows Server with a LUN, and the operating system treats the LUN like a single physical disk. Therefore, Windows and Windows Server do not know how many real spindles are behind the LUN. Microsoft Confidential

I/O request packets (IRPs)
An I/O request packet (IRP) is a data structure in Windows and Windows Server that controls how the I/O operation is processed at each stage Like a network packet, the structure is fully self- contained and has all routing information inside Most I/O requests are represented by an I/O request packet (IRP), which travels from one I/O system component to another. With this design, an individual application thread can manage multiple I/O requests concurrently. An IRP is a data structure that contains information completely describing an I/O request. The I/O manager creates an IRP that represents an I/O operation, passing a pointer to the IRP to the correct driver and disposing of the packet when the I/O operation is complete. In contrast, a driver receives an IRP, performs the operation the IRP specifies, and passes the IRP back to the I/O manager, either for completion or to be passed on to another driver for further processing. Microsoft Confidential

Section 2 review What is the difference between a physical disk and a logical disk from the perspective of Windows and Windows Server? What is the difference between software and hardware RAID? What is a data structure in Windows and Windows Server that controls how the I/O operation is processed at each stage? Microsoft Confidential

A physical disk is a direct attached storage device that is presented to Windows or Windows Server. It is also called a LUN. A logical disk is a disk volume that has a drive letter or that is a mount point. Hardware RAID consolidates multiple hard drives into a single LUN that is presented to Windows or Windows Server. Software RAID is where Windows or Windows Server consolidates multiple LUNs. An I/O request packet (IRP) is a data structure in Windows and Windows Server that controls how the I/O operation is processed at each stage. Microsoft Confidential

Section 3: Disk counters
PhysicalDisk and LogicalDisk counter objects Disk counters Troubleshooting Process I/O counters Disk free space counters Microsoft Confidential

PhysicalDisk and LogicalDisk counter objects
PhysicalDisk counters Measure LUN performance Hardware RAID, disk virtualization, and I/O distribution can make measurements difficult to assess Examples: 07 (D:, E:) LogicalDisk counters Measure performance of lettered drives Logical disk counters are usually more relevant to applications and services Example: drive D:, E:, or D:\Disk1\BackupDrive7 The PhysicalDisk counter object measures how many LUNs are attached to the computer. Hardware RAID can make many physical spindles appear as a single physical disk to Windows and Windows Server, so what appears to the computer to be a single physical disk might not actually be a single physical disk. This virtualization of physical disks can complicate the analysis of disk performance. The LogicalDisk counter object measures logical disk drive letters and mount points. A logical disk can span multiple physical disks, and there can be multiple logical disks on a single physical disk. If you do not know what the physical architecture of the disk subsystem looks like, generally start with the LogicalDisk counter object. This is because it is the closest measurement of file–to-process disk performance that you can get from performance counters. Microsoft Confidential

Disk Counters Primary counters Average Disk sec / Read Average Disk sec / Write These counters show disk latency. How long did the data take to access? Secondary counters Disk Transfers / sec Disk Bytes / sec Average Disk Queue Length Current Disk Queue Length These counters measure the amount of disk activity. How much data is being accessed? Counters are separated into two sets, Primary and Secondary. Primary counters are used to identify if a bottleneck exists. Secondary counters are used to help narrow down where the bottleneck may be (JBOD or Fabric). Microsoft Confidential

Avg. Disk Sec/Read|Write
Avg. Disk sec/Read and Avg. Disk sec/Write are the average times, in seconds, of read or write operations respectively Also called disk response times An initial indicator of disk performance Part of the LogicalDisk and PhysicalDisk counter objects Implemented at the disk driver class for high accuracy How long did it take? Disk queue Driver Hardware controller The best initial indicator of how the disk is performing is the average response time for servicing I/O request packets. The Avg. Disk sec/Read and the Avg. Disk sec/Write performance counters measure the response times of every I/O request packet and calculate the response times between counter collections. These counters are nearly as accurate as the disk driver class. These counters are often used synonymously with disk response times. Response time Bus|Network|Cable Physical media Microsoft Confidential (continued)

Avg. Disk Sec/Read|Write (continued)
Software and hardware disk cache can greatly affect disk latency Due to disk virtualization and I/O distribution, disk latency is a reliable measure of disk performance Compare to the access time in the hardware specification of the bus/network/cable and the physical media Example: A 7200 RPM disk drive should respond faster than 13 ms Measures average response times of IRPs Disk queue Driver Hardware controller If the disk hardware can keep up with the I/O request packet load, the response times are as fast or faster than the disk access times (the “worse case” amount of time needed to get data off of a disk). If there are more I/O requests in the disk queue than can be serviced by the disk in a timely manner, the average I/O request packet response times increase accordingly. This is why the Avg. Disk sec/Read and Avg. Disk sec/Write counters are important initial indicators of disk performance. Response time Bus|Network|Cable Physical media Microsoft Confidential

Thresholds for Avg. Disk sec/Read|Write
Response times longer than 15 ms (based on average I/O sizes of 64 KB or smaller) generally impact performance Quick spikes in response times are common and normal Common next steps: Check hardware for failures or faults Check I/O sizes more than 64 KB If the hardware is shared, consider using dedicated hardware If you are using spinning media, consider defragmenting Adjust the hardware cache Consider using faster media Reduce unnecessary I/O Threshold Effect 15 ms or less Informational Normal usage More than 15 ms Warning Performance might be affected More than 25 ms Critical Performance very likely affected The 15 ms (0.015) warning threshold for Avg. Disk sec/Read and Avg. Disk sec/Write is based on response times for 64 KB or smaller. A 7200 RPM disk drive can generally respond faster than 13 ms, based on its access time and assuming I/O sizes of 64 KB or smaller. Therefore, if the average response time is greater than its “worse case” response time and the I/O sizes are 64 KB or smaller, the disk is performing poorly. If you suspect poor disk performance, follow the common next steps listed in the slide. Microsoft Confidential

Example of Avg. Disk sec/Write
In this case, drive E consistently has response times of more than 25 ms, which greatly affects the application and system performance Drive E consistently has response times of more than 25 ms In the example in this slide, the counters are in the unit of seconds, so the value of equals 25 ms on the Avg. Disk sec/Read and Avg. Disk sec/Write performance counters. The disk response times are greater than 25 ms on average. Microsoft Confidential

Disk queue counter Disk queue Avg. Disk Queue Length counter Example of Avg. Disk Queue Length Microsoft Confidential

Disk queue Each disk has a disk queue
The disk queue length is the number of “in-flight” I/O request packets (IRPs) One or more IRPs can be serviced at the same time, depending on the physical media IRPs might be reorganized by a driver to optimize the physical media Example: The HBA queue depth determines how many I/Os a SAN can service simultaneously The number of “in-flight” IRPs waiting for service Disk queue Driver Hardware controller Each disk has a disk queue, which lists the “in-flight” I/O request packets that have not been serviced yet. Some disk drivers might retrieve the I/Os in a different order than the order they were received in. For example, if the disk queue received three I/O requests in the order 1-2-3, the disk driver might retrieve the I/Os from the physical media in the order This might be for various reasons, such as disk rotation or location on the spindle. The point is that the I/O requests in the disk queue might not be serviced in a first in first out (FIFO) order, causing some of the I/O requests to wait longer than others. Also, some disk hardware (such as a host bus adapter (HBA) of a SAN) might be able to handle more than one I/O request at a time. Bus|Network|Cable Physical media Microsoft Confidential

Avg. Disk Queue Length counter
The Avg. Disk Queue Length counter is the average number of both read and write requests that are queued or “in-flight” for the selected disk during the sample interval Part of LogicalDisk and PhysicalDisk counter objects Because of disk virtualization and I/O distribution, no threshold can be assigned Look for increasing trends The number of “in-flight” IRPs waiting for service The Avg. Disk Queue Length counter is a calculated value that measures the average number of both read and write requests that are queued or “in-flight” for a selected disk during a sample interval. Traditionally, performance counters that relate to the disk queue, such as Avg. Disk Queue Length, % Idle Time, and % Disk Time, have been staples in the IT professional’s tool belt. The counters have great value when analyzing single spindle disks, but they are less effective when more spindles are added to a LUN or when spindles are shared between LUNs. For example, if Avg. Disk Queue Length is 2 and there are 10 spindles behind the LUN, the LUN should have no problem with handling the load (assuming an equal distribution of those I/O requests to physical spindles). This is like having 10 checkout lines with an average of only two people wanting to check out. Even if the number of physical spindles is known, it is still difficult to apply a threshold to the Avg. Disk Queue Length because of disk virtualization, hardware cache, and many other variables. Many system administrators use a warning threshold of 2 + number of spindles, but the response times still must be considered. Similarly, if you are in line to buy a cup of coffee and there are 40 people in front of you, you might assume that the length of the line (the queue) predicts your wait time for service. But what if you receive your cup of coffee within a few seconds? Does it matter that there were 40 people in the line? The point is that knowing the length of the disk queue is helpful as supplemental information, but with disk virtualization you cannot afford to make assumptions. This is why disk response times are the best indicators of disk performance. Disk queue Driver Hardware controller Bus|Network|Cable Physical media Microsoft Confidential

Example of Avg. Disk Queue Length
In this example, the average length of the disk queue of drive F is increasing Might be because of an increased load or an overwhelmed disk Queue length of drive F (orange line) is increasing over time This slide shows an example where the disk queue is increasing over a long period of time. This increase could be for many reasons, such as increased load or the disk’s inability to service the demand fast enough to empty the queue for any significant length of time. Microsoft Confidential

Disk throughput counter
Disk Transfers/sec is the rate of read and write operations on the disk Use this counter to establish a performance baseline Shows the number of serviced IRPs per second Part of the LogicalDisk and PhysicalDisk counter objects How many per second? Disk queue Driver Hardware controller Disk Transfers/sec is the rate of read and write operations on the disk. This is from the perspective of the operating system, and it might correlate with more I/Os on the hardware. This counter can help determine the amount of I/O throughput on the disk. Disk hardware and performance can differ greatly, so there is no generic threshold for this counter. With that said, for a given disk, try to establish the baseline performance, and then compare that baseline if you suspect poor performance. For example, if you know the disk can sustain 1000 disk transfers per second and stay within 10 ms response time, but you later find a response time of 50 ms with only 200 disk transfers, this might indicate a problem with the disk hardware, such as can be the case with physical spindles shared between multiple servers. Therefore, if you suspect poor disk performance, you can compare the performance to the baseline. Use this counter to establish a performance baseline that you can compare to if you suspect poor performance. Disk transfers Bus|Network|Cable Physical media Microsoft Confidential

Disk Transfers/sec and I/Os
Disk Transfers/sec is not the same as I/O operations per second (IOPS) A read or write operation might result in more than one I/O on the hardware Conversions of read and write operations to RAID I/O: RAID 0: (reads) + (writes) RAID 1: (reads) + (2 x writes) RAID 0+1: (reads) + (2 x writes) RAID 5: (reads) + (4 x writes) RAID 10: (reads) + (2 x writes) This one write operation results in two I/Os Windows write operation Disk queue RAID 1 (mirror) Disk transfers/sec measures the read and write operations of the operating system, but it is not the same as the actual I/O operations per second (IOPS) on the disk hardware. For example, a single write operation on a hardware RAID 1 mirror set might result in 2 or more I/Os. Regardless of the number of physical spindles in a RAID 5 set, it is generally about 4 I/Os (2 reads, 2 writes), depending on the implementation, to complete a write operation: one I/O to read the data bit, one I/O to read the parity bit, modify data, one I/O to write the data bit, and one I/O to write the parity bit. In summary, the hardware implementation impacts the true number of I/Os necessary to complete a read or write operation. Therefore, consult your hardware vendor to understand the ratio for your implementation. The conversion table in this slide is based on standard implementations of RAID. Microsoft Confidential

Disk read/write ratio counters
The ratio of read and write operations can help you in adjusting the disk cache Compare the Reads/sec and Writes/sec counters to determine the read/write ratio Part of the LogicalDisk and PhysicalDisk counter objects Disk cache is generally significantly faster media in front of slower media storage Adjusting the disk cache can greatly affect performance 50% writes reads Disk queue R R W W R W Hardware disk cache The Reads/sec and Writes/sec performance counters help you determine the read:write ratio of operations to the disk. Most hardware RAID solutions have an adjustable high speed cache that can significantly improve performance by servicing the I/O requests directly from the cache. Disk cache performance might improve significantly if the percentage of cache can be aligned with the actual read:write ratio. Physical media Microsoft Confidential

Larger I/Os might take longer Microsoft Confidential
I/O size counters Larger I/Os might take longer The size of the read and write operations (IRPs) can affect the response times Use Avg. Disk Bytes/Read and Avg. Disk Bytes/Write counters to determine the I/O sizes The threshold of a 15 ms response time is based on I/O sizes of 64 KB or smaller Part of LogicalDisk and PhysicalDisk counter objects Disk queue 30 ms 10 ms 1 MB 4 ms 64K 4K The size of the I/Os can significantly affect the response times of the I/O requests. A 1 MB I/O request is 256 times larger than a 4 KB I/O request, so it needs more time to be serviced. For this reason, if you observe disk response times greater than 15 ms, check the average I/O size by using the Avg. Disk Bytes/Read and Avg. Disk Bytes/Write counters. The threshold of 15 ms disk response time is based on I/O sizes of 64 KB or smaller, so average I/O sizes larger than 64 KB might take longer than 15 ms and still be considered a good response time. If you regularly deal with average I/O sizes larger than 64 KB, consider establishing a disk performance baseline while performance is good. Use the baseline to compare to suspected poor performance. Also, consider looking at the actual data throughput of the disk by using the Disk Bytes/sec performance counters. A throughput of 10 MB of data is generally expected from a hardware RAID 1 mirror set of 7200 RPM disk drives. Disk transfers Physical media Microsoft Confidential

Troubleshooting flow To determine if you have a disk issue, use the following set of questions: Do I have disk latency? (Is Average Disk sec / Read or Average Disk sec / Write greater than 0.025?) If there is no latency there is no need to go further. Are all or most of the disks showing latency? If only one disk has latency then there might be a bottleneck in the JBOD. If all disks show latency then it might indicate a bottleneck in the fabric These questions help identify if we have a disk bottleneck and where the bottleneck is. Microsoft Confidential

Troubleshooting flow continued…
Is there substantial queuing? (Is Average Disk Queue Length greater than 32?) Is my HBA unable to keep up? Queuing my indicate a fabric bottleneck Compare the disk activity during the time of latency with the latency during the time of maximum disk activity If the Disk Transfers / sec are much higher at another time when latency is very low it would point to a fabric bottleneck. Microsoft Confidential

Troubleshooting flow example
Apply the list of questions from the previous slide to the following information: Users complain and enter trouble tickets for the helpdesk. Disk latency peaks at 1100 MS. Disk transfers / sec equal 819 Disk bytes / sec equal 9 MB Avg disk queue length equals 504 Challenge questions: At this point, Do you suspect this is a JBOD or fabric issue? How many spindles do I need to support this load? Challenge question 1 – answer: At this point the issue is starting to look like it’s in the communication path (fabric); however, we still need more information. We assume this because the queue length is very high. Remember, most HBA settings are defaulted to 32. These numbers represent a large I/O bottleneck. Challenge question 2 – answer: If these are 15k spindles, you would need at least 5. Maybe we have latency because we have less than 5? What if this was only 10k spindles? It’s important to understand why we need a certain amount of disks and what we are doing at this time. Discuss this with your instructor. Microsoft Confidential

Troubleshooting flow example continued…
At 11:50 AM there is latency of 1109 ms and there is very little activity. Microsoft Confidential

Troubleshooting flow example
Now compare the activity during the time of latency with other times to help confirm your theory. 10 minutes later we have the following data: Disk latency is only 29 MS. *this is still high but not as unacceptable Disk transfers / sec equal 2300 Disk bytes / sec equal 54 MB This would indicate that the bottleneck is in the fabric because the disk is capable of doing more at this time. It can support up to 12 disks worth of I/O. Earlier it couldn’t do 5 disks worth. *Note: This does not account for shared spindles on a SAN but we cannot tell that from perfmon. It’s up to the SAN Admin to tell us if this issue was fabric or shared spindle. Microsoft Confidential

Troubleshooting flow example continued…
10 minutes later we can see that the disk is able to do 2300 transfers and 54 MB of disk bytes with mild queuing and latency. This helps prove out our case that this is a fabric bottleneck. Earlier one fifth of the activity caused huge latency. We could assume that the disk could not communicate with the server. Again please remember that this does not account for a shared spindle issue. The SAN administrator will need to identify the absolute root cause. Microsoft Confidential

Process I/O counters Process I/O operations Avg. Disk Sec/Read|Write Thresholds for Avg. Disk sec/Read|Write Example of Avg. Disk sec/Write Microsoft Confidential

Process I/O operations
\Process(*)\IO Read Operations/sec and \Process(*)\IO Write Operations/sec measure the rate at which the process is issuing read or write (respectively) I/O operations This includes file, network, and device I/Os Use as supplemental data of which process might be issuing disk I/O Sqlservr.exe \Process(sqlservr)\IO Write Operations/sec Disk queue The \Process(*)\IO Read Operations/sec and \Process(*)\IO Write Operations/sec counters measure the rate at which a process is issuing read or write (respectively) I/O operations. This I/O can be file (disk), network, or device I/O. This counter does not differentiate the I/O resource type, so it is not a direct indicator of how a process is using disk I/O. These counters are best used to supplement data from other disk related performance counters. Physical media Microsoft Confidential

Disk free space counters
% Free Space measures the percentage of total usable space on a disk drive Free Megabytes displays the amount of unallocated space on the disk If the free disk space is low, defragmentation failures or failed writes can result These counters are only part of the LogicalDisk counter object and do not measure free space for the LUN. If a disk is close to full, the Seek Time will be greater because the read/write heads generally travel more. Again, this counter does not measure the actual free space on each disk. It measures by partition. Data stored on the inner portion of a platter is generally slower to access. This only applies to a single spindle (workstations or very small servers) When the capacity of a disk is full, it can cause many functional issues with the operating system and disk performance might become slower as the disk fills up. Use the % Free Space and Free Megabytes performance counters to determine whether the disk is nearing capacity. These counters are part of the LogicalDisk counter object only. They are not part of the PhysicalDisk counter object. This is another reason to consider using LogicalDisk counter objects first. Short stroking is a technique that was used prior to the wide spread availability of solid state drives. The technique allowed the administrator to configure the disk to only present the outer, faster portion of the disk. Disk free space may be usable to identify performance issues on a single disk (not virtualized storage). Microsoft Confidential

Thresholds for % Free Space
Effect 10% or more Informational Normal usage Less than 10% Warning Disk latency might occur Less than 5% Critical Disk latency is very likely. The system might not be able to write to the disk Common next steps: Delete or move any unneeded files Increase disk capacity if you can Since disk performance and functionality of a disk might degrade as it fills up monitoring disk free space is important. Therefore, as an initial indicator, monitor whether % Free Space is less than 10. If it is, next look at the Free Megabytes counter to see exactly how much space is free. If the disk is nearly full, consider deleting unneeded files. Microsoft Confidential

How often is the queue empty? Microsoft Confidential
% Idle Time counter The % Idle Time counter reports the percentage of time that the disk queue is empty Part of LogicalDisk and PhysicalDisk counter objects A sustained average of less than 10 might warrant investigation, but no threshold can be assigned because of disk virtualization and I/O distribution How often is the queue empty? Disk queue Driver Hardware controller The % Idle Time performance counter is a popular counter, which is used in many technical articles. Despite its popularity, be careful about relying on it too much. Like the Avg. Disk Queue Length counter, % Idle Time is based on the number of I/O requests that are waiting for service from the disk driver. It measures the percentage of time that the disk queue has nothing in it. This means that, if the average disk queue length is 1 or more, % Idle Time is zero. Remember, if Avg. Disk Queue Length is 2 and there are 10 spindles behind the LUN, the LUN should have no problem with handling the load (assuming an equal distribution of those I/O requests to physical spindles). Therefore, you should not use % Idle Time as an initial indicator when you are dealing with hardware RAID, SANs, or any kind of disk virtualization. With that said, % Idle Time is good for telling you how often the disk queue is not busy which can still serve as a supplemental data in disk performance analysis. Bus|Network|Cable Physical media Microsoft Confidential

Thresholds for % Idle Time
% Idle Time is a good indicator of how idle the disk queue is. This counter could be used an initial indicator that warrants more investigation. Correlate this counter with other disk related counters. Threshold Effect 10% or more Informational Normal usage Less than 10% Warning Disk latency might occur Common next steps: Correlate with disk response time counters such as Avg. Disk sec/Read and Avg. Disk sec/Write Look for sustained averages of % Idle Time of less than 10 which indicates that 90% of the time the disk queue has work to do. This counter could be used an initial indicator that warrants more investigation, but by itself, does not indicate a disk performance problem unless there is only one physical disk with no virtualization. Therefore, correlate this performance counter with other disk related performance counters such as Avg. Disk sec/Read and Avg. Disk sec/Write which are discussed later. Microsoft Confidential

Example of % Idle Time In this example, drive E goes down to 0% idle time. This indicates that there were one or more IRPs constantly in the disk queue % Idle Time of drive E (orange line) shows constant I/O in the disk queue In this example, the % Idle Time counter drops to 0 for a sustained amount of time. This warrants more investigation into the disk performance, but it does not by itself indicate a performance problem. Microsoft Confidential

Section 3 review True or false: IRPs in the disk queue are always served first in first out (FIFO). What is the general warning threshold for disk response times? The response time thresholds are based on what I/O size? How many I/O operations per second (IOPS) occur on a RAID 5 disk array with 100 write operations per second? What kind of I/O do the \Process(*)\IO Read Operations/sec and \Process(*)\IO Write Operations/sec counters measure? Microsoft Confidential

False: IRPs in the disk queue can be reorganized by the disk driver for speed and efficiency. The general warning threshold for disk response times is 15 ms. The response time thresholds are based on an I/O size of 64 KB or smaller. 400 I/O operations per second (IOPS) occur on a RAID 5 disk array with 100 write operations per second. RAID 5: (read) + (4 x write). The \Process(*)\IO Read Operations/sec and \Process(*)\IO Write Operations/sec counters measure disk, network, and device I/O. Microsoft Confidential

Section 4: Disk analysis tools
Resource Monitor Process Monitor Microsoft xperf Microsoft Confidential

Launch Resource Monitor Microsoft Confidential
Provides live disk usage information Resource Monitor is built in to Windows Vista, Windows Server 2008, and later More detailed information in Windows Server 2008 R2 and Windows 7 Launch Resource Monitor Resource Monitor was first introduced in Windows Vista and Windows Server 2008, but the individual disk response times for processes and files on the disk was first introduced in Windows 7 and Windows Server 2008 R2. This level of detail is not available in Perfmon counters. Resource Monitor uses Event Tracing for Windows (ETW) as the source of the data—no performance counter data is used. Resource Monitor is best used to analyze disk performance issues that are occurring live. It can only be used live—it cannot capture the data for later analysis. Other tools, such as Microsoft xperf, can capture ETW data in an Event Tracing for Windows log (.etl) file for later analysis. You can launch Resource Monitor from the Performance tab of Task Manager. Microsoft Confidential

Process Monitor Process Monitor (Microsoft SysInternals) can show details of disk I/O, such as process and files, number of I/Os, and aggregate call stacks of the I/Os Can show whether drivers, such as anti-virus, are slowing I/O performance Process Monitor is another great tool for disk analysis. It is part of the suite of SysInternals tools. SysInternals is owned by Microsoft. Process Monitor captures all access to the registry, file system, network, process and thread activity, and profiling events. In the context of this module, it captures detailed information about disk I/O. Process Monitor can aggregate and sort the data by process, file, or other filter criteria. It can also show the performance of disk filter drivers by showing aggregated call stacks of the I/O. This helps identify poor performance from anti-virus and anti-intrusion software and from any other drivers. Process Monitor uses Event Tracing for Windows (ETW) and its own driver as its data sources. You can download Process Monitor directly from the SysInternals website, You do not need to install it, and it can run on nearly all versions of Windows and Windows Server. Microsoft Confidential

Microsoft xperf Microsoft xperf can show disk I/O details Part of the Windows Server Performance Tool Kit For more information, go to the “Windows Performance Analysis Developer Center” at us/performance/default.aspx Microsoft xperf is another great tool that can show details of disk I/O. It can show specifically which files and processes are the most active on a given disk, and it can provide details of the disk offsets that are used to read and write to the disk. The chart on the left of the slide shows where a disk head was reading and writing, with details of the process and file involved in each I/O. Xperf gathers a lot of data quickly, so use it only if you suspect a disk performance problem or problems with other resources (discussed in other modules). Xperf uses Event Tracing for Windows (ETW) as its data source, it is part of the Windows Server 2008 Performance Toolkit, and it is free to use. Microsoft Confidential

Section 4 review Which disk analysis tool is built in to Windows 7 and Windows Server 2008 R2? Which disk analysis tool visually shows detailed I/O patterns? Which disk analysis tool can be run directly from Microsoft Confidential

Resource Monitor is built in to Windows 7 and Windows Server 2008 R2. Microsoft xperf visually shows detailed I/O patterns. Process Monitor can be run directly from Microsoft Confidential

Lab 2a: The tale of two SQLs
Scenario Between 10:40 A.M. and 10:50 A.M., users submitted trouble tickets regarding the performance of the database server. Lab files \Labs\Lab2a_TheTaleofTwoSQLs\Lab2a_SQL01.blg \Labs\Lab2a_TheTaleofTwoSQLs\Lab2a_SQL09.blg Purpose This lab focuses on analyzing disk performance. Configuration Two, 2-node clusters running 64-bit version of the Windows Server® 2003 operating system Microsoft® SQL Server Symptoms Poor response times: Between 10:40 A.M. and 10:50 A.M., users submitted trouble tickets regarding the poor response times of the database server SQL01. Goals Use the LogicalDisk and PhysicalDisk counter objects. Investigate the disk performance of the database servers. Use the Process counters to identify which processes are consuming the most I/O Microsoft Confidential

Notes Continue Problem What is the root cause of the poor performance on SQL01 during the stated time? The following are hints to help you resolve this lab: \LogicalDisk(*)\Avg. Disk sec/Read: Use this counter to determine the average read response times of each logical disk. Values greater than (25 ms) are considered too long. \LogicalDisk(*)\Avg. Disk sec/Write: Use this counter to determine the average write response times of each logical disk. Values greater than (25 ms) are considered too long. \LogicalDisk(*)\Disk Transfers/sec: Use this counter to determine the number of read and write operations on each logical disk. If the I/O sizes are 64 KB or less, a 7200 RPM disk should be able to do more than 100 disk transfers per second while maintaining disk latency at under 10 ms. \Process(*)\IO Data Operations/sec: Use this counter to determine which processes are involved in I/O data operations. Keep in mind that this includes all I/O, including disk and network. \Process(*)\IO Other Operations/sec: Use this counter to determine which processes are involved in other I/O operations. Keep in mind that this includes all I/O, including disk and network. Microsoft Confidential

Questions? Microsoft Confidential

References Windows Internals, 5th edition, by Mark Russinovich and David A. Solomon Taking Your Server’s Pulse Windows Sysinternals Administator’s Reference “Key OS Performance Metrics_v7.xps” In the Supporting Documents folder with your classroom material. Microsoft Confidential

Vital Signs: Performance Monitoring Windows Server Module 2: Disk

Similar presentations

Presentation on theme: "Vital Signs: Performance Monitoring Windows Server Module 2: Disk"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Vital Signs: Performance Monitoring Windows Server Module 2: Disk

Similar presentations

Presentation on theme: "Vital Signs: Performance Monitoring Windows Server Module 2: Disk"— Presentation transcript:

Similar presentations

About project

Feedback