1. VSP3868 VMware vStorage Best Practices Name, Title, Company

2. 2 Disclaimer  This session may contain product features that are currently under development.  This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product.  Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.  Technical feasibility and market demand will affect final delivery.  Pricing and packaging for any new technologies or features discussed or presented have not been determined.

3. 3 Agenda VMware Storage Best Practices Storage Environment Performance Improvements and Tuning Performance Monitoring Common Storage Issues

4. 4 VMware Storage Best Practices Overview Storage Environment Business Continuity and Disaster Recovery Vendor Knowledge and Partnership Hard Disk Performance

5. 5 Business Continuity and Disaster Recovery  Backups Number and depth based on application RPO/RTO Test your backups  Support considerations Virtual Machine Snapshots are not backups! Available backups allow for a wider variety of options in data loss scenarios

6. 6 Vendor Knowledge and Partnership  Know your SAN environment Open communication channel to your internal storage team Follow storage vendor best practice guidelines for ESX(i)  Know your storage vendor Support relationship – proactive versus reactive Software/Firmware updates for storage hardware  Know your storage network or fabric HBA and switch software/firmware updates

7. 7 Hard Disk Performance  Areas of Concern Disk subsystem bottlenecks Spikes and sustained latency  Performance versus Capacity Disk performance does not scale with drive size Larger drives generally equate lower performance  For example: 1 TB of space is required for an app 2 x 500GB 15K RPM SAS drives = ~300 IOPS Capacity needs satisfied, Performance low 8 x 146GB 15K RPM SAS drives = ~1,240 IOPS Capacity needs satisfied, Performance high  More spindles generally equals greater performance - check with your SAN array vendor for technology details

8. 8 VMware Storage Best Practices Overview Storage Environment Performance Improvements and Tuning vStorage APIs for Array Integration (VAAI) Multipathing Storage I/O Control (SIOC)

9. 9 vStorage APIs for Array Integration (VAAI)  VAAI Enables your host to offload specific virtual machine and storage management operations to compliant storage hardware.  Performance Benefits ESX(i) hosts consume less CPU and memory resources, and less storage fabric and network bandwidth. Improved Migration of virtual machines with Storage vMotion Faster deployment and cloning of virtual machines Reduces SCSI Reservation conflicts Faster zeroing operations (creation of Fault Tolerant VMs)  Benefits derived through new SCSI primitives  Full Copy  Block Zeroing  Atomic Test and Set (ATS) Locking

10. 10 vStorage APIs for Array Integration (VAAI) – Requirements  ESX(i) version 4.1 or later.  VAAI compatible storage solution  VAAI hardware acceleration is enabled by default on ESX(i)  ESX(i) 4.1 – block devices only (NAS not yet supported)  ESXi 5.0 – NFS and block device support available for vendor implementation  Consult storage vendor for array configuration requirements

11. 11 vStorage APIs for Array Integration (VAAI) - Summary  VAAI Is configured by default, can be disabled through host Advanced Settings, if required by storage vendor Storage array must support VAAI SCSI primitives Tested VAAI functionality can be reviewed via the vSphere GUI

12. 12 Multipathing - Overview  Pluggable Storage Architecture - PSA  Native Multipathing Plugin – NMP Storage Array Type Plugin – SATP Path Selection Policy – PSP  Fixed Path  Most Recently Used  Round Robin

13. 13 Multipathing – Path Selection Plugins  Most Recently Used – MRU Active/Passive arrays  Fixed Active/Active arrays Asymmetrical LUN Unit Access – ALUA  Round Robin Load balance I/O for improved performance Uses Active or Optimized paths, not Standby paths  Vendor/Partner Multipathing Plugin solutions

14. 14 Multipathing – Best Practices  Consistency One PSP per LUN across all ESX(i) hosts  Configuration Match to storage array type (AP, AA, AA-ALUA) All LUNs on AP arrays to all ESX(i) hosts through same storage processor  Multipathing – iSCSI One vmkernel port group per physical NIC Follow VMware and/or vendor best practice specifics VMware iSCSI Configuration Guide (pages 29-51) https://www.vmware.com/pdf/vsphere4/r41/vsp_41_iscsi_san_cfg.pdf

15. 15 Storage I/O Control - Overview  Storage I/O Control Monitors I/O latency to datastores at each ESX host sharing a physical device. When the average normalized latency exceeds a set threshold (30ms by default), the datastore is considered to be congested.  If congested, SIOC distributes available storage resources to virtual machines in proportion to their configured shares.  Used to determine migration needs with Storage DRS in ESXi 5.0

16. 16 Storage I/O Control – Usage Scenario  Without SIOC, Disk shares do not provide a fair distributed load on a shared VMFS.

17. 17 VMware Storage Best Practices Overview Storage Environment Performance Improvements and Tuning Performance Monitoring vCenter/vSphere Client esxtop/resxtop Latency Values Partner Vendor Tools

18. 18 Performance Monitoring – vCenter/vSphere Client  Performance Charts Performance  Advanced  Chart Options  Datastore/Disk Latency and IOPS statistic counters for disks Historical availability depends on vCenter Statistics level (vCenter Advanced Settings) Statistics available for ESX(i) host objects Datastore Read/Write Latency = device average in milliseconds Disk Physical Device Command/Write/Read Latency = device average in milliseconds  Additional detail on vCenter Performance Metrics vSphere Datacenter Administration Guide, Appendix B, page 265: http://www.vmware.com/pdf/vsphere4/r41/vsp_41_dc_admin_guide.pdf

19. 19 Performance Monitoring – vCenter/vSphere Client

20. 20 Performance Monitoring – esxtop/resxtop  esxtop/resxtop local/remote More depth than vSphere Client  Storage statistics CMDS/s – I/Ops DAVG, KAVG, GAVG – device/kernel/guest latency in milliseconds NFS latency, vSphere 4.1 or later Statistics for disk device, disk adapter, virtual machine  Using esxtop to identify storage performance issues http://kb.vmware.com/kb/1008205  Additional detail on esxtop/resxtop vSphere Resource Management Guide, Appendix A, page 89 http://www.vmware.com/pdf/vsphere4/r41/vsp_41_resource_mgmt.pdf

21. 21 Performance Monitoring – esxtop/resxtop

22. 22 Performance Monitoring – Latency Values  Good latency values are subjective based on application performance needs and storage environment capabilities  Lower is better!  General rules of thumb: 10 milliseconds or less is adequate 20 milliseconds or higher, sustained, should be investigated Sustained spikes in latency may need to be investigated  Commands not acknowledged by the SAN within 5000 ms will be aborted This may lead to performance issues as the commands are retried Multiple aborts should be investigated

23. 23 Performance Monitoring – Partner Vendor Tools  Partner Tools vSphere APIs  vCenter Plugins Consult with storage vendor

24. 24 VMware Storage Best Practices Storage Storage Environment Performance Improvements and Tuning Performance Monitoring Common Storage Issues Snapshots Misconfiguration Improper Device Removal

25. 25 Common Storage Issues – VM Snapshots  Best Practices Snapshots are not backups Regular monitoring Configure vCenter Snapshot alarms – KB 1018029KB 1018029 Limit to 2-3 snapshots in a chain to prevent performance degradation Delete all snapshots before Virtual Machine disk changes Confirm via command line if uncertainty of snapshot state exists As a general rule, use Snapshots for no more than 24-72 hours  Improvements ESX(i) 4.0 U2 – Snapshot deletion takes up less space on disk ESXi 5.0 – New functionality to monitor snapshots and provide warning if snapshots need consolidation.  Snapshot Best Practices – KB 1025279KB 1025279  Understanding Virtual Machine Snapshots in ESX – KB 1015180KB 1015180

26. 26 Common Storage Issues - Misconfiguration  Firmware and Driver issues SAN array firmware does not match to ESX(i) version HBA, FC switch, physical NIC firmware and/or drivers out of date VMware Hardware Compatibility Guide http://www.vmware.com/resources/guides.html http://www.vmware.com/resources/guides.html  Inconsistent Path Selection Policy (example 1) Host A, LUN A – MRU; Host B, LUN A – Fixed (example 2) Host A, LUN A – vendor MMP; Host B, LUN A - Fixed  LUN detected as a snapshot Missing datastores resolved via force-mount Resignature for long-term solution Consistent LUN presentation to all hosts from SAN

27. 27 Common Storage Issues – Improper Device Removal  Improper removal of a physical device containing a VMFS datastore or RDM APD (All Paths Dead) state in ESX(i) 4.x Upgrade to 4.0 U2 / 4.1 U1 or more recent Follow steps outlined in KB 1015084 when removing a datastoreKB 1015084  Rescan after making changes to the storage environment! VMFS RDM

28. 28 VMware Storage Best Practices Summary Storage Environment Performance Improvements and Tuning Performance Monitoring Common Storage Issues

29. 29 Appendix - VMware Storage Best Practices  ESX FC SAN configuration guide https://www.vmware.com/pdf/vsphere4/r41/vsp_41_san_cfg.pdf  ESX iSCSI Configuration Guide https://www.vmware.com/pdf/vsphere4/r41/vsp_41_iscsi_san_cfg.pdf  MSCS Configuration Guide https://www.vmware.com/pdf/vsphere4/r41/vsp_41_mscs.pdf  VMware Performance Best Practices 4.1 http://www.vmware.com/pdf/Perf_Best_Practices_vSphere4.1.pdf

30. COMING DECEMBER 2011 By Mike Laverick ISBN: 9780321799920 Complete guide to using SRM 5.0 with both VMware and third-party products. Available in print and eBook formats and online through Safari Books Online. VMWORLD 2011 EXCLUSIVE Buy the Pre-Publication Edition, Receive the eBook edition for FREE – a $39.99 Value. Visit VMworld Bookstore for details. VMWORLD 2011 EXCLUSIVE Buy the Pre-Publication Edition, Receive the eBook edition for FREE – a $39.99 Value. Visit VMworld Bookstore for details.

31. COMING DECEMBER 2011 By Cody Bunch ISBN: 9780321799913 Complete with real-world case studies and code. Available in print and eBook formats and online through Safari Books Online. VMWORLD 2011 EXCLUSIVE Get a FREE preview chapter from this and two other VMware Press releases. PLUS: PRE-ORDER and Save 30% Visit VMworld Bookstore for details VMWORLD 2011 EXCLUSIVE Get a FREE preview chapter from this and two other VMware Press releases. PLUS: PRE-ORDER and Save 30% Visit VMworld Bookstore for details

32. COMING FEBRUARY 2012 By Mostafa Khalil ISBN: 9780321799937 Demystifies the "black box" of vSphere storage. Available in print and eBook formats and online through Safari Books Online. VMWORLD 2011 EXCLUSIVE Get a FREE preview chapter from this and two other VMware Press releases. PLUS: PRE-ORDER and Save 30% Visit VMworld Bookstore for details VMWORLD 2011 EXCLUSIVE Get a FREE preview chapter from this and two other VMware Press releases. PLUS: PRE-ORDER and Save 30% Visit VMworld Bookstore for details