1. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems Chuck Laing- Senior Technical Staff Member 23 October 2013

2. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 2 http://ibmtechu.com/ent pSG539

3. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 3 3 The Top Ten Things SA’s should know about Storage  Storage Overview - what's inside? 1.What is the Physical makeup? 2.What is the Virtual makeup (good throughput design tips) 3.What is a Storage Pool - where do I place data? 4.What should I be aware of/what should I avoid? (Tips & Pitfalls-Tuning) To Stripe or not Stripe, that is the question!  5. Connectivity- Picking the right drivers –Host Attachment Kits  6.Documentation - why it matters –7. Topology Diagrams –8. Disk Mapping (view at a glance) –9. Easy Storage Inquiry Tools  10. How to Improve Performance –Bottlenecks

4. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 4 4 Knowledge is POWER! As System’s Administrators – we don’t always KNOW what we don’t know about storage  Ask for storage leveraging what you know  Avoid bottlenecks  Use tools available  Speed problem isolation  Make more informed architectural decisions What we are NOT going to do today:  Try to turn you into storage administrators  Boil the ocean

5. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 5 The Big Picture Other!

6. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 6 What is SVC? 6 Storage Subsystems SAN HDS DS8000 Volume SAN Volume Controller DS4000 HP EMC Combine the capacity from multiple arrays on frames into storage pools Advanced Copy Services Apply copy services across the storage pool Manage the storage pool from a central point Make changes to the storage without disrupting host applications Volume SATA 15K rpm RAID 5 JBODRAID 1  SVC provides flexibility across the entire storage infrastructure ! NO… Saying SAN Volume Controller doesn't count! NO… Saying SAN Volume Controller doesn't count!

7. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 7 7 Combine the capacity from multiple arrays into a single pool of storage Apply common copy services across the storage pool Manage the storage pool from a central point Make changes to the storage without disrupting host applications Virtual Disk Virtual Disk Virtual Disk Virtual Disk SAN SAN Volume Controller / V7000 Advanced Copy Services Storage Pool HP EMC DS3000 DS8000 HDS Flexibility to auto or manually migrate full volumes to meet needs Automated use of SSDs through Sub- LUN tiering with Easy Tier IBM builds virtualization into the storage infrastructure Store more with what’s on the floor What is V7000?

8. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 8 8 DS8700 DB2 Brokerage Workload w/Easy Tier Breakthrough Performance with Smart Management of Data IBM Easy Tier provides two basic features: –Easy Tier Automatic Mode Automatically migrates sub-volume data to the right tier in mixed SSD/HDD extent pools (so called hybrid pools) by moving hot extents to SSD and cold extents to HDD. –Easy Tier Manual Mode Manually merge extent pools or migrate full volumes non- disruptively between extent pools to easily optimize data placement across pools and different storage tiers as well as the restriping of volumes within an extent pool Dynamic Volume Relocation Dynamic Extent Pool Merge IBM Storage Tier Advisor Tool (STAT) provides guidance for SSD capacity planning based on existing workloads. By showing the hot extents on each volume it allows to easily evaluate the current SSD usage and to identify data which would benefit from SSD. Easy Tier operates on DS8000, SVC, Storwize V7000 Increase of 241% in transactions! No Migration 1 st hr Avg: 9.13ms 1 st hr SD: 1.89ms Migration Active 2 nd +3 rd hr Avg: 7.61ms 2 nd +3 rd hr SD: 1.31ms What does “Easy Tier” allow you to do?

9. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 9 9 Virtual Server Infrastructure Virtual Storage Infrastructure Tivoli Storage Productivity Center Storage Hypervisor Midrange Storwize V7000 Enterprise SAN Volume Controller Manage Virtual Storage Infrastructure Store more with what’s on the floor

10. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 10 “If we need to add a new application, we can provision the storage very easily without worrying about the technical side of things. That’s really the ‘wow factor’…” Uli Müller IT Director EURONICS Deutschland eG IBM XIV Storage System IBM XIV builds virtualization into the storage system Store more with what’s on the floor Functionality –Thin Provisioning, snapshot, replication, migration, reporting –Built-in Data Migration Facility –All inclusive Management –Ease of management, GUI, single tier, single configuration Reliability –No downtime, scalability, 60 minute redundancy, self healing Performance –Consistent performance, no tuning, no hot spots, performance increases with capacity Cost –Low acquisition costs, low management costs, low operating costs

11. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 11 DS8800 Benefits Up to 40% better performance Save floor space with nearly twice the drive density Reduce costs with up to 36% less energy consumed Management Console High Density Enclosures, 2.5 in. Drives Front view (cover removed) Batteries POWER 6+ Controllers I/O drawers Primary Power Supplies Host over 1,000 drives on up to 40% less floor space than IBM DS8700 with 3.5 in. disks IBM System Storage DS8800 DS8800 Enhancements New 2.5” small form factor drives 50% more drives per enclosure IBM POWER6+ processors 8Gb/sec host and device adapters Store more with what’s on the floor

12. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 12 A deeper dive Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems

13. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 13 DS8000 Hardware Physical Makeup Summary - DS8700 (2-way/4 way base frame) 242x model 941 Familiar Layout 99,999% = ¼ day in 72 years MTBF Is it important to know the physical makeup?; Does it really matter?

14. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 14 Physical to Logical - peeling back the layers Arrays across Enclosures A B Even numbered extpools Primary IO Data flow ownership Balance Odd numbered extpools Primary IO Data flow ownership Just like an onion -- virtualization has many layers

15. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems How does the DS8000 virtual layer work? Raid-0 only 15

16. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 16 DS8000 Virtualization layers (Concepts & Architecture) How logical extents in ranks are formed from the DS8000, 6+P+S type array format EXT 1 EXT 3 EXT 4 EXT 5 EXT 2 EXT 3 EXT 4 EXT 5 EXT 1 1GB EXT 2 1GB EXT 3 1GB EXT 4 1GB EXT 5 1GB

17. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 17 Volume Storage Pool Stripe 16MB – 2GB Managed Disk LUN mdisk0 100GB mdisk1 100GB mdisk2 100GB mdisk3 100GB mdisk6 200GB mdisk5 200GB mdisk4 200GB EMC 100GB EMC 100GB EMC 100GB EMC 100GB IBM 200GB IBM 200GB IBM 200GB mdiskgrp0 [EMC Group] 400GB mdiskgrp1 [IBM Group] 600GB vdisk0 125GB vdisk2 525GB vdisk3 1500GB vdisk4 275GB vdisk5 5GB Mapping to Hosts w/SDD or supported MultiPath Driver SVC - From Physical to Logical View vdisk1 10GB Space-efficient

18. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 18 Preferred path for vdisk1 is SVC N1P2 & N1P3 Non Preferred path for vdisk1 is SVC N2P2 &N2P3 Preferred path for vdisk2 is SVC N2P2 & N2P3 Non Preferred path for vdisk2 is SVC N1P2 &N1P3 Examples of correct Host to SVC Cluster zoning

19. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 19  Virtualized grid SAN storage: Data distribution across all drives No RAID groups to manage Raid Groups Disk Tuning Complex Mgmt. IBM XIV Grid Storage Standard Features Automatic load balancing Consistent performance Integrated software: Thin Provisioning High Performance, flexible SNAPshots Remote Replication Consistency Groups Easy to use GUI based management Data migration What is XIV? XIV a Massively Parallel – Grid Architecture

20. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 20 IBM XIV Storage Distribution Algorithm  Each volume is spread across all drives  Data is “cut” into 1MB “partitions” and stored on the disks  XIV algorithm automatically distributes partitions across all disks in the system pseudo-randomly Data Module Interface Data Module Interface Data Module Interface Switching XIV disks behave like connected vessels, as the distribution algorithm aims for constant disk equilibrium. Thus, IBM XIV’s Storage overall disk usage could approach 100% utilization when loaded

21. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 21  Data distribution only changes when the system changes –Equilibrium is kept when new hardware is added –Equilibrium is kept when old hardware is removed –Equilibrium is kept after a hardware failure Node 2 Node 3 Node 1 Data Module 2Data Module 1Data Module 3 XIV Distribution Algorithm on System Changes

22. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 22 Node 4  Data distribution only changes when the system changes –Equilibrium is kept when new hardware is added –Equilibrium is kept when old hardware is removed –Equilibrium is kept after a hardware failure Data Module 2 Data Module 3 Data Module 1 [ hardware upgrade ] Data Module 4 XIV Distribution Algorithm on System Changes

23. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 23  Data distribution only changes when the system changes –Equilibrium is kept when new hardware is added –Equilibrium is kept when old hardware is removed –Equilibrium is kept after a hardware failure Data Module 2 Data Module 3Data Module 4 Data Module 1 [ hardware failure ] The fact that distribution is full and automatic ensures that all spindles join the effort of data re-distribution after configuration change. Tremendous performance gains are seen in recovery/optimization times thanks to this fact. XIV Distribution Algorithm on System Changes

24. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 24 What is a V7000? 99,999% = ¼ day in 72 years MTBF Modular Hardware Building Blocks Software inherited from prior Offerings plus Enhancements Software inherited from SVC and DS8000 RAID RAID 0, 1, 5, 6, 10 Storage Virtualization (Internal and external disks) Non-disruptive Data Migration Global & Metro Mirror (Multi-site) FlashCopy Thin Provisioning New and enhanced Software functions New GUI (Easy-to-use, web based, XIV like) Easy Tier TM SSD exploitation RAID & enclosure RAS services and diagnostics Additional scalability improvements Integration with IBM Systems Director Enhancements to TPC, FCM and TSM support IBM Storwize V7000

25. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 25 What are the most common/Important OS I/O tuning parameters?  Device Queue Depth –Queue Depth can help or hurt performance per LUN Be aware of Queue Depth when planning system layout, adjust only if necessary To calculate - best thing to do is go to each device “Information Center” URLs listed in link slide What are the default Queue Depths? ___  HBA transfer rates –FC adapters  LVM striping vs spreading  Data Placement –Random versus sequential –Spreading versus Isolation Queue Depth is central to the following fundamental performance formula: IO Rate = Number of Commands * Response Time per Command For example: –IO Rate = 32 Commands per Second /.01 Seconds (10 milliseconds) per Command = 3200 IOPs Some real-world examples: OS=Default Queue Depth= Expected IO Rate AIX Standalone = 16 per LUN = 1600 IOPs per LUN AIX VIOS = 20 per LUN = 2000 IOPs per LUN AIX VIOC = 3 per LUN = 300 IOPs per LUN Windows = 32 per Disk = 3200 IOPS per LUN Content provided by Mark Chitti Tips –

26. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 26 Data Placement and Host Vdisk mapping  Spreading versus Isolation –Spreading the I/O across MDGs exploits the aggregate throughput offered by more physical resources working together –Spreading I/O across the hardware resources will also render more throughput than isolating the I/O to only a subset of hardware resource –You may reason that the more hardware resources you can spread across, the better the throughput Don’t spread file systems across multiple frames Makes it more difficult to manage code upgrades, etc.  Should you ever isolate data to specific hardware resources?  Name a circumstance! Isolation –In some cases more isolation on dedicated resources may produce better I/O throughput by eliminating I/O contention –Separate FlashCopy – Source and Target LUNs – on isolated spindles

27. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 27 Since the time to complete an I/O operation depends on: EXT 1 EXT 3 EXT 4 EXT 5 EXT 2 EXT 3 EXT 4 EXT 5 LV 1LV 2LV3LV4LV5 LUN 1 LUN 3 LUN 1 Made up of strips from the outer section/edge of each physical Disk RAID-5 6+P LUN 3 Made up of strips from the middle sections of each physical Disk RAID-5 6+P What cause disk latency? Does Cache mitigate disk latency? a. Seek time- time to position the read/write head b. Rotational delay- time waiting for disk to spin to proper starting point c. Transfer time You could deduce that: a)Logical-disk3 would be a better place to store data that will be randomly accessed since the read/write heads would most likely have shorter seek times to the middle of the disks. b) Logical-disk1 would provide greater sequential throughput since it is on the outer edge of the disks.

28. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 28 Data Placement Strip2 Strip4 Strip5 Strip1 Strip3 LUN1 made of strips on the outer edge of the DDMs (1s) also could have App A Raid-5 7+P LUN3 made of strips in the middle of the DDMs (3s) also could have App B Raid-5 7+P Placing Applications on the same LUNs/Pools result in IO contention Extent pool or 8 Ranks 4 2 5 3 1 4 2 5 3 1 4 2 5 3 1 4 2 5 3 1 4 2 5 3 1 4 2 5 3 1 4 2 5 3 1 4 2 5 3 1 For existing applications, use storage and server performance monitoring tools to understand current application workload characteristics such as: Read/Write ratio Random/sequential ratio Average transfer size (blocksize) Peak workload (I/Os per second for random access, and MB per second for sequential access) Peak workload periods (time of day, time of month) Copy services requirements (Point-in-Time Copy, Remote Mirroring) Host connection utilization and throughput (HBA Host connections) Remote mirroring link utilization and throughput What causes THRASHING? Most commonly when workloads peak at the same time or log files and data files share physical spindles

29. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 29 Data Placement - Why Storage Tiering? – Key Drivers Storage is not used cost efficiently Top Customer Pain Points  70+% in Tier1  <40% utilized Slide and content provided by Laura Richardson

30. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 30 Sequential Pools Striped Pools  Should you ever stripe with previrtualized volumes?  We recommend not striping or spreading in SVC, V7000 and XIV Storage Pools  Avoid LVM spreading with any striped storage pool  You can use file system striping with DS8000 storage pools –Across storage pools with a finer granularity stripe –Within DS8000 storage pools but on separate spindles when volumes are created sequentially Host Stripe - Raid-0 only Host Stripe No Host Stripe Host Stripe StripeStripe Data Placement - Storage Pools and Striping

31. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 31 RAID array LUN or logical disk 1 2 3 4 5 1 PV 2345 datavg # mklv lv1 –e x hdisk1 hdisk2 … hdisk5 # mklv lv2 –e x hdisk3 hdisk1 …. hdisk4 ….. Use a random order for the hdisks for each LV Disk subsystem Slide Provided by Dan Braden What does random LV creation order, help prevent? Random IO Data layout

32. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 32 Sequential IO Data layout  Does understanding the backend enable good front-end configuration?  Sequential IO (with no random IOs) best practice: –Create RAID arrays with data stripes a power of 2 RAID 5 arrays of 5 or 9 disks RAID 10 arrays of 2, 4, 8, or 16 disks –Create VGs with one LUN per array –Create LVs that are spread across all PVs in the VG using a PP or LV strip size >= a full stripe on the RAID array –Do application IOs equal to, or a multiple of, a full stripe on the RAID array –Avoid LV Striping Reason: Can’t dynamically change the stripe width for LV striping –Use PP Striping Reason: Can dynamically change the stripe width for PP striping Slide Provided by Dan Braden

33. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 33 Data Layout - OS Spreading versus Striping  Is there is a difference? What’s the diff? – Do you know what are your volumes made of! File system spread

34. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 34 Data Layout Summary  Does data layout affect IO performance more than any tunable IO parameter?  Good data layout avoids dealing with disk hot spots –An ongoing management issue and cost  Data layout must be planned in advance –Changes are generally painful  iostat might and filemon can show unbalanced IO  Best practice: evenly balance IOs across all physical disks unless TIERING  Random IO best practice: –Spread IOs evenly across all physical disks unless dedicated resources are needed to isolate specific performance sensitive data For disk subsystems Create RAID arrays of equal size and type Create VGs with one LUN from every array Spread all LVs across all PVs in the VG

35. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 35 Supported Connectivity - Does zoning matter? – SVC example  Symmetry  Balance  Performance Throughput SVC to Host Storage to SVC TIP – These make good topology diagrams

36. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 36 Connectivity – topology and diagrams facilitate resiliency Can you have too many paths? Preferred path for vdisk1 is SVC N1P2 & N1P3 Non Preferred path for vdisk1 is SVC N2P2 &N2P3 Preferred path for vdisk2 is SVC N2P2 & N2P3 Non Preferred path for vdisk2 is SVC N1P2 &N1P3 Server/Host view of the datapaths * Non Preferred Paths Preferred Paths # cat rmmissing_path lsdev -Cc disk |grep -i MPIO |awk '{print $1}' | \ while read LINE; do lspath -l $LINE -F "name path_id parent connection status" |grep -i Missing | \ while read ELINE; do echo $ELINE |awk '{print $3, $4}' |read PRNTDVS CONECTSET rmpath -d -l $LINE -p $PRNTDVS -w "$CONECTSET" done echo "hdisk $LINE done" Done Script and content provided by Aydin Tasdeler If you need to clean up the pathing by reducing to the supported number for SVC then here is a useful script How to remove missing paths on VIO server using a script

37. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 37 Host Attachment Kits  SVC -To attach an IBM® System p® AIX® host, you must install the AIX host attachment script. –www.ibm.com/servers/storage/support/software/sdd/downloading.htmlwww.ibm.com/servers/storage/support/software/sdd/downloading.html  DS8000 - For the most current information on supported hosts, operating systems, adapters, and switches, go to the System Storage® Interoperation Center (SSIC) website at www.ibm.com/systems/support/storage/config/ssic/. –For additional SDD information, see the IBM System Storage Multipath Subsystem Device Driver User's Guide at –www.ibm.com/systems/support/storage/software/sddwww.ibm.com/systems/support/storage/software/sdd –http://www.redbooks.ibm.com/redbooks/pdfs/sg248887.pdf  XIV -Install the proper Host Attachments Kit (Multipathing drivers) –http://www-01.ibm.com/support/search.wss?q=ssg1*&tc=STJTAG+HW3E0&rs=1319&dc=D400&dtmhttp://www-01.ibm.com/support/search.wss?q=ssg1*&tc=STJTAG+HW3E0&rs=1319&dc=D400&dtm –ftp://ftp.software.ibm.com/storage/XIV/Host_Attachment/AIX/  V7000 – For the most current information on supported hosts –Confirm the adapter driver and firmware levels are current by checking the System Storage Interoperation Center. http://www- 03.ibm.com/systems/support/storage/config/ssic/displayesssearchwithoutjs.wss?start_over=yes –For host connectivity-Install the host attachment script as specified by the SDDPCM Readme file. http://www-01.ibm.com/support/docview.wss?rs=540&context=ST52G7&uid=ssg1S7000303

38. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 38 Documentation – Does it matter? Why?  Disk mapping at a glance –Mapping becomes important  Spreading versus isolation Isolation Spreading Track data placement and Host Vdisk mapping

39. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 39 Documentation – Why it matters  How do I achieve SVC node to Server Balance? Spreadsheet developed by Keith Williams Use the SVCQTOOL listed under the tools section of this slide deck to produce a spread sheet similar to this Or Use the script found in the speaker notes of this slide Add a column for preferred node to host client

40. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems Are there any automated storage inquiry tools out there that will help me understand my setup?  Storage tools –Gathers information such as, but not limited to: LUN layout LUN to Host mapping Storage Pool maps Fabric connectivity –DS8QTOOL Go to the following Website to download the tool: http://congsa.ibm.com/~dlutz/public/ds8qtool/index.htm –SVCQTOOL Go to the following Website to download the tool: http://congsa.ibm.com/~dlutz/public/svcqtool/index.htm 40 How do I get the information

41. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 41 Systems Administrator - How do I improve disk performance on the Host?  Reduce the number of IOs –Bigger caches Application, file system, disk subsystem Use caches more efficiently –No file system logging –No access time updates  Improve average IO service times –Better data layout –Reduce locking for IOs –Buffer/queue tuning –Use SSDs or RAM disk –Faster disks/interfaces, more disks –Short stroke the disks and use the outer edge –Smooth the IOs out over time  Reduce the overhead to handle IOs

42. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 42 Troubleshooting – What’s the most common thing that changes over time? Rank Pool 1 Vdisk1 Vdisk2 App A App B Host A Host B An ounce of prevention is worth a pound of cure Data Migration Rank Pool 2 Map Apps sharing the same physical spindles on traditional arrays may peak at the same time Depending on the work load characteristics, isolating the workload may prove to be more beneficial and out perform a larger array. –There are 3 important principles for creating a logical configuration for the Storage Pools to optimize performance: Workload isolation Workload resource-sharing Workload spreading Some examples of I/O workloads or files/datasets which may have heavy and continuous I/O access patterns are: Sequential workloads (especially those with large blocksize transfers) Log files or datasets Sort/work datasets or files Business Intelligence and Data Mining Disk copies (including Point in Time Copy background copies, remote mirroring target volumes, and tape simulation on disk) Video/imaging applications Engineering/scientific applications Certain batch workloads I always separate Log files from Data files for best performance.

43. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 43 010101010101010101 010010101101001000 ESS CPQ LCRB IBM SAN VOLUME CONTROLLER Node IO-Group SDD Drivers Decoupling the relationship between servers and storage systems makes for efficient use of storage resources. FlashCopy and Mirroring can be between dissimilar disk systems. Designed to offer flexible, SAN wide copy services. FlashCopy 010101010101010101 010010101101001000 Copy Data Remote Copy How can this help me with data placement? Advanced Features – plan careful data placement

44. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 44 StorAdmin – How do I improve disk performance?  Data layout affects IO performance more than any tunable IO parameter If a bottleneck is discovered, then some of the things you need to do are: –Identify the hardware resources the heavy hitting volumes are on Identify which D/A pair the rank resides on Identify which I/O enclosure the D/A pair resides on Identify which host adapters the heavy hitting volumes are using Identify which host server the problem volumes reside on Identify empty non used volumes on other ranks – storage pools –Move data off the saturated I/O enclosures to empty volumes residing on less used ranks/storage pools –Move data off the heavy hitting volumes to empty volumes residing on less used hardware resources and perhaps to the another Storage Device –Balance LUN mapping across Backend and host HBAs SVC iogrps SVC preferred nodes –Change Raid type.

45. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 45 Troubleshooting: What are some Storage Bottlenecks? After verifying that the disk subsystem is causing a system bottleneck, a number of solutions are possible. These solutions include the following: Consider using faster disks SDD will out perform HDD, etc. Eventually change the RAID implementation if this is relevant to the server’s I/O workload characteristics. –For example, going to RAID-10 if the activity is heavy random writes may show observable gains. Add more arrays/ranks to the Storage pool. –This will allow you to spread the data across more physical disks and thus improve performance for both reads and writes. Add more RAM –Adding memory will increase system memory disk cache, which in effect improves disk response times. Finally, if the previous actions do not provide the desired application performance: –Off-load/migrate - processing to another host system in the network (either users, applications, or services).

46. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems  For Redbooks on the topics discussed here go to: http://www.redbooks.ibm.com/ –In the Serach field – type in the Keyword you want to pull up information on, such as SVC, DS8000, XIV, V7000  SVC Information Center http://publib.boulder.ibm.com/infocenter/svcic/v3r1m0/index.jsp –V7000 - Install the host attachment script as specified by the SDDPCM Readme file. http://www-01.ibm.com/support/docview.wss?rs=540&context=ST52G7&uid=ssg1S7000303  V7000 -Support Information Center http://publib.boulder.ibm.com/infocenter/storwize/ic/index.jsp –Support for Storwize V7000 website at www.ibm.com/storage/support/storwize/v7000  XIV Information Center http://publib.boulder.ibm.com/infocenter/ibmxiv/r2/index.jsp –XIV Redbooks: Architecture, Implementation, & usage http://www.redbooks.ibm.com/abstracts/sg247659.html http://www.redbooks.ibm.com/redpieces/abstracts/sg247904.html?Open  DS8000 Information Center –http://publib.boulder.ibm.com/infocenter/dsichelp/ds8000ic/index.jsphttp://publib.boulder.ibm.com/infocenter/dsichelp/ds8000ic/index.jsp –Redbook - http://www.redbooks.ibm.com/redbooks/pdfs/sg248887.pdf  Details about the what the RAID levels actually mean is available at URL below: – http://en.wikipedia.org/wiki/RAIDhttp://en.wikipedia.org/wiki/RAID Helpful URLs- One URL is missing, what is it? And………… IBM Systems Magazine –If any of the above links do not work for you, copy and paste this URL into your browser: www.ibmsystemsmagpowersystemsdigital.com www.ibmsystemsmagpowersystemsdigital.com page 34 and 35

47. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 47 Summary  Knowing - what's inside will help you make informed decisions?  You should make a list of the things you don’t know –Talk to the Storage Administrator or those who do know  A better Admin understands 1.The backend physical makeup 2.The backend virtual makeup 3.What's in a Storage Pool for better data placement 4.Avoid the Pitfalls associated with IO Tuning  5. Know where to go to get right device drivers  6. Know why documentation matters –7. Keep Topology Diagrams –8. keep Disk Mapping documentation –9. Be able to use Storage Inquiry Tools to find answers –10. Understand how to troubleshoot storage performance bottlenecks

48. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems Questions- What’s the bottom line? 48

49. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems Thank you! For you interest and attendance

50. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 50 Extras Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems

51. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 51 You can make Tiers on the same Storage Platform Manual Mode (Volume Level) and Automatic Mode (Extent Level) Automatic extent level data relocation enabled in a Merged Extent Pool SSD Rank Pools FC or SATA Rank Pools Merged Pool (SSD/FC or SSD/SATA) Manual Mode Volume Based Data Relocation Automatic Mode Extent Level Data Relocation Change disk class Change RAID type Change disk RPM

52. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 52 Tier0 Ultra High Performance Tier1 High Performance Mission Critical Tier2 Medium Performance Non-Mission Critical Tier3 Low Performance Archival/Tape Performance You should consider: Cost versus Performance Cost Per Gigabyte Tiered Storage Strategy Overview Tier 0: Ultra High Performance Applications 1-3% Tier 1: Mission critical, revenue generating applications 15-20% Tier 2: Backup, recovery and vital data 20-25% Tier 3: Archives and long term retention 50-60% Storage Pyramid

53. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems Tiered Storage Classification TIERDescription Technical Examples – High level Guidance – Local variations on technology exist IBM (block) SVC Recommended for OpenIBM (file) Nseries/SoNASI/O DensityPerformance range capability TIER0 – Solid State drives only Ultra High Performance. Meet QoS for High End DS8870/SVC 400GB recommended *** RAID 5 -- Small Block Recommended – 720/820 FlashSystem >1 DS8870 -> Greater than 250,000 IOPs/5,300+ MBs 720/820 FlashSystem  Greater than 400,000 IOPs mixed work load (70/30). 100% write the 720 should be used as it can maintain this while the 820 drops down significantly. TIER1(a) TIER1(b) High Performance. Drive up utilization of high- ended storage subsystems and still maintain performance QoS objectives. For low capacity requirements smaller less powerful devices may meet tier definition DS8870 w/SAS 300GB 15K disk drive RAID5/RAID10 arrays *** -- DS8700/DS8800 removed from strategy -- N7000 series/FAS6 series units with HDD disk drive tech. 146GB/300GB SAS drives in RAID-DP arrays. 0.5 - 1 DS8870 -> 250,000+ IOPs / 5800 MBs N7000/FAS6 -> less than 60,000 IOPs or less than 1,500MBs XIV* (GEN3) model 214 with 2TB SAS drives (11 Module or greater unit) XIV* (GEN3) model 214 with 3TB SAS drives (11 Module or greater unit) --- 11.2 code version required --- --- SSDs (solid state drives) required --- --- XIV GEN2 removed from strategy --- XIV 2TB XIV (GEN3 15 mod) less than 120,000 IOPs/3,000 MBs XIV (GEN3 11 mod) less than 80,000 IOPs/2,600 MBs XIV 3TB XIV (GEN3 15 mod) less than 100,000 IOPs/2,800 MBs XIV (GEN3 11 mod) less than 70,000 IOPs/2,400 MBs TIER2 Medium Performance. Meet QoS for applications/data that resides here. For low capacity requirements smaller less powerful devices may meet tier definition DS8870 w/SAS 600GB 10k disk drive RAID5/RAID10 arrays V7000 w/SAS 600GB using RAID5 V7000 (Pure) Integrated Chassis Platform w/450GB SAS N6000 series/FAS3 series units with HDD disk drive tech. 146GB/300GB/450GB SAS drives in RAID-DP arrays. SoNAS** with SAS V7000 Unified (Pending) <0.5 DS8870 -> less than 50,000 IOPs or less than 1,500 MBs V7000 (block) -> less than 50,000 IOPs or less than 1,500 MBs N6000/FAS3 -> less than 50,000 IOPs or less than 1,500 MBs SoNAS -> up to 25,000 IOPs/1,000 MBs XIV* (GEN3) with 2TB SAS drives (9 or 10 modules) XIV* (GEN3) with 3TB SAS drives (9 or 10 modules) -- No XIVs less than 9 modules recom -- XIV 2TB XIV (GEN3) less than 70,000 IOPs/2,400 MBs XIV 3TB XIV (GEN3) less than 50,000 IOPs/2,000 MBs * XIV should not be used for transaction intensive workloads (under 5ms response) and CACHE unfriendly workloads. Use DS88 for this need. ** SoNAS to be used as part of Cloud offering only *** Recommendation is to run DISK Magic to confirm drive selection/amount needed/DA Pairs needed COST / PERFORMANCE / AVAILABILITY Keep within I/O density ranges to maintain good (eg <10ms) response times Response times may increase exponentially above these ranges Tier rating is based on performance AND reliability

54. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems Tiered Storage Classification TIERDescription Technical Examples – High level Guidance – Local variations on technology exist IBM (block) SVC Recommended for Open IBM (file) Nseries/SoNAS I/O Density Performance range capability TIER3 Low Performance. Meet QoS for applications/data that resides here. DS8870 with NL-SAS tech using RAID6 V7000 with NL-SAS using RAID6 N3000 series/FAS2 series. Any unit with SATA using RAID DP SoNAS** with NL-SAS V7000 Unified (Pending) <0.1 DS8870 -> less than 30,000 IOPs or less than 1,000 MBs N3000/FAS2 -> 11,000 IOPs/500 MBs thru 30,000 IOPs/750 MBs SoNAS -> up to 5,000 IOPs/400 MBs V7000 (block) -> less than 30,000 IOPs or less than 300 MBs TIER4 Archival, long term retention, backup Virtual Engines, Tape ATLs, ProtecTIER N/A Tier based on features. * XIV should not be used for transaction intensive workloads (under 5ms response) and CACHE unfriendly workloads. Use DS88 for this need. ** SoNAS to be used as part of Cloud offering only *** Recommendation is to run DISK Magic to confirm drive selection/amount needed/DA Pairs needed COST / PERFORMANCE / AVAILABILITY Keep within I/O density ranges to maintain good (eg <10ms) response times Response times may increase exponentially above these ranges Tier rating is based on performance AND reliability

55. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 55  Users define volumes with any logical size  Users acquire only the physical capacity of XIV Storage needed for data that is actually written –The part of the volume that contains no data does not consume any physical space Thin Provisioning – What is it? Actual Data Written 10GB Actual Data Written 30GB Actual Data Written 20GB Volume 30GB Volume 50GB Volume 70GB Fat provisioned Thin provisioned Amount of physical storage required Actual Data Written 10GB Actual Data Written 30GB Actual Data Written 20GB Volume 30GB Volume 50GB Volume 70GB 150 GB 60 GB Real Capacity Virtual Capacity Used Capacity Used Capacity Used Capacity

56. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 56 IBM XIV Storage: Thin Provisioning  Defining logical volumes bigger than physical capacity  Installing physical capacity only if and when needed  No space consumed when data is 0  Pools are used to manage quota  Results: –Reduced overall direct storage cost –Storage expenses spread over time, exploiting price reductions –Easier management –Save 20-50% of storage capacity

57. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 57 SDD Drivers EMC Drivers RDAC Drivers EMC DS4000DS8000 Remote Copy ? Flashcopy ?  Proprietary, non-interoperable Copy Services  Static relationship between servers and storage systems  Can standardize on one vendor – usually very expensive  Or restrict servers to vendors  Use SVC and TPC to address Out of Space Out of Space Out of Space Storage Problems and Limitations Today Free capacity 010101010101010101 010010101101001000 Data Migration Inefficient use, storage not available to all servers  Migration of data disruptive and time consuming Traditional SAN storage environments

58. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 58 Connectivity - Host to SVC Zoning Best Practices Good communication between the SA and the Storadmin, can uncover issues quickly –Correct datapathing has 3 factors Proper zoning Proper SVC Host definitions (SVC logical config of the host def) Proper redundancy for the SVC preferred /non preferred pathing DEV#: 3 DEVICE NAME: hdisk3 TYPE: 2145 ALGORITHM: Load Balance SERIAL: 600507680181059BA000000000000005 ============================================================ Path# Adapter/Path Name State Mode Select Errors 0 fscsi0/path0 OPEN NORMAL 558254 0 1* fscsi0/path1 OPEN NORMAL 197 0 2* fscsi0/path2 OPEN NORMAL 197 0 3 fscsi0/path3 OPEN NORMAL 493559 0 4 fscsi2/path4 OPEN NORMAL 493330 0 5* fscsi2/path5 OPEN NORMAL 197 0 6* fscsi2/path6 OPEN NORMAL 197 0 7 fscsi2/path7 OPEN NORMAL 493451 0 8 fscsi5/path8 OPEN NORMAL 492225 0 9* fscsi5/path9 OPEN NORMAL 197 0 10* fscsi5/path10 OPEN NORMAL 197 0 11 fscsi5/path11 OPEN NORMAL 492660 0 12 fscsi7/path12 OPEN NORMAL 491988 0 13* fscsi7/path13 OPEN NORMAL 197 0 14* fscsi7/path14 OPEN NORMAL 197 0 15 fscsi7/path15 OPEN NORMAL 492943 0 Correct Incorrect

59. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 59 Tip - Queue Depth Tuning  Take some measurements  Make some calculations –(Storage port depth / total LUNs per host = queue depth) If a single host with 10 assigned LUNs, is accessing the storage port supporting 4096 then calculate as (4096/10 = 409) or 256 in this case –Are there different calculations for the different storage devices? For volumes on homogeneous hosts examples: SVC q = ((n ×7000) / (v×p×c)) DS8000 = 2048 XIV= 1400 V7000 q = ((n * 4000) / (v * p * c)) Best thing to do is go to each device “Information Center” URLs listed in link slide –Don’t increase queue depths beyond what the disk can handle! IOs will be lost and will have to be retried, which reduced performance  Note: –For more information on the info needed to make the calculations, please refer to the “Dan Braden” in the Extra slides at the end of this deck

60. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 60 More on Host Disk IO Tuning  Please refer to the following PPTs provided by Dan Braden  Disk IO Tuning  SANBoot

61. © 2013 IBM Corporation Best Practices for Performance, Design and Troubleshooting IBM Storage connected to Power Systems 61 Document VIO to LPAR mapping  Script Output sample to produce documentation Content provided by Aydin Y. Tasdeler