Presentation is loading. Please wait.

Presentation is loading. Please wait.

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Benchmarking Swift Eamonn.

Similar presentations


Presentation on theme: "© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Benchmarking Swift Eamonn."— Presentation transcript:

1 © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Benchmarking Swift Eamonn O’Toole Mark Seger

2 © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Agenda Benchmarking with HP’s getput – Procedure, tools and operation Case study – Selecting servers for HP’s public cloud

3 © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The Benchmarking Bible Scripts work best for repeatability – Both for load generation and measurement Test from the bottom of the stack up Longer runs tend to reduce cache effects The middle of the test is as important as the duration Avoid changing more than 1 thing at a time It will take as long as it will take There’s no such thing as a coincidence!

4 © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Size Matters Large Objects – IOPS are small, so pay attention to MB/sec – These use a lot of bandwidth so make sure network wide enough – Use a lot of CPU so could need ~1core/stream/client Small Objects – MB/sec is low, so pay attention to IOPS – Network bandwidth is less of a concern but latency is – CPU requirements are relatively low as well

5 © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Collectl Developed about a dozen years ago Open Source on sourceforge Collects fine-grained metrics – CPU, Disk, Network, Memory and more – Process level, including I/O Can generate stats in real-time or record for later playback – In playback mode can summarize metrics for each process Colplot generates plots for visualizing overall performance

6 © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Getput Tools Designed exclusively for Swift Benchmarking Lots of options for simulating lots of behaviors – Puts, Gets, Deletes – Object sizes – Number of clients – Number of processes – Level of container sharing Options for running tests – Ranges for numbers of objects, processes and clients – Define pre/post test initialization/analysis scripts The complete list beyond the scope of this talk

7 © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Getting Started Need swift credential exported to your environment If swift stat works, getput will work and if it doesn’t it won’t! $./getput.py -cc -oo -n1 -s1k -tp,g,d Rank Test Clts Proc OSize Start End MB/Sec Ops Ops/Sec Errs Latency Median LatRange 0 put 1 1 1k 20:28:04 20:28: get 1 1 1k 20:28:04 20:28: del 1 1 1k 20:28:04 20:28: Simple put, get, del $./getput.py -cc -oo -n1 -s1k,2k -tp --procs 1,2 Rank Test Clts Proc OSize Start End MB/Sec Ops Ops/Sec Errs Latency Median LatRange 0 put 1 1 1k 20:32:55 20:32: put 1 1 2k 20:32:55 20:32: Rank Test Clts Proc OSize Start End MB/Sec Ops Ops/Sec Errs Latency Median LatRange 0 put 1 2 1k 20:32:55 20:32: put 1 2 2k 20:32:56 20:32: Multiple sizes, multiple number of processes Note that 1KB PUTs are a lot slower than 2KB PUTs

8 © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Watching with collectl $./getput.py -cc -oo -n1 -s1g -tp Rank Test Clts Proc OSize Start End MB/Sec Ops Ops/Sec Errs Latency Median LatRange 0 put 1 1 1g 20:43:51 20:44: Large object upload p# #Time cpu sys inter ctxsw KBRead Reads KBWrit Writes KBIn PktIn KBOut PktOut 20:52: :52: :52: :52: :52: :52: :52: :52: :52: :52: :52: :52: :52: :52: :52: :52: :52: :52: :52: Collectl Network rate is NOT smooth

9 © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Running a Benchmark PUTs scale linearly through 16 process, rate increases are slower at 32 and 48 GETs look read good through 32 processes and slow down a bit at 48 DELs had some irregular latencies in upper range $ gpsuite --suite 1kobjs Test Clts Proc OSize Start End MB/Sec Ops Ops/Sec Errs Latency LatRange put 1 1 1k 11:36:24 11:38: get 1 1 1k 11:38:54 11:39: del 1 1 1k 11:39:41 11:40: put 1 2 1k 11:40:45 11:42: get 1 2 1k 11:43:15 11:43: del 1 2 1k 11:44:01 11:44: put 1 4 1k 11:45:03 11:47: get 1 4 1k 11:47:33 11:47: del 1 4 1k 11:48:18 11:49: put 1 8 1k 11:49:31 11:51: get 1 8 1k 11:52:01 11:52: del 1 8 1k 11:52:46 11:53: put k 11:54:07 11:56: get k 11:56:37 11:56: del k 11:57:23 11:58: put k 11:59:01 12:01: get k 12:01:31 12:01: del k 12:02:14 12:03: put k 12:04:11 12:06: get k 12:06:41 12:06: del k 12:07:26 12:09: gpsuite –suite 1kobjs

10 © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Example of getput maxing out Test Clts Proc OSize Start End MB/Sec Ops Ops/Sec Errs Latency LatRange put 1 1 1k 16:43:50 16:48: put k 16:50:17 16:55: put k 16:56:51 17:01: put k 17:03:40 17:08: put k 17:09:36 17:14: put k 17:15:41 17:20: gpsuite –suite 1kobjs Look at the latencies growing in both average and range Also notice we’ve hit the wall at a little <150 IOPS BUT swift did keep on chugging along Wow! Note – this cluster only had 1 object server

11 © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Selecting servers for HP’s public cloud Get better understanding of Swift performance and optimise hardware/Swift combination Two different hardware configurations – 12-disk data servers Dedicated proxy servers Data servers host account/container/object services 5:1 data-servers:proxy-servers – 60-disk data servers Dedicated proxy servers Data servers host object services only Container/account services on separate servers to object services 1:1 data-servers:proxy-servers Concentrate on transaction rates, especially PUTs of small objects (1KB to 10KB) – Most objects in production are small (50% <= 20KB) – High transaction rate exercises CPU, container & proxy services

12 © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Configuration 1... Disk1 Disk Disk 1 Disk Disk2 Proxy servers 12 physical cores, 2666MHz 96GB RAM 10 GigE 2*2TB 7200 RPM drives (mirror) ½ U width Data servers 12 physical cores, 2666MHz 24GB RAM 1 GigE 12*2TB 7200RPM drives 1U high Run object, container & account services... Server 1 Server 5

13 © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. First set of measurements: “idle” system Idle: no external PUTs, GETs, DELETEs etc This system has 123K containers & 17M objects per data-server Measurements with different services turned on and off in graph Significant “idle” CPU load Biggest contributor to “idle” CPU burn is container replicator

14 © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. CPU measurements: 1KB object PUTs

15 © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. I/O measurements: 1KB object PUTs

16 © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Observations on Configuration 1 measurements Idle CPU burn is 34%, increases to 38% at the maximum-achieved PUT rate (approx 338 PUTs/s) Container services are the major CPU hogs Small amount of memory hurts performance - most of the reads go to disk as opposed to cache – Major source of reads: object auditor – Object server reads grow approx. linearly with PUT rate (read 6x as much as write for 1KB PUTs) Running the container service in conjunction with the object service hurts I/O - the container data flushes object data from cache

17 © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Conclusions from Configuration 1 measurements PUT throughput (1KB) is limited by READ IOPs Keep container and object services separate The object services consumes relatively little CPU Large amounts of RAM for buffer cache will help increase performance

18 © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Configuration 2 Disk 1 Disk 4... Disk 1 Disk Proxy/Container & Account Servers Same server type for proxy services & container/account services 12 physical cores, 2666MHz GB RAM 10 GigE 4*1TB 7200 RPM disks, in a variety of RAID configurations ½ U width Object servers 12 physical cores, 2666MHz 96GB RAM 10 GigE 60*2TB 7200 RPM disks 4.3U high... Note Used many combinations of server & Swift services Used many variations of server details – e.g. RAID config Report results for a specific server/Swift service config

19 © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Performance measurements: 4KB object PUTs

20 © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Observations on Configuration 2 measurements We achieved a maximum throughput of approx 1600 PUTs/s using 1KB objects, and 2000 PUTs/s using 4KB objects Dramatic jump in CPU usage particularly on the Object-server for the 2000 PUTs/s run – Benefiting from hyperthreading On the Proxy/Account&Container-server, the dominant processes are the proxy-server and the container-server. All reads are satisfied from cache on Object- server and Proxy/Account&Container-server

21 © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Conclusions from Configuration 2 measurements Massive increase in operation throughput – 5x System 1 (per rack U) Proxy services and account/container services can coexist Object auditing time probably an issue with 60 disks – Estimate over 200 days for auditor to walk that many disks on “full” system – Possible solution: parallel object auditor Patch under review https://review.openstack.org/#/c/59778/https://review.openstack.org/#/c/59778/ Next steps – Detailed object auditor measurements – Large container measurements using SSDs, striped disks

22 © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Links collectl : getput: https://github.com/markseger/getputhttps://github.com/markseger/getput


Download ppt "© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Benchmarking Swift Eamonn."

Similar presentations


Ads by Google