Presentation is loading. Please wait.

Presentation is loading. Please wait.

Optimizing Performance of HPC Storage Systems

Similar presentations


Presentation on theme: "Optimizing Performance of HPC Storage Systems"— Presentation transcript:

1 Optimizing Performance of HPC Storage Systems
Torben Kling Petersen, PhD Principal Architect High Performance Computing

2 Current Top10 ….. All Top10 systems are running Lustre 1.8.x Rank Name
Computer Site Total Cores Rmax Rpeak Power (KW) File system Size Perf 1 Tianhe-2 TH-IVB-FEP Cluster, Xeon E C 2.2GHz, TH Express-2, Intel Xeon Phi National Super Computer Center in Guangzhou 17808 Lustre/H2FS 12.4 PB ~750 GB/s 2 Titan Cray XK7 , Opteron C 2.2GHz, Cray Gemini interconnect, NVIDIA K20x DOE/SC/Oak Ridge National Laboratory 560640 8209 Lustre 10.5 PB 240 GB/s 3 Sequoia BlueGene/Q, Power BQC 16C 1.60 GHz, Custom Interconnect DOE/NNSA/LLNL 7890 55 PB 850 GB/s 4 K computer Fujitsu, SPARC64 VIIIfx 2.0GHz, Tofu interconnect RIKEN AICS 705024 12659 40 PB 965 GB/s 5 Mira BlueGene/Q, Power BQC 16C 1.60GHz, Custom DOE/SC/Argonne National Lab. 786432 3945 GPFS 7.6 PB 88 GB/s N/A BlueWaters Cray XK7, Opteron 16C 2.2GHz, Cray Gemini interconnect, NVIDIA K20x NCSA - 24 PB 1100 GB/s 6 Piz Daint Cray XC30, Xeon E C 2.600GHz, Aries interconnect , NVIDIA K20x Swiss National Supercomputing Centre (CSCS) 115984 2325 2.5 PB 138 GB/s 7 Stampede PowerEdge C8220, Xeon E C 2.7GHz, IB FDR, Intel Xeon Phi TACC/ Univ. of Texas 462462 4510 14 PB 150 GB/s 8 JUQUEEN BlueGene/Q, Power BQC 16C 1.600GHz, Custom Interconnect Forschungs zentrum Juelich (FZJ) 458752 2301 5.6 PB 33 GB/s 9 Vulcan BlueGene/Q, Power 16C 1.6GHz, Custom Intercon. 393216 1972 10 SuperMUC iDataPlex DX360M4, Xeon E C 2.70GHz, Infiniband FDR Leibniz Rechenzentrum 147456 3423 10 PB 200 GB/s All Top10 systems are running Lustre 1.8.x

3 Performance testing Lies Bigger lies Benchmarks

4 Storage benchmarks IOR IOzone Bonnie++ spgdd-survey obdfilter-survey
FIO dd/xdd Filebench dbench Iometer MDstat metarates …….

5 Lustre® Architecture – High Level
Object Storage Servers (OSS) 1-1,000s Object Storage Target (OST) Client Router OSS Disk arrays & SAN Fabric disk CIFS Client Gateway NFS Client OSS disk Support multiple network types IB, X-GigE Client OSS disk Client Metadata Servers (MDS) MDS MDS Lustre Client 1-100,000 Metadata Target (MDT) disk

6 Dissecting benchmarking

7 The chain and the weakest link …
Non-blocking fabric ? TCP-IP Overhead ?? Routing ? SAS port over subscription Cabling … RAID controller (SW/HW) Client Memory OSS Memory disk interconnect bus network bus bus CPU Memory PCI-E bus FS client MPI stack SAS Controller/Expander PCI-E bus CPU Memory OS Interconnect Disk drive perf RAID sets SAS or S-ATA File system ?? Only a balanced system will deliver performance …..

8 Server Side Benchmark Using obdfilter-survey is a Lustre benchmark tool that measures OSS and backend OST performance and does not measure LNET or Client performance This is a good benchmark to isolate network and client from the server. Example of obdfilter-survey parameters ~]# nobjlo=1 nobjhi=1 thrlo=256 thrhi=256 size=65536 obdfilter-survey Parameters Defined size=65536         // file size (2x Controller Memory is good practice) nobjhi=1 nobjlo=1     // number of files thrhi=256 thrlo=256     // number of worker threads when testing OSS If you see results significantly lower than what is expected, rerun the test multiple times to ensure those results are not consistent. This benchmark can also target individual OSTs if we see an OSS node performing lower than expected, it can be because of a single OST performing lower due to drive issue, RAID array rebuilds, etc. ~]# targets=“fsname-OST0000 fsname-OST0002” nobjlo=1 nobjhi= thrlo=256 thrhi=256 size=65536 obdfilter-survey

9 Client Side Benchmark IOR uses MPI-IO to execute the benchmark tool across all nodes and mimics typical HPC applications running on Clients Within IOR, one can configure the benchmark for File-Per-Process, and Single-Shared-File File-Per-Process: Creates a unique file per task and most common way to measure peak throughput of a Lustre parallel Filesystem Single-Shared-File: Creates a Single File across all tasks running on all clients Two primary modes for IOR Buffered: This is default and takes advantage of Linux page caches on the Client DirectIO: Bypasses Linux page caching and writes directly to the filesystem

10 Typical Client Configuration
At customer sites, typically all clients have the same architecture, same number of CPU cores, and same amount of memory. With a uniform client architecture, the parameters for IOR are simpler to tune and optimize for benchmarking Example for 200 Clients Number of Cores per Client: 16 (# nproc) Amount of Memory per Client 32GB (cat /proc/meminfo)

11 IOR Rule of Thumb Always want to transfer 2x the memory size of the total number of clients used to avoid any client side caching effect In our example: (200 Clients*32 GB)*2 = 12,800 GB Total file size for the IOR benchmark will be 12.8 TB NOTE: Typically all nodes are uniform.

12 Lustre Configuration

13 Lustre Server Caching Description
Lustre read_cache_enable controls whether data read from disk during a read request is kept in memory and available for later read requests for the same data, without having to re-read it from disk. By default, read cache is enabled (read_cache_enable = 1). Lustre writethrough_cache_enable controls whether data sent to the OSS as a write request is kept in the read cache and available for later reads, or if it is discarded from cache when the write is completed. By default, writethrough cache is enabled (writethrough_cache_enable = 1) Lustre readcache_max_filesize controls the maximum size of a file that both the read cache and writethrough cache will try to keep in memory. Files larger than readcache_max_filesize will not be kept in cache for either reads or writes. Default is all file sizes are cached.

14 Client Lustre Parameters
Network Checksums Default is turned on and impacts performance. Disabling this is first thing we do for performance LRU Size Typically we disable this parameter Parameter used to control the number of client-side locks in an LRU queue Max RPCs in Flight Default is 8, increase to 32 RPC is remote procedure call This tunable is the maximum number of concurrent RPCs in flight from clients. Max Dirty MB Default is 32, good rule of thumb is 4x the value of max_rpcs_in_flight. Defines the amount of MBs of dirty data can be written and queued up on the client

15 Lustre Striping Default Lustre Stripe size is 1M and stripe count is 1
Each file is written to 1 OST with a stripe size of 1M When multiple files are created and written, MDS will do best effort to distribute the load across all available OSTs The default stripe size and count can be changed. Smallest Stripe size is 64K and can be increased by 64K and stripe count can be increased to include all OSTs Changing stripe count to all OSTs indicates each file will be created using all OSTs. This is best when creating a single shared file from multiple Lustre Clients One can create multiple directories with various stripe sizes and counts to optimize for performance

16 Experimental setup & Results

17 Equipment used ClusterStor with 2x CS6000 SSUs Clients Lustre
2TB NL-SAS Hitachi Drives 4U CMU Neo 1.2.1, HF applied Clients 64 Clients, 12 Cores, 24GB Memory, QDR Mellanox FDR core switch Lustre Client: 1.8.7 Lustre Server version: 2.1.3 4 OSSes 16 OSTs (RAID 6)

18 Subset of test parameters
Disk backend testing – obdfilter-survey Client based testing – IOR I/O mode I/O Slots per client IOR transfer size Number of Client threads Lustre tunings writethrough cache enabled read cache enabled read cache max filesize = 1M Client Settings LRU Disabled Checksums Disabled MAX RPCs in Flight = 32

19 Lustre obdfilter-survey
# pdsh -g oss "TERM=linux thrlo=256 thrhi=256 nobjlo=1 nobjhi=1 rsz=1024K size=32768 obdfilter-survey" cstor01n04: ost 4 sz K rsz 1024K obj 4 thr 1024 write [ , ] rewrite [ , ] read [ , ] cstor01n05: ost 4 sz K rsz 1024K obj 4 thr 1024 write [ , ] rewrite [ , ] read [ , ] This means that a single SSU have a write performance of 6,055 MB/s (75,9 MB/s per disk) read performance of 7,904 MB/s (98.8 MB/s per disk )

20 Buffered I/O

21 Direct I/O np=512 = 8 threads per client

22 Summary

23 Reflections on the results
Never trust marketing numbers … Testing all stage of the data pipeline is essential Optimal parameters and/or methodology for read and write are seldom the same Real life applications can often be configured accordingly Balanced architectures will deliver performance Client based IOR performs within 5% of backend In excess of 750 MB/s per OST … -> 36 GB/s per rack … A well designed solution will scale linearly using Lustre cf. NCSA BlueWaters

24 Optimizing Performance of HPC Storage Systems
Thank You


Download ppt "Optimizing Performance of HPC Storage Systems"

Similar presentations


Ads by Google