Download presentation
Presentation is loading. Please wait.
Published byAnastasia Short Modified over 8 years ago
1
Experiences with VI Communication for Database Storage Yuanyuan Zhou, Angelos Bilas, Suresh Jagannathan, Cezary Dubnicki, Jammes F. Philbin, Kai Li
2
Presentation Outline Motivation Use VI-based interconnects to improve I/O path performance between a database server and the storage subsystem. Approach Design a block-level storage architecture V3 – VI-attached Volume Vault Implement a software layer between the application and VI DSA – Direct Storage Access Perform measurement by using Microsoft SQL Server 2000 on a 4-cpu and 32-cpu database server.
3
V3 Architecture V3 client Small scale uniprocessor SMP system Large scale multiprocessor server V3 server V3 volume A volume provides more than 2TB storage Span multiple nodes by combinations of RAID More than 250TB storage in V3 back-end VI interconnect Customer software
4
DSA Implementations Block-level I/O module specification layer between the application and VI. Take advantage of VI-specific features RDMA Minimize kernel involvement minimize multiple copying Large number of Overlapping I/O requests
5
DSA Implementation Provide new features Flow control, retransmission, reconnection Optimization Memory registration and deregistration Interrupt handling Lock synchronization One kernel-level implementation Two user-level implementations
6
Kernel-level DSA Implementation Implemented on top of a kernel-level version of the VI specification. Provide the standard I/O interface for storage driver. Support user-level / kernel- level application without modification. Leverage the benefits of VI in kernel-level storage APIs
7
User-level DSA Implementation Replace all I/O calls to V3 storage Support standard windows I/O interface. Issue I/O requests without kernel involvement. Need kernel involvement for I/O completion. Notify wDSA by the interrupt to complete the corresponding I/O requests.
8
User-level DSA Implementation A new I/O API Minimize kernel involvement, context switches and interrupt Trade off transparency for performance. May need modification of the application. Application-controlled I/O completion mode either polling or interrupt
9
System Optimization for DSA VI registration and deregistration Not feasible to pre-registering all I/O buffers at startup in database systems. Dynamically manage registered memory. kDSA – Windows I/O manager cDSA – Address Windowing Extensions Batched deregistration
10
System Optimization Interrupts Notify the database for a completed asynchronous I/O request. Interrupt batching to reduce the high cost kDSA –When I/O requests > the threshold, use polling. cDSA – Set a completion flag by RDMA. Polling the flag during a time interval; switch to wait for interrupt after then.
11
System Optimization Lock Synchronization Count a significant percentage on the I/O path. Reduce the lock/unlock operations cDSA has more control of the lock synchronization pairs.
12
Experiment Platform A mid-size, 4-cpu SMP A large-size, 32-CPU SMP Giganet network Maximum end-to-end user-level bandwidth is about 110MB/s One way latency for a 64-bytes message is about 7 us.
13
Micro-benchmark Results DSA Overhead V3 configuration A single application client runs the micro- benchmark. A single storage node presents the virtual disk Raw VI A locally attached disk, without any V3 software Reading a data block from the storage server.
14
DSA Overhead A round-trip delay for a read I/O request cDSA has the lowest CPU overhead
15
V3 Cached-block Performance
16
V3 vs. Local Case
17
On-line Transaction Processing BenchMark – TPC-C Commercial databases issue multiple concurrent I/Os and tolerant high I/O response times and low-throughput disks. Large database configuration Mid-size database configuration
18
Large Database Configuration
19
Mid-size Database Configuration
20
Conclusion cDSA provides a 18% transaction rate improvement over FC for large database configuration. Effective use of VI in I/O intensive environments requires substantial enhancements to flow control, reconnection, interrupt handling, memory registration and lock synchronization. New storage APIs that help minimize kernel involvement in the I/O path are needed to fully exploit the benefits of VI-based communication.
Similar presentations
© 2024 SlidePlayer.com Inc.
All rights reserved.