Presentation is loading. Please wait.

Presentation is loading. Please wait.

Experiences with VI Communication for Database Storage Yuanyuan Zhou, Angelos Bilas, Suresh Jagannathan, Cezary Dubnicki, Jammes F. Philbin, Kai Li.

Similar presentations


Presentation on theme: "Experiences with VI Communication for Database Storage Yuanyuan Zhou, Angelos Bilas, Suresh Jagannathan, Cezary Dubnicki, Jammes F. Philbin, Kai Li."— Presentation transcript:

1 Experiences with VI Communication for Database Storage Yuanyuan Zhou, Angelos Bilas, Suresh Jagannathan, Cezary Dubnicki, Jammes F. Philbin, Kai Li

2 Presentation Outline Motivation  Use VI-based interconnects to improve I/O path performance between a database server and the storage subsystem. Approach  Design a block-level storage architecture V3 – VI-attached Volume Vault  Implement a software layer between the application and VI DSA – Direct Storage Access  Perform measurement by using Microsoft SQL Server 2000 on a 4-cpu and 32-cpu database server.

3 V3 Architecture V3 client  Small scale uniprocessor  SMP system  Large scale multiprocessor server V3 server  V3 volume  A volume provides more than 2TB storage  Span multiple nodes by combinations of RAID  More than 250TB storage in V3 back-end VI interconnect Customer software

4 DSA Implementations Block-level I/O module specification layer between the application and VI. Take advantage of VI-specific features  RDMA  Minimize kernel involvement  minimize multiple copying  Large number of Overlapping I/O requests

5 DSA Implementation Provide new features  Flow control, retransmission, reconnection  Optimization Memory registration and deregistration Interrupt handling Lock synchronization  One kernel-level implementation  Two user-level implementations

6 Kernel-level DSA Implementation Implemented on top of a kernel-level version of the VI specification. Provide the standard I/O interface for storage driver. Support user-level / kernel- level application without modification. Leverage the benefits of VI in kernel-level storage APIs

7 User-level DSA Implementation Replace all I/O calls to V3 storage Support standard windows I/O interface. Issue I/O requests without kernel involvement. Need kernel involvement for I/O completion. Notify wDSA by the interrupt to complete the corresponding I/O requests.

8 User-level DSA Implementation A new I/O API Minimize kernel involvement, context switches and interrupt Trade off transparency for performance. May need modification of the application. Application-controlled I/O completion mode either polling or interrupt

9 System Optimization for DSA VI registration and deregistration  Not feasible to pre-registering all I/O buffers at startup in database systems. Dynamically manage registered memory. kDSA – Windows I/O manager cDSA – Address Windowing Extensions  Batched deregistration

10 System Optimization Interrupts  Notify the database for a completed asynchronous I/O request.  Interrupt batching to reduce the high cost kDSA –When I/O requests > the threshold, use polling. cDSA – Set a completion flag by RDMA. Polling the flag during a time interval; switch to wait for interrupt after then.

11 System Optimization Lock Synchronization  Count a significant percentage on the I/O path.  Reduce the lock/unlock operations cDSA has more control of the lock synchronization pairs.

12 Experiment Platform A mid-size, 4-cpu SMP A large-size, 32-CPU SMP Giganet network Maximum end-to-end user-level bandwidth is about 110MB/s One way latency for a 64-bytes message is about 7 us.

13 Micro-benchmark Results DSA Overhead V3 configuration  A single application client runs the micro- benchmark.  A single storage node presents the virtual disk Raw VI  A locally attached disk, without any V3 software Reading a data block from the storage server.

14 DSA Overhead A round-trip delay for a read I/O request cDSA has the lowest CPU overhead

15 V3 Cached-block Performance

16 V3 vs. Local Case

17 On-line Transaction Processing BenchMark – TPC-C Commercial databases issue multiple concurrent I/Os and tolerant high I/O response times and low-throughput disks. Large database configuration Mid-size database configuration

18 Large Database Configuration

19 Mid-size Database Configuration

20 Conclusion cDSA provides a 18% transaction rate improvement over FC for large database configuration. Effective use of VI in I/O intensive environments requires substantial enhancements to flow control, reconnection, interrupt handling, memory registration and lock synchronization. New storage APIs that help minimize kernel involvement in the I/O path are needed to fully exploit the benefits of VI-based communication.


Download ppt "Experiences with VI Communication for Database Storage Yuanyuan Zhou, Angelos Bilas, Suresh Jagannathan, Cezary Dubnicki, Jammes F. Philbin, Kai Li."

Similar presentations


Ads by Google