1. Balazs Voneki CERN/EP/LHCb Online group
RDMA optimizations on top of 100 Gbps Ethernet for the upgraded data acquisition system of LHCb
Balazs Voneki
CERN/EP/LHCb Online group
TIPP2017, Beijing

2. LHCb Upgrade
To improve detectors and electronics such that the experiment can run at higher instantaneous luminosity
Increase the event rate from 1 MHz to 40 MHz
Selection in software
Key challenges:
Relatively large chunks
Everything goes through the network
RU=Readout Unit BU=Builder Unit
Target instantaneous luminosity = 2 * 10^33 cm^-2 s^-1
The Event Builder Network and the EB-EFF (Event Filter Farm) does not have to be the same technology
Early results of RDMA optimizations on top of 100 Gbps Ethernet, TIPP2017, Beijing, – Balazs Voneki

3. Network technologies Possible 100 Gbps solutions: Intel® Omni-Path
EDR InfiniBand
100 Gbps Ethernet
Possible 200 Gbps solution:
HDR InfiniBand
Arguments for Ethernet:
Widely used
Old, mature, well-tried
OPA and IB are single vendor technologies, Ethernet is multi vendor
Ethernet is challenging at these speeds
Early results of RDMA optimizations on top of 100 Gbps Ethernet, TIPP2017, Beijing, – Balazs Voneki

4. Iperf result charts TCP
Early results of RDMA optimizations on top of 100 Gbps Ethernet, TIPP2017, Beijing, – Balazs Voneki

5. Iperf result charts TCP UDP High CPU load
Major difference between vendors
Early results of RDMA optimizations on top of 100 Gbps Ethernet, TIPP2017, Beijing, – Balazs Voneki

6. Linux network stack
Source:
Early results of RDMA optimizations on top of 100 Gbps Ethernet, TIPP2017, Beijing, – Balazs Voneki

7. What is RDMA? Remote Direct Memory Access
DMA from the memory of one node into the memory of another node without involving either one’s operating system
Performed by the network adapter itself, no work needs to be done by the CPUs, caches or context switches
Benefits:
High throughput
Low latency
These are especially interesting for
High Performance Computing!
RoCE (pronounced: Rocky) provides a seamless, low overhead, scalable way to solve the TCP/IP I/O bottleneck with minimal extra infrastructure.
iWARP = another alternative RDMA offering. It doesn’t need custom switch settings. How is that so? The implementation of iWARP is using TCP.
Both NICs are PCIe Gen 3.0 compliant
Source:
Early results of RDMA optimizations on top of 100 Gbps Ethernet, TIPP2017, Beijing, – Balazs Voneki

8. RDMA Technologies Available solutions:
RoCE (RDMA over Converged Ethernet)
iWARP (internet Wide Area RDMA Protocol)
RoCE needs custom settings
on the switch (priority
queues to guarantee lossless
L2 delivery). iWARP does not
need that, only the NICs has
to support it.
Test made using:
Chelsio T62100-LP-CR NIC
Mellanox CX455A ConnectX-4 NIC
Mellanox SN G Ethernet switch
RoCE (pronounced: Rocky) provides a seamless, low overhead, scalable way to solve the TCP/IP I/O bottleneck with minimal extra infrastructure.
iWARP = another alternative RDMA offering. It doesn’t need custom switch settings. How is that so? The implementation of iWARP is using TCP.
Both NICs are PCIe Gen 3.0 compliant
Source:
Early results of RDMA optimizations on top of 100 Gbps Ethernet, TIPP2017, Beijing, – Balazs Voneki

9. Testbed details iWARP testbench elements: 4 nodes of:
Dell PowerEdge C6220
2x Intel® Xeon® CPU E at 2.60GHz (8 cores, 16 threads)
32 GB DDR3 memory at 1333 MHz
Chelsio 100G T62100-LP-CR NIC
RoCE testbench elements:
Intel® S2600KP
2x Intel® Xeon® CPU E v3 at 2.30GHz (10 cores, 20 threads)
64 GB DDR4 memory at 2134 MHz
Mellanox CX455A ConnectX-4 100G NIC
Early results of RDMA optimizations on top of 100 Gbps Ethernet, TIPP2017, Beijing, – Balazs Voneki

10. Result charts iWARP RoCE
The iWARP results with a single thread run did not go beyond 90 Gbps (top-left figure).
Multiple threads needed to be tried to get closer to the theoretical maximum 100 Gbps (top-right figure).
The RoCE tests showed nice throughput numbers even with one thread, so in this case there was no need to test the multi-thread case.
Early results of RDMA optimizations on top of 100 Gbps Ethernet, TIPP2017, Beijing, – Balazs Voneki

11. Result charts iWARP iWARP RoCE
The iWARP results with a single thread run did not go beyond 90 Gbps (top-left figure).
Multiple threads needed to be tried to get closer to the theoretical maximum 100 Gbps (top-right figure).
The RoCE tests showed nice throughput numbers even with one thread, so in this case there was no need to test the multi-thread case.
Early results of RDMA optimizations on top of 100 Gbps Ethernet, TIPP2017, Beijing, – Balazs Voneki

12. Result charts iWARP iWARP RoCE By 1 single thread 2.5% CPU for
The iWARP results with a single thread run did not go beyond 90 Gbps (top-left figure).
Multiple threads needed to be tried to get closer to the theoretical maximum 100 Gbps (top-right figure).
The RoCE tests showed nice throughput numbers even with one thread, so in this case there was no need to test the multi-thread case.
By 1 single thread
2.5% CPU for
all message
sizes
Early results of RDMA optimizations on top of 100 Gbps Ethernet, TIPP2017, Beijing, – Balazs Voneki

13. Result charts UDP TCP RoCE By 1 single thread 30 Gbps 2.5% CPU 17 Gbps
The iWARP results with a single thread run did not go beyond 90 Gbps (top-left figure).
Multiple threads needed to be tried to get closer to the theoretical maximum 100 Gbps (top-right figure).
The RoCE tests showed nice throughput numbers even with one thread, so in this case there was no need to test the multi-thread case.
By 1 single thread
2.5% CPU for
all message
sizes
Early results of RDMA optimizations on top of 100 Gbps Ethernet, TIPP2017, Beijing, – Balazs Voneki

14. Result charts UDP TCP RoCE By 1 single thread 94.4 Gbps 98 Gbps
8.6% CPU
98 Gbps
18% CPU
30 Gbps
2.5% CPU
17 Gbps
2.5% CPU
UDP
TCP
RoCE
The iWARP results with a single thread run did not go beyond 90 Gbps (top-left figure).
Multiple threads needed to be tried to get closer to the theoretical maximum 100 Gbps (top-right figure).
The RoCE tests showed nice throughput numbers even with one thread, so in this case there was no need to test the multi-thread case.
By 1 single thread
2.5% CPU for
all message
sizes
Early results of RDMA optimizations on top of 100 Gbps Ethernet, TIPP2017, Beijing, – Balazs Voneki

15. MPI result charts with RoCE
Early results of RDMA optimizations on top of 100 Gbps Ethernet, TIPP2017, Beijing, – Balazs Voneki

16. Result charts 2 with RoCE
Purpose of heat maps:
To check stability
Early results of RDMA optimizations on top of 100 Gbps Ethernet, TIPP2017, Beijing, – Balazs Voneki

17. Summary Promising results Pure TCP/IP is inefficient
Zero-copy approach is needed
Need to understand why the bidirectional heat map is not homogeneous
Early results of RDMA optimizations on top of 100 Gbps Ethernet, TIPP2017, Beijing, – Balazs Voneki

18. Thank you!
Early results of RDMA optimizations on top of 100 Gbps Ethernet, TIPP2017, Beijing, – Balazs Voneki