1. iWARP Ethernet Key to Driving Ethernet into the Future Brian Hausauer Chief Architect NetEffect, Inc.

2. Agenda Situation overview Performance considerations NetworkingApplications New generation of adapters Performance discussion and demo Wrap up

3. Clustering iWARP Ethernet Clustering Myrinet, Quadrics, InfiniBand, etc. Storage iWARP Ethernet SAN Block Storage Fibre Channel networking storage clustering storage LAN iWARP Ethernet LAN Ethernet Data Center Evolution Separate Fabrics for Networking, Storage, and Clustering ▪ ▪ ▪ ▪ ▪ ▪ ▪ switch networking storage clustering networking Applications adapter networking storage clustering adapter ▪ ▪ ▪ ▪ ▪ ▪ ▪ switch NAS Users ▪ ▪ ▪ ▪ ▪ ▪ ▪ switch

4. Converged iWARP Ethernet SAN Single Adapter for All Traffic Converged Fabric for Networking, Storage, and Clustering Users Smaller footprint Lower complexity Higher bandwidth Lower power Lower heat dissipation NAS Server Blade iWARP Server Blade iWARP Server Blade iWARP Server Blade iWARP Server Blade iWARP Switch ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ switch networking storage clustering adapter Applications

5. device driver OS Networking Performance Barriers application I/O library user server software kernel TCP/IP software hardware standard Ethernet TCP/IP packet I/O cmd % CPU Overhead 100% application to OS context switches 40% transport processing 40% Intermediate buffer copies 20% context switch Packet Processing Intermediate Buffer Copies Command Context Switches adapter buffer app buffer OS buffer driver buffer I/O Adapter

6. device driver OS application I/O library user device driver OS NetEffect NE010 iWARP Ethernet server software kernel TCP/IP software hardware standard Ethernet TCP/IP packet adapter buffer app buffer OS buffer driver buffer I/O cmd % CPU Overhead 100% application to OS context switches 40% transport processing 40% Intermediate buffer copies 20% context switch TCP/IP app buffer application to OS context switches 40% Intermediate buffer copies 20% 60% 40% application to OS context switches 40% Transport (TCP) offload RDMA / DDP User-Level Direct Access/ OS Bypass Packet Processing Intermediate Buffer Copies Command Context Switches I/O Adapter standard Ethernet TCP/IP packet Eliminate Networking Performance Barriers With iWARP

7. Application Performance Barriers In Today’s Data Center Non-overlapped socket send() Usually means data is copied before transmit on wire On receive, transaction control info and data payloads are usually multiplexed on a single byte stream To avoid an additional buffer copy on receive, application often does not pre-post receive buffers

8. Application Performance Solutions in Tomorrow’s Data Center Windows already provides overlapped I/O to solve copy-on-transmit problem Elimination of copy-on-receive requires application to be RDMA-aware for typical transaction protocols

9. App performs socket send #2 and blocks OS builds TCP/IP data packets App performs socket send #1 and blocks NIC Tx OS receives TCP/IP data packets and builds ACK packets OS receives ACK packets and unblocks App NIC Rx NIC Tx NIC Rx Local Server Network Remote Server Time ApplicationBlocked! Legacy Sockets App Performance Barrier Non-Overlapped Socket send() OSes typically eliminate application blocking by copying application socket send data into kernel buffers

10. Local Server Network Remote Server Time OS builds TCP/IP data packets App performs Winsock2 Overlapped socket send #1 NIC Tx OS receives TCP/IP data packets and builds ACK packets OS receives ACK packets and notifies App of completion NIC Rx NIC Tx NIC Rx Enhanced Sockets App Performance Fix Winsock2 Overlapped Socket send() OS builds TCP/IP data packets App performs Winsock2 Overlapped socket send #2 NIC Tx OS receives TCP/IP data packets and builds ACK packets OS receives ACK packets and notifies App of completion NIC Rx NIC Tx NIC Rx ApplicationBlocked!

11. Application buffers in Host memory Data Payload d Ctrl Msg #4 Legacy Sockets App Performance Barrier No Pre-Posted Socket recv() Data Payload s+1 Ctrl Msg #3 Data Payload p Ctrl Msg #2 Data Payload s Ctrl Msg #1 // Pseudocode showing legacy sockets app receive algorithm while (1) { post socket recv() to obtain transaction control message; identify pre-allocated app buffer pertaining to received control message; post socket recv() to move transaction data payload into identified buffer; } p d s Transaction Protocols such as iSCSI multiplex control info and data payloads on a single byte stream

12. Application buffers in Host memory Data Payload d Ctrl Msg #4 RDMA Aware Sockets App Performance Fix Use Direct Data Placement (DDP) Intelligent NIC uses iWARP headers embedded in the packets to directly place data payloads in pre-allocated app buffers Eliminates software latency loop from legacy sockets apps Data Payload q+1 Ctrl Msg #3 Data Payload p Ctrl Msg #2 Data Payload q Ctrl Msg #1 p d q iWARP Receive Queue Preposted buffers for Control Messages

13. Networking Performance Continuum Application Characteristics Networking Offloads Legacy Sockets App Enhanced Sockets App Layer 2 traditional NIC only Legacy Sockets App Enhanced Sockets App RDMA-enabled NIC supporting WSD RDMA aware Sockets App RDMA-enabled NIC supporting RDMA Chimney Availability Now Future Windows Server release

14. Ethernet Adapters Are Evolving To Require... Networking offloads defined by RDMAC and IETF iWARP extensions to TCP/IP Transport (TCP) offload RDMA / DDP User-Level Direct Access/OS Bypass Ability to eliminate both networking and application performance barriers Simultaneous support for traditional sockets and RDMA-aware applications Industry standard h/w and s/w interfaces Performance > 1 million messages per second < 10% CPU utilization < 10us end-to-end application latency Scalability 100k’s of simultaneous connections Architecture that scales to multiple 10 Gb Ports NE010 10 Gb iWARP Ethernet Channel Adapter

15. iWARP Demonstration Enhanced sockets application running on iWARP hardware through Winsock Direct RDMA-enabled application running on iWARP hardware through iWARP Verbs emulating RDMA-aware sockets application adapter NE010 iWARP Ethernet adapter NE010 iWARP Ethernet

16. Network Application Performance Unidirectional B/W vs. Message Size NetEffect WSD Overlapped I/ONetEffect WSD Non-Overlapped I/ONetEffect RDMA-aware App Host Stack Overlapped I/OHost Stack Non-Overlapped I/O 0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 1101001000 Message Size (KB) Gb/s PCI-X Bus B/W Limit

17. Network Application CPU Utilization GBits per CPU GHz versus Message Size 0.00 10.00 20.00 30.00 40.00 50.00 60.00 70.00 80.00 90.00 1101001000 Message Size (KB) GBits per CPU GHz Host Stack Overlapped I/ONetEffect WSD Overlapped I/ONetEffect RDMA-aware App Conventional wisdom: Traditional NIC with Host Stack capable of 1 Gb per x86 CPU GHz

18. Takeaways iWARP Ethernet Channel Adapters Eliminate networking barriers Support Microsoft’s advanced APIs enabling application evolution for performance NetEffect iWARP Ethernet Channel Adapters Industry leading 10 GB Ethernet throughput, CPU utilization and latency Available now

19. Call To Action Deploy Winsock Direct with iWARP RDMA to boost performance of existing applications Plan for convergence of networking, storage and clustering enabled by 10 GB iWARP Ethernet Channel Adapters Develop RDMA-aware applications for optimal performance

20. Additional Resources Web Resources NetEffect: www.neteffect.com www.neteffect.com iWARP Consortium: www.iol.unh.edu/consortiums/iwarp/ www.iol.unh.edu/consortiums/iwarp/ Specs RDMA Consortium: www.rdmaconsortium.org www.rdmaconsortium.org IETF RDDP WG: www.ietf.org/html.charters/rddp- charter.html www.ietf.org/html.charters/rddp- charter.htmlwww.ietf.org/html.charters/rddp- charter.html White Papers Asynchronous Zero-copy Communication for Synchronous Sockets nowlab.cse.ohio-state.edu/publications/conf- papers/2006/balaji-cac06.pdf Contact info bh2006 @ neteffect.com