iWARP Ethernet Key to Driving Ethernet into the Future Brian Hausauer Chief Architect NetEffect, Inc.

Agenda Situation overview Performance considerations NetworkingApplications New generation of adapters Performance discussion and demo Wrap up

Clustering iWARP Ethernet Clustering Myrinet, Quadrics, InfiniBand, etc. Storage iWARP Ethernet SAN Block Storage Fibre Channel networking storage clustering storage LAN iWARP Ethernet LAN Ethernet Data Center Evolution Separate Fabrics for Networking, Storage, and Clustering ▪ ▪ ▪ ▪ ▪ ▪ ▪ switch networking storage clustering networking Applications adapter networking storage clustering adapter ▪ ▪ ▪ ▪ ▪ ▪ ▪ switch NAS Users ▪ ▪ ▪ ▪ ▪ ▪ ▪ switch

Converged iWARP Ethernet SAN Single Adapter for All Traffic Converged Fabric for Networking, Storage, and Clustering Users Smaller footprint Lower complexity Higher bandwidth Lower power Lower heat dissipation NAS Server Blade iWARP Server Blade iWARP Server Blade iWARP Server Blade iWARP Server Blade iWARP Switch ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ switch networking storage clustering adapter Applications

device driver OS Networking Performance Barriers application I/O library user server software kernel TCP/IP software hardware standard Ethernet TCP/IP packet I/O cmd % CPU Overhead 100% application to OS context switches 40% transport processing 40% Intermediate buffer copies 20% context switch Packet Processing Intermediate Buffer Copies Command Context Switches adapter buffer app buffer OS buffer driver buffer I/O Adapter

device driver OS application I/O library user device driver OS NetEffect NE010 iWARP Ethernet server software kernel TCP/IP software hardware standard Ethernet TCP/IP packet adapter buffer app buffer OS buffer driver buffer I/O cmd % CPU Overhead 100% application to OS context switches 40% transport processing 40% Intermediate buffer copies 20% context switch TCP/IP app buffer application to OS context switches 40% Intermediate buffer copies 20% 60% 40% application to OS context switches 40% Transport (TCP) offload RDMA / DDP User-Level Direct Access/ OS Bypass Packet Processing Intermediate Buffer Copies Command Context Switches I/O Adapter standard Ethernet TCP/IP packet Eliminate Networking Performance Barriers With iWARP

Application Performance Barriers In Today’s Data Center Non-overlapped socket send() Usually means data is copied before transmit on wire On receive, transaction control info and data payloads are usually multiplexed on a single byte stream To avoid an additional buffer copy on receive, application often does not pre-post receive buffers

Application Performance Solutions in Tomorrow’s Data Center Windows already provides overlapped I/O to solve copy-on-transmit problem Elimination of copy-on-receive requires application to be RDMA-aware for typical transaction protocols

App performs socket send #2 and blocks OS builds TCP/IP data packets App performs socket send #1 and blocks NIC Tx OS receives TCP/IP data packets and builds ACK packets OS receives ACK packets and unblocks App NIC Rx NIC Tx NIC Rx Local Server Network Remote Server Time ApplicationBlocked! Legacy Sockets App Performance Barrier Non-Overlapped Socket send() OSes typically eliminate application blocking by copying application socket send data into kernel buffers

Local Server Network Remote Server Time OS builds TCP/IP data packets App performs Winsock2 Overlapped socket send #1 NIC Tx OS receives TCP/IP data packets and builds ACK packets OS receives ACK packets and notifies App of completion NIC Rx NIC Tx NIC Rx Enhanced Sockets App Performance Fix Winsock2 Overlapped Socket send() OS builds TCP/IP data packets App performs Winsock2 Overlapped socket send #2 NIC Tx OS receives TCP/IP data packets and builds ACK packets OS receives ACK packets and notifies App of completion NIC Rx NIC Tx NIC Rx ApplicationBlocked!

Application buffers in Host memory Data Payload d Ctrl Msg #4 Legacy Sockets App Performance Barrier No Pre-Posted Socket recv() Data Payload s+1 Ctrl Msg #3 Data Payload p Ctrl Msg #2 Data Payload s Ctrl Msg #1 // Pseudocode showing legacy sockets app receive algorithm while (1) { post socket recv() to obtain transaction control message; identify pre-allocated app buffer pertaining to received control message; post socket recv() to move transaction data payload into identified buffer; } p d s Transaction Protocols such as iSCSI multiplex control info and data payloads on a single byte stream

Application buffers in Host memory Data Payload d Ctrl Msg #4 RDMA Aware Sockets App Performance Fix Use Direct Data Placement (DDP) Intelligent NIC uses iWARP headers embedded in the packets to directly place data payloads in pre-allocated app buffers Eliminates software latency loop from legacy sockets apps Data Payload q+1 Ctrl Msg #3 Data Payload p Ctrl Msg #2 Data Payload q Ctrl Msg #1 p d q iWARP Receive Queue Preposted buffers for Control Messages

Networking Performance Continuum Application Characteristics Networking Offloads Legacy Sockets App Enhanced Sockets App Layer 2 traditional NIC only Legacy Sockets App Enhanced Sockets App RDMA-enabled NIC supporting WSD RDMA aware Sockets App RDMA-enabled NIC supporting RDMA Chimney Availability Now Future Windows Server release

Ethernet Adapters Are Evolving To Require... Networking offloads defined by RDMAC and IETF iWARP extensions to TCP/IP Transport (TCP) offload RDMA / DDP User-Level Direct Access/OS Bypass Ability to eliminate both networking and application performance barriers Simultaneous support for traditional sockets and RDMA-aware applications Industry standard h/w and s/w interfaces Performance > 1 million messages per second < 10% CPU utilization < 10us end-to-end application latency Scalability 100k’s of simultaneous connections Architecture that scales to multiple 10 Gb Ports NE Gb iWARP Ethernet Channel Adapter

iWARP Demonstration Enhanced sockets application running on iWARP hardware through Winsock Direct RDMA-enabled application running on iWARP hardware through iWARP Verbs emulating RDMA-aware sockets application adapter NE010 iWARP Ethernet adapter NE010 iWARP Ethernet

Network Application Performance Unidirectional B/W vs. Message Size NetEffect WSD Overlapped I/ONetEffect WSD Non-Overlapped I/ONetEffect RDMA-aware App Host Stack Overlapped I/OHost Stack Non-Overlapped I/O Message Size (KB) Gb/s PCI-X Bus B/W Limit

Network Application CPU Utilization GBits per CPU GHz versus Message Size Message Size (KB) GBits per CPU GHz Host Stack Overlapped I/ONetEffect WSD Overlapped I/ONetEffect RDMA-aware App Conventional wisdom: Traditional NIC with Host Stack capable of 1 Gb per x86 CPU GHz

Takeaways iWARP Ethernet Channel Adapters Eliminate networking barriers Support Microsoft’s advanced APIs enabling application evolution for performance NetEffect iWARP Ethernet Channel Adapters Industry leading 10 GB Ethernet throughput, CPU utilization and latency Available now

Call To Action Deploy Winsock Direct with iWARP RDMA to boost performance of existing applications Plan for convergence of networking, storage and clustering enabled by 10 GB iWARP Ethernet Channel Adapters Develop RDMA-aware applications for optimal performance

Additional Resources Web Resources NetEffect: iWARP Consortium: Specs RDMA Consortium: IETF RDDP WG: charter.html charter.htmlwww.ietf.org/html.charters/rddp- charter.html White Papers Asynchronous Zero-copy Communication for Synchronous Sockets nowlab.cse.ohio-state.edu/publications/conf- papers/2006/balaji-cac06.pdf Contact info neteffect.com