InfiniBand support for Socket- based connection model by CM Arkady Kanevsky November 16, 2005 version 4

2 Work Items for InfiniBand CM extension Maximize Consumer-usable private data Standard IBTA CM REQ with private data format extensions –CM REQ provides IP address and port (analog) of the requestor, and IP address of the destination TCP Port of the destination is available from receiver Service ID –Use SID range to identify that private data is formatted Define mapping between TCP port space and IB Service IDs for RDMA aware ULPs.

3 CM REQ Message Private Data Format Source IP Address (31 – 0) Source IP Address (63 – 32) Source IP Address (95 – 64) Source IP Address (127 – 96) Source Port IP Version Consumer Private Data Destination IP Address (31 – 0) Destination IP Address (63 – 32) Destination IP Address (95 – 64) Destination IP Address (127 – 96) reserved

4 CM REQ Message Private Data Fields Version (4 bits) – protocol version IP Version (4 bits) –0100b: IPv4 –0110b: IPv6 –other values reserved reserved (8 bits) –set to 0 byte aligned Source Port (2 bytes) 16-byte aligned Source IP Address (16 bytes) –IP address of initiator 16-byte aligned Destination IP Address (16 bytes) –IP address of destination 16-byte aligned Consumer Private Data (56 bytes)

5 CM REQ Message Private Data formatting indicator SID range indicates that private data is formatted API or verbs that use IETF addressing indicate that IETF protocol # and port need to be mapped into IB SID

6 Mapping between TCP ports and IB Service IDs Use simple mapping –predefine upper 48 bits to form “RDMA aware ULPs service IDs” Use IBTA Service ID range –byte 1 – 0x00 –byte 2-4 – 0x00s –byte 5 – 0x01 – indicator that SID is in RDMA aware SID range –byte 6 - IETF Protocol number (8 bits) »TCP=0x06 (6), UDP=0x11 (17), SCTP= 0x83 (132) –byte 7-8 – protocol ports –1-to-1 map protocol ports to lower 16 bits of SID ensures that all well known ports are mapped the same way for each IETF protocol

Rationale and Discussions

8 Mapping Rationale - I Support for a ULP running over raw IB, IPoIB, and intermediate protocols (SDP) –SDP defines its own Service ID space and port mappings. Proposal shall not alter existing SDP protocol or its implementations. SDP ULPs are not RDMA aware. –IPoIB does not expose IB ports and all IP traffic goes over a UD IB over a single IB QP. All port multiplexing is done by IPoIB. ULP is not RDMA aware. –RDMA aware ULPs need port mapping.

9 Mapping Rationale - II What is RDMA aware ULP? –ULP that explicitly use RDMA semantic: memory registration, preposting recv buffers, RDMA operations and so on. RDMA transport independent –run on IB and iWARP without transport dependent code SDP ULPs are not RDMA aware –SDP provides port mapping and startup protocol SDP is transport dependent –ULPs do not use RDMA semantic

10 Alternatives Port mapping Use port mapper –designated Service IDs for SDP, RDMA aware ULPs. in-line port mapper for RDMA aware ULPs –Service on top of CM will redirect connection requests based on 5-tuple: IETF protocol, DST IP address and Port, SRC IP address and Port –designated Service ID for port mapper

11 Discussions - I Do we need include 0-bazed VA (ZB) into this proposal? iSER protocol does not advertise VA and relies implicitly on 0-based VA –This is ULP specific protocol/assumption and should be handled by the ULP –Outside the scope of this proposal

12 Discussions - II Do we need include support for Remote Send with Invalidate (SI) into this proposal? Rationale why it was remove for v3: –QP does not have an attribute for turning on or off SI support Hence CM can not convey this info on its own, nor take any action based on this field in CM messages –Remote Invalidate usage support request and response must come from ULP Different ULPs can take difference actions based on its ability to adapt to remote side usage of SI as well as IB support for it: –example 1: Responder refuses the connection if requestor ask for SI and responder either will not do it or its IB Provider does not support it. –example 2: Responder accepts the connection if requestor ask for SI but convey to the requestor whether or not it will use it –example 3: Responder refuses the connection if requestor does not ask for SI –example 4: Responder accepts the connection if requestor do not ask for SI but convey to the requestor whether or not it will use it Thus, it is ULP specific connection management and outside the scope of this proposal IBTA iSER protocol should define the protocol to use Consumer Private Data for SI. IETF iSER does not expose it Current IETF NFS-RDMA protocol suite does not use SI

13 Discussions - III Do we need to support for IETF Protocol number? IETF iWARP is only defined for TCP and SCTP Since IETF post spaces are independently managed based on protocol number it is better to support it to avoid issues in the future when RDMA will be defined/used over non-TCP protocol Protocol # is in SID – so no bandwidth savings by not supporting it Clean and complete IETF 5-tuple support

14 Discussions - IV Do we need to pass Destination Port? CM can do the mapping between Service IDs and ports to provide Destination Port to a Responder. –The conversion is an API issue is outside the scope of the spec The same is true for populating Private Data CM formatted fields. –IBTA does not define CM verbs.

15 Discussions - V The use of IP addresses reserved for privileged/kernel users Not an issue since CM REQ message Q- key is setup by CM appropriately. IBTA spec already covers it. –If packet is tampered on a wire then it is analogous to the Ethernet situation. Nothing special to be done. –provided by IB security model

16 Discussion VI 64 bit IP addresses alignment private data starts at 140 byte –IP addresses start at 144 byte –This is 8 byte align IP addresses are 8 byte aligned