Copyright © 2007 InfiniBand ® Trade Association. Other names and brands are properties of their respective owners. IB Cross-Subnet Communication OpenFabrics Alliance Workshop May 1 st, 2007

Copyright © 2007 InfiniBand ® Trade Association. Other names and brands are properties of their respective owners. Why IB Routing? l Scalability l Local address space reuse l Containment of failures and topology changes l Confinement of fabric management scope to subnets H1H1 H2H2 HMHM … Subnet M SM/SA M H1H1 H2H2 HNHN … Subnet N SM/SA N H1H1 H2H2 HQHQ … Subnet Q SM/SA Q Router-3 Router-1 Router-2 H1H1 H2H2 HNHN … Subnet P SM/SA P

Copyright © 2007 InfiniBand ® Trade Association. Other names and brands are properties of their respective owners. Introduction and Agenda l IBTA Working on IB Cross-Subnet Communication Spec – Past Work l Agenda – Scope – Guidelines – Problem Partitioning – Advanced Topics

Copyright © 2007 InfiniBand ® Trade Association. Other names and brands are properties of their respective owners. Scope and Phased Approach l Limiting Scope via.. – Topology Restrictions – Intermediate Solution Simplifications l e.g. Static Routing Tables l Market Guidance – OFA Feedback Requested H1H1 H2H2 HMHM … Subnet M SM/SA M H1H1 H2H2 HNHN … Subnet N SM/SA N H1H1 H2H2 HQHQ … Subnet Q SM/SA Q Router-3 Router-1 Router-2 H1H1 H2H2 HNHN … Subnet P SM/SA P Router-1

Copyright © 2007 InfiniBand ® Trade Association. Other names and brands are properties of their respective owners. IB Routing Guidelines 5 l Layered Architecture l Leverage from the IP world whenever possible l SM is a Subnet Local Entity l Preserve Existing IB Management Infrastructure l Interoperability H1H1 H2H2 HMHM … SM/SA M H1H1 H2H2 HNHN … SM/SA N Router Switch

Copyright © 2007 InfiniBand ® Trade Association. Other names and brands are properties of their respective owners. l Assume Network is Setup and Configured l Connection Setup – Endnode Needs l GID for destination (DGID) l Local Subnet GID (next hop for DGID) l Local Subnet Path Parameters for next hop GID (DLID, SL) l Global Path Parameters (P_Key, MTU, Static Rate, Q_Key) l End to End CM – Router l No special involvement. Cross-subnet CM packets are treated just like regular packets A Day in the Life of Cross-Subnet IB Traffic

Copyright © 2007 InfiniBand ® Trade Association. Other names and brands are properties of their respective owners. A Day in the Life… (cont’d) l Mainstream Operation – Endnode: No changes from intra-subnet case l Address Vector is agnostic to whether parameters are local or global – Router: l Receive Packet l Strip LRH l Compute next hop GID (from packet DGID) l Obtain Local Subnet Path Parameters for next hop GID l Append new LRH l Transmit Packet

Copyright © 2007 InfiniBand ® Trade Association. Other names and brands are properties of their respective owners. Problem Partitioning l Global Network Setup and Management – Global Parameters Management – Establishing Routes – Monitoring, Statistics, Debugging l Connection Setup – Global Name Resolution – Global Path Parameters Discovery – Cross Subnet CM l Mainstream Data Transfer – Routing Decision – L2 Address Resolution – Cross Subnet IB Transport Protocol

Copyright © 2007 InfiniBand ® Trade Association. Other names and brands are properties of their respective owners. Global Network Setup and Management l Global Parameters Management – MTU, P_Key, Q_Key, Static Rate l Establishing Routes – Deadlock Avoidance – Routing Protocol l Including Inter-router messaging format and protocol l …or Static Routing l Monitoring, Statistics, Debugging

Copyright © 2007 InfiniBand ® Trade Association. Other names and brands are properties of their respective owners. Global Name Resolution l Goal: Map Service Name to GID l Impacts – Routers l None – Endnode l IB DNS Query – Management Infrastructure l Name Resolution Service

Copyright © 2007 InfiniBand ® Trade Association. Other names and brands are properties of their respective owners. Global Path Parameters Discovery l Goal: Obtain Global Path Parameters for Inter-Subnet Traffic – Path MTU, Static Rate, P_Key, Q_Key l Some Possible Approaches – Global Manager Query l Requires GM Sync with Routing Protocol – Probed Discovery l Limits Runtime Global Policing l Orthogonal to whether global or distributed l Impacts – Router l GMQ: N/A l PD: participates in the protocol – Endnode l GMQ: Query GM l PD: initiates pre-CM probe – Management Infrastructure l GMQ: Responds to Queries l PD: Setup only

Copyright © 2007 InfiniBand ® Trade Association. Other names and brands are properties of their respective owners. Routing Decision l Goal: Given a DGID obtain next hop GID – Amazingly similar to IPv6 routing function l Impacts – Router l Routing Table Lookup – Endnode l Routing Table Lookup or Query External Agent – Management Infrastructure l None

Copyright © 2007 InfiniBand ® Trade Association. Other names and brands are properties of their respective owners. L2 Address Resolution l Goal: Map Subnet Local GID to IB L2 Address l Impacts – Router l PathQuery to the SA l Caching – Endnode l PathQuery to the SA – Management Infrastructure l None

Copyright © 2007 InfiniBand ® Trade Association. Other names and brands are properties of their respective owners. Advanced Topics l Multicast – Setup and Management l Group Membership Protocol – Mainstream Operation l Packet Replication l Global QoS l Load Balancing l High Availability l Congestion Control l Fault Containment

Copyright © 2007 InfiniBand ® Trade Association. Other names and brands are properties of their respective owners. Call to Action l IBTA LWG Charter (joint work with MgtWG) l Plan – Step 0: Scoping – Step 1: Near Time Solution – Step 2: Full Blown Solution l Join the IBTA to Participate in the Definition –