1. OFED 1.2 Status and Contents
April 07
Tziporet Koren 1

2. Agenda What is OFED? OFED Architecture and Components
OFED 1.2 New Features
OFED 1.2 Status
What’s next?

3. OFED - Open Fabrics Enterprise Distribution
Enterprise Working Group (EWG) within Open Fabrics Alliance (OFA)
Collaborative effort to test & release OFA software
Broader test participation
Multi-vendor interoperability
Ready for OS vendor adoption
Support for many distributions
Components – Kernel & User Space
Add-on components for vendors to differentiate above OFA
Reduces deployment complexity and cost

4. Agenda What is OFED? OFED Architecture and Components
OFED 1.2 New Features
OFED 1.2 Status
What’s next?

5. Apps & Access Methods for using OF Stack
OFA Linux Software Stack SA
Subnet Administrator
MAD
Management Datagram
SMA
Subnet Manager Agent
PMA
Performance Manager Agent
IPoIB
IP over InfiniBand
SDP
Sockets Direct Protocol
SRP
SCSI RDMA Protocol (Initiator)
iSER
iSCSI RDMA Protocol (Initiator)
RDS
Reliable Datagram Service
VNIC
Virtual NIC
UDAPL
User Direct Access Programming Lib
HCA
Host Channel Adapter
R-NIC
RDMA NIC
Application Level
IP Based
App
Access
Sockets
Based Access
Various
MPIs
Block
Storage
Access
Clustered
DB Access
Access to
File
Systems
Diag
Tools
Open SM
User Level MAD API
UDAPL
User APIs
OpenFabrics User Level Verbs & CMA / API
User Space
SDP Lib
Kernel Space
Upper Layer Protocol
VNIC
IPoIB
SDP
SRP
iSER
RDS
NFS-RDMA
RPC
Cluster
File Sys
Connection Manager
Abstraction (CMA)
Mid-Layer
SA Client
MAD
SMA
Connection Manager
Connection
Manager
Kernel bypass
Kernel bypass
OpenFabrics Kernel Level Verbs / API
Provider
Hardware
Specific Driver
Hardware Specific
Driver
Common
Apps & Access Methods for using OF Stack
Key
Hardware
InfiniBand HCA
iWARP R-NIC
InfiniBand
iWARP

6. OFED 1.2 Components OFA development Add on HCA/NIC Drivers
Bonding module
MPI Components:
MVAPICH
Open MPI
MVAPICH2
MPI tests: OSU benchmarks, Intel MPI benchmarks, Presta
HCA/NIC Drivers
Mellanox, QLogic, IBM, Chelsio
Core: verbs, MAD, SMA, CMA, SA cache
IPoIB
SDP
SRP, iSER
RDS
VNIC
UDAPL
OSM
Diagnostic tools
New in 1.2

7. Agenda What is OFED? OFED Architecture and Components
OFED 1.2 New Features
Kernel
High Availability
User Level
Management
iWARP
MPI
OFED 1.2 Status
What’s next?

8. Main New Features - Kernel
GA level for the EDC market:
SDP, RDS, High Availability, Storage (iSER and SRP)
Stability improvements
Performance improvements:
New - IPoIB Connected Mode (~1000 MB/Sec)
New - RDS for Oracle
SDP message BW:
10X for small messages
5X for medium messages
Scalability:
SDP memory consumption limit

9. High Availability - IPoIB
Fails over from one interface to another on carrier off
Two solutions:
User space: A script that detects carrier off/on events, reconfigures interfaces, sends rarp to notify remote side
Status: GA
Kernel module: Bonding
Covered in a separate talk
Status: Beta

10. High Availability - SRP
Failover between two ports/HCAs
SRP HA is built of two parts:
srp_daemon – discovery and sets up for all possible paths to SRP targets on the fabric
Multipath tool – switches to a different path when a path fails. Current version uses Device Mapper multi-path
Device Mapper (DM) driver from Linux kernel
The persistent binding and HA are provided by user space apps (dm-multipath & dm-multipathd)
Solution works for RedHat EL4 and SLES10
Status: Beta

11. High Availability – RDS & DSP
RDS implementation fully support HA
Requires IPoIB HA to work
SDP:
Does not support HA
Can be implemented for same HCA using APM
For multi-HCA support requires protocol change

12. Main New Features - User Level
libibverbs 1.1:
Fork support (requires apps change)
Better low-level driver handling, including multiple drivers linked in statically
Documentation: man pages
librdmacm (uCMA) 1.0:
Multicast joining from user space
UD support

13. Main New Features - Management
OpenSM:
Routing improvements
Performance improvement to min hop and up/down of over an order of magnitude
New fat-tree and LASH algorithms
SA optional record support “virtually” complete
IB router enablement
SA database dump/restore
Many diagnostic improvements since OFED 1.1
Covered in DoE tools talk
ibdiagui
GUI for ibdiagnet
Used at SC06

14. OFED 1.2 - iWARP Support Chelsio NIC supported
Verbs and CMA APIs are the same as Infiniband
ULPs that are supported:
MPI (mvapich2 tested)
uDAPL
Basic Testing:
mvapich2
NFS-RDMA
Status: Beta

15. Main New Features - MPI MPI implementations:
MVAPICH: version 0.9.9
Open MPI: version 1.2.1
MVAPICH2: version (New)
Common MPI setup sourcing:
Simple menu-driven interface to choose which MPI implementation to set as the default on a per-user and/or system-wide basis

16. Main New Features - MVAPICH
MVAPICH – Version 0.9.9
Improved message coalescing:
Reduction of per QP send queues for reduction in memory requirement
Increases the small message messaging rate significantly
Multi-core optimizations:
Optimized scalable shared memory design
Optimized, high-performance shared memory aware collective operations
Multi-port support for enabling user processes to bind to different IB ports for balanced communication performance
On-demand connection management using native IB UD support
Multi-path support for hot-spot avoidance in large scale clusters using LMC
Memory Hook Support provided by integration with ptmalloc2 library

17. Main New Features - Open MPI
Improvements to scalability of launching applications on large numbers of nodes
"Installdirs" functionality (install OMPI into one place and then be able to move it elsewhere; good for ISV's)
Support for fork() when using the OF libibverbs
Support for setting fixed limits on registered memory
Fixes for heterogeneous network environments (e.g., different number of IB ports on different hosts)

18. MPI - MVAPICH2 Includes most of the features of MVAPICH
Performance and scalability comparable to MVAPICH for two-sided communication
Added MPI-2 features (one-sided communication, collectives and datatype)
Integrated Multi-rail support
Multi-threading support (MPI_Thread_Multiple)
RDMACM support for InfiniBand and iWARP
Checkpoint/Restart support for application transparent systems-level fault tolerance

19. Agenda What is OFED? OFED Architecture and Components
OFED 1.2 New Features
OFED 1.2 Status
OFED 1.2 Release Status
OFED 1.2 System Matrix
Third Party Components Testing
What’s next?

20. OFED 1.2 Release Status Feature freeze: Feb 2 Alpha: Feb 14
Beta: Mar 14
RC1: Apr 4
RC2: Apr 18
RC3: May 3
Release: May 16

21. OFED 1.2 System Matrix CPU Arch: kernel.org: kernel 2.6.20 and 2.6.19
X86, x86_64, PPC64, ia64 (IB only)
kernel.org: kernel and
Novell:
SLES 9 SP3
SLES 10 (SP1)
Redhat:
RHEL 4 (up3 and up4)
RHEL 5
Free distros (Fedora, SuSE Pro, Ubuntu)
Basic testing only

22. Third Party Components Testing
Proprietary MPIs:
Intel
HP (over uDAPL)
Proprietary SMs:
Cisco, Voltaire, Qlogic
Storage Targets:
iSER: IP Store (FalconStore), Voltaire FC GW
SRP: Engenio, MTD2000, Areca-1220, DDN, Cisco GW

23. Agenda What is OFED? OFED Architecture and Components
OFED 1.2 New Features
OFED 1.2 Status
What’s next?

24. What’s Next? OFED 1.3 – Oct/Nov 07
Features that didn’t make it in 1.2:
Minimize integration effort into OS distribution
Definition immediately after 1.2 is out
QoS - collaborate with IBTA to align schedule of software delivery
IPoIB: NAPI
NFS over RDMA integration
Mellanox ConnectX IB HCA support
Including new features
Other features to be agreed upon by OFA and EWG at the conference
Effort into OS distribution, QoS - collaborate with IBTA to align schedule of software delivery with OFED 1.3 (make sure all end users understand what is the hold up here), Mellanox ConnectX IB HCA support, including new features (subject to discussion and approval by OFA and EWG - to be discussed and prelim plan put in place at this conference)

25. Summary OFED becomes the industry standard
OFED 1.2 for the EDC market:
Stability
Performance
High Availability
Scalability
OFED 1.2 for the HPC market:
Scalable for large clusters
Multi-core support
Multi-rail
Performance improvements
Successful collaboration between all participants

26. Thank You