OFED 1.2 Status and Contents April 07 Tziporet Koren 1

Agenda What is OFED? OFED Architecture and Components OFED 1.2 New Features OFED 1.2 Status What’s next?

OFED - Open Fabrics Enterprise Distribution Enterprise Working Group (EWG) within Open Fabrics Alliance (OFA) Collaborative effort to test & release OFA software Broader test participation Multi-vendor interoperability Ready for OS vendor adoption Support for many distributions Components – Kernel & User Space Add-on components for vendors to differentiate above OFA Reduces deployment complexity and cost

Agenda What is OFED? OFED Architecture and Components OFED 1.2 New Features OFED 1.2 Status What’s next?

Apps & Access Methods for using OF Stack OFA Linux Software Stack SA Subnet Administrator MAD Management Datagram SMA Subnet Manager Agent PMA Performance Manager Agent IPoIB IP over InfiniBand SDP Sockets Direct Protocol SRP SCSI RDMA Protocol (Initiator) iSER iSCSI RDMA Protocol (Initiator) RDS Reliable Datagram Service VNIC Virtual NIC UDAPL User Direct Access Programming Lib HCA Host Channel Adapter R-NIC RDMA NIC Application Level IP Based App Access Sockets Based Access Various MPIs Block Storage Access Clustered DB Access Access to File Systems Diag Tools Open SM User Level MAD API UDAPL User APIs OpenFabrics User Level Verbs & CMA / API User Space SDP Lib Kernel Space Upper Layer Protocol VNIC IPoIB SDP SRP iSER RDS NFS-RDMA RPC Cluster File Sys Connection Manager Abstraction (CMA) Mid-Layer SA Client MAD SMA Connection Manager Connection Manager Kernel bypass Kernel bypass OpenFabrics Kernel Level Verbs / API Provider Hardware Specific Driver Hardware Specific Driver Common Apps & Access Methods for using OF Stack Key Hardware InfiniBand HCA iWARP R-NIC InfiniBand iWARP

OFED 1.2 Components OFA development Add on HCA/NIC Drivers Bonding module MPI Components: MVAPICH Open MPI MVAPICH2 MPI tests: OSU benchmarks, Intel MPI benchmarks, Presta HCA/NIC Drivers Mellanox, QLogic, IBM, Chelsio Core: verbs, MAD, SMA, CMA, SA cache IPoIB SDP SRP, iSER RDS VNIC UDAPL OSM Diagnostic tools New in 1.2

Agenda What is OFED? OFED Architecture and Components OFED 1.2 New Features Kernel High Availability User Level Management iWARP MPI OFED 1.2 Status What’s next?

Main New Features - Kernel GA level for the EDC market: SDP, RDS, High Availability, Storage (iSER and SRP) Stability improvements Performance improvements: New - IPoIB Connected Mode (~1000 MB/Sec) New - RDS for Oracle SDP message BW: 10X for small messages 5X for medium messages Scalability: SDP memory consumption limit

High Availability - IPoIB Fails over from one interface to another on carrier off Two solutions: User space: A script that detects carrier off/on events, reconfigures interfaces, sends rarp to notify remote side Status: GA Kernel module: Bonding Covered in a separate talk Status: Beta

High Availability - SRP Failover between two ports/HCAs SRP HA is built of two parts: srp_daemon – discovery and sets up for all possible paths to SRP targets on the fabric Multipath tool – switches to a different path when a path fails. Current version uses Device Mapper multi-path Device Mapper (DM) driver from Linux kernel The persistent binding and HA are provided by user space apps (dm-multipath & dm-multipathd) Solution works for RedHat EL4 and SLES10 Status: Beta

High Availability – RDS & DSP RDS implementation fully support HA Requires IPoIB HA to work SDP: Does not support HA Can be implemented for same HCA using APM For multi-HCA support requires protocol change

Main New Features - User Level libibverbs 1.1: Fork support (requires apps change) Better low-level driver handling, including multiple drivers linked in statically Documentation: man pages librdmacm (uCMA) 1.0: Multicast joining from user space UD support

Main New Features - Management OpenSM: Routing improvements Performance improvement to min hop and up/down of over an order of magnitude New fat-tree and LASH algorithms SA optional record support “virtually” complete IB router enablement SA database dump/restore Many diagnostic improvements since OFED 1.1 Covered in DoE tools talk ibdiagui GUI for ibdiagnet Used at SC06

OFED 1.2 - iWARP Support Chelsio NIC supported Verbs and CMA APIs are the same as Infiniband ULPs that are supported: MPI (mvapich2 tested) uDAPL Basic Testing: mvapich2 NFS-RDMA Status: Beta

Main New Features - MPI MPI implementations: MVAPICH: version 0.9.9 Open MPI: version 1.2.1 MVAPICH2: version 0.9.8 (New) Common MPI setup sourcing: Simple menu-driven interface to choose which MPI implementation to set as the default on a per-user and/or system-wide basis

Main New Features - MVAPICH MVAPICH – Version 0.9.9 Improved message coalescing: Reduction of per QP send queues for reduction in memory requirement Increases the small message messaging rate significantly Multi-core optimizations: Optimized scalable shared memory design Optimized, high-performance shared memory aware collective operations Multi-port support for enabling user processes to bind to different IB ports for balanced communication performance On-demand connection management using native IB UD support Multi-path support for hot-spot avoidance in large scale clusters using LMC Memory Hook Support provided by integration with ptmalloc2 library

Main New Features - Open MPI Improvements to scalability of launching applications on large numbers of nodes "Installdirs" functionality (install OMPI into one place and then be able to move it elsewhere; good for ISV's) Support for fork() when using the OF libibverbs Support for setting fixed limits on registered memory Fixes for heterogeneous network environments (e.g., different number of IB ports on different hosts)

MPI - MVAPICH2 Includes most of the features of MVAPICH Performance and scalability comparable to MVAPICH for two-sided communication Added MPI-2 features (one-sided communication, collectives and datatype) Integrated Multi-rail support Multi-threading support (MPI_Thread_Multiple) RDMACM support for InfiniBand and iWARP Checkpoint/Restart support for application transparent systems-level fault tolerance

Agenda What is OFED? OFED Architecture and Components OFED 1.2 New Features OFED 1.2 Status OFED 1.2 Release Status OFED 1.2 System Matrix Third Party Components Testing What’s next?

OFED 1.2 Release Status Feature freeze: Feb 2 Alpha: Feb 14 Beta: Mar 14 RC1: Apr 4 RC2: Apr 18 RC3: May 3 Release: May 16

OFED 1.2 System Matrix CPU Arch: kernel.org: kernel 2.6.20 and 2.6.19 X86, x86_64, PPC64, ia64 (IB only) kernel.org: kernel 2.6.20 and 2.6.19 Novell: SLES 9 SP3 SLES 10 (SP1) Redhat: RHEL 4 (up3 and up4) RHEL 5 Free distros (Fedora, SuSE Pro, Ubuntu) Basic testing only

Third Party Components Testing Proprietary MPIs: Intel HP (over uDAPL) Proprietary SMs: Cisco, Voltaire, Qlogic Storage Targets: iSER: IP Store (FalconStore), Voltaire FC GW SRP: Engenio, MTD2000, Areca-1220, DDN, Cisco GW

Agenda What is OFED? OFED Architecture and Components OFED 1.2 New Features OFED 1.2 Status What’s next?

What’s Next? OFED 1.3 – Oct/Nov 07 Features that didn’t make it in 1.2: Minimize integration effort into OS distribution Definition immediately after 1.2 is out QoS - collaborate with IBTA to align schedule of software delivery IPoIB: NAPI NFS over RDMA integration Mellanox ConnectX IB HCA support Including new features Other features to be agreed upon by OFA and EWG at the conference Effort into OS distribution, QoS - collaborate with IBTA to align schedule of software delivery with OFED 1.3 (make sure all end users understand what is the hold up here), Mellanox ConnectX IB HCA support, including new features (subject to discussion and approval by OFA and EWG - to be discussed and prelim plan put in place at this conference)

Summary OFED becomes the industry standard OFED 1.2 for the EDC market: Stability Performance High Availability Scalability OFED 1.2 for the HPC market: Scalable for large clusters Multi-core support Multi-rail Performance improvements Successful collaboration between all participants

Thank You