Open Fabrics BOF Supercomputing 2008 Tziporet Koren, Gilad Shainer, Yiftah Shahar, Bob Woodruff, Betsy Zeller Rev. 0.9

2 Agenda  Introduction – 5 minutes  OFA objectives and goals  Open Fabrics Linux Update (15 – minutes)  OFED 1.3, 1.3.1, 1.4 releases  OFED 1.5 plans and roadmap  Open Fabrics Windows Update (15 – minutes)  WinOF 2.0 release  WinOF 2.1 plans and roadmap  Open Discussion – 25 minutes

3 Open Fabrics BOF - Introduction  Open Fabrics “The mission of the OpenFabrics Alliance (OFA) is to develop, distribute and promote a unified, transport-independent, open-source software stack for RDMA-capable fabrics and networks, including InfiniBand and Ethernet “  Support for both Microsoft Windows and Linux  Development happens using a typical open source development model Developer’s Working Group code developed and reviewed using lists  Validation and release by the Enterprise Working Group and the Windows Working Group OFED – for Linux, WinOF – for Windows,

4 Open Fabrics BOF - Introduction  Open Fabrics – working groups (cont.)  XWG – Executive Working group overall steering committee,  IWG – Interoperability Working Group Open Fabrics logo program,  LWG – Legal Working Group handles code licensing and other legal issues,  MWG – Marketing Working Group Promotes the work of the alliance and recruits new members  UWG – Users Working Group define end user requirements

5 Open Fabrics BOF - Introduction  Open Fabrics – End User Lists         (general mail list for all end users)

6 Agenda  Introduction – 5 minutes  OFA objectives and goals  Open Fabrics Linux Update (15 – minutes)  OFED 1.3, 1.3.1, 1.4 releases  OFED 1.5 plans and roadmap  Open Fabrics Windows Update (15 – minutes)  WinOF 2.0 release  WinOF 2.1 plans and roadmap  Open Discussion – 25 minutes

7 Linux: OFED Components  HCA/NIC Drivers  IB: IBM, Mellanox, QLogic  iWARP: Chelsio, NetEffect (now Intel)  Core: Verbs, mad, SMA, CMA, SA cache  IPoIB  SDP  SRP and SRP Target  iSER  RDS  Qlogic_VNIC  uDAPL  OSM  Diagnostic tools OFED 1.4 additions:  iSER Target  NFS-RDMA  Bonding module  Open iSCSI  MPI Components  MVAPICH  Open MPI  MVAPICH2  Benchmark tests  Proprietary MPIs: Intel, HP, Platform mpi  Proprietary SMs: Cisco, Voltaire, Qlogic OFA DevelopmentAdd on Tested with

8 Update from Sonoma ’08 Session Progress:  Developers try to get SW into kernel first. If not, submit to next available kernel, and pull separately into OFE.  Warnings greatly reduced, also resulting in potential bugs found  Interoperability “pre-test” event was run on OFED 1.4 RC2, allowing vendors to make bug fixes before GA  Identified and published list of libraries which might “change without notice”  was successful, and reasonably low impact on community. Met goal of not changing kernel, and accepting only bug fixes.  Evaluating and tracking buglist carefully and in detail. Community members are improving at tracking their bugs.  Acquired and published list of what each vendor plans to test Still improving:  Enabling distros  Evaluating impact of changes to existing protocols  Get community “on-board” with feature freeze date. Large changes still coming in late for some features. However, on the plus side, process enabled changing on new feature to “optional” because it wasn’t quite ready.  Reviewing copyrights and licenses  As we provide OFED diagnostics on Windows, we need to re-think having libraries change without notice.

9 OFED 1.3 and  OFED 1.3 released on Feb  Improved performance and scalability  Quality of Service (QOS) support  New OpenSM routing algorithms  IPoIB connect mode – improved performance  MPI enhancements to improve scalability and reduce memory footprint  Enhanced diagnostic tools to improve network management  Support for additional hardware, added additional iWarp NIC drivers  Expanded interoperability with Storage and Networking hardware  uDAPL 2.0  OFED release on June  Added support for RedHat EL 5.2 and SLES 10 SP2  fixed several critical bugs  Distro integration:  Red Hat AS 4.7 and RHEL 5.2, SLES10 SP2  Used in Intel ® Cluster Ready Solutions

10 OFED 1.4  General Info  Released in Nov 2008  Used in the Interoperability event in Nov 2008  Features:  Kernel base  VPI support: Eth and IB for ConnectX  NFS-RDMA – as technology preview  ISer Target  New BMME verbs (fast memory thru send queue (FRWR); Local invalidate send work requests; Read with invalidate)  IPoIB – LRO  RDS – GA with RDMA API  SDP – GA  uDAPL (socket CM for scalability; UD extensions)

11 OFED 1.4 (cont.)  Features (cont)  OpenSM: Cached routing; APM - disjoint paths; Path balancing for LMC + console diagnostics; OpenSM configuration unification; MGID to MLID mapping for IPv6 SNM; Chained routing engines; IBA additions; Failover/Handover improvements  Congestion Control in ibutils  MPI: MVAPICH 1.1; MVAPICH2 1.2; OpenMPI  Likely Distro Integration  Red Hat EL 5.4, SLES 11, SLERT11  Used in Intel ® Cluster Ready Solutions

12 OFED 1.4 OS Matrix  kernel.org: kernel and  Novell  SLES 10  SLES 10 SP1 (up1)  SLES 10 SP2  Redhat  RHEL 4 (up4, up5, up6, up7)  RHEL 5 (up1, up2)  OEL  OEL 5  Free distros (with limited QA):  Open SuSE 10.3  Fedora Core 9  Ubuntu 6.06 (with RPM package installed)  CentOS 5.2 * new for OFED 1.4 in bold

13 OFED 1.5 Plans  Preliminary Schedule  Feature Freeze: 3/20/09  Alpha Release: 3/20/09  Beta Release: 4/20/09  RC1: 5/5/09  RC2-RCx: About every 2 weeks as needed  Release: June 2009  Features:  Kernel.org: and  Multiple Event Queues to support Multi-core CPUs  NFS/RDMA – GA  RDS support for iWarp  OpenMPI 1.3  Add support/backports for RedHat EL 5.3 and EL 4.8  Support for Mellanox vNIC (EoIB) and FCoIB with BridgeX device  more TBD…

14 OFED 1.5 OS Matrix  kernel.org: kernel and  Novell  SLES 10  SLES 10 SP1 (up1)  SLES 10 SP2  SLES 11  Redhat  RHEL 4 ( up4, up5, up6, up7, u8)  RHEL5 no updates,  RHEL 5 (,up1, up2, u3),  OEL  OEL 5  Free distros (with limited QA):  Open SuSE 10.3  Fedora Core 9  Ubuntu 6.06 (with RPM package installed)  CentOS 5.2 new for OFED 1.5 in bold drop support for items in blue

15 Agenda  Introduction – 5 minutes  OFA objectives and goals  Open Fabrics Linux Update (15 – minutes)  OFED 1.3, 1.3.1, 1.4 releases  OFED 1.5 plans and roadmap  Open Fabrics Windows Update (15 – minutes)  WinOF 2.0 release  WinOF 2.1 plans and roadmap  Open Discussion – 25 minutes

16 Windows OpenFabrics (WinOF)  Collaborative effort to develop, test and release OFA software for Windows  Components – Kernel and User Space  Broader test participation  Add-on components for vendors to differentiate above WinOF

17 Supported Platforms  Architectures  x86, x86_64, IA64  Operating systems  Windows XP 32&64  Windows Server 2003  Windows Cluster Compute Server 2003  Windows Vista  Windows Server 2008  Windows HPC Server 2008  WHQL’ed  Windows Server 2003  Windows Cluster Compute Server 2003

18 WinOF Software Stack WinOF

19 WinOF Maintainers  Release coordinators - Stan Smith (Intel), Ishai Rabinovitz (Mellanox)  IBAL, HCA drivers - Leonid Keller (Mellanox)  NetworkDirect – Leonid Keller (Mellanox)  SRP – Leonid Keller (Mellanox) & Eleanor Witiak (QLogic)  IPoIB – Tzachi Dar (Mellanox) & Anh Duong (QLogic)  WSD – Tzachi Dar (Mellanox)  OpenSM – Yevgeny Kliteynik (Mellanox)  DAPL/DAT, Installer – Stan Smith (Intel)  VNIC – Alex Estrin (QLogic)  Winverbs, WinMad, OFED compat libraries – Sean Hefty (Intel)  Qualification sites  Intel, Mellanox, Microsoft, QLogic, Voltaire and others

20 WinOF 1.1 Release  HCA Drivers  Access Layer (IBAL)  NDIS 5.1 (IPoIB)  WSD provider *  SRP initiator **  uDAPL  OpenSM  VNIC  MSI installer  Tools  Verbs perf Benchmarks  MPI Components  Microsoft  Intel  HP  SDP  Cluster management ComponentsADD-ONs Components supported by individual vendors may vary * Not available on Windows XP ** Not available on Windows XP 32 bit

21 WinOF 2.0 Release Add-on Features  Server 2008 & Vista support  WinVerbs framework in place  Will enable RDMA over InfiniBand and Ethernet  Supports OFED libibverbs APIs.  Network Direct Interface support  Improving performance for Microsoft MPI based applications (compared to Windows CCS 2003)  Performance tuning  IPoIB partitioning support  Checked/debug drivers available

22 Futures  IPoIB Connected mode  Higher performance capabilities for TCP/UDP over InfiniBand  IPoIB enhanced error logging  WinVerbs fully supported  Support OFED librdmacm  InfiniBand diagnostic tools  InfiniBand boot device support  All InfiniBand-based HCAs supported

23 How to Participate?  Developing code  Sending patches and comments to the mailing list  Doing QA  Opening bugs in Bugzilla (  When opening a new bug you can choose OpenFabrics WindowsOpenFabrics Windows

24 Agenda  Introduction – 5 minutes  OFA objectives and goals  Open Fabrics Linux Update (15 – minutes)  OFED 1.3, 1.3.1, 1.4 releases  OFED 1.5 plans and roadmap  Open Fabrics Windows Update (15 – minutes)  WinOF 2.0 release  WinOF 2.1 plans and roadmap  Open Discussion – 25 minutes