August 20, 2006 Delivering Low Latency With IB SUSE ® Linux Enterprise Real Time Moiz Kohari Suse Labs Director Real-Time Systems

© Novell Inc. All rights reserved 2 Agenda Why Real-Time What Is Real-Time Benefits IB – End To End QoS Future Development/Roadmap

© Novell Inc. All rights reserved 3 Why Companies Want Real Time Faster, predictable data delivery Eliminate data latency Applications that must always respond in a finite period of time – Market data distribution, algorithmic trading, process control, data acquisition... Some distributed clusters benefit from highly accurate time synchronization, low-latency, and deterministic I/O Some applications, like Visualization, Simulation and VOD, need consistent processing times for smooth graphics displays

© Novell Inc. All rights reserved 4 What is SUSE ® Linux Enterprise Real Time? An enriched distribution of the SUSE Linux kernel from Novell ® that provides guaranteed performance in time- critical environments. An industry-standard real-time version of Linux for x86 architectures providing the following key features: – CPU Shielding: Dedicated resources (such as cpu, memory, fabric) for high priority processes – Kernel Preemption: Fast response to external events, optimized kernel critical sections (aggressive lock breaking) – Priority Inheritance: Solves priority inversion problem for mission critical processes – Frequency Based Scheduling: Mechanism for coordinating multiple execution threads within predefined CPU cycles

© Novell Inc. All rights reserved 5 When Is Real-Time The Appropriate Platform? Use standard kernel for throughput sensitive workloads and real-time kernel for latency sensitive workloads (for example): When fair scheduling algorithms are required across compute resources (educational deployments – time sharing) Web farms serving FIFO based requests Enterprise mail servers When certain mission/business critical jobs demand the fastest response time possible: Missile Defence Systems Air traffic control Trading systems Process control applications Standard Kernel: Real-Time Kernel:

© Novell Inc. All rights reserved 6 What is Meant by Interrupt Response Time? Standard Linux Kernel Measured 44,759,417 samples Max latency: 92.3 ms samples < 0.1ms ( %) samples < 0.2ms ( %) samples < 1.0ms ( %) samples < 10.ms ( %) samples < 30.ms ( %) 10761samples < 60.ms ( %) 166 samples < 90.ms ( %) 1 samples < 100ms (100%) Interrupts fire at 490 us /dev/rtc SUSE Linux Enterprise Real Time Kernel Measured 28,800,882 samples Max latency: 27 us Min latency: 11 us Average latency: 11.3 us samples < 0.02 ms ( %) 12samples < 0.03 ms (100 %) Kernel.org – rcl Interrupt Response SUSE Linux Enterprise Real Time

© Novell Inc. All rights reserved 7 SLERT Performance

© Novell Inc. All rights reserved 8 Leveraging IB To Double Wombat Feed Handler Capacity Server 1 Wombat Feed Handler Voltaire 9096 Switch Server 2 Wombat Client Server 3 Wombat Client 4 Consumer Application Connections Subscribing to 100,000 Symbols Each 50,000/second per connection ➔ 2 Consumer Applications/Client ➔ Handling A Total Of 200,000 msgs/sec

© Novell Inc. All rights reserved 9 Real Time Tests at a Major Investment Bank

© September 08, 2005 Novell Inc, Confidential & Proprietary 10 InfiniBand End-to-end QoS Granular Scalable End-to-end Hardware-based Deterministic

© September 08, 2005 Novell Inc, Confidential & Proprietary 11 Development/Roadmap Real-Time Kernel Fabric QoS Ultra-Responsive, Real-time Clusters QoS-Based Virtual Compute Platform

© September 08, 2005 Novell Inc, Confidential & Proprietary 12 Xen and IB ● Guest needs the performance of direct access to HCA for data transfer ● Guest needs to appear as endpoint on ib fabric ● XEN-IB prototype of VMM bypass for guest domains ● only supports Unreliable Datagram service type ● service discovery not supported ● no support for QP1/0 access at guest

© September 08, 2005 Novell Inc, Confidential & Proprietary 13 Xen and IB SA CM VERBS mthca driver ib core Domain 0Guest Domain libmthca libibverbs user app SRPIPoIB Hardware XEN kernel SA CM VERBS ib core libmthca libibverbs user app HCA provider vendor specific driver Guest ib resouces proxy HCA provider provider- front provider- back madfrontmadback MAD SRPIPoIB

Unpublished Work of Novell, Inc. All Rights Reserved. This work is an unpublished work and contains confidential, proprietary, and trade secret information of Novell, Inc. Access to this work is restricted to Novell employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability. General Disclaimer This document is not to be construed as a promise by any participating company to develop, deliver, or market a product. Novell, Inc., makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further, Novell, Inc., reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.