1. Realtime Linux Clark Williams <williams@redhat.com> Tech Lead
MRG/RHEL-RT Realtime Kernel

2. Overview
Some customers require consistent response time rather than high-throughput
“low latency” is the commonly used term
These applications are deadline-oriented
Stock kernel cannot prevent latency spikes

3. Who Are these customers?
Financial institutions
may be penalized if a transaction takes longer than a specified threshold
Government/Military
Distributed Simulations
C³ Applications aggregating sensor data
Telcos
Base-station gear

4. What is “latency”?
The time between an event occurring and the code that will respond to that event executing
Most common is time between interrupt firing and user-space thread running
Cyclictest program measures scheduler latency

5. How does RT reduce “latency”?
Adds PREEMPT_RT_FULL config
Converts all spinlock_t -> rt_mutex_t
Thread most IRQ handlers
rt_mutex_t
prevents in-kernel priority inversion
Increases preemption opportunities

6. Spinlock Family Tree Stock kernel (e.g. RHEL7) spinlock_t
raw_spinlock_t
arch_spinlock_t
RT kernel (e.g. RHEL-RT)
rt_mutex_t

7. raw_spinlock_t Disables preemption while held Does not sleep
Lightweight operation

8. rt_mutex_t Does not disable preemption while held
Does disable migration while held
Can sleep if contended
Has priority-inheritance chain
Heavyweight operation

9. Spinlocks on PREEMPT_RT
Sections protected by spinlock_t on RT may be preempted
Preemption no longer off while lock held
Contended spinlock_t may sleep
Sleeping spinlock in interrupt context is BAD
Enter threaded interrupts...

10. Interrupt Handling on RT
Most IRQs are threaded
Share a common handler that
Acks the interrupt
Wakes the thread associated with the IRQ
Timer tick only IRQ that defaults to traditional model
Have to be sure that nothing in the IRQ handler(s) takes a spinlock
Having threaded handlers means you can prioritize service order
Run as SCHED_FIFO thread, can be prioritized and affined

11. RHEL-RT compared to RHEL
RHEL-RT is RHEL 7 tree with RT patches applied
rt_mutex_t is heavier weight than a raw_spinlock_t
Typical performance hit is 10-15% slowdown on RT
RT Breaks kABI
doesn't break it so much as it smashes it into teeny-tiny-bits, then incinerates the remains.
Conversion of spinlock_t to rt_mutex_t is the culprit

12. What RT Can't Fix
BIOS weirdness
SMIs
Bad Programming

13. Summary RT Linux is for deadline oriented applications
rt_mutex_t prevents long processing delays
Prevents priority-inversion
Increases preemption opportunities
Added code means RT is slower for throughput oriented applications