1. Presented by: Jeff Schaffer Sr
Presented by: Jeff Schaffer Sr. Field Applications Engineer QNX Software Systems
“Embedded Operating Systems:
The State of the Art”
QNX is a leading provider of real time operating system (RTOS) software, development tools, and services for mission critical embedded applications.

2. Role of the Embedded OS Traditional
Permit sharing of common resources of the computer (disks, printers, CPU)
Provide low-level control of I/O devices that may be complex, time dependent, and non-portable
Provide device-independent abstractions (e.g. files, filenames, directories)
Additional Roles
Prevent common causes of system failure and instability; minimize impact when they occur
Extend system life cycles
Isolate problems during development and at runtime

3. Architecture Comparison
REAL TIME EXECUTIVE
Advantage: single address space
Disadvantage: single address space, different binary images
Failure: means reboot
MONOLITHIC KERNEL
Advantage: apps run in own memory space
Disadvantage: kernel not protected, kernel testing
Failure: might mean reboot
TRUE MICROKERNEL
Advantage
Modules run in own memory space
Add/replace services on the fly
Reusable modules
Direct hardware access
Disadvantage: context switching
Failure: usually does not mean reboot

4. MicroKernel – Neutrino
Flash
fsys
Audio
driver
TCP/IP
Process
Manager
App
Serial
driver
Http
server
Java
Photon GUI
Microkernel
X86, PPC, MIPS, SH4, ARM, StrongARM, XScale
Philosophy: a trusted kernel running a system of untrusted software components
Dynamic architecture makes hot-start and upgrades easy, even with drivers
Processes communicate via messages or other methods, such as shared memory. Permits loose inter-module coupling.
Processes provide a reusable component model with well defined message interfaces
No requirement for filesystem, GUI, etc.

5. Typical Forms of IPC Mailboxes Pipes Message Queues Shared Memory
Process 1
Process 2
Mailboxes
Kernel
Pipes
Process 1
Process 2
Message Queues
msg 4
msg 3
msg 2
msg 5
Process address
map
Process address
map
Shared memory
object
Shared Memory
map
map

6. Which Architecture for me?
Depends on your application and processor!
Simple apps (such as single control loops) generally only need a real-time executive
As system becomes more complex, typically need a more complex operating system architecture
Need to look at factors such as scalability and reliability
Do standards matter?

7. Advantages of standards
API’s
Two most common standards
Advantages of standards
Portability of code
Hiring of programmers

8. Do I need Real-Time? Maybe ... What is Real Time?
Less than 1 second response?
Less than 1 millisecond response?
Less than 1 microsecond response?

9. Real-Time
"A real-time system is one in which the correctness of the computations not only depends upon the logical correctness of the computation but also upon the time at which the result is produced. If the timing constraints of the system are not met, system failure is said to have occurred."
Donald Gillies (comp.realtime FAQ)

10. Bill O. Gallmeister - POSIX.4 Programming for the Real World
A Simple Example...
“it doesn’t do you any good if the signal that cuts fuel to the jet engine arrives a millisecond after the engine has exploded”
Bill O. Gallmeister - POSIX.4 Programming for the Real World

11. “Hard” vs. “Soft” Real Time
absolute deadlines
late responses cannot be tolerated and may have a catastrophic effect on the system
example: flight control
Soft
systems which have reduced constraints on "lateness”; e.g. late responses may still have some value
still must operate very quickly and repeatably
example: cardiac pacemaker
ATM

12. Real-time OS Requirements
Operating system factors that permit real-time:
Thread Scheduling
Control of Priority Inversion
Time Spent in Kernel
Interrupt Processing

13. Factor #1: Scheduling Non real-time scheduling round-robin FIFO
adaptive
Real-time scheduling
priority based
sporadic

14. Factor #2: Priority Inversion
Information Bus Manager
Communications Task
Meteorological Data Gathering Task
Source: Embedded Systems Programming
Sequence:
1. Low priority task acquires bus mutex to transfer data
2. High priority task blocks until mutex released
3. Medium priority task pre-empts low priority task
4. Watchdog timer resets since Bus Manager has not run in some time

15. Factor #3: Kernel Time Kernel operations must be pre-emptible
if they are not, an unknown amount of time can be spent in the kernel performing an operation on behalf of a user process
can cause real-time process to miss deadline
All kernels have some window (or multiple windows) of time where pre-emption cannot occur
Some operating systems attempt to provide real-time capability by adding “checkpoints” within the kernel so they can be interrupted at these points

16. Example usecs usecs usecs int KER a few opcodes Interrupts off
Entry
a few opcodes
Interrupts off
Unlocked
Kernel
Operation
which
may
include
message
pass
usecs to
msecs
Pre-emptable
A Kernel call is a
software interrupt
Locked
usecs
No pre-emption
Interrupts on
Unlocked
usecs
Pre-emptable
iret
Exit
a few opcodes
Interrupts off

17. Split Out Long Operations
Nto
Proc
Thread
Sync
Message
Sched
Signal
Channel
Clock
Timer
Intr
Fork
Exec
Pathname
Spawn
Mmap
Waitpid
Session
UID/GID
Debug
Process
Manager

18. Factor #4: Interrupts This is broken down into the following areas:
Method of handling the interrupt processing chain
Handling of Nested Interrupts

19. Interrupt Processing Chain
ISR
INT x
INT y
IST
IST scheduled by normal OS scheduling,
deterministic
ISR
INT x
INT y
IST
IST scheduled whenever queue emptied, non-deterministic

20. Can I Make Any Conventional OS Real-Time
Method
Add real-time layer below conventional OS, running conventional OS as a low priority real-time process
Add real-time layer to hardware service layer
Conventional OS
Problems
different API’s
real-time layer proprietary
existing OS apps not R/T
poor communication between operating systems
loss of control issue
Real-time kernel

21. Scalability
Title of presentation
Title 2

22. Scaling Solution #1: Single Board, Single Node
Mem.
Bridge
PCI
Bus
CPU
Peripherals
The only scaling possible is a CPU replacement

23. Scaling Solution #2: Single Board, Multiple Nodes
Mem.
Bridge
PCI
Bus
CPU
Peripherals
Node 1
Bridge
PCI
Bus
CPU
Peripherals
Node 2
Relatively simple to implement
Allows “scaling-on-demand”
Suitable if nodes have independent “work”
Inter-node IPC slower than memory access
Complexity in maintaining global view of data
Difficult to break-up computationally-intensive tasks

24. Scaling Solution #3: Single Board, Multiple Processors
Mem.
Bridge
CPU
PCI
Bus
CPU 1
Peripherals
Tightly-coupled symmetric multiprocessing (SMP)
All processors have a symmetric and consistent view of physical memory and peripherals
Scales processing power
Need software (RTOS) support

25. The SMP OS Dilemma
SMP systems to date use desktop operating systems; not responsive enough for real-time requirements
Application servers
Databases
Web servers
Typical real-time operating systems (home-built or commercial), such as are commonly used in routers and switches today, do not have SMP support
SMP capable real-time operating systems run the CPU’s as independent processors with independent operating systems

26. SMP Support True (tightly coupled) SMP support
Only the kernel needs SMP awareness
Transparent to application software and drivers - identical binaries for UP and SMP systems
Automatic scheduling across all CPU’s

27. QNX “True” SMP STATE_RUNNING thread on each processor
Priority-based ready queues
Each thread can be locked to a specific CPU by using a processor affinity mask
Scheduler remembers last CPU thread ran on
Minimize thread migration
Optimize cache usage
Highest-priority READY thread always immediately scheduled
Thread
Running
CPU 0
Process
CPU 1
Ready queues 63
Priority 62 61
...
Blocked states

28. Why Is Cache Important?
Cache efficiency is probably the single largest determinant of performance on SMP
Coherent view of physical memory is maintained using cache snooping
Cache snooping is done at the CPU bus level and so operates at lower speeds than core
Coherency is “invisible” to software

29. Performance Implications
Snoop traffic expected on SMP
Cache hits generally cause no bus transaction
Multiple processors writing to same location degrades performance (ping-pong effect)
Performance degrades when large amount of data modified on one processor and read on the other
Sometimes it is better to have specific threads in a process run on same CPU

30. Designing for SMP: One Big task
Will not work with SMP
Single thread
Giant App

31. Designing for SMP: Single Threaded Tasks
App 1
App 2
Works with SMP
Process data can be shared with shared memory
Good concurrency, some complexity
IPC not usually as efficient as memory sharing

32. Designing for SMP: Scaling Software with Threads
Single copy server
All process data is implicitly shared and accessible
Can achieve good concurrency with less complexity
POSIX synchronization used
Mutexes
Semaphores
Condition variables
Usually more efficient than inter-process synchronization
Threads
Server
Note: SMP finds concurrency problems fast!

33. Optimizing Compute-intensive Applications
Pool of worker threads
Dispatch “work” to worker threads
Scales very well with SMP
The tricky part is “breaking up” the problem
Worker thread
Worker thread
Main thread
Threads
Application

34. Interrupt Handling IST ISR
Interrupt processed on CPU that was targeted
Can distribute load by handling interrupts on different processors
Sometimes not the optimal strategy due to cache effects
CPU 0
CPU 1
IRQ CPU 1
IRQ 7
IRQ 10
IRQ 8
IRQ 9

35. Scaling Solution #4: Multiple Processors/Nodes
Mem.
Bridge
CPU
PCI
Bus
CPU 1
Peripherals
Node 1
Bridge
CPU
PCI
Bus
CPU 1
Peripherals
Node 2

36. Example Chassis ... Network Network Network High-speed interconnect
Line card
Low-speed bus
Network
High-speed interconnect
Network
Network
Line card
...

37. The QNET MicroNetwork Microkernel Flash Fsys CDROM TCP/IP Process
Manager
QNET
Audio
Photon
App
LAN or
Internet or
Backplane
Messages flow transparently through QNET from one message bus to another.
QNET
Microkernel
App
All applications and servers become network distributed without any special code.

38. QNX Qnet Manager
Extends message passing across multiple QNX microkernels
Over anything with a packet driver:
Ethernet, RapidIO, 3GIO, InfiniBand, Stargen, etc.
Class of service
Use symbolic prefixes to make client code independent of location of resource manager
Line
card
Control

39. One or multiple links can connect different nodes.
QNET Class of Service
Control
card
Line
card
Line
card
One or multiple links can connect different nodes.

40. QNET: Load-Balanced Distribution
Line
card
Control
Data is sent out the link which will deliver it the fastest. This is based upon link speed and queue length for each link.

41. QNET: Ordered Distribution
Line
card
Control
Data is sent out a primary link. If it fails, data is diverted to a secondary link. The primary link is probed and when it comes back online, data is diverted back to it.

42. QNET: Parallel Distribution
Line
card
Control
Data is sent out both links at the same time. A failure on either of the links is handled gracefully.

43. Designing for Networked SMP: Single/Multi Threaded Tasks
Multiple threads
Single thread
App 1
App 2
Different processes necessary for different nodes
Works with SMP
Process data can be shared with shared memory
IPC for networked communication

44. Transparent Redirection
Client Node
/net/a/dev/service A
Client
/service B
/net/b/dev/service
Simple link provides transparent redirection
Process has to monitor status of link
Switch over is not transparent to client

45. Transparent Redirection
Client Node
/net/a/dev/service A
Client
Service
mgr
/dev/service B
/net/b/dev/service
Service manager acts as a proxy
Monitors health of and/or load on services/nodes
Switch over is transparent to client

46. Redundant Links Requests serviced redundantly
Client Node
/net/a/dev/service A
Client
Service
mgr
/dev/service B
/net/b/dev/service
Requests serviced redundantly
First/majority/best result
Different implementations

47. MOST BUS Qnet FLASH FSYS TCP/IP App Blue Tooth Graphics Browser Audio
Photon
Qnet
FLASH
FSYS
Graphics
Browser
Audio
Photon
Qnet
CDROM

48. MOST BUS Qnet FLASH FSYS TCP/IP App Blue Tooth FLASH FSYS Photon
Graphics
CDROM
FSYS
Browser
Browser
Audio
Qnet
Graphics
Qnet

49. Reliability and Availability
Title of presentation
Title 2

50. Why? Embedded systems are different!
Failure in an embedded system can have severe effects - like death …
“Pilots really hate to be told they have
to reboot their plane while in flight”
Walter Shawlee

51. Definitions MTBF: Mean Time Between Failure
The average number of hours between failures for a large number of components over a long time. (e.g. MIL-HDBK-217)
MTTR: Mean Time To Repair
Total amount of time spent performing all corrective maintenance repairs divided by the number of repairs
MTBI: Mean Time Between Interruptions.
The average number of hours between failures while a redundant component is down.

52. Defining HA Reliability Availability 5 Nines
Quantified by failure rate (MTBF) Time to resume service after failure is MTTR
Reliability
Allows for failure, with quick service restoration. As MTTR  0, Availability  100%
Availability
< 5 minutes downtime / year (> % uptime)
Assume faults exist: design to contain, notify, recover and restore rapidly
5 Nines

53. Costs speak for themselves
Annual Cost of Downtime versus Availability
Source: Gartner Group ($13,000/minute Cross-industry Average)

54. Availability via Reliability and Repair
low MTTR -> high availability
System is composed of reliable components, that are protected from each other, and that communicate ONLY through well known interfaces.
this leads to
fault isolation
speedy recovery
reset a component not a board/system
dynamic control
stop/start
upgrade

55. Software vs Hardware HA
utilizes redundancy of key components
a single fault cannot cause all redundant components to fail (No SPOF). e.g. mirrored disks, multiple system boards, I/O cards
Active/active, active/spare, active/standby
But that’s only part of the problem!!!
Software is a Significant Cause of Downtime

56. Comparison

57. High Level Look at a Core Router/Switch
OCLD (1W)
OCLD (2W)
OCLD (3W)
OCLD (4W)
OCI (1A)
OCI (1B)
OCI (2A)
OCI (2B)
OCM (A)
OCM (B)
OCI (3A)
OCI (3B)
OCI (4A)
OCI (4B)
OCLD (4E)
OCLD (3E)
OCLD (2E)
OCLD (1E)
Shelf Processor
Filler I O
OFF ON
Maintenance Panel 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Fiber Management Trough
Optical Multiplexer Tray (OMX)
Cooling Unit
One or more control elements

58. Isolate Fault to a Board
Handling Failures
OCLD (1W)
OCLD (2W)
OCLD (3W)
OCLD (4W)
OCI (1A)
OCI (1B)
OCI (2A)
OCI (2B)
OCM (A)
OCM (B)
OCI (3A)
OCI (3B)
OCI (4A)
OCI (4B)
OCLD (4E)
OCLD (3E)
OCLD (2E)
OCLD (1E)
Shelf Processor
Filler I O
OFF ON
Maintenance Panel 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Fiber Management Trough
Optical Multiplexer Tray (OMX)
Cooling Unit
Isolate Fault to a Board
Switch to Backup

59. May not be in the Hardware
OCLD (1W)
OCLD (2W)
OCLD (3W)
OCLD (4W)
OCI (1A)
OCI (1B)
OCI (2A)
OCI (2B)
OCM (A)
OCM (B)
OCI (3A)
OCI (3B)
OCI (4A)
OCI (4B)
OCLD (4E)
OCLD (3E)
OCLD (2E)
OCLD (1E)
Shelf Processor
Filler I O
OFF ON
Maintenance Panel 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Fiber Management Trough
Optical Multiplexer Tray (OMX)
Cooling Unit
Isolate fault to a SW component
Route Manager
TCP/IP stack
SNMP Manager
Application
Flash Drivers
Device Manager
Network
Manager
RTOS
Hardware

60. Ideal: Identify and Fix
Faulty Software Component
Isolate and contain
Repair (e.g. restart)
Notify
Diagnose
Upgrade
Application
Network
Manager
SNMP Manager
Application
Device Manager
Route Manager
Application
TCP/IP stack
Flash Drivers
Application
RTOS

61. Component-level recovery rarely done
Lack of suitable protection and isolation
Lack of modularity
Tight component coupling
Few dynamic capabilities
Software failures normally handled by:
Hardware watchdogs
Redundant boards

62. Repair Time Board Replacement Hours Reboot Minutes Failover to Standby
Seconds
SW Component Restart
10’s Milliseconds
SW Failover
Milliseconds

63. High Availability Manager
Process Memory Violation
DISK
FSYS
Kernel notifies HA Manager
FLASH
FSYS
TCP/IP
Dump file for
post-mortem
analysis
TCP/IP
HA Manager
restarts
service
Microkernel HA
Manager
ATM

64. Notification and Recovery
HA Manager (HAM) monitors components, sends notification of component failure
Heart-beat services detect component hangs
Core file on crash can be created for debugging and analysis
Checkpointing permits recovering current state
HAM Checkpointed State
HAM
HAM
Guardian
Stack
Driver
App
Checkpointed State

65. Recovery
A second “shadow” server attaches to the same name

66. Recovery If primary faults, new clients connect to shadow server
A second “shadow” server attaches to the same name
If primary faults, new clients connect to shadow server
Old clients can re-connect to shadow server.

67. Recovery
Start a new “shadow” server

68. Service Upgrades New version of server attaches to same name
Client
Server
v 1.0
/dev/service
New
Client
/dev/service
Server
v 1.1
New version of server attaches to same name
New clients connect to new server
Old server exits when all old clients have exited

69. QNX Momentics Tools

70. Design Goals Tools needed to be easy to learn
Tools which could take advantage of QNX
Tools which could integrate tools from other vendors, company designed tools, and industry specific tools and have them work with our tools and each other
Tools needed to be customizable to the user or the company

71. The Best Tools and the Best RTOS
QNX® Momentics
QNX® Neutrino® RTOS
Invoke command-line tools
C/C++ code
developer
Source
debugger
Command-
line
tools
Ethernet, Serial,
JTAG, ROMulator
3rd-Party
Tools
Java code
developer
Profiler
BSPs
Rational
Target
information
Memory
analysis
DDKs
Flash
fsys
TCP/IP
Virtio
Target
agent
Neutrino
runtime
System
builder
Photon
app builder
…TBA
Microkernel
IDE Workbench
(Eclipse framework)
Photon microGUI
Java
Http
server
Windows, Solaris, QNX Neutrino
XScale

72. QNX IDE: Standards based
IBM donated Framework
Java IDE
200 person-years of effort
Open Source
Consortium founding members include

73. System Profiling

74. Systems Analysis Toolkit
Event
Log
System Events
interrupts,
scheduler,
messages,
system calls
System Characterization
Performance analysis
Field diagnostic
Live or post-mortem
Statistical &
Numerical
Analysis
Application
Protocol
Trace
Instrumented
MicroKernel
Device
Driver
TCP/IP
Printer
Data display

75. Providing Technology for Today… Architecture for Tomorrow
Irvine Office
David Weintraub - Regional Sales Manager
Woodland Hills Office
Jeff Schaffer - Sr. Field Applications Engineer