Presentation is loading. Please wait.

Presentation is loading. Please wait.

Presented by: Jeff Schaffer Sr

Similar presentations

Presentation on theme: "Presented by: Jeff Schaffer Sr"— Presentation transcript:

1 Presented by: Jeff Schaffer Sr
Presented by: Jeff Schaffer Sr. Field Applications Engineer QNX Software Systems “Embedded Operating Systems: The State of the Art” QNX is a leading provider of real time operating system (RTOS) software, development tools, and services for mission critical embedded applications.

2 Role of the Embedded OS Traditional
Permit sharing of common resources of the computer (disks, printers, CPU) Provide low-level control of I/O devices that may be complex, time dependent, and non-portable Provide device-independent abstractions (e.g. files, filenames, directories) Additional Roles Prevent common causes of system failure and instability; minimize impact when they occur Extend system life cycles Isolate problems during development and at runtime

3 Architecture Comparison
REAL TIME EXECUTIVE Advantage: single address space Disadvantage: single address space, different binary images Failure: means reboot MONOLITHIC KERNEL Advantage: apps run in own memory space Disadvantage: kernel not protected, kernel testing Failure: might mean reboot TRUE MICROKERNEL Advantage Modules run in own memory space Add/replace services on the fly Reusable modules Direct hardware access Disadvantage: context switching Failure: usually does not mean reboot

4 MicroKernel – Neutrino
Flash fsys Audio driver TCP/IP Process Manager App Serial driver Http server Java Photon GUI Microkernel X86, PPC, MIPS, SH4, ARM, StrongARM, XScale Philosophy: a trusted kernel running a system of untrusted software components Dynamic architecture makes hot-start and upgrades easy, even with drivers Processes communicate via messages or other methods, such as shared memory. Permits loose inter-module coupling. Processes provide a reusable component model with well defined message interfaces No requirement for filesystem, GUI, etc.

5 Typical Forms of IPC Mailboxes Pipes Message Queues Shared Memory
Process 1 Process 2 Mailboxes Kernel Pipes Process 1 Process 2 Message Queues msg 4 msg 3 msg 2 msg 5 Process address map Process address map Shared memory object Shared Memory map map

6 Which Architecture for me?
Depends on your application and processor! Simple apps (such as single control loops) generally only need a real-time executive As system becomes more complex, typically need a more complex operating system architecture Need to look at factors such as scalability and reliability Do standards matter?

7 Advantages of standards
API’s Two most common standards Advantages of standards Portability of code Hiring of programmers

8 Do I need Real-Time? Maybe ... What is Real Time?
Less than 1 second response? Less than 1 millisecond response? Less than 1 microsecond response?

9 Real-Time "A real-time system is one in which the correctness of the computations not only depends upon the logical correctness of the computation but also upon the time at which the result is produced. If the timing constraints of the system are not met, system failure is said to have occurred." Donald Gillies (comp.realtime FAQ)

10 Bill O. Gallmeister - POSIX.4 Programming for the Real World
A Simple Example... “it doesn’t do you any good if the signal that cuts fuel to the jet engine arrives a millisecond after the engine has exploded” Bill O. Gallmeister - POSIX.4 Programming for the Real World

11 “Hard” vs. “Soft” Real Time
absolute deadlines late responses cannot be tolerated and may have a catastrophic effect on the system example: flight control Soft systems which have reduced constraints on "lateness”; e.g. late responses may still have some value still must operate very quickly and repeatably example: cardiac pacemaker ATM

12 Real-time OS Requirements
Operating system factors that permit real-time: Thread Scheduling Control of Priority Inversion Time Spent in Kernel Interrupt Processing

13 Factor #1: Scheduling Non real-time scheduling round-robin FIFO
adaptive Real-time scheduling priority based sporadic

14 Factor #2: Priority Inversion
Information Bus Manager Communications Task Meteorological Data Gathering Task Source: Embedded Systems Programming Sequence: 1. Low priority task acquires bus mutex to transfer data 2. High priority task blocks until mutex released 3. Medium priority task pre-empts low priority task 4. Watchdog timer resets since Bus Manager has not run in some time

15 Factor #3: Kernel Time Kernel operations must be pre-emptible
if they are not, an unknown amount of time can be spent in the kernel performing an operation on behalf of a user process can cause real-time process to miss deadline All kernels have some window (or multiple windows) of time where pre-emption cannot occur Some operating systems attempt to provide real-time capability by adding “checkpoints” within the kernel so they can be interrupted at these points

16 Example usecs usecs usecs int KER a few opcodes Interrupts off
Entry a few opcodes Interrupts off Unlocked Kernel Operation which may include message pass usecs to msecs Pre-emptable A Kernel call is a software interrupt Locked usecs No pre-emption Interrupts on Unlocked usecs Pre-emptable iret Exit a few opcodes Interrupts off

17 Split Out Long Operations
Nto Proc Thread Sync Message Sched Signal Channel Clock Timer Intr Fork Exec Pathname Spawn Mmap Waitpid Session UID/GID Debug Process Manager

18 Factor #4: Interrupts This is broken down into the following areas:
Method of handling the interrupt processing chain Handling of Nested Interrupts

19 Interrupt Processing Chain
ISR INT x INT y IST IST scheduled by normal OS scheduling, deterministic ISR INT x INT y IST IST scheduled whenever queue emptied, non-deterministic

20 Can I Make Any Conventional OS Real-Time
Method Add real-time layer below conventional OS, running conventional OS as a low priority real-time process Add real-time layer to hardware service layer Conventional OS Problems different API’s real-time layer proprietary existing OS apps not R/T poor communication between operating systems loss of control issue Real-time kernel

21 Scalability Title of presentation Title 2

22 Scaling Solution #1: Single Board, Single Node
Mem. Bridge PCI Bus CPU Peripherals The only scaling possible is a CPU replacement

23 Scaling Solution #2: Single Board, Multiple Nodes
Mem. Bridge PCI Bus CPU Peripherals Node 1 Bridge PCI Bus CPU Peripherals Node 2 Relatively simple to implement Allows “scaling-on-demand” Suitable if nodes have independent “work” Inter-node IPC slower than memory access Complexity in maintaining global view of data Difficult to break-up computationally-intensive tasks

24 Scaling Solution #3: Single Board, Multiple Processors
Mem. Bridge CPU PCI Bus CPU 1 Peripherals Tightly-coupled symmetric multiprocessing (SMP) All processors have a symmetric and consistent view of physical memory and peripherals Scales processing power Need software (RTOS) support

25 The SMP OS Dilemma SMP systems to date use desktop operating systems; not responsive enough for real-time requirements Application servers Databases Web servers Typical real-time operating systems (home-built or commercial), such as are commonly used in routers and switches today, do not have SMP support SMP capable real-time operating systems run the CPU’s as independent processors with independent operating systems

26 SMP Support True (tightly coupled) SMP support
Only the kernel needs SMP awareness Transparent to application software and drivers - identical binaries for UP and SMP systems Automatic scheduling across all CPU’s

27 QNX “True” SMP STATE_RUNNING thread on each processor
Priority-based ready queues Each thread can be locked to a specific CPU by using a processor affinity mask Scheduler remembers last CPU thread ran on Minimize thread migration Optimize cache usage Highest-priority READY thread always immediately scheduled Thread Running CPU 0 Process CPU 1 Ready queues 63 Priority 62 61 ... Blocked states

28 Why Is Cache Important? Cache efficiency is probably the single largest determinant of performance on SMP Coherent view of physical memory is maintained using cache snooping Cache snooping is done at the CPU bus level and so operates at lower speeds than core Coherency is “invisible” to software

29 Performance Implications
Snoop traffic expected on SMP Cache hits generally cause no bus transaction Multiple processors writing to same location degrades performance (ping-pong effect) Performance degrades when large amount of data modified on one processor and read on the other Sometimes it is better to have specific threads in a process run on same CPU

30 Designing for SMP: One Big task
Will not work with SMP Single thread Giant App

31 Designing for SMP: Single Threaded Tasks
App 1 App 2 Works with SMP Process data can be shared with shared memory Good concurrency, some complexity IPC not usually as efficient as memory sharing

32 Designing for SMP: Scaling Software with Threads
Single copy server All process data is implicitly shared and accessible Can achieve good concurrency with less complexity POSIX synchronization used Mutexes Semaphores Condition variables Usually more efficient than inter-process synchronization Threads Server Note: SMP finds concurrency problems fast!

33 Optimizing Compute-intensive Applications
Pool of worker threads Dispatch “work” to worker threads Scales very well with SMP The tricky part is “breaking up” the problem Worker thread Worker thread Main thread Threads Application

34 Interrupt Handling IST ISR
Interrupt processed on CPU that was targeted Can distribute load by handling interrupts on different processors Sometimes not the optimal strategy due to cache effects CPU 0 CPU 1 IRQ CPU 1 IRQ 7 IRQ 10 IRQ 8 IRQ 9

35 Scaling Solution #4: Multiple Processors/Nodes
Mem. Bridge CPU PCI Bus CPU 1 Peripherals Node 1 Bridge CPU PCI Bus CPU 1 Peripherals Node 2

36 Example Chassis ... Network Network Network High-speed interconnect
Line card Low-speed bus Network High-speed interconnect Network Network Line card ...

37 The QNET MicroNetwork Microkernel Flash Fsys CDROM TCP/IP Process
Manager QNET Audio Photon App LAN or Internet or Backplane Messages flow transparently through QNET from one message bus to another. QNET Microkernel App All applications and servers become network distributed without any special code.

38 QNX Qnet Manager Extends message passing across multiple QNX microkernels Over anything with a packet driver: Ethernet, RapidIO, 3GIO, InfiniBand, Stargen, etc. Class of service Use symbolic prefixes to make client code independent of location of resource manager Line card Control

39 One or multiple links can connect different nodes.
QNET Class of Service Control card Line card Line card One or multiple links can connect different nodes.

40 QNET: Load-Balanced Distribution
Line card Control Data is sent out the link which will deliver it the fastest. This is based upon link speed and queue length for each link.

41 QNET: Ordered Distribution
Line card Control Data is sent out a primary link. If it fails, data is diverted to a secondary link. The primary link is probed and when it comes back online, data is diverted back to it.

42 QNET: Parallel Distribution
Line card Control Data is sent out both links at the same time. A failure on either of the links is handled gracefully.

43 Designing for Networked SMP: Single/Multi Threaded Tasks
Multiple threads Single thread App 1 App 2 Different processes necessary for different nodes Works with SMP Process data can be shared with shared memory IPC for networked communication

44 Transparent Redirection
Client Node /net/a/dev/service A Client /service B /net/b/dev/service Simple link provides transparent redirection Process has to monitor status of link Switch over is not transparent to client

45 Transparent Redirection
Client Node /net/a/dev/service A Client Service mgr /dev/service B /net/b/dev/service Service manager acts as a proxy Monitors health of and/or load on services/nodes Switch over is transparent to client

46 Redundant Links Requests serviced redundantly
Client Node /net/a/dev/service A Client Service mgr /dev/service B /net/b/dev/service Requests serviced redundantly First/majority/best result Different implementations

47 MOST BUS Qnet FLASH FSYS TCP/IP App Blue Tooth Graphics Browser Audio
Photon Qnet FLASH FSYS Graphics Browser Audio Photon Qnet CDROM

Graphics CDROM FSYS Browser Browser Audio Qnet Graphics Qnet

49 Reliability and Availability
Title of presentation Title 2

50 Why? Embedded systems are different!
Failure in an embedded system can have severe effects - like death … “Pilots really hate to be told they have to reboot their plane while in flight” Walter Shawlee

51 Definitions MTBF: Mean Time Between Failure
The average number of hours between failures for a large number of components over a long time. (e.g. MIL-HDBK-217) MTTR: Mean Time To Repair Total amount of time spent performing all corrective maintenance repairs divided by the number of repairs MTBI: Mean Time Between Interruptions. The average number of hours between failures while a redundant component is down.

52 Defining HA Reliability Availability 5 Nines
Quantified by failure rate (MTBF) Time to resume service after failure is MTTR Reliability Allows for failure, with quick service restoration. As MTTR  0, Availability  100% Availability < 5 minutes downtime / year (> % uptime) Assume faults exist: design to contain, notify, recover and restore rapidly 5 Nines

53 Costs speak for themselves
Annual Cost of Downtime versus Availability Source: Gartner Group ($13,000/minute Cross-industry Average)

54 Availability via Reliability and Repair
low MTTR -> high availability System is composed of reliable components, that are protected from each other, and that communicate ONLY through well known interfaces. this leads to fault isolation speedy recovery reset a component not a board/system dynamic control stop/start upgrade

55 Software vs Hardware HA
utilizes redundancy of key components a single fault cannot cause all redundant components to fail (No SPOF). e.g. mirrored disks, multiple system boards, I/O cards Active/active, active/spare, active/standby But that’s only part of the problem!!! Software is a Significant Cause of Downtime

56 Comparison

57 High Level Look at a Core Router/Switch
OCLD (1W) OCLD (2W) OCLD (3W) OCLD (4W) OCI (1A) OCI (1B) OCI (2A) OCI (2B) OCM (A) OCM (B) OCI (3A) OCI (3B) OCI (4A) OCI (4B) OCLD (4E) OCLD (3E) OCLD (2E) OCLD (1E) Shelf Processor Filler I O OFF ON Maintenance Panel 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Fiber Management Trough Optical Multiplexer Tray (OMX) Cooling Unit One or more control elements

58 Isolate Fault to a Board
Handling Failures OCLD (1W) OCLD (2W) OCLD (3W) OCLD (4W) OCI (1A) OCI (1B) OCI (2A) OCI (2B) OCM (A) OCM (B) OCI (3A) OCI (3B) OCI (4A) OCI (4B) OCLD (4E) OCLD (3E) OCLD (2E) OCLD (1E) Shelf Processor Filler I O OFF ON Maintenance Panel 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Fiber Management Trough Optical Multiplexer Tray (OMX) Cooling Unit Isolate Fault to a Board Switch to Backup

59 May not be in the Hardware
OCLD (1W) OCLD (2W) OCLD (3W) OCLD (4W) OCI (1A) OCI (1B) OCI (2A) OCI (2B) OCM (A) OCM (B) OCI (3A) OCI (3B) OCI (4A) OCI (4B) OCLD (4E) OCLD (3E) OCLD (2E) OCLD (1E) Shelf Processor Filler I O OFF ON Maintenance Panel 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Fiber Management Trough Optical Multiplexer Tray (OMX) Cooling Unit Isolate fault to a SW component Route Manager TCP/IP stack SNMP Manager Application Flash Drivers Device Manager Network Manager RTOS Hardware

60 Ideal: Identify and Fix
Faulty Software Component Isolate and contain Repair (e.g. restart) Notify Diagnose Upgrade Application Network Manager SNMP Manager Application Device Manager Route Manager Application TCP/IP stack Flash Drivers Application RTOS

61 Component-level recovery rarely done
Lack of suitable protection and isolation Lack of modularity Tight component coupling Few dynamic capabilities Software failures normally handled by: Hardware watchdogs Redundant boards

62 Repair Time Board Replacement Hours Reboot Minutes Failover to Standby
Seconds SW Component Restart 10’s Milliseconds SW Failover Milliseconds

63 High Availability Manager
Process Memory Violation DISK FSYS Kernel notifies HA Manager FLASH FSYS TCP/IP Dump file for post-mortem analysis TCP/IP HA Manager restarts service Microkernel HA Manager ATM

64 Notification and Recovery
HA Manager (HAM) monitors components, sends notification of component failure Heart-beat services detect component hangs Core file on crash can be created for debugging and analysis Checkpointing permits recovering current state HAM Checkpointed State HAM HAM Guardian Stack Driver App Checkpointed State

65 Recovery A second “shadow” server attaches to the same name

66 Recovery If primary faults, new clients connect to shadow server
A second “shadow” server attaches to the same name If primary faults, new clients connect to shadow server Old clients can re-connect to shadow server.

67 Recovery Start a new “shadow” server

68 Service Upgrades New version of server attaches to same name
Client Server v 1.0 /dev/service New Client /dev/service Server v 1.1 New version of server attaches to same name New clients connect to new server Old server exits when all old clients have exited

69 QNX Momentics Tools

70 Design Goals Tools needed to be easy to learn
Tools which could take advantage of QNX Tools which could integrate tools from other vendors, company designed tools, and industry specific tools and have them work with our tools and each other Tools needed to be customizable to the user or the company

71 The Best Tools and the Best RTOS
QNX® Momentics QNX® Neutrino® RTOS Invoke command-line tools C/C++ code developer Source debugger Command- line tools Ethernet, Serial, JTAG, ROMulator 3rd-Party Tools Java code developer Profiler BSPs Rational Target information Memory analysis DDKs Flash fsys TCP/IP Virtio Target agent Neutrino runtime System builder Photon app builder …TBA Microkernel IDE Workbench (Eclipse framework) Photon microGUI Java Http server Windows, Solaris, QNX Neutrino XScale

72 QNX IDE: Standards based
IBM donated Framework Java IDE 200 person-years of effort Open Source Consortium founding members include

73 System Profiling

74 Systems Analysis Toolkit
Event Log System Events interrupts, scheduler, messages, system calls System Characterization Performance analysis Field diagnostic Live or post-mortem Statistical & Numerical Analysis Application Protocol Trace Instrumented MicroKernel Device Driver TCP/IP Printer Data display

75 Providing Technology for Today… Architecture for Tomorrow
Irvine Office David Weintraub - Regional Sales Manager Woodland Hills Office Jeff Schaffer - Sr. Field Applications Engineer

Download ppt "Presented by: Jeff Schaffer Sr"

Similar presentations

Ads by Google