Presentation is loading. Please wait.

Presentation is loading. Please wait.

Adaptive Partition Scheduling Part 1: Why we did it Cool stuff from QNX A.Danko November 27, 2018.

Similar presentations


Presentation on theme: "Adaptive Partition Scheduling Part 1: Why we did it Cool stuff from QNX A.Danko November 27, 2018."— Presentation transcript:

1 Adaptive Partition Scheduling Part 1: Why we did it Cool stuff from QNX
A.Danko November 27, 2018

2 Yet another thread scheduler. Why?
The story begins with a customer: “We can use QNX! We need ARINC653!!!!!! HELP!” November 27, 2018

3 ARINC 653 Partition Scheduler and “special” IPC
Why? Shiny New Toy Partition scheduler (ARINC 653) Very popular in fixed military systems Each partition is guaranteed a percentage of CPU Priorities are only meaningful within a partition Shortcomings include Detailed RMA required to verify system Overload of IPC FIFO input queue Failures include denial of service and CPU quota exhaustion Monolithic design within one partition Hard to retrofit to existing 1-cpu applications. Inefficient use of total CPU. Runs idle when tasks are ready. Increased interrupt latency Does not address shared entities such as a file system Restrictive programming model. No DMA POSIX 30% OTHER 50% JAVA 20% ARINC 653 Partition Scheduler and “special” IPC November 27, 2018

4 Real-world examples of partitioning for QNX customers
Why? Real-world examples of partitioning for QNX customers Selling a portion of throughput Security: Untrusted Applications Router Customer 1 Customer 2 Car NAV 3rd party (malware?) etc … TCP/IP TCP/IP Protocol Protocol Radio 80% 20% Application Router Application Downloaded applications from the WEB cannot hurt the system Application Protocol 50% 50% Locked System Recovery Customer 2’s network load cannot hurt customer 1 HOG App bash 90% 10% Hard-wall scheduler not-required. Do we need any new scheduler? Emergency recovery shell November 27, 2018

5 Evolution of schedulers
Why? Evolution of schedulers Timeline priority pre-emptive Timeslicing Time-varying priority Really clever time-varying Fair Share scheduling Adaptive configuration Yes, but: System locks up Backhoes and Mother’s day Untuneable for more than 1 application. US Military Satcom Hard to manage share interactions. Not invented – until now. SCHED_FIFO SCHED_RR SCHED_SPORADIC November 27, 2018

6 Evolution: Lessons learned
Why? Evolution: Lessons learned Numerical priorities are chosen by applications but system scheduling behavior must be designed globally Degradation and overload: Priorities are not constants. Importance of work depends on circumstances. Modes: normal operation, restart, emergency maintenance Scheduling strategy needs to be based on unit of work, but what we have is communicating threads. must measure real-time behavior. 0.1 % accuracy Want to specify shares as global percentages Applications don’t get to pick their importance or shares. System engineers do. Need to throttle cpu usage without losing realtime latencies. November 27, 2018

7 Adaptive Partition Scheduling
Design What is Partitioning? General Answer Separation of work To isolate: cpu usage memory usage system resource usage Failures QNX Answer POSIX compatible design which can be applied to existing systems with little or no recoding A global hard real-time scheduler with overload protection and CPU guarantees Separation of work based on “working for common purpose” Runtime typed memory and kernel object guarantees and limits With full inheritance and accounting for all children Persistent storage (file system) guarantees and limits Process model for fault isolation Dynamic configuration Adaptive Partition Scheduling November 27, 2018

8 Principles Scheduler must not trigger an overload
Design Principles Scheduler must not trigger an overload Overhead may not increase with # of threads Real-time during underload Same behavior as today Real-time during overload At least for interrupt handling Must also be a fair-share scheduler global scheduler algorithm globally configured Must mesh with current QNX architecture Preemptive priority, individual thread scheduling Heavy use of message passing Easy to drop onto existing applications Can’t be a “bag on the side” Simple enough for customers to use Engineerable Reconfigure on the fly Offered load Throughput Insert picture of Juggling Watermelons here November 27, 2018

9 Overconstrained problem?
Nope: Implemented in QNX 6.3.2 Actually Works See “How it Works” in Part2. November 27, 2018

10 Adaptive Partition Scheduling
Design Adaptive Partition Scheduling Part 2: How it works. What it does: Counting time Who’s got time Real time Out of time Free time Borrowed time Equal time How it does it API Why is it secure? Why is it cool? November 27, 2018

11 Counting time What does 14% cpu mean? Accuracy:
Design Counting time What does 14% cpu mean? CPU usage is calculated over a sliding window. Accuracy: Counting ticks is not enough. “Micro-billing” is used to track actual CPU utilization even when threads don’t use their whole timeslice. micro- and nano-second resolution Threads are billed based on real usage, not statistics “windowsize” is configurable as an argument to kernel at boot Tradeoff maximum READY-state latency with accuracy of CPU budgeting 100ms window -> 1% accuracy or better. Internal arithmetic accurate to 0.5% or better Partition usage ns cpu time executed, during last sliding window, expressed as percentage Partition budget Guaranteed percentage of cpu time, balanced over sliding window T= now T= -100ms November 27, 2018

12 Who’s got time: Partition Membership
QNX Scheduler Partition Set of threads working for a common purpose Set of initial processes/threads designated by customer + all subsequent children Guest members Server’s cpu time billed to client Resmgr threads temporarily join partition of sender thread Not locked to a static set of code. OS services are part of whatever partition they need to be. hence the name “adaptive partition” November 27, 2018

13 Who’s got time: Partition Inheritance
Design Who’s got time: Partition Inheritance File System Process 6 6 6 7 - 11 10 Message 9 8 - 10 9 Message 4 9 - CPU budget available Receive Threads CPU budget available Adaptive Partition 1 (Multi-media) Adaptive Partition 2 (Java application) Resource manager threads work on behalf of sender Priority and adaptive partition in inherited on receive Execution time in server billed to client’s partition This allows proper accounting for shared resources November 27, 2018

14 Real time: Behavior under normal load
Design Real time: Behavior under normal load Blocked Ready 6 6 6 7 11 8 Running 10 10 9 9 4 CPU budget available CPU budget available Adaptive Partition 1 (Multi-media) Adaptive Partition 2 (Java application) Hard real-time scheduler under normal load Running thread selected as highest priority READY thread No delay on scheduling if adaptive partition has budget November 27, 2018

15 Out of time: Behavior under overload
Design Out of time: Behavior under overload Blocked Ready 6 6 6 7 11 8 Running 10 9 4 CPU budget available CPU budget exceeded Adaptive Partition 1 (Multi-media) Adaptive Partition 2 (Java application) Highest priority READY thread in Partition with budget runs No delay on scheduling if adaptive partition has budget November 27, 2018

16 Free Time: Behavior with unused CPU
Design Free Time: Behavior with unused CPU Blocked 6 6 6 6 Running 7 11 10 8 10 9 10 8 9 4 CPU budget exceeded CPU budget exceeded CPU budget available Adaptive Partition 1 (Multi-media) Adaptive Partition 2 (Java application) Adaptive Partition 3 If no partitions with remaining budget have READY threads, highest priority READY thread is selected to run from other partitions This allows “free” time to be given based upon priority “Free” time is still accounted and may have to be paid back (for example, if partition 3 becomes ready within 1 averaging window) November 27, 2018

17 Borrowed Time: Critical Threads
Design Borrowed Time: Critical Threads Blocked Ready 6 6 6 7 11 Critical Thread 8 Running 30 30 11 11 4 CPU budget available CPU budget exceeded Adaptive Partition 1 (Multi-media) Adaptive Partition 2 (Air Bag Control) Critical threads still run (based on priority) even if partition has no budget Critical threads provide deterministic scheduling even in overload Critical threads are given critical budget and can go into short-term debt Critical time is accounted and has to be repaid Exceeding critical budget is considered an error and causes notification/action November 27, 2018

18 Equal time. How to choose between partitions of equal priority
Design Equal time. How to choose between partitions of equal priority Unimportant? Many threads run at default priority, therefore equal priority Possible algorithms: - round robin - favor partition with most free time - favor longest waiter Requirement: Minimize latencies during underload WBN: divide free time by % cpu share. Solution: Interleave partitions by ratio of partition shares We found a clever way to do that, so it’s in the patent. November 27, 2018

19 How it does it uKernel libmod_aps.a Scheduler messaging Process
creation libmod_aps.a Per-partition Ready Q messaging for all partitions, p Def m(p) -> (bud(p)||crit(p), prio(p), run_t/wsize/bud(p)) Then schedule ps Def ps -> rdy(ps) and (m(ps) < m(pi)) For all i != s Scheduler clock intr handler ready() block() select_thread() November 27, 2018

20 Algorithm summary A partition sees real-time behaviour when under budget Only limited when another partition must get its guarantee Fair-share scheduling at or over budget Equal prio partitions are interleaved Budgets balanced in much less than windowsize Free time (above budget) is given out: By default: in real-time mode Optionally: by ratio of budgets Critical Thread run even if out of budget Criticality is inherited November 27, 2018

21 Overhead: Fancy, but is it fast?
Scheduling overhead increases with: - number of partitions - number of messages/sec - number of clock interrupts/sec, i.e. ClockPeriod() * does not increase with number of threads * Free or almost free operations: Inheriting partition as part of message receive Joining a thread to a partition Dynamically changing budgets Computational requirements 32 bit multiply, 64bit add *no floating point* *no divides* *no address space swapping* *short-circuit calculation of merit function* *no inter-cpu msging on SMP* *history-less algorithm* Overhead typically 1% of total cpu November 27, 2018

22 Design APIs Control of Adaptive Partitioning Scheduler is done through a kernel API API allows associating a thread with a partition Used to launch processes within a partition Children inherit parent’s partition Dynamic capabilities part of design Budgets may be changed at run time – instant effect Threads may join/unjoin partitions freely APIs to attach event triggered on critical budget overrun Selectable security API is restricted to privileged processes (root) Must be called from within default (system) partition Partitions are created with budget (normal and possibly critical) API provided to “lock down” partition configuration Prevent creation of new partitions or modification of budgets November 27, 2018

23 API 2: Launching applications
1. Build File schedaps MyPartition 20 [schedaps=MyPartition] /bin/myApp 2. Command line aps create –b20 MyPartition on –Xaps=MyPartition /bin/myApp 3. Momentics IDE4 Drag and drop 4. include <sys/sched_aps.h> Full programmatic interface: configure, get stats, launch, secure November 27, 2018

24 Why is AP Secure? AP enforces budgets every clock interrupt
Root can be required to do configuration changes Partition creation by subdivision of parent It’s not possible to create a sub-partition greater than a parent Not even root can violate this rule Configuration can be locked November 27, 2018

25 Why is this cool?: Engineerable
Design Why is this cool?: Engineerable Identifying units of work: Partition Inheritance Identify code that starts up applications Inheritance figures out the rest Filesystems etc do not require separately engineered cpu share Customer need not analyze budgets for OS components Global share management: % cpu cpu shares defined in units customers are used to: Percentage gets us off the hook for accounting for different clock speeds. Realtime when you need it: Critical Threads Interrupts and important event still get handled on time. Secure Budgets, especially critical budgets, are set globally by root, not by applications “to err is human, but …” November 27, 2018

26 Adaptive Partition Scheduling
Part 3. The Slick Demo November 27, 2018


Download ppt "Adaptive Partition Scheduling Part 1: Why we did it Cool stuff from QNX A.Danko November 27, 2018."

Similar presentations


Ads by Google