1. Joe Loyall, Rick Schantz, Gary Duzan
Detection and Reaction to Unplanned Operational Events in Large Scale Distributed Real-Time Embedded Systems
Jianming Ye
Joe Loyall, Rick Schantz, Gary Duzan
BBN Technologies, Cambridge, MA
{jye, jloyall, schantz,

2. Outline Distributed Realtime Embedded (DRE) System Overview
Unplanned Events in DRE Systems
Multi-layer Architecture to Handle Unplanned Events
QuO as Underlying Technology
Conclusions

3. DRE System Overview Applications are distributed and network centric
Operate in dynamic environments
Stringent QoS requirements
Resources are constrained and shared
Potentially large number of unplanned events
Military Systems of Systems
Industrial
Production
Signal
Analysis
And
Geolocation
Shipboard
Systems
Disaster Response Systems
FEMA

4. Unplanned Events in DRE Systems
Definitions:
Unplanned events are events that are not expected to happen during normal operation and thus they cannot be specified in pre-planned mission requirements.
We can further divide unplanned events into two subtypes: unpredictable events and unexpected events.
Unpredictable events are those that we can envision might happen, but that we cannot predict when, where, and how they will manifest. For example, hardware defects or failures, unexpected loads, or security breach.
Unexpected events are those that cannot even be envisioned. We normally don’t know the nature of these events.
Our discussion on unplanned events will primarily focus on unpredictable events rather than unexpected events.

5. Unplanned Events in DRE Systems cont.
Characteristics of Unplanned Events
Unplanned events have key symptoms that can be detected, e.g. network hardware defects causing decreased bandwidth capacity and prolonged delay.
Each symptom can have multiple causes. For example diminished bandwidth capacity can be caused by hardware defects, unexpected loads, or security breach.
Key symptoms of events can lead us to the right remedies to a large number of unplanned events.
Need to react quickly to symptoms to minimize adverse effects
Determine the cause can be slower and require a larger view of the system
Unplanned event handling can be treated as part of overall QoS management strategy

6. Multi-layer Architecture for Unplanned Events
Overview of Multi-layer Architecture (MLA)
QoS management is divided into multiple layers
Unplanned event handling is part of overall QoS management
Rapid symptom handling at lower levels
Diagnosis and cause analysis at higher levels
The number of QoS management layers can scale up and down based on the size and characteristics of the DRE system.
QoS Manager
layer i+2
Feedback/Control
QoS Manager
QoS Manager
layer i+1 …
QoS Manager
QoS Manager
QoS Manager
QoS Manager … …
layer i

7. Multi-layer Architecture for Unplanned Events cont.
QoS manager can be divided into three logical units:
Detection - Monitor local conditions and events through probes in the system and operating environment,
Decision - Determine the adaptation strategies based on the information from the detection unit as well as the control/policy signals pushed down from higher layer QoS managers and feedback from the lower layer QoS managers.
Reaction - Execute and push the adaptation strategies down to the lower layer QoS manager for enforcement.

8. Multi-layer Architecture for Unplanned Events cont.
Handling unplanned events
Step
Example
Detection units probe for symptoms of unplanned events
Decreased bandwidth capacity, prolonged delay
Decision and reaction units provide rapid reaction to the symptoms of unplanned events based on local information
Compress data and/or reduce data rate
Higher layer manager(s) figures out a corrective strategy to bring the system back to normal operation
If the cause is the defect in the router, then higher level manager may decide to redirect the traffic through a different path; if the cause is external loads, then it may decide to elevate the privilege of the intended traffic

9. QuO as Underlying Technology
QuO middleware framework
Works with CORBA, Java RMI
Provides measurement, control, and contract-driven adaptation
Provides a QoS behavior encapsulation model, called Qosket
CORBA Component Model (CCM) support

10. QuO as Underlying Technology cont.
Using QuO in MLA
Elements in MLA
QuO Components
Detection unit
System condition objects
Decision unit
Contracts
Controllers
Resource Managers
Reaction unit
Callback objects
Delegates
Multi-layer feedback/control

11. Conclusions
The Multi-layer Architecture provides the flexibility to handle unplanned events at multiple layers
Unplanned event handling using this architecture is analogous to the way exceptions are handled in various programming languages
Various QoS management approaches, from centralized to highly distributed and combined strategies in between, can be accommodated within the same architecture
QuO adaptive middleware provides support to key functionalities of the architecture
For more info visit: