The 14 th IEEE Real-Time and Embedded Technology and Applications Symposium, April 2008 Real-Time Distributed Discrete-Event Execution with Fault Tolerance Thomas Huining Feng and Edward A. Lee Center for Hybrid and Embedded Software Systems EECS, UC Berkeley {tfeng,

RTAS 2008Feng & Lee, CHESS, EECS, UC Berkeley …, e 2 = (null, t 2 ) Distributed Discrete-Event Execution Strategy Execution strategy decides whether/when it is safe to process an input event. Conventional: Compute can process top event e 1 if e 2 has a greater time stamp. Null message (null, t 2 ) Cons: overhead, sensitive to faults, lack of real-time property 2 A A B B Sensor Source Merge … … …, e 1 = (v 1, t 1 ) C C Compute Actuator … …, e 2 = (v 2, t 2 ) i1i1 i2i2

RTAS 2008Feng & Lee, CHESS, EECS, UC Berkeley Overview of Our Approach Leverage time-synchronized platforms Eliminate null messages Potentially improves concurrency Decompose assertions of real-time properties Recover software components from faults 3 A A B B Sensor Source Merge C C Compute Actuator

RTAS 2008Feng & Lee, CHESS, EECS, UC Berkeley n cameras located around a football field, all connected to a central computer. Events at blue ports satisfy t ≤ τ (t – time stamp of any event; τ – real time) Events at red ports satisfy t ≥ τ Reference Application: Distributed Cameras 4

RTAS 2008Feng & Lee, CHESS, EECS, UC Berkeley Reference Application: Distributed Cameras Problems to solve: – Make event-processing decisions locally – Guarantee timely command delivery to the Devices – Guarantee real-time update at the Display – Tolerate images loss or corruption at Image Processor 5

RTAS 2008Feng & Lee, CHESS, EECS, UC Berkeley Minimum Model-Time Delay δ δ : P × P → R + U {∞} returns the minimum model-time delay between any two ports. (P – set of ports; R + – set of non-negative reals.) Example: δ(i 5, o 1 ) = min{δ 5 +δ 1, δ 5 +δ 4 +δ 2 }, where δ 1, …, δ 6  R + are pre-defined. 6 i1i1 i2i2 i3i3 o1o1 o2o2 i5i5 o5o5 δ5δ5 δ6δ6 i6i6 i4i4 δ4δ4 δ4'δ4' o3o3 o4o4 δ1δ1 δ2δ2 δ3δ3

RTAS 2008Feng & Lee, CHESS, EECS, UC Berkeley Intuition of Execution Strategy When is it safe to process e = (v, t) at i 1 ? 1.future events at i 1, i 2 and i 3 have time stamps ≥ t (conventional), or 2.future events at i 1 and i 2 have time stamps ≥ t, or 3.future events at i 1 have time stamps ≥ t, and future events at i 2 depend on events at i 4 with time stamps ≥ t – δ 4, or 4.future events at i 1 and i 2 depend on events at i 5 and i 6 with time stamps ≥ t – min{δ 5, δ 6, δ 5 + δ 4, δ 6 + δ 4 }. e = (v, t) 7 i1i1 i2i2 i3i3 o1o1 o2o2 i5i5 o5o5 δ5δ5 δ6δ6 i6i6 i4i4 δ4δ4 δ4'δ4' o3o3 o4o4 δ1δ1 δ2δ2 δ3δ3

RTAS 2008Feng & Lee, CHESS, EECS, UC Berkeley Relevant Dependency i ~ i' iff they are input of the same actor and affect a common output. An equivalence class is a transitive closure of ~. Construct a collapsed graph, and compute relevant dependency between equivalence classes. d(ε', ε) = min i'  ε', i  ε {δ(i', i)} 8 i1i1 i2i2 i3i3 o1o1 o2o2 i5i5 o5o5 δ5δ5 δ6δ6 i6i6 i4i4 δ4δ4 δ4'δ4' o3o3 o4o4 δ1δ1 δ2δ2 δ3δ3 min{δ 5, δ 6 } ε3ε3 ε1ε1 ε2ε2 min{δ 5, δ 6, δ 5 + δ 4, δ 6 + δ 4 } ε4ε4 δ4δ4 δ4'δ4' min{δ 5 + δ 4 ', δ 6 + δ 4 '}

RTAS 2008Feng & Lee, CHESS, EECS, UC Berkeley Dependency Cut A dependency cut for ε is a minimal but complete set of equivalence classes that needs to be considered to process an event at ε. Example: C 1 and C 2 are both dependency cuts for ε 1. 9 min{δ 5, δ 6 } ε3ε3 ε1ε1 ε2ε2 min{δ 5, δ 6, δ 5 + δ 4, δ 6 + δ 4 } ε4ε4 δ4δ4 δ4'δ4' min{δ 5 + δ 4 ', δ 6 + δ 4 '} C1C1 C2C2

RTAS 2008Feng & Lee, CHESS, EECS, UC Berkeley Execution Strategy Determine top event e = (v, t) at ε 1 safe to process If we choose C 1 : future events at ε 1 have time stamps ≥ t. If we choose C 2 : for any ε  C 2, future events at in ε 1 depend on events at ε with time stamps ≥ t – d(ε, ε 1 ). In general, we can freely choose any dependency cut. 10 min{δ 5, δ 6 } ε3ε3 ε1ε1 ε2ε2 min{δ 5, δ 6, δ 5 + δ 4, δ 6 + δ 4 } ε4ε4 δ4δ4 δ4'δ4' min{δ 5 + δ 4 ', δ 6 + δ 4 '} C1C1 C2C2

RTAS 2008Feng & Lee, CHESS, EECS, UC Berkeley Implementation of the Execution Strategy n + 1 platforms with synchronized clocks (IEEE 1588). Choose dependency cuts at platform boundary. A queue stores events local to the platform. At real time τ, future events have time stamps ≥ τ – d n. 11 network latency d n … Queue

RTAS 2008Feng & Lee, CHESS, EECS, UC Berkeley Tolerating Loss of Images Start the composition as soon as the starting packets are received. Create a checkpoint at the beginning (small constant overhead) Backtrack when fault is detected (linear in memory locations) In most cases, discard the checkpoint (garbage collection) connection lost Camera 1 Camera 2 12 start

RTAS 2008Feng & Lee, CHESS, EECS, UC Berkeley A Program Transformation Approach Before TransformationAfter Transformation int s; void f(int i) { s = i; } int s; void f(int i) { $ASSIGN$s(i); } An assignment is transformed into a function call to record the old value: private final int $ASSIGN$s(int newValue) { if ($CHECKPOINT != null && $CHECKPOINT.getTimestamp() > 0) { $RECORD$s.add(null, s, $CHECKPOINT.getTimestamp()); } return s = newValue; } This incurs a constant overhead. 13

RTAS 2008Feng & Lee, CHESS, EECS, UC Berkeley Observation: The overhead for each basic operation is constant. A Program Transformation Approach Before TransformationAfter Transformation int s; void f(int i) { s = i; } int s; void f(int i) { $ASSIGN$s(i); } Image img; int partNum; void consume(Packet p1, Packet p2) { if (img == null) { img = new Image(); partNum = 0; } img.parts[partNum] = compose(p1, p2); partNum++; } Image img; int partNum; void consume(Packet p1, Packet p2) { if (img == null) { $ASSIGN$img(new Image()); $ASSIGN$partNum(0); } img.$ASSIGN$parts(partNum, compose(p1, p2)); $ASSIGN$SPECIAL$partNum(11, -1); } 0:+= 1:-=... 11:++ 12:-- 14 Value, not used for ++.

RTAS 2008Feng & Lee, CHESS, EECS, UC Berkeley Conclusion and Future Work Advantages – Eliminate null messages – Decompose real-time schedulability analysis – Advance the system even when some platforms fail – Tolerate faults without sacrificing real-time properties Future Work – Examine different choices of dependency cuts – Develop static WCET (worst-case execution time) analysis to guarantee real-time properties on each platform – Build an implementation to support a variety of real applications – Exploit parallelism with multi-core platforms 15