1. Linear Time Byzantine Self- Stabilizing Clock Synchronization Ariel Daliot 1, Danny Dolev 1, Hanna Parnas 2 1 School of Engineering and Computer Science, 2 Department of Neurobiology and the Otto Loewi Center for Cellular and Molecular Neurobiology, The Hebrew University of Jerusalem, Israel This research is supported in part by Intel COMM Grant - Internet Network/Transport Layer & QoS Environment (IXA)

2. Lecture Outline What is “Pulse Synchronization” Examples of pulse synchronization in nature A biologically inspired pulse synchronization algorithm for distributed computer networks Efficient Byzantine Self-Stabilizing clock synchronization above pulse synchronization

3. The target is to synchronize pulses from any state and any faults.....|.............|..................|.....................|...................|.... ……...|.............|..................|.....................|..............|.................|.............|..................|.....................|..................|.....t ……………......|.............|..................|.....................|....................... …......|.............|..................|.....................|................|........ …………….|.............|..................|.....................|.....|....................……......|.............|..................|.....................|...........|..................||||||........||.....|||......||......||......|.......||.||.||.....|......||.......…… …….....|.............|||.||.||.||||...............|.......|||||||||||||||||||||...||||.||||…... cycle Synchronized state Arbitrary state

4. Self-Stabilizing “Pulse Synchronization” Convergence: Starting from an arbitrary state s, the system reaches a synchronized state in finite time Closure: If s is a synchronized state of the system at real-time t 0 then  real-time t≥ t 0 : 1.The system state at time t is a synchronized state 2.«Linear Envelope», for every correct node p: a[t-t 0 ] + b  ψ p (t, t 0 )  g[ t-t 0 ] + hLinear Envelope Ψ p (t 1,t 2 ) is is the number of pulses a correct node p i invoked during a real time interval [t 1,t 2 ] within which p i was continuously correct

5. “Pulse Synchronization”, in distributed computer systems The computers are required to: Invoke regular pulses in tight synchrony Synchronize from ANY state (self-stabilization) Limit the pulse frequency Tolerate a number of node and network faults Examples of other synchronization problems: “Firing Squad”, “Clock Synchronization”, etc.

6. Fault Models Many problems trivial with no faults, some unsolvable with a single fault (E.g. Byzantine Generals) Common fault models: Crash/Link/Message faults Byzantine failures (“malicious” faults) –Usually proven to require n>3f to tolerate f faults –Not solvable for some problems Transient faults (system in arbitrary state or total chaos) –Requires Self-Stabilizing algorithms in order to overcome –Not solvable for some problems (Clock Synchronization)

7. Self-Stabilization Addresses the situation when ALL nodes can concurrently be faulty for a limited period of time A SS algorithm realizes its task once the system is back within the assumption boundaries Is orthogonal to Byzantine failures, i.e. these are uncorrelated fault models Byzantine algorithms typically focus on limiting the influence of faulty nodes once the task has been realized Self-stabilizing algorithms focus on realizing the task following a “catastrophic” state

8. Synchrony phenomena in biology The phenomenon of synchronization is displayed by many biological systems –Synchronized flashing of the male malaccae fireflies –Oscillations of the neurons in the circadian pacemaker, determining the day-night rhythm –Crickets that chirp in unison –Coordinated mass spawning in corals –Audience clapping together after a “good” performance We were inspired by the pacemaker network in the cardiac ganglion of lobsters

9. Synchrony phenomena in biology The phenomenon of synchronization is displayed by many biological systems –Synchronized flashing of the male malaccae fireflies –Oscillations of the neurons in the circadian pacemaker, determining the day-night rhythm –Crickets that chirp in unison –Coordinated mass spawning in corals –Audience clapping together after a “good” performance We were inspired by the pacemaker network in the cardiac ganglion of lobsters

10. Synchrony phenomena in biology The phenomenon of synchronization is displayed by many biological systems –Synchronized flashing of the male malaccae fireflies –Oscillations of the neurons in the circadian pacemaker, determining the day-night rhythm –Crickets that chirp in unison –Coordinated mass spawning in corals –Audience clapping together after a “good” performance We were inspired by the pacemaker network in the cardiac ganglion of lobsters

11. Synchrony phenomena in biology The phenomenon of synchronization is displayed by many biological systems –Synchronized flashing of the male malaccae fireflies –Oscillations of the neurons in the circadian pacemaker, determining the day-night rhythm –Crickets that chirp in unison –Coordinated mass spawning in corals –Audience clapping together after a “good” performance We were inspired by the pacemaker network in the cardiac ganglion of lobsters

12. Synchrony phenomena in biology The phenomenon of synchronization is displayed by many biological systems –Synchronized flashing of the male malaccae fireflies –Oscillations of the neurons in the circadian pacemaker, determining the day-night rhythm –Crickets that chirp in unison –Coordinated mass spawning in corals –Audience clapping together after a “good” performance We were inspired by the pacemaker network in the cardiac ganglion of lobsters

13. Synchrony phenomena in biology The phenomenon of synchronization is displayed by many biological systems –Synchronized flashing of the male malaccae fireflies –Oscillations of the neurons in the circadian pacemaker, determining the day-night rhythm –Crickets that chirp in unison –Coordinated mass spawning in corals –Audience clapping together after a “good” performance We were inspired by the pacemaker network in the cardiac ganglion of lobsters

14. Synchrony phenomena in biology The phenomenon of synchronization is displayed by many biological systems –Synchronized flashing of the male malaccae fireflies –Oscillations of the neurons in the circadian pacemaker, determining the day-night rhythm –Crickets that chirp in unison –Coordinated mass spawning in corals –Audience clapping together after a “good” performance We were inspired by the pacemaker network in the cardiac ganglion of lobsters

15. Synchrony phenomena in biology Synchronization typically attained in spite of: –Inherent difference among the biological elements –High levels of noise from external sources or from the biological elements Displays high degree of “Self-Stabilization”

16. Cardiac ganglion of the lobster (Sivan, Dolev & Parnas, 2000) Four interneurons tightly synchronize their pulses in order to give the heart its optimal pulse rate (though one is enough for activation) Able to adjust the synchronized firing pace, up to a certain bound (e.g. while escaping a predator) motor neurons |..|.. |.|.||. |..|..

17. Cardiac ganglion of the lobster (Sivan, Dolev & Parnas, 2000) Must not fire out of synchrony for prolonged times in spite of –Noise –Single neuron death –Inherent variations in the firing rate –Firing frequency regulating Neurohormones –Temperature change The vitality of the cardiac ganglion suggests it has evolved to be optimized for –Fault tolerance –Re-synchronization from any state (“self-stabilization”) –Tight synchronization –Fast re-synchronization

18. We present a Byzantine self-stabilizing pulse synchronization algorithm which works similarly to the cardiac pacemaker Input to the algorithm: n, f (f<⅓. n) and cycle Output of the algorithm: tells the node when to invoke a pulse The algorithm is proven to solve the pulse synchronization problem

19. The neurobiological principles «borrowed» Endogenous Periodic Spiking Time dependent Refractory Function, R Summation Nodes communicate with Messages which are then Summated to generate the Counter that is sent in the message and compared to the Refractory Function

20. “Pulse Synchronization” Algorithm

21. .....|.............|..................|.....................|...................|..............|. ……...|.............|..................|.....................|..............|.............|........|.............|..................|.....................|..................|.............|. t ……………......|.............|..................|.....................|.............|. …......|.............|..................|.....................|................|............|. …………….|.............|..................|.....................|.....|............|..……......|.............|..................|.....................|...........|............|......||||||........||.....|||......||......||......|.......||.||.||.....|......||.......…… …….....|.............|||.||.||.||||...............|.......|||||||||||||||||||||...||||.||||… Algorithm Analysis and Complexity arbitrary initial state tightness = d (near optimal) f < ⅓. n log(n) cycles½-1 cycle

22. Algorithm Complexity Tolerates f concurrent Byzantine faults, f < ⅓. n Self-stabilizing, synchronizes from an arbitrary state Faults can cause the synchronized pulse frequency to be up to twice the original frequency Synchronization is VERY tight, d time units, theoretical bound is –The best non-stabilizing Byzantine CS reaches d time units –Self-Stabilizing Byzantine CS reaches (n-f). d Time complexity is O(logn) –Best Self-stabilizing Byzantine Digital Clock Synchronization converges in expected (n-f). n 6(n-f) pulses –Non-stabilizing Byzantine Clock Synchronization use O(f)-O(n 2 ) time

23. A related problem – real-time-Clock Synchronization (rCS) There exists γ, t 0, ν, a and b such that  t≥t 0 : Agreement. For any correct nodes p, q |C p (t) - C q (t)| ≤ γ, (precision) Validity. For every correct node p (1+ν) -1 t +a ≤ C p (t) ≤ (1+ν)t + b, (accuracy) Optimal precision is d. (1-1/n) Optimal accuracy is ν = 

24. real-time-Clock Synchronization rCS has two additional constraints over pulse synchronization: –The pulses have labels (“the time”) –The time needs to approximate real time Most Byzantine rCS use the following principles: –At every time the computers exchange clock values –They operate some function on the received values (which seeks to neutralize the effect of the Byzantine values and set the clocks close to each other)

25. Clock Synchronization simple example… All computers exchange clocks at time X They must reach time X at roughly the same time Assume some computer receives the values: [435, 2, 2, 3, 2, 935, 3, 3, 2, 2, 6984] Throw away the values that look bad Do some statistical function on what is left and set the clock accordingly It can be proven this leaves all the computers with very close clock values that are good estimates of real time

26. real-time-Clock Synchronization impossibility result with no external time source => This works only if the clocks initially have close values => Which implies rCS cannot be solved when all clocks hold arbitrary times =>Which means there is no self-stabilizing algorithm for rCS I.e. if the clocks are initially far apart they cannot both synchronize AND estimate real time => Internal rCS assume clocks are initially synched

27. FAB8 AIT WSR - ww16-17/2003 Main outages and issues: …. synchronize problem in VAX - On WW16.5 - Impact: 8 CW SC's unable to introduce lots for a period of 4 hr and 15 min.( from 23:00 until 03:15). Root cause: - the job which synchronize the time between the VAXes failed on 22:00 (Thursday Night) and created gaps between the machines clocks. This gap caused the remotes which worked with CW* to get to loop status, with error message of FCM message. Solution: Time synchronized. Helpdesk will get alerts when this job will fail again.

28. A Distributed System according to Lamport “A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable.”

29. Applicability of logical clocks Many algorithms depending on clock synchronization actually only require synchronized logical clocks E.g.: TDMA, Kerberos tickets, DHCP leases, global snapshots, data base time stamps and many others Why then not use a self-stabilizing Byzantine clock synchronization algorithm that synchronizes logical time?

30. Self-Stabilizing Byzantine Clock Synchronization Because the known previously best self- stabilizing Byzantine clock synchronization algorithm converges in expected (n-f). n 6(n-f) time! (Dolev-Welch, 95) The difficulty lies in the fact that: –the initial clock values can differ arbitrarily –there is no agreed time for exchanging the values and setting the clock according to the values received –the clocks can wrap around

31. Clock Synchronization using synchronized pulses We assume no outside source of real-time At every pulse exchange clock values and operate some clock adjustment function on the received multiset If clocks were initially close to real time then they will stay close to real time If not then the clocks will proceed synchronously close to logical time This scheme yields a Byzantine self-stabilizing clock synchronization algorithm with convergence time, accuracy and precision on the order of existing rCS

32. The Byzantine Self-Stabilizing Clock Synchronization Algorithm At “pulse” event Begin Clock := ET; Wait for every correct node to invoke a pulse; ET := SS-Strong-Byz-Agreement(ET + cycle mod M); End ET - Expected Time of next pulse Cycle - Expected elapsed logical time until next pulse M - Bound on clock value

33. The Byzantine Self-Stabilizing Pulse Synchronization Algorithm if (cycle_countdown = 0) then send “Propose-Pulse” message to all; if (received f+1 distinct “Propose-Pulse” messages) then send “propose-Pulse” message to all; if (received n-f distinct “Propose-Pulse” messages) then invoke “pulse” event; cycle_countdown := cycle; flush “Propose-Pulse” message counter; ignore “Propose-Pulse” messages for 2d(1+  ) time;

34. The Self-Stabilizing Byzantine Strong Agreement Algorithm Any Strong Byzantine Agreement algorithm can be used Agreement and validity is not ensured until the pulses synchronize Self-stabilization is supported by counting recovering nodes as correct only following cycle+time-for-BA of correct behavior We use a slightly modified version of the Toueg, Perry and Srikanth (1987) Strong Byzantine Agreement algorithm It has the advantage of “early stopping”: if all correct nodes start with identical values then termination is within 2 rounds Hence, during continuous correct system behavior clock synchronization is maintained with very little overhead

35. *global pulse Comparison chart AlgorithmSelf- Stabilizing /Byzantine Precision  AccuracyConvergence Time Messages PULSE-CSSS+BYZ 2d + O(  ) cycle+2(2f+5)dO(nf 2 ) NOADJUST- CS SS+BYZ 2d + O(  ) cycleO(n 2 ) DHSSBYZ d+O(  )(f+1)d+ O(  ) 2(f+1)dO(n 2 ) LL-APPROXBYZ 5  + O(  )  + O(  )d+O(  ) O(n 2 ) DW-SYNCH*SS+BYZ00M. 2 2(n-f) n 2. M. 2 2(n-f) DW-BYZ-SSSS+BYZ 4(n-f)  + O(  )(n-f)  +O(  ) O(n O(n) ) PT-SYNC*SS004n 2 O(n 2 )

36. The teaching of Pythagoras “Evolution is the law of life” “Unity is the law of God” “Number is the law of the universe”

37. “Everything in the universe is subject to predictable progressive cycles”. Pythagoras,

38. Future research: Noise tolerance of reciprocal cyclic/oscillatory behavior Things are not static, events happen again and again… Almost everything displays some degree of oscillatory behavior Examples: earth rotation, the moon spins around its axis with exactly the same period as its period of rotation around the earth, yearly seasons, day-night, human metabolic cycle, electron rotation, moods, fashion, culture, hormone levels, animal population levels, menstruation period, virus epidemics, animal migration, trends in politics, stocks, etc, etc, … The phenomena have a reciprocal effect on each other What mechanisms prevent from entering a chaotic mess? Refractoriness can be used to explain this point in simple terms It is a previously unused element in non-linear dynamics

39. Questions?

40. “ Synchronization is rewarding… ”

42. Related Problems Digital Clock Synchronization –Agreement on pulse counters, with or without a global pulse Clock Synchronization –Common notion of real time, high precision and accuracy Phase Clocks –Agreement on pulse counters in asynchronous settings Synchronized Rates –Clocks progress at approximately the same rate, the times may differ Firing Squad –All nodes enter the same state in step k after a process has initialized fire Pulse Synchronization –Precise synchronization of regular pulses, slack linear envelope accuracy

47. “Pulse Synchronization” algorithm (n, f, cycle) 1. Send a Message consisting of the Counter (“Pulse”) 2. Summate messages from the other computers received within a «time window» and update the Counter accordingly 3. If (Counter ≥ current Threshold) Then Goto 1 Else Goto 2

48. “Linear Envelope”

49. The hunting of the R k Given cycle, f and n, the solution to the whole problem is by solving the set of linear constraints on R

50. The hunting of the R k Given cycle, f and n, the solution to the whole problem is by solving the set of linear constraints on R How many equations (partitions) are there?

51. The hunting of the R k Given cycle, f and n, the solution to the whole problem is by solving the set of linear constraints on R How many equations (partitions) are there? Turns out there is no analytic solution to this question, (integer partitioning), the number is denoted P(k)

52. The hunting of the R k Given cycle, f and n, the solution to the whole problem is by solving the set of linear constraints on R How many equations (partitions) are there? Turns out there is no analytic solution to this question, (integer partitioning), the number is denoted P(k) Asymptotically equals e.g. P(50) = 204226 P(1000) = 24061467864032622473692149727991

53. Still hunting R k No computer can solve this exponential set of constraints or do Gaussian elimination Some other method for reaching a solution (if exists) is sought after