Correctness of Gossip-Based Membership under Message Loss Maxim Gurevich, Idit Keidar Technion

The Setting Many nodes – n ▫ 10,000s, 100,000s, 1,000,000s, … Come and go ▫ Churn Fully connected network ▫ Like the Internet Every joining node knows some others ▫ (Initial) Connectivity

Membership: Each Node Needs To Know Some Live Nodes Applications ▫ Gossip partners ▫ Unstructured overlay networks ▫ Gathering statistics Work best with random node samples ▫ Gossip algorithms converge fast ▫ Overlay networks are robust, good expanders ▫ Statistics are accurate

Membership Protocols Each node has a view ▫ Set of node ids ▫ Supplied to the application ▫ Used by membership protocol for maintenance ▫ Modeled as a directed graph uv w vyw… y

Desirable Properties Randomness… Holy grail for samples: IID ▫ Each sample uniformly distributed ▫ Each sample independent of other samples  Avoid spatial dependencies among view entries  Avoid correlations between nodes ▫ Good load balance among nodes

What About Churn? Desirable Properties Cont’d Views should constantly evolve ▫ Remove failed nodes, add joining ones Views should evolve to IID from any state Minimize temporal dependencies ▫ Dependence on the past should decay quickly ▫ Useful for application requiring fresh samples

Do Existing Protocols Measure Up?

……w… Existing Work: Practical Protocols Studied only empirically ▫ Good load balance [Lpbcast, Jelasity et al 07] ▫ Fast decay of temporal dependencies [Jelasity et al 07] ▫ Induces spatial dependence  ……z… uv w v…w… w z Example: Push protocol

v…z… Existing Work: Analysis Analyzed theoretically [Allavena et al 05, Mahlmann et al 06] ▫ Uniformity, load balance, spatial independence ▫ Unrealistic assumptions   Atomic actions with bi-directional communication  No message loss ▫ No bounds on decay of temporal dependencies  ……z… ……w… uv w v…w… w z Shuffle protocol z

Our Contribution: Bridge This Gap Formally specify desirable properties outlined above A practical protocol ▫ Tolerates message loss, churn, failures ▫ No complex bookkeeping for atomic actions Formally prove the desirable properties ▫ Including under message loss

…… Send & Forget Membership The best of push and shuffle Some view entries may be empty uv w v…w… uw uw

S&F: Message Loss Message loss ▫ Or no empty entries in v’s view uv w u v w

S&F: Compensating for Loss Edges (view entries) disappear due to loss Need to prevent views from emptying out Keep the sent ids when too little ids in view uv w u v w

S&F: Advantages over Other Protocols No bi-directional communication ▫ No complex bookkeeping ▫ Tolerates message loss Simple ▫ Amenable to formal analysis Easy to implement

Proving all desirable properties ▫ Analytical: degrees distribution w/out loss  Used in setting duplication threshold ▫ Markov 1: degree distribution with loss ▫ Markov 2: Markov Chain of reachable global states  IID samples, Temporal Independence Hold even under (reasonable) message loss! Key Contribution: Analysis

Analytic Degree Distribution Similar (better) to that of a random graph Validated by a more accurate Markov model

Proving all desirable properties ▫ Analytical: degrees distribution w/out loss  Used in setting duplication threshold ▫ Markov 1: degree distribution with loss ▫ Markov 2: Markov Chain of reachable global states  IID samples, Temporal Independence Hold even under (reasonable) message loss! Key Contribution: Analysis

… Node Degree Markov Chain Numerically compute the stationary distribution Transitions without loss Transitions due to loss State corresponding to isolated node outdegree indegree … … … … … … …

Results Outdegree is bounded by the protocol Decreases with increasing loss Indegree is not bounded Low variance even under loss Typical overload at most 2x

Proving all desirable properties ▫ Analytical: degrees distribution w/out loss  Used in setting duplication threshold ▫ Markov 1: degree distribution with loss ▫ Markov 2: Markov Chain of reachable global states  IID samples, Temporal Independence Hold even under (reasonable) message loss! Key Contribution: Analysis

Decay of Spatial Dependencies For uniform loss < 15%, dependencies decay faster than they are created 1 – 2  loss rate fraction of view entries are independent ▫ E.g., for loss rate of 3%  more than 90% of entries are independent uv w u v w u does not delete the sent ids …

Temporal Independence Dependence on past views decays within O(log n  view size) time Use “expected conductance” Ids travel fast enough ▫ Reach random nodes in O(log n) hops ▫ Due to “sufficiently many” independent ids in views - previous slide

Conclusions Formalized the desired properties of a membership protocol Send & Forget protocol ▫ Simple for both implementation and analysis Analysis under message loss ▫ Load balance ▫ Uniformity ▫ Spatial Independence ▫ Temporal Independence

Thank You