Eventual Consistency Jinyang

Review: Sequential consistency Sequential consistency properties: –All read/write ops follow some total ordering –Read must see result of latest write Realizing sequential consistency: –Reads/writes from a single node execute one at a time –All reads/writes to address X must be ordered by one memory/storage module responsible for X

Realizing sequential consistency W(A)1 W(A)2 Cache or replica Cache Or replica W(B)3 Invalidate, R(B)

Disadvantages of sequential consistency Requires highly available connections –Lots of chatter between clients/servers Not suitable for certain scenarios: –Disconnected clients (e.g. your laptop) –Apps might prefer potential inconsistency to loss of availability

Why (not) eventual consistency? Support disconnected operations –Better to read a stale value than nothing –Better to save writes somewhere than nothing 8Potentially anomalous application behavior –Stale reads and conflicting writes…

Sequential vs. eventual consistency Sequential: pessimistic conflict handling –Updates cannot take effect unless they are serialized first Eventual: optimistic conflict handling –Let updates happen, worry about whether they can be serialized later

Operating w/o total connectivity replica Client writes to its local replica W(A)1 W(A)2 Sync w/ server resolves non-conflicting changes, reports conflicting ones to user No sync between clients

Pair-wise synchronization replica W(A)1 W(A)2 W(B)3 Pair-wise sync resolves non-conflicting changes, reports conflicting ones to users

Examples usages? File synchronizers –One user, many gadgets

File synchronizer Goal 1.All replica contents eventually become identical 2.No lost updates –Do not replace new version with old ones

Prevent lost updates Detect if updates were sequential –If so, replace old version with new one –If not, detect conflict

How to prevent lost updates? Strawman: use mtime to decide which version should replace the other Problem? H1 H2 W(f)a mtime: W(f)c f W(f)b f f 15648

Strawman fix Carry the entire modification history If history X is a prefix of Y, Y is newer H1 W(f)a W(f)b W(f)c H1:15648 H1:16679 H1:15648 H2:23657

Compress version history H1 W(f)a W(f)b W(f)c H1:1 H1:2 H1:1 H1:2 H2:1 H1:1 H1:2 H1:2 implies H1:1, so we only need one number per host H1:1H1:2 H1:1H1:2 H2:1 H2

Compare vector timestamp H1:1 H2:3 H3:2 H1:1 H2:5 H3:7 H1:1 H2:3 H3:2 H1:2 H2:1 H3:7 < <

Using vector timestamp H1 W(f)a W(f)b W(f)c H1:1H1:2 H1:1 H2:1 H1:2 H2:1 H2

Using vector timestamp H1 W(f)a W(f)b W(f)c H1:1H1:2 H1:1 H2:1 H1:1 H2:1 H2

How to deal w/ conflicts? Easy: mailboxes w/ two different set of messages Medium: changes to different lines of a C source file Hard: changes to same line of a C source file After conflict resolution, what should the vector timestamp be?

What about file deletion? Can we forget about the vector timestamp for deleted files? Simple solution: treat deletion as a write –Conflicts involving a deleted file is easy Downside: –Need to remember vector timestamp for deleted files indefinitely

Tra [Cox, Josephson] What are Tra’s novel properties? –Easy to compress storage of vector timestamps –No need to check every file’s version vector during sync –Allows partial sync of subtrees –No need to keep timestamp for deleted files forever

Tra’s key technique Two vector timestamps: 1.One represents modification time –Tracks what a host has 2.One represents synchronization time –Tracks what a host knows Sync time implies no modification happens since mod time H1:1 H2:5 H3:7 H1:10 H2:20 H3:25

f1 f2 H1:0 H2:0 H1:0 H2:0 Using sync time H1 W(f1)a W(f2)b H1:1 H2:0 H2 H1:2 H2:0 f1 f2 H1:1 H1:2 H2:0 H1:2 H2:0 f2

Compress mtime and synctime dir synctime = element-wise min of child sync times dir mtime = element-wise max of child mod times Sync(d1  d1’) –Skip d1 if mtime of d1 is less than synctime of d1’ Can we achieve this with single mtime? –Skip d1 if mtime of d1 is less than mtime of d1’

Synctime enables partial synchronization Directory d1 contains f1 and f2, suppose host sync a subtree (d1/f1) –With synctime+mtime: synctime of d1 does not change. Mtime of d1 increases –With mtime-only: Mtime of d1 increases Host later syncs subtree d1/f2 –With synctime+mtime: will pull in modifications in f2 because synctime of d1 is smaller –With mtime-only: skips d1 because mtime is high enough

f2 H1:0 H2:0 Using sync time H1 W(f1)a W(f2)b H1:1 H2 H1:2f1f2 H1:2 H2:0 d Sync f1 only f1 H1:0 H2:0 H1:2 H1:0 H2:0 d f1 H1:1 H1:2 H2:0 H1:2 H1:0 H2:0 d Sync f2 only f1 H1:1 H1:2 H2:0 d f2 H1:2

f2 H1:0 How to deal w/ deletion H1 W(f1)a D(f2) H1:1 H2 f1f2 H1:2 H2:0 d f1 H1:0 H2:0 d H1:2 H2:0 Deletion notice for a deleted file contains its sync time f1 H1:1 H1:2 H2:0 d

f2 How to deal w/ deletion H1 W(f1)a D(f2) H1:1 H2 f1f2 H1:2 H2:0 d f1 H1:0 H2:1 d H1:2 H2:0 Deletion notice for a deleted file contains its sync time H2:1 f1 H1:1 H1:2 H2:1 d f2

Another definition of eventual consistency Eventual consistency (Tra) –All replica contents are eventually identical –Do not care about individual writes, just overwrite old replica w/ new one Eventual consistency (Bayou) –Writes are eventually applied in some total order –Reads might not see most recent writes in total order

Bayou Version Vector Write log N0:0 N1:0 N2:0 N0:0 N1:0 N2:0 N0:0 N1:0 N2:0 N0 N1 N2

Bayou propagation Version Vector Write log N0:3 N1:0 N2:0 N0 N1 N2 0:N0 W(x) 1:N0 W(y) 2:N0 W(z) N0:0 N1:1 N2:0 N0:0 N1:0 N2:0 0:N1 W(x) 0:N0 W(x) 1:N0 W(y) 2:N0 W(z) N0:3 N1:0 N2:0

Bayou propagation Version Vector Write log N0:3 N1:0 N2:0 N0 N1 N2 0:N0 W(x) 1:N0 W(y) 2:N0 W(z) N0:3 N1:4 N2:0 0:0 1:0 2:0 0:N0 W(x) 0:N1 W(x) 1:N0 W(y) 2:N0 W(z) 0:N1 W(x) N0:3 N1:4 N2:0

Bayou propagation Version Vector Write log N0 N1 N2 N0:3 N1:4 N2:0 N0:0 N1:0 N2:0 0:N0 W(x) 0:N1 W(x) 1:N0 W(y) 2:N0 W(z) N0:4 N1:4 N2:0 0:N0 W(x) 0:N1 W(x) 1:N0 W(y) 2:N0 W(z) Which portion of The log is stable?

Bayou propagation Version Vector Write log N0 N1 N2 N0:3 N1:4 N2:0 0:N0 W(x) 0:N1 W(x) 1:N0 W(y) 2:N0 W(z) N0:4 N1:4 N2:0 0:N0 W(x) 0:N1 W(x) 1:N0 W(y) 2:N0 W(z) N0:3 N1:4 N2:4 0:N0 W(x) 0:N1 W(x) 1:N0 W(y) 2:N0 W(z)

Bayou propagation Version Vector Write log N0 N1 N2 N0:3 N1:4 N2:4 0:N0 W(x) 0:N1 W(x) 1:N0 W(y) 2:N0 W(z) N0:4 N1:4 N2:0 0:N0 W(x) 0:N1 W(x) 1:N0 W(y) 2:N0 W(z) N0:3 N1:4 N2:4 0:N0 W(x) 0:N1 W(x) 1:N0 W(y) 2:N0 W(z) N0:3 N1:4 N2:4

Bayou uses a primary to commit a total order Why is it important to make log stable? –Stable writes can be committed –Stable portion of the log can be truncated Problem: If any node is offline, the stable portion of all logs stops growing Bayou’s solution: –A designated primary defines a total commit order –Primary assigns CSNs (commit-seq-no) –Any write with a known CSN is stable –All stable writes are ordered before tentative writes

Bayou propagation Version Vector Write log N0:3 N1:0 N2:0 N0 N1 N2 0:0:N0 W(x) 1:1:N0 W(y) 2:2:N0 W(z) N0:0 N1:1 N2:0 N0:0 N1:0 N2:0 ∞:0:N1 W(x) N0:0 N1:1 N2:0

Bayou propagation Version Vector Write log N0:4 N1:1 N2:0 N0 N1 N2 0:0:N0 W(x) 1:1:N0 W(y) 2:2:N0 W(z) N0:0 N1:1 N2:0 N0:0 N1:0 N2:0 ∞:0:N1 W(x) 3:0:N1 W(x) 0:0:N0 W(x) 1:1:N0 W(y) 2:2:N0 W(z) 3:3:N1 W(x) N0:4 N1:1 N2:0

Bayou’s limitations Primary cannot fail Server creation & retirement makes nodeID grow arbitrarily long Anomalous behaviors for apps? –Calendar app