1. EECS 498 Introduction to Distributed Systems Fall 2017
Harsha V. Madhyastha

2. November 29, 2017
EECS 498 – Lecture 20

3. November 29, 2017
EECS 498 – Lecture 20

4. Context for Spanner Distributed database that supports transactions
BigTable: Database spread across servers within a datacenter
Spanner: Database spread across data centers
Employee ID
Date of Joining
Age
Title
Salary
November 29, 2017
EECS 498 – Lecture 20

5. Support for Transactions
Serializability
Externally visible effects equivalent to some serial order of execution
How is this different from linearizability?
Serializability does not guarantee real time order
Spanner offers strict serializability
Serializability + Linearizability
Previously considered impractical at global scale
November 29, 2017
EECS 498 – Lecture 20

6. Example: Social Network
Transaction T1:
Remove supervisor from friend’s list
Make a negative post about supervisor
Transaction T2:
Read friend list
Read posts made by friends
November 29, 2017
EECS 498 – Lecture 20

7. Executing Transactions
Transaction latency = 4 rounds of wide-area communication
2 rounds for each phase of 2PL
November 29, 2017
EECS 498 – Lecture 20

8. Executing Transactions
Leader-based Paxos  each phase of 2PL in 1 round-trip
Still too slow for read-only transactions
November 29, 2017
EECS 498 – Lecture 20

9. Read-Only Txns with 2PL TC P1 P2 P3 Lock Commit November 29, 2017
EECS 498 – Lecture 20

10. Lock-free Read-Only Txns
How to ensure T1’s read from P1 returns data from before T2?
TC1
Read P1 P2
Lock
Commit
TC2
November 29, 2017
EECS 498 – Lecture 20

11. Snapshot Reads Every transaction has a commit timestamp
When a row is modified by transaction, create new version
Transaction’s timestamp is version number
Read-only transaction reads latest version of data committed before its timestamp
What property must be true for linearizability?
Commit timestamps should monotonically increase
November 29, 2017
EECS 498 – Lecture 20

12. Executing Transactions
How to ensure commit timestamps monotonically increase?
November 29, 2017
EECS 498 – Lecture 20

13. Synchronizing Clocks Accurate sources of time:
GPS
Atomic clocks
Why not put these into every server?
Spanner deploys a few of these in every DC
All servers periodically sync with timemasters
November 29, 2017
EECS 498 – Lecture 20

14. TrueTime Architecture
GPS timemaster
GPS timemaster
GPS timemaster
GPS timemaster
Atomic-clock timemaster
GPS timemaster
Client
Datacenter 1
Datacenter 2 …
Datacenter n
Current time = now ± ε
November 29, 2017
EECS 498 – Lecture 20

15. TrueTime Implementation
Between syncs, assume worst case clock drift ε
+6ms
200 μs/sec
reference
uncertainty
time
0sec
30sec
60sec
90sec
November 29, 2017
EECS 498 – Lecture 20

16. TrueTime TT.now() returns a range: [earliest, latest]
Current time is …
… not less than earliest
… not greater than latest
High-level principle:
Known unknowns better than unknown unknowns
November 29, 2017
EECS 498 – Lecture 20

17. Timestamps and TrueTime
Acquired locks
Release locks T
Pick commit timestamp s s
Wait until TT.now().earliest > s
= TT.now().latest
average ε
average ε
November 29, 2017
EECS 498 – Lecture 20

18. Leader Leases
Must ensure at most one leader in any replica group at any point in time
In Spanner, leader gets a lease on how long it gets to be the leader
How can group members come to consensus on leader leases to ensure leader disjointness?
Assume clocks are perfectly synchronized, but network has arbitrary delays
How to leverage TrueTime?
November 29, 2017
EECS 498 – Lecture 20

19. Paxos-based Leader Leases
When a replica responds to a proposer at time t, it grants lease from [t, t + 10s]
t = TT.now().earliest()
Proposer assumes leader lease until minimum of replica leases
Lease valid until TT.now().latest > min lease
November 29, 2017
EECS 498 – Lecture 20

20. Schema-Change Transactions
Not scalable to acquire locks on all shards
How can we leverage TrueTime?
Schedule transaction for time in future
Employee ID
Date of Joining
Age
Title
Salary
Gender
November 29, 2017
EECS 498 – Lecture 20

21. Cloud Spanner Spanner recently introduced to Google Cloud
Claims to offer high availability
Spanner’s communication over Google’s private network
Well engineered over last decade
Partitions are very uncommon
Spanner unavailability << VM unavailability
Don’t need app logic to deal with Spanner outage
November 29, 2017
EECS 498 – Lecture 20

22. Causes of Spanner Outages
November 29, 2017
EECS 498 – Lecture 20