Consistency-Based Service Level Agreements for Cloud Storage Douglas B. Terry, Vijayan Prabhakaran, Ramakrishna Kotla, Mahesh Balakrishnan, Marcos K. Aguilera, Hussam Abu-Libdeh Microsoft Research

“A foolish consistency is the hobgoblin of little minds” -- Ralph Waldo Emerson (1841) “… and of large clouds” -- Douglas Brian Terry (2013) 2

Today’s Cloud Storage Providers Replicate data widely Offer choice of strong or eventual consistency e.g. Amazon DynamoDB, Yahoo PNUTS, Google App Engine, Oracle NoSQL, Cassandra, … Microsoft Windows Azure Tradeoff consistency, availability and performance 3

Problem Developers must choose consistency No single choice is best for all clients and situations 4 Client Consistency U.S. (secondary) England (primary) India (secondary) China (client only) strong eventual roundtrip times in milliseconds

Pileus key features Replicated, partitioned key-value store Choice of consistency Consistency-based service level agreements (SLAs) 5 a cap cloud

Pileus System Model 6 primary core secondary nodes Get(key, SLA) Put(key, value) Get(key, SLA) sync replication lazy replication Get(key, SLA) API BeginSession (SLA) BeginTx (SLA) Put (key, value) Get (key, SLA) returns value, consistency EndTx () EndSession ()

Read Consistency Guarantees 7 Strong ConsistencyReturn value of latest Put. Causal Consistency Return value of latest causally preceding Put. Bounded Staleness (t) Return value that is stale by at most t seconds. Read My Writes Return value of latest Put in client session or a later value. Monotonic Reads Return same or later value as earlier Get in client session. Eventual ConsistencyReturn value of any Put. [COPS 2011 ] [TACT 2002 ] [Bayou 1994 ]

Read Latencies 8 Client/ Consistency U.S. (secondary) England (primary) India (secondary) China (client only) strong causal bounded(30) read-my-writes monotonic eventual roundtrip times in milliseconds consistency affects latency client location affects latency

Consistency-based SLA Applications declare desired consistency/latency 9 strong300 ms. consistencylatency read my writes300 ms utility 0.5 eventual300 ms Shopping Cart:

SLA Enforcement: Client Monitoring NodePrimary?RTTsHigh Timestamp Ayes210 Bno166 Cno For each tablet: from configuration service measured on Gets, Puts, and pings returned from Gets, Puts, and pings

SLA Enforcement: Node Selection 1.For each subSLA and node, a.compute P latency b.compute P consistency c.compute P latency x P consistency x utility 2.Select node with maximum expected utility 3.Send Get operation to node 4.Measure RTT and update records 5.Return data and delivered consistency to caller 11 On Get (key, SLA):

Experimental Setup System configuration: Primary: England Secondaries: U.S., India Clients: U.S., England, India, China Benchmark: YCSB with 50/50 Gets/Puts 500-op sessions Node selection schemes: Primary = get from primary Random = get from random node Closest = get from closest node Pileus = get from node with highest expected utility Measurement: Average utility for Get operations U.S. England China India

Experiment #1: SLA Simplified shopping cart SLA: 13 consistencylatency read my writes300 ms utility 1.0 eventual300 ms.0.5

Experiment #1: Delivered Utility 14 Average utility per Get Client datacenter (secondary) (primary)(client only)

Experiment #1: Delivered Utility 15 Average utility per Get Client datacenter (secondary) (primary)(client only) Primary selection works well when close to the primary, but poorly when distant

Experiment #1: Delivered Utility 16 Average utility per Get Client datacenter (secondary) (primary)(client only) Random selection rarely works well

Experiment #1: Delivered Utility 17 Average utility per Get Client datacenter (secondary) (primary)(client only) 100% Gets from England; 100% meet top subSLA

Average utility per Get Experiment #1: Delivered Utility 18 Client datacenter (secondary) (primary)(client only) 91% from U.S.; 9% from England; 100% meets top subSLA 14.5 ms. avg. latency vs. 148 ms. for primary

Experiment #1: Delivered Utility 19 Average utility per Get Client datacenter (secondary) (primary)(client only) 99.6% from U.S.; 0.4% from India; 96% meets top subSLA

(secondary) (primary)(client only) Client datacenter Experiment #1: Delivered Utility 20 Pileus always delivers the most utility! Average utility per Get

Experiment #1: Delivered Utility 21 Average utility per Get Client datacenter (secondary) (primary)(client only) 9% fail to meet read- my-write

Experiment #2: SLA Password checking SLA: 22 consistencylatency strong150 ms utility 1.0 eventual150 ms strong1000 ms.0.25

Experiment #2: Delivered Utility 23 Average utility per Get Client datacenter (secondary) (primary)(client only)

Conclusions: Main Contributions Our Pileus system provides a broad choice of consistency guarantees and range of delivered read latency allows declarative specification of desired consistency and latency through consistency-based SLAs selects nodes to maximize expected utility while adapting to varying conditions