Dynamo: Amazon's Highly Available Key-value Store Guiseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swami Sivasubramanian, Peter Vosshall, and Werner Vogels Presented by Steve Schlosser Big Data Reading Group October 1, 2007

What Dynamo is Dynamo is a highly available distributed key- value storage system  put(), get() interface  Sacrifices consistency for availability  Provides storage for some of Amazon's key products (e.g., shopping carts, best seller lists, etc.)‏  Uses “synthesis of well known techniques to achieve scalability and availability” Consistent hashing, object versioning, conflict resolution, etc.

Scale Amazon is busy during the holidays  Shopping cart: tens of millions of requests for 3 million checkouts in a single day  Session state system: 100,000s of concurrently active sessions Failure is common  Small but significant number of server and network failures at all times “Customers should be able to view and add items to their shopping cart even if disks are failing, network routes are flapping, or data centers are being destroyed by tornados.”

Flexibility Minimal need for manual administration Nodes can be added or removed without manual partitioning or redistribution Apps can control availability, consistency, cost- effectiveness, performance  Can developers know this up front?  Can it be changed over time?

Assumptions & requirements Simple query model  values are small (<1MB) binary objects No ACID properties  Weaker consistency  No isolation guarantees  Single key updates Stringent latency requirements  99.9th percentile Non-hostile environment

Service level agreements SLAs are used widely at Amazon Sub-services must meet strict SLAs  e.g., 300ms response time for 99.9% of requests at peak load of 500 requests/s  Average-case SLAs are not good enough  Mentioned a cost-benefit analysis that said 99.9% is the right number Rendering a single page can make requests to 150 services

Consistency Eventual consistency “Always writable”  Can always write to shopping cart  Pushes conflict resolution to reads Application-driven conflict resolution  e.g., merge conflicting shopping carts  Or Dynamo enforces last-writer-wins  How often does this work?

Other stuff Incremental scalability  Minimal management overhead Symmetry  No master/slave nodes Decentralized  Centralized control leads to too many failures Heterogeneity  Exploit capabilities of different nodes

Interface get(key) returns object replica(s) for key, plus a context object  context encodes metadata, opaque to caller put(key, context, object) stores object

Variant of consistent hashing A B C D E F G Key K Each node is assigned to multiple points in the ring (e.g., B, C, D store keyrange (A, B) # of points can be assigned based on node’s capacity If node becomes unavailable, load is distributed to others

Replication A B C D E F G Key K Coordinator for key K D stores (A, B], (B, C], (C, D] B maintains a preference list for each data item specifying nodes storing that item Preference list skips virtual nodes in favor of physical nodes

Data versioning put() can return before update is applied to all replicas Subsequent get()s can return older versions  This is okay for shopping carts  Branched versions are collapsed  Deleted items can resurface A vector clock is associated with each object version  Comparing vector clocks can determine whether two versions are parallel branches or causally ordered  Vector clocks passed by the context object in get()/put() Application must maintain this metadata?

Vector clock example

“Quorum-likeness” get() & put() driven by two parameters:  R: the minimum number of replicas to read  W: the minimum number of replicas to write R + W > N yields a “quorum-like” system Latency is dictated by the slowest R (or W) replicas Sloppy quorum to tolerate failures  Replicas can be stored on healthy nodes downstream in the ring, with metadata specifying that the replica should be sent to the intended recipient later

Adding and removing nodes Explicit commands issued via CLI or browser Gossip-style protocol propagates changes among nodes  New node chooses virtual nodes in the hash space

Implementation Persistent store either Berkeley DB Transactional Data Store, BDB Java Edition, MySQL, or in-memory buffer w/ persistent backend All in Java! Common N, R, W setting is (3, 2, 2) Results are from several hundred nodes configured as (3, 2, 2)  Not clear whether they run in a single datacenter…

One tick = 12 hours

One tick = 1 hour

One tick = 30 minutes During periods of high load popular objects dominate During periods of low load, fewer popular objects are accessed

Quantifying divergent versions In a 24 hour trace  99.94% of requests saw exactly one version  % received 2 versions  % received 3 versions  % received 4 versions Experience showed that diversion came usually from concurrent writers due to automated client programs (robots), not humans

Conclusions Scalable:  Easy to shovel in more capacity at Christmas Simple:  get()/put() maps well to Amazon’s workload Flexible:  Apps can set N, R, W to match their needs Inflexible:  Apps have to set N, R, W to match their needs  Apps may have to do their own conflict resolution  They claim it’s easy to set these – does this mean that there aren’t many interesting points? Interesting?