1. Dynamo: Amazon's Highly Available Key-value Store
Guiseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swami Sivasubramanian, Peter Vosshall, and Werner Vogels

2. Cloud Data Services & Relational Database Systems go hand in hand
Oracle, Microsoft SQL Server and even MySQL have traditionally powered enterprise and online data clouds
Clustered - Traditional Enterprise RDBMS provide the ability to cluster and replicate data over multiple servers – providing reliability
Highly Available – Provide Synchronization (“Always Consistent”), Load-Balancing and High-Availability features to provide nearly 100% Service Uptime
Structured Querying – Allow for complex data models and structured querying – It is possible to off-load much of data processing and manipulation to the back-end database

3. Traditional RDBMS clouds are: EXPENSIVE
Traditional RDBMS clouds are: EXPENSIVE! To maintain, license and store large amounts of data
The service guarantees of traditional enterprise relational databases like Oracle, put high overheads on the cloud
Complex data models make the cloud more expensive to maintain, update and keep synchronized
Load distribution often requires expensive networking equipment
To maintain the “elasticity” of the cloud, often requires expensive upgrades to the network

4. The Solution
Downgrade some of the service guarantees of traditional RDBMS
Replace the highly complex data models Oracle and SQL Server offer, with a simpler one – This means classifying service data models based on the complexity of the data model they may required
Replace the “Always Consistent” guarantee synchronization model with an “Eventually Consistent” model – This means classifying services based on how “updated” its data set must be
Redesign or distinguish between services that require a simpler data model and lower expectations on consistency.
We could then offer something different from traditional RDBMS!

5. The Solution
Amazon’s Dynamo – Used by Amazon’s EC2 Cloud Hosting Service. Powers their Elastic Storage Service called S2 as well as their E-commerce platform
Offers a simple Primary-key based data model. Stores vast amounts of information on distributed, low- cost virtualized nodes
Google’s BigTable – Google’s principle data cloud, for their services – Uses a more complex column-family data model compared to Dynamo, yet much simpler than traditional RMDBS Google’s underlying file-system provides the distributed architecture on low-cost nodes
Facebook’s Cassandra – Facebook’s principle data cloud, for their services. This project was recently open-sourced. Provides a data-model similar to Google’s BigTable, but the distributed characteristics of Amazon’s Dynamo

6. What is Dynamo
Dynamo is a highly available distributed key- value storage system
put(), get() interface
Sacrifices consistency for availability
Provides storage for some of Amazon's key products (e.g., shopping carts, best seller lists, etc.)‏
Uses “synthesis of well known techniques to achieve scalability and availability”
Consistent hashing, object versioning, conflict resolution, etc.

7. Scale Amazon is busy during the holidays Failure is common
Shopping cart: tens of millions of requests for 3 million checkouts in a single day
Session state system: 100,000s of concurrently active sessions
Failure is common
Small but significant number of server and network failures at all times
“Customers should be able to view and add items to their shopping cart even if disks are failing, network routes are flapping, or data centers are being destroyed by tornados.”

8. Flexibility Minimal need for manual administration
Nodes can be added or removed without manual partitioning or redistribution
Apps can control availability, consistency, cost- effectiveness, performance
Can developers know this up front?
Can it be changed over time?

9. System Assumptions & Requirements
Query Model: simple read and write operations to a data item that is uniquely identified by a key.
values are small (<1MB) binary objects
No ACID Properties: Atomicity, Consistency, Isolation, Durability.
Efficiency: latency requirements which are in general measured at the 99.9th percentile of the distribution.
Other Assumptions: operation environment is assumed to be non-hostile and there are no security related requirements such as authentication and authorization.

10. Service level agreements
SLAs are used widely at Amazon
Sub-services must meet strict SLAs
e.g., 300ms response time for 99.9% of requests at peak load of 500 requests/s
Average-case SLAs are not good enough
Mentioned a cost-benefit analysis that said 99.9% is the right number
Rendering a single page can make requests to 150 services
Service-oriented architecture of
Amazon’s platform

11. Sacrifice strong consistency for availability
Eventual consistency
“Always writable”
Can always write to shopping cart
Pushes conflict resolution to reads
Application-driven conflict resolution
e.g., merge conflicting shopping carts
Or Dynamo enforces last-writer-wins
How often does this work?

12. Other Design Principles
Incremental scalability
Minimal management overhead
Symmetry
No master/slave nodes
Decentralized
Centralized control leads to too many failures
Heterogeneity
Exploit capabilities of different nodes

13. Summary of techniques used in Dynamo and their advantages
Problem
Technique
Advantage
Partitioning
Consistent Hashing
Incremental Scalability
High Availability for writes
Vector clocks with reconciliation during reads
Version size is decoupled from update rates.
Handling temporary failures
Sloppy Quorum and hinted handoff
Provides high availability and durability guarantee when some of the replicas are not available.
Recovering from permanent failures
Anti-entropy using Merkle trees
Synchronizes divergent replicas in the background.
Membership and failure detection
Gossip-based membership protocol and failure detection.
Preserves symmetry and avoids having a centralized registry for storing membership and node liveness information.

14. Interface
get(key) returns object replica(s) for key, plus a context object
context encodes metadata, opaque to caller
put(key, context, object) stores object

15. Variant of consistent hashing
Key K A G B
Each node is assigned to multiple points in the ring
(e.g., B, C, D store keyrange (A, B) F C
# of points can be assigned based on node’s capacity E
If node becomes unavailable, load is distributed to others D

16. Replication Key K Coordinator for key K A G B
B maintains a preference list for each data item specifying nodes storing that item F C
Preference list skips virtual nodes in favor of physical nodes E D
D stores (A, B], (B, C], (C, D]

17. Data versioning
put() can return before update is applied to all replicas
Subsequent get()s can return older versions
This is okay for shopping carts
Branched versions are collapsed
Deleted items can resurface
A vector clock is associated with each object version
Comparing vector clocks can determine whether two versions are parallel branches or causally ordered
Vector clocks passed by the context object in get()/put()
Application must maintain this metadata?

18. Vector Clock A vector clock is a list of (node, counter) pairs.
Every version of every object is associated with one vector clock.
If the counters on the first object’s clock are less-than-or-equal to all of the nodes in the second clock, then the first is an ancestor of the second and can be forgotten.

19. Vector clock example

20. “Quorum-likeness” get() & put() driven by two parameters:
R: the minimum number of replicas to read
W: the minimum number of replicas to write
R + W > N yields a “quorum-like” system
Latency is dictated by the slowest R (or W) replicas
Sloppy quorum to tolerate failures
Replicas can be stored on healthy nodes downstream in the ring, with metadata specifying that the replica should be sent to the intended recipient later

21. Adding and removing nodes
Explicit commands issued via CLI or browser
Gossip-style protocol propagates changes among nodes
New node chooses virtual nodes in the hash space

22. Implementation
Persistent store either Berkeley DB Transactional Data Store, BDB Java Edition, MySQL, or in-memory buffer w/ persistent backend
All in Java!
Common N, R, W setting is (3, 2, 2)
Results are from several hundred nodes configured as (3, 2, 2)
Not clear whether they run in a single datacenter…

23. One tick = 12 hours

24. One tick = 1 hour

25. During periods of high load popular objects dominate
During periods of low load, fewer popular objects are accessed
One tick = 30 minutes

26. Quantifying divergent versions
In a 24 hour trace
99.94% of requests saw exactly one version
% received 2 versions
% received 3 versions
% received 4 versions
Experience showed that diversion came usually from concurrent writers due to automated client programs (robots), not humans

27. Conclusions Scalable: Simple: Flexible: Inflexible: Interesting?
Easy to shovel in more capacity at Christmas
Simple:
get()/put() maps well to Amazon’s workload
Flexible:
Apps can set N, R, W to match their needs
Inflexible:
Apps have to set N, R, W to match their needs
Apps may have to do their own conflict resolution
They claim it’s easy to set these – does this mean that there aren’t many interesting points?
Interesting?