1. Amazon’s Dynamo Simple Cloud Storage

2. Foundations 1970 – E.F. Codd “A Relational Model of Data for Large Shared Data Banks”E.F. Codd –Idea of tabular data –SQL Foundations Codd’s 12 rules –How database structured and what is available to user –Application not dependent on physical or logical levels of database –Insert, Update, Delete operators

3. Foundations, continued First Relational database management systems (RDBMS) –Oracle in 1979, first SQL based system –Microsoft SQL server, etc –Open source software would follow later (mySQL) Follows Codd’s ideas –Complexity on the server side, let the query do all the work –Very stringent requirements

4. Drawbacks Licensing fees on a per processor rate – High end Oracle system is mind-numbingly expensivemind-numbingly expensive Load Distribution requires specific nodes to handle – Some servers have specific roles, failure point in network Complexity on servers creates difficultly with maintenance, upgrades – Upgrades often all at once as result, not incremental

5. A New Direction Simplify services that the database provides – Easier scaling and error handling Take a more pragmatic approach – Tailor system to sacrifice some aspects of the traditional RDBMS to gain performance in others – Systems less general, specific end requirements in mind when creating

6. Examples Amazon Dynamo – Simple primary key – Highly available, end user based model – Low cost virtualized nodes Facebook Cassandra – Similar goals to Amazon’s Dynamo – Highly avaiable, incremental scalablilty Google File System – Master node – Data distributed across low cost nodes

7. Dynamo Goals Scale – adding systems to network causes minimal impact Symmetry – No special roles, all features in all nodes Decentralization – No Master node(s) Highly Available – Focus on end user experience SPEED – A system can only be as fast as the lowest level Service Level Agreements – System can be adapted to an application’s specific needs, allows flexibility

8. Dynamo Assumptions Query Model – Simple interface exposed to application level – Get(), Put() – No Delete() Atomicity, Consistency, Isolation, Durability – Operations either succeed or fail, no middle ground – System will be eventually consistent, no sacrifice of availability to assure consistency – Conflicts can occur while updates propagate through system – System can still function while entire sections of network are down Efficiency – Measure system by the 99.9 th percentile – Important with millions of users, 0.1% can be in the 10,000s Non Hostile Environment - No need to authenticate query, speed boost

9. Wanted Results Deliver requests in a bounded time Always writable – Highly available to users No dedicated roles Work split between nodes fairly

10. Techniques

11. Partitioning Consistent Hashing – Changing the number of slots in hash table results in only a small number of keys to remap – More info More info A ring of virtual nodes – Node responsible for region between it and its predecessor

12. Virtual Node Physical Machine has # of virtual nodes based on performance Can adapt load more easily if a machine goes down Likewise, assign nodes to a new machine in network

13. Replication Application provided parameter N Replication on different physical nodes – Data still available if nodes go down – Makes part of preference list for query

14. Versioning and Vector Clocks Updates propagate asynchronously, need a way of distinguishing conflicts – Possible reason for absence of Delete() Vector Clock – List of (node, counter) – Limited size, limit overhead for data – If all fields are less than or equal, first can be updated by second

15. Sloppy Quorum and Hinted Handoff W and R parameter set min # of nodes in a read or write Read and write on the first N healthy nodes, no strict membership, can vary over time Hint in metadata for intended node, will update once that node is again available Allows for temporary failure in nodes or entire networks

16. Synchronization and Gossip Merkle Trees - InfoInfo Use common key values between two nodes – Traverse tree and check vector clocks to see if updates are needed – Exchange information on most current version of the data if inconsistencies are found Gossip – Nodes select neighbors at random and reconcile membership change histories Use seed nodes to initialize Detect failures

17. Routing get() and put() Two Techniques – Route request through a load balancer Slower Simpler application level code – Partition aware client, route directly to appropriate nodes Faster More complicated application level First node routed to is “coordinator” node – Generates vector clock for put and gives data to N highest healthy nodes – Queries N highest nodes for all versions, returned all versions found

18. Implementation Java based – Hardware independent, JVM Allows different back-end systems to be used, based on size of data needed to be stored – Berkeley Database Transactional Data Store – BDB Java Edition – MySQL, can handle large objects Coordinator node is a state machine for read/writes for client – Coordinator for a write determined by fastest read

19. Flexibility Changing W, R, N – Business logic specific reconciliation Data replicated over nodes Application level reconciliation fro conflicting objects – Timestamp Reconciliation Similar to above, last write wins – High performance read engine By setting R = 1, W = N Reads fast and numerous, few updates

20. Observed Results - Speed

21. Observed Results – Load Balancing Higher traffic causes load to be balanced more evenly – Requests of popular keys let system to balance more easily In lower traffic, less important to balance load

22. Observed Results - Coordination Client coordination can provide a speed boost Read and write latency nearly identical Results as expected

23. Observed Results - Versions Measured over 24 hour period for shopping cart – 99.94% of users saw 1 version – 0.00057% saw 2 versions – 0.00047% saw 3 versions – 0.00009% saw 4 versions Increase caused by increase in number of concurrent writers, most likely

24. Conclusions Dynamo allows Amazon’s customers to have a consistent experience even in face of server and network errors Gives a scalable solution with millions of data points to be queried quickly and efficiently Offloads complexity to the application to provide a simple, flexible, and fast server-side implementation

25. Thanks for listening!