We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byClifford Criddle
Modified over 2 years ago
Database replication for commodity database services Gustavo Alonso Department of Computer Science ETH Zürich
©Gustavo Alonso. ETH Zürich.2 Replication as a problem
©Gustavo Alonso. ETH Zürich.3 How to replicate data? o Depending on when the updates are propagated: Synchronous (eager) Asynchronous (lazy) o Depending on where the updates can take place: Primary Copy (master) Update Everywhere (group) Synchronous Asynchronous Primary copy Update everywhere
©Gustavo Alonso. ETH Zürich.4 Theory … o The name of the game is correctness and consistency o Synchronous replication is preferred: copies are always consistent (1-copy serializability) programming model is trivial (replication is transparent) o Update everywhere is preferred: system is symmetric (load balancing) avoids single point of failure o Other options are ugly: inconsistencies centralized formally incorrect Synchronous Asynchronous Primary copy Update everywhere
©Gustavo Alonso. ETH Zürich.5 … and practice o The name of the game is throughput and response time o Asynchronous replication is preferred: avoid transactional coordination (throughput) avoid 2PC overhead (response time) o Primary copy is preferred: design is simpler (centralized) trust the primary copy o Other options are not feasible: overhead deadlocks do not scale Synchronous Asynchronous Primary copy Update everywhere
©Gustavo Alonso. ETH Zürich.6 The dangers of replication... SYNCHRONOUS o Coordination overhead distributed 2PL is expensive 2PC is expensive prefer performance to correctness o Communication overhead 5 nodes, 100 tps, 10 w/txn = 5’000 messages per second !! UPDATE EVERYWHERE o Deadlock/Reconciliation rates the probability of conflicts becomes so high, the system is unstable and does not scale o Useless work the same work is done by all administrative costs paid by everybody all nodes must understand replication (not trivial)
©Gustavo Alonso. ETH Zürich.7 Text book replication (BHG’87) Read One, Write All o Each site uses 2PL o Atomic commitment through 2PC o Read operations are performed locally o Write operations involve locking all copies of the data item (request a lock, obtain the lock, receive an acknowledgement) o Optimizations are based on the idea of quorums SITE ASITE BSITE C BOT R(x) W(x) Lock Upd... request ack change
©Gustavo Alonso. ETH Zürich.8 Response Time centralized databaseupdate T= replicated database update: 2N messages 2PC The way replication takes place (one operation at a time), increases the response time and, thereby, the conflict profile of the transaction. The message overhead is too high (even if broadcast facilities are available).
©Gustavo Alonso. ETH Zürich.9 o Approximated deadlock rate: if the database size remains constant, or if the database size grows with the number of nodes. o Optimistic approaches result in too many aborts. ABC BOT R(x) W(x) Lock D W(x) BOT Deadlocks (Gray et al. SIGMOD’96)
©Gustavo Alonso. ETH Zürich.10 Commercial systems Replication using distributed locking Number of Replicas Response Time in ms
©Gustavo Alonso. ETH Zürich.11 Cost of Replication o Overall computing power of the system: o No gain with large ws factor (rate of updates and fraction of the database that is replicated) o Quorums are questionable, reads must be local to get performance advantages.
©Gustavo Alonso. ETH Zürich.12 GANYMED: Solving the replication problem
©Gustavo Alonso. ETH Zürich.13 What can be done? o Are these fundamental limitations or side effects of the way databases work? Consistency vs. Performance: is this a real trade-off? Cost seems to be inherent: if all copies do the same, no performance gain Deadlocks: typical synchronization problem when using locks Communication overhead: ignored in theory, a real show- stopper in practice o If there are no fundamental limitations, can we do better? In particular, is there a reasonable implementation of synchronous, update everywhere replication? Consistency is a good idea Performance is also a good idea Nobody disagrees that it would be nice … … but commercial systems have given up on having both !!
©Gustavo Alonso. ETH Zürich.14 Consistency vs. Peformance o We want both: Consistency is good for the application Performance is good for the system o Then: Let the application see a consistent state... ... although the system is asynchronous and primary copy o This is done through: A middleware layer that offers a consistent view Using snapshot isolation as correctnes criteria REPLICATION MIDDLEWARE I see a consistent state Asynchronous Primary copy
©Gustavo Alonso. ETH Zürich.15 Two sides of the same coin SNAPSHOT ISOLATION o To the clients, the middleware offers snapshot isolation: Queries get their own consistent snapshot (version) of the database Update transactions work with the latest data Queries and updates do not conflict (operate of different data) First committer wins for conflicting updates o PostgreSQL, Oracle, MS SQL Server ASYNCH – PRIMARY COPY o Primary copy: master site where all updates are performed o Slaves: copies where only reads are peformed o A client gets a snapshot by running its queries on a copy o Middleware makes sure that a client sees its own updates and only newer snapshots o Updates go to primary copy and conflicts are resolved there (not by the middleware) o Updates to master site are propagated lazily to the slaves
©Gustavo Alonso. ETH Zürich.16 Ganymed: Putting it together Based on JDBC drivers Only scheduling, no concurrency control, no query processing... Simple messaging, no group communication Very much stateless (easy to make fault tolerant) Acts as traffic controller and bookkeeper Route queries to a copy where a consistent snapshot is available Keep track of what updates have been done where (propagation is not uniform)
©Gustavo Alonso. ETH Zürich.17 Linear scalability
©Gustavo Alonso. ETH Zürich.18 Improvements in response time (!!!)
©Gustavo Alonso. ETH Zürich.19 Fault tolerance (slave failure)
©Gustavo Alonso. ETH Zürich.20 Fault tolerance (master failure)
©Gustavo Alonso. ETH Zürich.21 GANYMED: Beyond conventional replication
©Gustavo Alonso. ETH Zürich.22 Oracle master – PostgreSQL slaves
©Gustavo Alonso. ETH Zürich.23 Oracle master – PostgreSQL slaves
©Gustavo Alonso. ETH Zürich.24 Updates through SQL (Oracle-Postgres)
©Gustavo Alonso. ETH Zürich.25 Updates through SQL (Oracle-Postgres)
©Gustavo Alonso. ETH Zürich.26 DB2 master – PostreSQL slaves
©Gustavo Alonso. ETH Zürich.27 DB2 master – PostreSQL slaves
©Gustavo Alonso. ETH Zürich.28 Critical issues o The results with a commercial master and open source slaves is still a proof of concept but a very powerful one o More work needs to be done (in progress) Update extraction from the master Trigger based = attach triggers to tables to report updates (low overhead at slaves, high overhead at master) Generic = propagate update SQL statements to copies (high overhead at slaves, no overhead at master, limitations with hidden updates) Update propagation = tuple based vs SQL based SQL is not standard (particularly optimized SQL) Understanding workloads (how much write load is really present in a database workload) Replicate only parts of the database (table fragments, tables, materialized views, indexes, specialized indexes on copies...)
©Gustavo Alonso. ETH Zürich.29 Query optimizations (DB2 example) SELECT J.i_id, J.i_thumbnail FROM item I, item J WHERE (I.i_related1 = j.i_id OR I.i_related2 = j.i_id OR I.i_related3 = j.i_id OR I.i_related4 = j.i_id OR I.i_related5 = j.i_id) AND i.i_id = 839;
©Gustavo Alonso. ETH Zürich.30 Query optimization (DB2 example) SELECT J.i_id, J.i_thumbnail FROM item J WHERE J.i_id IN ( (SELECT i_related1 FROM item WHERE i_id = 839) UNION (SELECT i_related2 FROM item WHERE i_id = 839) UNION (SELECT i_related3 FROM item WHERE i_id = 839) UNION (SELECT i_related4 FROM item WHERE i_id = 839) UNION (SELECT i_related5 FROM item WHERE i_id = 839) );
©Gustavo Alonso. ETH Zürich.31 Understanding workloads TPC-WUpdatesRead-onlyRatio Browsing3.21 %96.79 %1 : Shopping10.77 %89.23 %1 : 8.29 Ordering24.34 %75.66 %1 : 3.11 COSTRatio (avg) updates : read only Ratio (total) updates : read only Browsing7:50 : : Shopping6.38 : : Ordering7.70 : : Ratio (avg) updates : read only Ratio (total) updates : read only 6.92 : : : : : : NON-OPTIMIZED SQLOPTIMIZED SQLPOSTGRES
©Gustavo Alonso. ETH Zürich.32 A new twist to Moore´s Law o What is the cost of optimization? SQL rewriting = several days two/three (expert) people (improvement ratio between 5 and 10) Ganymed = a few PCs with open source software (improvement factor between 2 and 5 for optimized SQL, for non-optimized SQL multiply by X) o Keep in mind: Copies do not need to be used, they can be kept dormant until increasing load demands more capacity Several database instances can share a machine (database scavenging) We do not need to replicate everything (less overhead for extraction)
©Gustavo Alonso. ETH Zürich.33 SQL is not SQL SELECT * FROM ( SELECT i_id, i_title, a_fname, a_lname, SUM(ol_qty) AS orderkey FROM item, author, order_line WHERE i_id = ol_i_id AND i_a_id = a_id AND ol_o_id > (SELECT MAX(o_id)-3333 FROM orders) AND i_subject = 'CHILDREN' GROUP BY i_id, i_title, a_fname, a_lname ORDER BY orderkey DESC ) WHERE ROWNUM <= 50 Amongst the 3333 most recent orders, the query performs a TOP-50 search to list a category's most popular books based on the quantity sold Virtual column specific to Oracle. In PostgreSQL = LIMIT 50 Use of MAX leads to sequential scan in Postgres, change to: SELECT o_id-3333 FROM orders ORDER BY o_id DESC LIMIT 1 Current version does very basic optimizations on the slave side. Further work on optimizations at the middleware layer will boost performance even more Optimizations can be very specific to the local data
©Gustavo Alonso. ETH Zürich.34 GANYMED: Our real goals
©Gustavo Alonso. ETH Zürich.35 Database scavenging o Ideas similar to cluster/grid computing tools that place computing jobs in a pool of computers o We want to dynamically place database slaves for master databases in a pool of computers o The goal is to have a true low cost, autonomic database cluster GANYMED DB-MASTER A DB-MASTER B DB-MASTER C DB-MASTER D SLAVE CLUSTER
©Gustavo Alonso. ETH Zürich.36 Steps to get there o We already have the performance and scalability gain o We already have the ability to replicate commercial engines (Oracle, DB2, SQL Server) o What is missing Optimization of write set extraction or SQL update propagation Optimization of SQL statements forwarded to slaves Optimization of replication strategies in slaves Dynamic creation of slaves (many possibilities) Autonomic strategies for dynamic creation/deletion of slaves Grid engine for resource management
©Gustavo Alonso. ETH Zürich.37 Databases as commodity service o Remote applications use the database through a web services enabled JDBC driver (WS-JDBC) GANYMED DB-MASTER A DB-MASTER B DB-MASTER C DB-MASTER D SLAVE CLUSTER WEB SERVICES INTERFACE (WS-JDBC) INTERNET
©Gustavo Alonso. ETH Zürich.38 Conclusions o Ganymed synthesizes a lot of previous work in DB replication Postgres-R (McGill) Middle-R (Madrid Technical Uni.) Middleware based approaches (U. Of T.) C-JDBC (INRIA Grenoble, Object Web) ... o Contributions There is nothing comparable in open source solutions Database independent Very small footprint Easily extensible in many context Can be turned into a lazy replication engine Can be used for data caching across WANs Almost unlimited scalability for dynamic content \ web data o Very powerful platform to explore innovative approaches Databases as a commodity service Database scavenging Optimizations to commercial engines through open source slaves
1 Advanced Database Application Development Performance Tuning Performance Benchmarks Standardization E-Commerce Legacy Systems.
Copyright © 2007 Ramez Elmasri and Shamkant B. Navathe Slide Types of Distributed Database Systems Homogeneous All sites of the database system have.
Database System Concepts, 5th Ed. ©Silberschatz, Korth and Sudarshan See for conditions on re-usewww.db-book.com Chapter 20: Database System.
©Silberschatz, Korth and Sudarshan18.1Database System Concepts Chapter 18: Database System Architectures Centralized Systems Client--Server Systems Parallel.
1 Scaleable Replicated Databases Jim Gray (Microsoft) Pat Helland (Microsoft) Dennis Shasha (Columbia) Pat ONeil (U.Mass)
Database System Concepts, 6 th Ed. ©Silberschatz, Korth and Sudarshan See for conditions on re-usewww.db-book.com Chapter 17: Database.
Distributed databases 1. 2 Outline introduction principles / objectives problems.
The Architecture of Transaction Processing Systems Chapter 26.
1. Parallel Databases Introduction I/O Parallelism Interquery Parallelism Intraquery Parallelism Intraoperation Parallelism Interoperation Parallelism.
1 Database Systems: Design, Implementation, and Management CHAPTER 10 Distributed Database Management System.
1 Consistency in Distributed Systems Recall the fundamental DS properties – DS may be large in scale and widely distributed 1.concurrent execution of components.
© 2013 A. Haeberlen NETS 212: Scalable and Cloud Computing 1 University of Pennsylvania Storage at Facebook December 3, 2013.
1 System Busses / Networks-on-Chip EECE Advanced Topics in VLSI Design Spring 2009 Brad Quinton.
1 Data Warehousing Denis Manley Enterprise Systems FT228/3.
Chapter 14 Query Optimization. ©Silberschatz, Korth and Sudarshan14.2Database System Concepts 3 rd Edition Chapter 14: Query Optimization Introduction.
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 10Slide 1 Chapter 10 Architectural Design.
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 11Slide 1 Chapter 11 Distributed Systems Architectures.
Copyright © George Coulouris, Jean Dollimore, Tim Kindberg This material is made available for private study and for direct.
Copyright: Silberschatz, Korth and Sudarshan 1 Chapter 23: Advanced Data Types and New Applications.
1 SYSTEM MODEL From Chapter 2 of Distributed Systems Concepts and Design,4 th Edition, By G. Coulouris, J. Dollimore and T. Kindberg Published by Addison.
Database System Concepts, 6 th Ed. ©Silberschatz, Korth and Sudarshan See for conditions on re-usewww.db-book.com Chapter 12: Query Processing.
VMware vCenter Server High Availability Product Support Engineering VMware Confidential.
1 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved. Distributed Systems Architectures.
Database System Concepts, 6 th Ed. ©Silberschatz, Korth and Sudarshan See for conditions on re-usewww.db-book.com Chapter 26: Advanced.
Intelligent People. Uncommon Ideas. 1 Building a Scalable Architecture for Web Apps - Part I (Lessons Directi) By Bhavin Turakhia CEO, Directi.
1 Computer Networks: A Systems Approach, 5e Larry L. Peterson and Bruce S. Davie Chapter 9 Applications Copyright © 2010, Elsevier Inc. All rights Reserved.
Alter – Information Systems 4th ed. © 2002 Prentice Hall 1 Understanding Systems From a Business Viewpoint.
Transaction Indra Budi Fakultas Ilmu Komputer UI 2 Exercise A series of actions to be taken on the database such that either all actions.
1 CSE 380 Computer Operating Systems Instructor: Insup Lee University of Pennsylvania Fall 2003 Lecture Notes: Multiprocessors (updated version)
©Silberschatz, Korth and Sudarshan8.1Database System Concepts, 5 th Ed, slide version 5.0, August Chapter 8: Application Design and Development.
© 2016 SlidePlayer.com Inc. All rights reserved.