High-Availability MySQL DB based on DRBD-Heartbeat Ming Yue September 27, 2007 September 27, 2007

2 DRBD: Distributed Replication Block Device 1 Motivation 3 Integrating DRBD-Heartbeat-MySQL 4 Failovers Outline 5 Discussion

Outline 2 DRBD: Distributed Replication Block Device 1 Motivation 3 Integrating DRBD-Heartbeat-MySQL 4 Failovers 5 Discussion

1 Motivation To minimize service interruption time due to unacceptable long-time recovery; To maximize the uptime of our system; Solutions? Network Redundancy + Failover DRBD + Heartbeat High-Availability

Outline 2 DRBD: Distributed Replication Block Device 1 Motivation 3 Integrating DRBD-Heartbeat-MySQL 4 Failovers 5 Discussion

2 DRBD: Distributed Replication Block Device VFS Block Device Block Device... Mapping Layer Disk VFS Block Device Block Device... Mapping Layer Disk Network

2 DRBD: Distributed Replication Block Device (continued) A loadable kernel module of Linux providing real-time data replication to other block device on remote machine.

Block Device Disk Block Device Disk Network /dev/drbd0 Primary “devdrbd01” Secondary “devdrbd02 ” Real-time Replication - Primary/Secondary status is specified manually if no Heartbeat. - Create file system on /dev/drbd0 of Primary node at first. - Only Primary node’s /dev/drbd0 can be mounted to its local directories. /mydrbd (not mountable) mounted /dev/drbd0 2 DRBD: Distributed Replication Block Device (continued)

Block Device Disk Block Device Disk Network /dev/drbd0 (not mountable) Secondary “devdrbd01” Secondary “devdrbd02 ” 2 DRBD: Distributed Replication Block Device (continued)

Block Device Disk Block Device Disk Network /dev/drbd0 Secondary “devdrbd01” Primary “devdrbd02 ” /dev/drbd0 Real-time Replication After manual Primary/Secondary status switch mounted /mydrbd (not mountable) 2 DRBD: Distributed Replication Block Device (continued)

Outline 2 DRBD: Distributed Replication Block Device 1 Motivation 3 Integrating DRBD-Heartbeat-MySQL 4 Failovers 5 Discussion

Heartbeat: A well known high-availability resource manager 3.1 What does heartbeat serve here? In DRBD-Heartbeat-MySQL configuration: 1. Provides the Virtual IP address interface 2. Auto-starts MySQL server on the Primary 3. Talks with the peer’s heartbeat process and starts failover if the Primary’s heartbeat doesn’t respond

3.2 Integrating DRBD-Heartbeat-MySQL “devdrbd” (virtual IP address/hostname) Client /dev/drbd0 MySQL server Disk /var/lib/mysql Heartbeat /dev/drbd0 Disk Heartbeat mounted DRBD Replication Not mountable Secondary “devdrbd02” Primary “devdrbd01” Eth0 Interface Heartbeat Probing

2 DRBD: Distributed Replication Block Device 1 Motivation 3 Integrating DRBD-Heartbeat-MySQL Outline 4 Failovers 5 Discussion 4.1 Failover after Power/Network Interruption 4.2 DRBD Synchronization 4.3 Failover in MySQLstat Plot

4.1 Failover – Power/Network Interruption on the Primary “devdrbd” (virtual IP address/hostname) /dev/drbd0 MySQL server Disk /var/lib/mysql Heartbeat /dev/drbd0 Disk Heartbeat mounted Not mountable Secondary “devdrbd02” Primary “devdrbd01” Eth0 Interface Heartbeat Probing DRBD Replication Power Off No peer to Replicate Un-plug eth0 Cable “devdrbd01” of No peer responding No response from peer heartbeat. Switch!

“devdrbd” (virtual IP address/hostname) /dev/drbd0 Disk Heartbeat /var/lib/mysql MySQL server mounted Heartbeat Probing Eth0 Interface No DRBD Replication Primary “devdrbd02” 4.1 Failover – Power/Network Interruption on the Primary

“devdrbd” (virtual IP address/hostname) /dev/drbd0 Disk /dev/drbd0 Disk Heartbeat /var/lib/mysql Primary “devdrbd02” Secondary “devdrbd01” Eth0 Interface Heartbeat Probing Not mountable MySQL server mounted DRBD Replication Heartbeat DRBD Synchronization Recovered after some time 4.1 Failover – Power/Network Interruption on the Primary

4.2 Failover – DRBD Synchronization DRBD Synchronization: A DRBD procedure which updates data of the Inconsistent node by the data of the Up-to-Date node. When does inconsistency happen? If one machine is powered off or off-line, “write” operation is performed only on the on-line machine.

4.2 Failover – DRBD Synchronization (continued) The speed of DRBD Synchronization In our configuration, the bandwidth of both synchronization and normal replication is 10MB/s in average. Time needed for complete DRBD Synchronization Too much time to accept? DRBD Synchronization is smart. It chooses the recently updated data to synchronize first. It depends on disk size. For our 4TB 64-bit machine, almost 20 hours.

4.3 Workload Simulation/Failover in MySQLstat Plot Environment: Two 64-bit machines configured by DRBD-heartbeat Two 64-bit machines configured by DRBD-heartbeat Install MySQL community on both machines Install MySQL community on both machines Simulate the status and workload of PandaDB Use a typical PandaDB dump as the queried database Use a typical PandaDB dump as the queried database Write workload generator to simulate client queries Write workload generator to simulate client queries It performs balanced query to the PandaDB tables through virtual IP interface and multiple connections. It performs balanced query to the PandaDB tables through virtual IP interface and multiple connections. It has re-connect feature to detect the service interruption and save current query state. It re-creates MySQL sessions when service is available again, and resumes the latest interrupted session. It has re-connect feature to detect the service interruption and save current query state. It re-creates MySQL sessions when service is available again, and resumes the latest interrupted session.

1. General Queries Time Gap Workload >150 Client Connections 4.3 Workload Simulation/Failover in MySQLstat Plot

3. Input/Output 2. Query Type 5% Insert/Delete 48% Update 47% Select 8.8M/sec in average 4.3 Workload Simulation/Failover in MySQLstat Plot

2 DRBD: Distributed Replication Block Device 1 Motivation 3 Integrating DRBD-Heartbeat-MySQL Outline 4 Failovers 5 Discussion 5.1 The Comparison with Master/Slave Replication 5.2 What we have already done 5.3 What we will do next

5.1 The Comparison with Master/Slave Replication Master/Slave is based on MySQL – User Level; DRBD is based on block device – Kernel Level. Master/Slave is asynchronous; DRBD is synchronous. Master/Slave has higher probability of inconsistency. Master/Slave has load-balancing; DRBD doesn’t. Master/Slave is geographically more flexible; DRBD has to be located in the same subnet or neighbor subnets.

5.2 What we have already done now Testing InnoDB MySQL DB on the basis of DRBD-Heartbeat Configuration (on both 32-bit and 64-bit machines): Configuration for integrating DRBD-Heartbeat-MySQL Configuration for integrating DRBD-Heartbeat-MySQL Multi-connection/multi-type workload simulation according to the status of Panda Server Multi-connection/multi-type workload simulation according to the status of Panda Server Failover situations Failover situations Power interruption Power interruption Network interruption Network interruption Automatic/manual DRBD Synchronization Automatic/manual DRBD Synchronization Testing time gap of failover/synchronization by automatically re-connect load generator Testing time gap of failover/synchronization by automatically re-connect load generator

5.3 What we will do next Simulate the workload and failover feature through production Panda Server; Simulate the workload and failover feature through production Panda Server; Quantitative efficiency comparison with Master/Slave Replication; Quantitative efficiency comparison with Master/Slave Replication; Simulate the work load and failover feature according to the status of Archive MyISAM MySQL DB. It has small number of connections, but is highly insert-intensive. Simulate the work load and failover feature according to the status of Archive MyISAM MySQL DB. It has small number of connections, but is highly insert-intensive.

Thank you very much! Questions?