High-Availability MySQL with DR:BD and Heartbeat: MTV Japan mobile services ©2008 MTV Networks Japan K.K.

Patrick Bolduan System Manager MTV Networks Japan K.K. ©2008 MTV Networks Japan K.K.

Let’s get started ©2008 MTV Networks Japan K.K.

About MTVNJ Company Profile Company NameMTV Networks Japan K.K. PresidentPeter Robert Bullard Address2-8-2 Jingumae, Shibuya-ku, Tokyo TEL: FAX: Business Description MTV Networks Japan K.K. (MTVNJ) operates the mobile and PC digital media services for all MTV Networks brands including MTV and Nickelodeon in Japan. In addition to digital media services, MTVNJ also operates a 24-hour Nickelodeon channel on CS/CATV, and the consumer products business for Nickelodeon's character franchises in Japan. ©2008 MTV Networks Japan K.K.

MTVNJ Mobile Brands ©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project intro ©2008 MTV Networks Japan K.K.

{database} DB Selection MTV Flux/MTV Mobile DB project ©2008 MTV Networks Japan K.K.

Database requirements: flexibility performance scalability redundancy support cost Project summary: Merge all the database services for the MTV Flux and MTV Mobile services on to a single database platform. MTV Flux/MTV Mobile DB project ©2008 MTV Networks Japan K.K. DB Selection

reqs/DB OracleMS SQL MySQL Community MySQL Enterprise flexibility △／×△／× ×○○ performance○○○○ scalability ○／△○／△ ○○○ redundancy ○／△○／△ ○○○ support○○ × ○ cost×× ◎△ MTV Flux/MTV Mobile DB project ©2008 MTV Networks Japan K.K. DB Selection

MTV Flux/MTV Mobile DB project We chose MySQL Enterprise ©2008 MTV Networks Japan K.K. DB Selection

MTV Flux/MTV Mobile DB project What we looked for: flexibility performance scalability redundancy support cost MySQL Architectures: Single/multi Master Master/Slave Cluster DR:BD/Heartbeat HA Master ©2008 MTV Networks Japan K.K. MySQL Architectures

www Single/multi Master cheap(!) easy configuration load ↑ ⇒ performance ↓ high risk (if master dies, no more DB) flexibilityperformancescalabilityredundancysupportcost ○ ○/△○/△ ××○ ◎ Master × MTV Flux/MTV Mobile DB project ©2008 MTV Networks Japan K.K. MySQL Architectures

www Master/Slave some redundancy (read/writes separate) commonly used architecture still some risk (operator needed for outage recovery) manual failover for outages MasterSlave × MTV Flux/MTV Mobile DB project ©2008 MTV Networks Japan K.K. flexibilityperformancescalabilityredundancysupportcost ○ △ ○ △ ○ ◎ MySQL Architectures

www Cluster redundant! can scale complicated configuration limited support in Japan (2007) Cluster △ MTV Flux/MTV Mobile DB project ©2008 MTV Networks Japan K.K. flexibilityperformancescalabilityredundancysupportcost ○○ ◎◎ ○/△○/△△ MySQL Architectures

www DR:BD/Heartbeat HA Master redundant (master fully redundant) can scale (add salves) reduced outage risk can use failover for maintenance tasks complicated configuration PrimarySecondary ◎ MTV Flux/MTV Mobile DB project ©2008 MTV Networks Japan K.K. flexibilityperformancescalabilityredundancysupportcost ◎ ○/△○/△ ○/△○/△◎ ○○ MySQL Architectures

MTV Flux/MTV Mobile DB project MySQL HA Master ©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project HA Master VIP /etc/my.cnf local /var/lib/mysql DRBD /var/lib/mysql DRBD /etc/my.cnf local heartbeat drbd Application PrimarySecondary ©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project VIP DRBD heartbeat Application Web site or whatever Manages/monitors DB process and mounts: provides application connection point (Virtual IP) monitors network status watches the mysqld process manages the /var/lib/mysql mount point mysqld (the database) Synchronous device layer data replication ©2008 MTV Networks Japan K.K. HA Master

MTV Flux/MTV Mobile DB project VIP /etc/my.cnf local /var/lib/mysql DRBD /var/lib/mysql DRBD /etc/my.cnf local heartbeat drbd Application PrimarySecondary × Failover process network or mysqld failure 1 – Application looses connection to mysqld 2 – heartbeat detects the problem ©2008 MTV Networks Japan K.K. HA Master 4 – Failover complete! 3 – heartbeat does: pDB mysql dir unmount DRBD switch sDB mysql dir mount sDB mysqld start drbd

2 – drbd fails over 1 – primary server fails! 3 – heartbeat detects outage MTV Flux/MTV Mobile DB project VIP /etc/my.cnf local /var/lib/mysql DRBD /var/lib/mysql DRBD /etc/my.cnf local heartbeat drbd Application PrimarySecondary ©2008 MTV Networks Japan K.K. HA Master Failover process server outage 4 – Failover complete! 3 – heartbeat does: sDB mysql dir mount sDB mysqld start

MTV Flux/MTV Mobile DB project HA MySQL install ©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project HA Master Install local disk A Master local disk B Master Master Dual MasterSingle Master Old DB architecture ©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project VIP /etc/my.cnf local /var/lib/mysql DRBD /var/lib/mysql DRBD /etc/my.cnf local heartbeat drbd New DB architecture HA Master Install PrimarySecondary ©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project HA Master Install Database migration work plan: stop production services perform full data backup ( mysqldump ) Replace MySQL architecture restore data from backup test database failover test application re-open production services ©2008 MTV Networks Japan K.K.

5 – test application connectivity local disk × MTV Flux/MTV Mobile DB project A MasterB Master Master Migration process VIP /etc/my.cnf local /var/lib/mysql DRBD /var/lib/mysql DRBD /etc/my.cnf local drbd heartbeat 1 – remove old database components 2 – install/configure drbd and mysql 3 – install heartbeat 4 – configure VIP HA Master Install PrimarySecondary × ©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project HA Master Install Actual migration work times: production services stop – 00:10 full data backup ( mysqldump ) – 00:05 MySQL architecture replacement – 01:30 data restore – 00:10 database failover test – 00:15 application test – 03:00 production services fully restored! ©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project HA MySQL lessons learned ©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project Lessons learned Heartbeat: no room for configuration mistakes pay special attention to peer connectivity run lots of tests before going to production!!! Impact on our application: essentially the same as a single master can use same HA for: single master - master/slave - multi-master failover fast enough that DB outages don’t affect application ©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project Lessons learned DR:BD: requires separate partition (can be LVM) like heartbeat – test your configurations!!! always check your DRBD status after a failover peers sometime require a manual re-sync mysqld: can take advantage of redundancy to swap out nodes never try to run both primary and secondary at once!! leave all the redundancy to heartbeat ©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project Lessons learned Overall – good points: we were able to deploy HA in a short maintenance window commercial support available (MySQL – LINBIT) MySQL consultative services were a big help also usable with MySQL Community can be part of other architectures multi-master - master/slave Overall – weak points: lots to learn, lots to test (heartbeat and DRBD) configuration and failover is complex mysqld restart requires some extra steps all admins/vendors need to understand failover basics ©2008 MTV Networks Japan K.K.

HA MySQL in other MTVNJ environments ©2008 MTV Networks Japan K.K.

Questions? ©2008 MTV Networks Japan K.K. DRBD related links MTVNJ links Patrick Bolduan MTV Networks Japan KK Jingu-mae, Shibuya-ku, Tokyo