From Startup to Enterprise A Story of MySQL Evolution Vidur Apparao, CTO Stephen OSullivan, Manager of Data and Grid Technologies April 2009

Data Size Growth 2009 LiveOps, Inc.2 The data volume for a majority of companies increases % every year. (IDC) Common Solutions: Rely on Moores Law Spend more money But there are other ways…

About LiveOps Technology Platform for Contact Centers On-Demand, Multi-tenanted Contact Center Platform Virtual Call Center of 20,000 independent home agents Eight years of continuous growth Founded in 2000 Profitable since employees 2009 LiveOps, Inc.3

LiveOps Data Main data classes: Configuration data – low GBs, slow change Logging data – low TBs, ever increasing System state – MBs, rapid change Customer-specific data – high GBs, versioned Largest table has 1.4 billion rows Tenant key as a column on all tables Multi-site deployment for high availability 2009 LiveOps, Inc.4 Configuration DataTransaction dataSession data Configuration ToolsReporting ToolsMonitoring Tools Web ApplicationsTelephony Applications

Phase 1: The Basic Model Application servers connecting to a single DB Replication to a slave for backup & load balancing 2009 LiveOps, Inc.5 Web & Telephony Applications R/W Master MySQL Replication Slave/Backup

Primary Drivers for Change AvailabilityPerformanceScale

Options for Improving (Write) Scale Sharding Partition data into distinct databases based on a sharding key Functional Segmentation Separate functional data classes into distinct databases MySQL Partitioning LiveOps choice: Sharding, Functional Segmentation 2009 LiveOps, Inc.7

Options for Improving (Query) Performance Replication & Load balancing Distribute query load across multiple replicants Separation of DB roles Separate fast from slow, OLTP from OLAP Caching Reduce dependency on the database for queries Consistent query tuning/optimization LiveOps choice: Load balancing, separation of roles 2009 LiveOps, Inc.8

Options for Improving Availability Application resilience Remove requirement of direct write access and/or degrade gracefully MySQL Cluster Multi-master replication Split tables or databases between ring replicating masters DRBD or SAN HA LiveOps choice: Application resilience, multi-master 2009 LiveOps, Inc.9

2. Functional segmentation of data between multiple masters. 6. Separation of DB roles based on type and cost of query. 4. Replication to a farm of read-only replicants within and across data centers. 5. Load balancing using DB monitoring and pushed configuration files. 3. Multi-master replication and quick recovery processes on master failure. 1. Data writers use store- and-forward pattern for fault tolerance. Phase 2: A Pure MySQL Solution 2009 LiveOps, Inc.10 Reporting and Analytics Web & Telephony Applications R/W Masters Read-only Replicants w/ Roles Queries Writes DB Monitor/ Load Balancer Monitoring Config. Push Logging Config + Session All of these techniques still dont get us to horizontal data scalability

Horizontal Scalability Options Distributed Storage Systems DFS for unstructured file storage BigTable/HBase for structured data storage Various vendors with distributed RDBMSs Grid Processing MapReduce and Hadoop 2009 LiveOps, Inc.11

Our Approach Take logging data out of the transactional databases Reduce replication load Store logging data as text files in a DFS Use MapReduce for ETL into OLAP databases Leverage open source tools like ActiveMQ and Hadoop 2009 LiveOps, Inc.12

6. MySQL as a data mart. Phase 3: MySQL w/ Horizontal Scalability 2009 LiveOps, Inc.13 Broker ActiveMQ Brokers Repository Process Map Reduce DFS Hadoop Backup Storage Array Reporting and Analytics R/W Masters Read-only Replicants Web & Telephony Applications Data Marts Audit Process 1. MySQL continues as OLTP solution. 2. Logging data now written to log files on local disk. 3. Log files moved via ActiveMQ to a log repository and DFS. 5. Hadoop as MapReduce system for ETL. Horizontal scalability is now in reach! 4. Audit process to reconcile data between log files and DFS.

Learned Best Practices Know your data Build and enforce a data access layer Put your data only where you need it Experiment early and often 2009 LiveOps, Inc.14

Conclusions MySQL, other open source technology, and commodity hardware can be used to build a horizontally scalable data solution Companies today are left to chart their own evolutionary paths Collaboration and communication between companies in this area can help everyone 2009 LiveOps, Inc.15

Thank You. Vidur Apparao, Stephen OSullivan