1. Benchmarking Cloud Serving Systems with YCSB
Presenter: Zhujun Xiao
Oct. 25
Brian F. Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, Russell Sears (Yahoo! Research), SoCC’2010

2. Motivation (1)
New systems for data storage and management “in the cloud”
Open source systems: Cassandra, HBase…
Cloud services: Microsoft Azure, Amazon web service (S3, SimpleDB) …
Private systems: Yahoo!’s PNUTS …
Serving systems: provide online read/write access to data.
as oppose to batch or analytical systems.

3. Motivation (2) Hard to choose the appropriate system!
However, it is hard to compare their performances
Different data models
BigTable model in Cassandra and HBase
Simple hashtable model in Voldemort
Document model in CouchDB
Different design decisions (trade-offs)
Read or write optimized
Synchronous or asynchronous replication
Latency or durability: sync writes to disk at write time or not
Data partitioning: row-based or column-based storage
Evaluted on different workloads
These systems only report performance for the “sweet spot” workloads for their system.
Hard to choose the
appropriate system!
1. Appending all updates to a sequential disk-based log can achieve high write throughput, but is not efficient for reads.
2. Synchronous replication can ensure all the copies are up-to-date, but bring long latency on updates. Asynchronous replication can avoid long latency but risk losing the updates due to failure before it can be replicated.
3. row-based storage can quickly read an entire record, and column-based storage could read a subset of the columns.
4. Sync writes to disk latter can reduce latency, but risk losing the unsynched data if the server crashes.
Result: developers have to manually evaluate multiple systems for their target application, which is expensive and time-consuming.

4. Goal Build a standard benchmark for cloud serving systems
Evaluate different systems on different workloads
A package of standard workloads
A workload generator
Evaluate both performance and scalability
Future direction: availability and replication

5. Benchmark Tiers Tier 1: performance Tier 2: scaling
Better system = achieve target latency and throughput with fewer server
Sizeup: latency versus throughput as throughput increases
Tier 2: scaling
Scaleup: latency should remain constant when number of servers and amount of data scale proportionally.
Elastic speedup: how latency changes as more servers are added.
Scaleup: performance (e.g. latency) should remain constant as number of servers, amount of data scale proportionally.

6. YCSB client Define the dataset and load it into the database
Data records
A set of operations
Execute the operations while measuring performance

7. Workload (1) A table of data records Basic operations
Each with 𝐹 fields.
Each record is identified by a primary key.
The values of each record is a random string of length 𝐿.
E.g byte records, 𝐹 =10, 𝐿=100.
Basic operations
Insert, update, read, scan
Update a record by replacing the value of one field
Scan records in order, starting at a randomly chosen record key. The number of records to scan is randomly chosen.

8. Workload (2) One workload is a mix of basic operations with choices of
Which operations to perform
Which record to read or write
How many records to scan
Decisions are governed by random distributions
Uniform
Zipfian
Latest
Multinomial
Zipfian: some records are more popular. More likely to be chosen. Following a Zipfian distribution.
Latest: latest records are more likely to be chosen.
Multinomial: Give each item a specific probability.
The Latest distribution is meant to model applications where recency matters; for example, only recent blog posts or news stories are popular, and the popularity decays quickly.

9. YCSB client architecture
Java program
Client threads
Load the database
Execute the workload
Measure the latency and achieved throughput
Extensibility
Extensible: define new workloads.
Extensible: plug in new DB interface.
Figure 1. YCSB client architecture

10. Experimental setup (1) Servers Tested systems Database
Six storage servers
YCSB client runs on a separate server.
Tested systems
Cassandra, HBase, PNUTS, and Shared MySQL.
Database
120 million 1KB records = 20 GB per sever
dual 64-bit quad core 2.5 GHz Intel Xeon CPUs, 8 GB of RAM, 6 disk RAID-10 array and gigabit ethernet

11. Experimental setup (2)
YCSB provides a package of standard workloads.

12. Workload A: update heavy
50% reads and 50% updates.
for all systems, operation la- tency increased as offered throughput increased. Cassandra is optimized for write-heavy workload, achieved best throughput and the lowest latency for reads.
HBase had very low latency for updates, since updates are buffered in memory. HBase does not sync log updates to disk, which provides low latency updates and high throughput.

13. Workload B: Read heavy 95% reads and 5% updates
PNUTS and sharded MySQL are now able to provide lower latencies on reads than either Cassandra or HBase.
overwrites records when they are updated. reads are fast because a single I/O can retrieve the entire, up-to-date record

14. Workload E: short ranges
Scans of records of size 1KB
Our sharded MySQL implementation is a hash table and does not sup- port range scans.
Cas- sandra performs much worse. Cassandra’s range scan sup- port is relatively new in version 0.5.0, and is not yet heavily optimized
Hbase and PNUTS have similar scan performance.

15. Scalability (1)
Scaleup: Vary the number of storage severs from 2 to 12 while varying the data size and request rate proportionally.
Cassandra and PNUTS, indicating good elastic scaling properties
HBase is that its behavior is erratic for very small clusters

16. Scalability (2)
Elastic speedup: run a read-heavy workload on 2 servers; add a 3rd server, then 4th, then 5th, then 6th.
5 servers to 6 servers
this results in a sharp increase in latency, as well as a wide variance in the latency. repartitioning is a known issue with Cassandra
This performance degradation results from moving data to the 6th server; regular serving requests compete for disk and network resources with the data repartitioning process, resulting in high and highly variable latency.
HBase is able to shift read and write load to the new server, resulting in lower latency
also moves data to the new server, resulting in higher latency after the sixth server is added, as well as latency variability. However, PNUTS is able to stabilize more quickly, as its repartitioning scheme is more optimized.

17. Discussion
What is the challenging part of building a benchmark, compared with simply implementing several systems and reporting their performance metrics?
All the design decisions would contribute to the final performance. Could YCSB be used for contribution breakdown?
Is it reasonable to use the distribution listed in the paper? Rather, does the real request pattern follow these distribution? Maybe in some cases, there is not regular pattern.
For example, Hbase has low write latency. Because it performs sequential I/O for updates, and HBase does not sync log updates to disk. Which decision contributes more