1. Load Balancing Memcached Traffic Using SDN
Idan Moyal
Interdisciplinary Center Herzliya, Israel
Joint work with:
Anat Bremler-Barr
IDC Herzliya
David Hay
Hebrew University
Liron Schiff
Tel-Aviv University
Supported by the European Research Council under the European Union's Seventh Framework Programme

2. Memcached
Memcached is a widely-used in-memory key-value based distributed caching system
Data is distributed between multiple servers.
The most basic operations are: get & set.
Every client is initialized with a list of servers.
Keys can be no longer than 250 characters.
Stored values cannot exceed 1MB.
Supported networking protocols: TCP/UDP (ASCII/Binary).

3. Movie Reviews Web Application Example
Memcached
Memcached is a widely-used in-memory key value based distributed caching system
Movie Reviews Web Application Example
DBMS
Read review from DB
Movie Review
Slow Response
Application Server
Data is distributed between multiple servers.
The most basic operations are: get & set.
Every client is initialized with a list of servers.
Keys can be no longer than 250 characters.
Stored values cannot exceed 1MB.
Supported networking protocols: TCP/UDP (ASCII/Binary).
Get movie review
Get movie review
Application Server
Load Balancer
Movie review
Movie review
Application Server

4. Movie Reviews Web Application Example
Memcached
Memcached is a widely-used in-memory key value based distributed caching system
Movie Reviews Web Application Example
Memcached server is picked by applying a hash function on the requested key
DBMS
CACHE HIT
Application Server
Data is distributed between multiple servers.
The most basic operations are: get & set.
Every client is initialized with a list of servers.
Keys can be no longer than 250 characters.
Stored values cannot exceed 1MB.
Supported networking protocols: TCP/UDP (ASCII/Binary).
Memcached Server
Memcached Client
Get cache review
Get movie review
Cache Movie Review
Get movie review
Application Server
Load Balancer
Movie review
Movie review
Application Server

5. Movie Reviews Web Application Example
Memcached
Memcached is a widely-used in-memory key value based distributed caching system
Movie Reviews Web Application Example
Memcached server is picked by applying a hash function on the requested key
DBMS
Read review from DB
Movie Review
Slow Response
CACHE MISS
Application Server
Data is distributed between multiple servers.
The most basic operations are: get & set.
Every client is initialized with a list of servers.
Keys can be no longer than 250 characters.
Stored values cannot exceed 1MB.
Supported networking protocols: TCP/UDP (ASCII/Binary).
Memcached Server
Memcached Client
Get cached review
Get movie review
Cache review
Get movie review
Application Server
Load Balancer
Movie review
Movie review
Application Server

6. The Hot Keys Problem
Memcached ignores key popularity when assigning keys to servers 
DBMS
Application Server
Memcached Server
Memcached Client
Memcached Server
Memcached Server
Application Server
Load Balancer
Memcached Server
Memcached Client
Memcached Server
Application Server
High Response Time
Memcached Server
Memcached Client
memcached get “guardians-of-the-galaxy-review”

7. Key Popularity Follows Zipf Distribution
[Almeida et al., 1996] [Fan et al., 2013] [Fukuda et al., 2014]
Most popular key
How is the problem solved today?

8. Our Solution: MBalancer
Objectives:
Our Solution:
Balance the load between servers (e.g., minimize average load/max load)
Duplicate hot keys across many servers
Seamless to both memcached client and memcached server
Let the network handle the load balancing

9. Our Solution: MBalancer
DBMS
Application Server
Memcached Server
Memcached Client
Memcached Server
Memcached Server
Application Server
Load Balancer
Memcached Server
Memcached Client
Memcached Server
Application Server
Good Response Time
Memcached Server
Memcached Client
memcached get “guardians-of-the-galaxy-review”
memcached get “guardians-of-the-galaxy-review”

10. Our Solution: MBalancer
DBMS
Same request, different destinations?
Application Server
Memcached Server
Memcached Client
Memcached Server
Memcached Server
Application Server
Load Balancer
Memcached Server
Memcached Client
Memcached Server
Application Server
Good Response Time
Memcached Server
Memcached Client
memcached get “guardians-of-the-galaxy-review”
memcached get “guardians-of-the-galaxy-review”

11. Our Solution: MBalancer
Memcached statistics
Using Software Defined Networking (SDN), which separates control plane and data plane:
Data plane – match action rules
Control plane – insert/delete rules
Data Plane
Control Plane
SDN Controller
MBalancer Application
Application Server
Memcached Server
Memcached Client
Memcached Server
Memcached Server
Application Server
Load Balancer
Memcached Server
Memcached Client
SDN Switch
Memcached Server
Application Server
Good Response Time
Memcached Server
Memcached Client

12. Our Solution: MBalancer
Memcached statistics
Data Plane
Control Plane
SDN Controller
MBalancer Application
Payload matching capabilities of exact location.
Common in many SDN switches.
Application Server
Memcached Server
Memcached Client
Memcached Server
Memcached Server
Application Server
Load Balancer
Memcached Server
Memcached Client
SDN Switch
Memcached Server
Application Server
Good Response Time
Memcached Server
Memcached Client
memcached get “guardians-of-the-galaxy-review”

13. Why duplication works? If we look at very few highly-loaded keys:
Most of the system’s load is concentrated there
Very modest memory overhead to duplicate these keys’ values
Only a few switching rules suffices to support load balancing

14. Why duplication works? Imbalance factor = average load / max load
The closer the imbalance factor is to 1, the higher the throughput is.
First picture: 10 servers
Secnd picture: 1000 keys
40-60% improvement
Improvement is consistent as the system grows

15. MBalancer Tasks Hot keys identification & duplication.
Load-balance requests in the switch.
Hot Keys updating.
OpenFlow counters (SDN).
sFlow Monitoring (SDN).
Memcached packet sniffer (etsy/tumblr).
Begin with Step 3

16. Task 1: Hot Keys Identification
Identification using off–the-shelf techniques (not part of MBalancer)
Tumblr’s, Etsy’s
Can also use sFlow, OpenFlow counters

17. Task 2:Load Balancing in the Switch Using OpenFlow
Payload matching
Flow Table
Entry per hot key get request
Group 2. Type: Select
Servers containing the hot keys (e.g. all servers)
Group Type Select (Since OpenFlow 1.3): Packets are processed by a single bucket in the group. Bucket selection provides equal load sharing (e.g., simlple round robin)

18. Load Balancing in the Switch Using OpenFlow
Hot key request
Payload matching
Flow Table
Group 2. Type: Select
Hot key request redirected
(works only for UDP requests)
Load Balancing

19. Task 3: Key Updating and Duplication Using OpenFlow
Flow Table
Total number of rules: 
2 Rules per hot key get & set operations.
At most one rule per server containing hot keys duplicates.
Keys are installed and updated with set operations
Group 2. Type: Select
Hot Keys updates are captured and sent to the controller

20. Key Duplication and Updating
MBalancer application contains a memcached client
Re-duplicate hot keys on update
The switch is configured to match “set” hot keys packets and send also to the controller
MBalancer Application
Memcached Client
User performs a request which requires updating a hot key
SDN Controller
Memcached set “best-seller”
Control Plane
Memcached get “best-seller”
Data Plane
Application Server
Memcached Server
Memcached Client
Memcached Server
Application Server
Load Balancer
Memcached Server
Memcached Client
SDN Switch
Memcached Server
Memcached set
“best-seller”
Application Server
Memcached Server
Memcached Client

21. Experimental Results Setting:
NoviKit 250 SDN switch (1Gbit/s network).
RYU SDN controller.
64 memcached clients, 2 memcached servers.
1000 keys, 100KB values.
10 Hot Keys are duplicated to all servers

22. Experimental results
Throughput
Latency

23. Comparing with Proxy-Based Solutions
Additional servers dedicated for hot keys
Hot Keys arrive at the proxy and sent to one of the servers behind it.
MBalancer does not require additional servers and architecture changes.
Used by Facebook
The number of extra servers required to achieve the same imbalance factor of MBalancer

24. MBalancer: Limitations
MBalancer performs load balancing for Memcached UDP get requests.
UDP is usually preferred when the goal is better performance (Facebook’s solution)
SDN switch with payload matching capabilities is required.
Memcached key length is limited (NoviKit250 limitation is 46 bytes).
MBalancer can perform load balancing for Memcached operating over UDP.
A solution for TCP requires adaptation of TCP ack & seq for requests done by a specific client, and it is impossible* to keep this state in the switch.
RST based solution was considered, but is not scalable and requires making changing Memcached clients implementation.
SDN switch with payload matching capabilities is required.
Memcached key length is limited (NoviKit250 limitation).
-need to be in the end

25. Conclusions
MBalancer solves an higher level application issue, the hot keys problem in Memcached, using SDN.
Solution does not require a change in the application’s architecture.
Low overhead in the SDN switch and memory footprint.
An innovative way to perform L7 load balancing
Many extensions in the paper, including a P4 implementation.

26. Thank You!
Questions?

27. Lorelyn Medina      123RF.com