Load Balancing Memcached Traffic Using SDN Idan Moyal Interdisciplinary Center Herzliya, Israel Joint work with: Anat Bremler-Barr IDC Herzliya David Hay Hebrew University Liron Schiff Tel-Aviv University Supported by the European Research Council under the European Union's Seventh Framework Programme

Memcached Memcached is a widely-used in-memory key-value based distributed caching system Data is distributed between multiple servers. The most basic operations are: get & set. Every client is initialized with a list of servers. Keys can be no longer than 250 characters. Stored values cannot exceed 1MB. Supported networking protocols: TCP/UDP (ASCII/Binary).

Movie Reviews Web Application Example Memcached Memcached is a widely-used in-memory key value based distributed caching system Movie Reviews Web Application Example DBMS Read review from DB Movie Review Slow Response Application Server Data is distributed between multiple servers. The most basic operations are: get & set. Every client is initialized with a list of servers. Keys can be no longer than 250 characters. Stored values cannot exceed 1MB. Supported networking protocols: TCP/UDP (ASCII/Binary). Get movie review Get movie review Application Server Load Balancer Movie review Movie review Application Server

Movie Reviews Web Application Example Memcached Memcached is a widely-used in-memory key value based distributed caching system Movie Reviews Web Application Example Memcached server is picked by applying a hash function on the requested key DBMS CACHE HIT Application Server Data is distributed between multiple servers. The most basic operations are: get & set. Every client is initialized with a list of servers. Keys can be no longer than 250 characters. Stored values cannot exceed 1MB. Supported networking protocols: TCP/UDP (ASCII/Binary). Memcached Server Memcached Client Get cache review Get movie review Cache Movie Review Get movie review Application Server Load Balancer Movie review Movie review Application Server

Movie Reviews Web Application Example Memcached Memcached is a widely-used in-memory key value based distributed caching system Movie Reviews Web Application Example Memcached server is picked by applying a hash function on the requested key DBMS Read review from DB Movie Review Slow Response CACHE MISS Application Server Data is distributed between multiple servers. The most basic operations are: get & set. Every client is initialized with a list of servers. Keys can be no longer than 250 characters. Stored values cannot exceed 1MB. Supported networking protocols: TCP/UDP (ASCII/Binary). Memcached Server Memcached Client Get cached review Get movie review Cache review Get movie review Application Server Load Balancer Movie review Movie review Application Server

The Hot Keys Problem Memcached ignores key popularity when assigning keys to servers  DBMS Application Server Memcached Server Memcached Client Memcached Server Memcached Server Application Server Load Balancer Memcached Server Memcached Client Memcached Server Application Server High Response Time Memcached Server Memcached Client memcached get “guardians-of-the-galaxy-review”

Key Popularity Follows Zipf Distribution [Almeida et al., 1996] [Fan et al., 2013] [Fukuda et al., 2014] Most popular key How is the problem solved today?

Our Solution: MBalancer Objectives: Our Solution: Balance the load between servers (e.g., minimize average load/max load) Duplicate hot keys across many servers Seamless to both memcached client and memcached server Let the network handle the load balancing

Our Solution: MBalancer DBMS Application Server Memcached Server Memcached Client Memcached Server Memcached Server Application Server Load Balancer Memcached Server Memcached Client Memcached Server Application Server Good Response Time Memcached Server Memcached Client memcached get “guardians-of-the-galaxy-review” memcached get “guardians-of-the-galaxy-review”

Our Solution: MBalancer DBMS Same request, different destinations? Application Server Memcached Server Memcached Client Memcached Server Memcached Server Application Server Load Balancer Memcached Server Memcached Client Memcached Server Application Server Good Response Time Memcached Server Memcached Client memcached get “guardians-of-the-galaxy-review” memcached get “guardians-of-the-galaxy-review”

Our Solution: MBalancer Memcached statistics Using Software Defined Networking (SDN), which separates control plane and data plane: Data plane – match action rules Control plane – insert/delete rules Data Plane Control Plane SDN Controller MBalancer Application Application Server Memcached Server Memcached Client Memcached Server Memcached Server Application Server Load Balancer Memcached Server Memcached Client SDN Switch Memcached Server Application Server Good Response Time Memcached Server Memcached Client

Our Solution: MBalancer Memcached statistics Data Plane Control Plane SDN Controller MBalancer Application Payload matching capabilities of exact location. Common in many SDN switches. Application Server Memcached Server Memcached Client Memcached Server Memcached Server Application Server Load Balancer Memcached Server Memcached Client SDN Switch Memcached Server Application Server Good Response Time Memcached Server Memcached Client memcached get “guardians-of-the-galaxy-review”

Why duplication works? If we look at very few highly-loaded keys: Most of the system’s load is concentrated there Very modest memory overhead to duplicate these keys’ values Only a few switching rules suffices to support load balancing

Why duplication works? Imbalance factor = average load / max load The closer the imbalance factor is to 1, the higher the throughput is. First picture: 10 servers Secnd picture: 1000 keys 40-60% improvement Improvement is consistent as the system grows

MBalancer Tasks Hot keys identification & duplication. Load-balance requests in the switch. Hot Keys updating. OpenFlow counters (SDN). sFlow Monitoring (SDN). Memcached packet sniffer (etsy/tumblr). Begin with Step 3

Task 1: Hot Keys Identification Identification using off–the-shelf techniques (not part of MBalancer) Tumblr’s, Etsy’s Can also use sFlow, OpenFlow counters

Task 2:Load Balancing in the Switch Using OpenFlow Payload matching Flow Table Entry per hot key get request Group 2. Type: Select Servers containing the hot keys (e.g. all servers) Group Type Select (Since OpenFlow 1.3): Packets are processed by a single bucket in the group. Bucket selection provides equal load sharing (e.g., simlple round robin)

Load Balancing in the Switch Using OpenFlow Hot key request Payload matching Flow Table Group 2. Type: Select Hot key request redirected (works only for UDP requests) Load Balancing

Task 3: Key Updating and Duplication Using OpenFlow Flow Table Total number of rules:  2 Rules per hot key get & set operations. At most one rule per server containing hot keys duplicates. Keys are installed and updated with set operations Group 2. Type: Select Hot Keys updates are captured and sent to the controller

Key Duplication and Updating MBalancer application contains a memcached client Re-duplicate hot keys on update The switch is configured to match “set” hot keys packets and send also to the controller MBalancer Application Memcached Client User performs a request which requires updating a hot key SDN Controller Memcached set “best-seller” Control Plane Memcached get “best-seller” Data Plane Application Server Memcached Server Memcached Client Memcached Server Application Server Load Balancer Memcached Server Memcached Client SDN Switch Memcached Server Memcached set “best-seller” Application Server Memcached Server Memcached Client

Experimental Results Setting: NoviKit 250 SDN switch (1Gbit/s network). RYU SDN controller. 64 memcached clients, 2 memcached servers. 1000 keys, 100KB values. 10 Hot Keys are duplicated to all servers

Experimental results Throughput Latency

Comparing with Proxy-Based Solutions Additional servers dedicated for hot keys Hot Keys arrive at the proxy and sent to one of the servers behind it. MBalancer does not require additional servers and architecture changes. Used by Facebook The number of extra servers required to achieve the same imbalance factor of MBalancer

MBalancer: Limitations MBalancer performs load balancing for Memcached UDP get requests. UDP is usually preferred when the goal is better performance (Facebook’s solution) SDN switch with payload matching capabilities is required. Memcached key length is limited (NoviKit250 limitation is 46 bytes). MBalancer can perform load balancing for Memcached operating over UDP. A solution for TCP requires adaptation of TCP ack & seq for requests done by a specific client, and it is impossible* to keep this state in the switch. RST based solution was considered, but is not scalable and requires making changing Memcached clients implementation. SDN switch with payload matching capabilities is required. Memcached key length is limited (NoviKit250 limitation). -need to be in the end

Conclusions MBalancer solves an higher level application issue, the hot keys problem in Memcached, using SDN. Solution does not require a change in the application’s architecture. Low overhead in the SDN switch and memory footprint. An innovative way to perform L7 load balancing Many extensions in the paper, including a P4 implementation.

Thank You! Questions?

Lorelyn Medina      123RF.com