Symbiotic Routing in Future Data Centers Hussam Abu-Libdeh Paolo Costa Antony Rowstron Greg O’Shea Austin Donnelly MICROSOFT RESEARCH Presented By Deng Xiang Hussam Abu-Libdeh Paolo Costa Antony Rowstron Greg O’Shea Austin Donnelly MICROSOFT RESEARCH Presented By Deng Xiang

OUTLINE Part I: introduction and background Part II: four services that use customized routing protocols and their evaluations Part III: the extended routing service Part IV: conclusion Part V: some questions

Part I introduction and background Definition of symbiotic routing Multiple routing protocols running concurrently, all working with, and benefiting from the base routing protocol Services use their own routing protocols to handle the common cases and rely on the base routing protocol the handle the corner cases

Why we need symbiotic routing? The networks in large-scale data centers adopt principles evolved from Internet networking The end-systems are isolated from the network and have little control over packets routing Building efficient services is hard Part I introduction and background

Part I CamCube: a new data center architecture Targets at shipping container sized data centers Use a 3D torus Each server connects to other servers directly without using any switches or routers Each server is assigned an (x,y,z) coordinate and it is fixed for the life time of the server

the simplest CamCube Part I introduction and background

For each server in CamCube… CamCube API Core servicesother services Part I introduction and background

CamCube API Exposes the 3D coordinate space to services running on CamCube Allow sending and receiving of packets to/from one-hop routing Allow a service to queue packets on the queue and query the queue length Part I introduction and background

Core services Expose the server coordinates of local server and its one-hop neighbors Monitor liveness of one-hop neighbors and inform other services server or link failure The mutiple-hop routing service(the base routing service) The ability to perform per-hop inspection and modification of packets Key-based routing Part I introduction and background

Simulator Use 27(3*3*3) servers to simulate 8,000 servers(20*20*20) 1 Gbps links Part I introduction and background

Part II four services with their own routing protocols TCP/IP service Intercept all the packets from the OS’s TCP/IP stack, tunnel them across CamCube and inject them into the TCP/IP stack at the destination Aim at achieving the maximum throughput between source and destination

Part II four services with their own routing protocols If use the base routing service… Out-of-order packet delivery Congestion Packet loss Decrease end-to-end throughput

Part II four services with their own routing protocols A custom routing protocol Step 1: the source determines the set of outbound links that lies on a shortest path Step 2: all the links are used to route packets out of the source Step 3: at each hop a packet is greedily routed towards the same axis and if progress cannot be made, another axis is selected

Part II four services with their own routing protocols Evaluation The custom routing protocol versus the base routing service A single server acts as the source and 2,000 randomly selected servers act as the destination 1,500-byte packet and each destination receives 10,000 packets Iterate over 2,000 servers sequentially

Part II four services with their own routing protocols

Aggregation service Similar to the reduce step of MapReduce To aggregate results of data parallel computation distributed on servers

Part II four services with their own routing protocols MapReduce master node N servers R servers split a data set into chunks and makes them equally distributed to N servers Aggregates the results to R servers Sends back the total result

Part II four services with their own routing protocols N servers(worker nodes) Each server has a set of key-value pairs and the keys in the set may be different Multiple servers can have key-value pairs that have the same key After computation, send the key-value pairs to the servers to which keys map R servers(reducers) After receiving all the key-value pairs, combine all the values for the same key Send the total result back to the master node

Part II four services with their own routing protocols How to aggregate on path We know the coordinate of the source and the destination Route the packets from the source server to a destination server on a deterministic path Tha path is made deterministic by fixing each of the axes in turn At each hop, the service locally buffers the packet in case of congestion

Part II four services with their own routing protocols Advantages Reduce the amount of data transferred by performing on-path aggregation Ensure a good distribution of computational load over the servers Use less links and reduce the link load

Part II four services with their own routing protocols Evaluation The aggregation service versus the base routing service Each server generates a single packet with a key-value pair The Full situation The None situation

Part II four services with their own routing protocols

Part III the extended routing service The extended routing service makes it simple and efficient for services to specify their own routing protocols.

Services themselves need to provide: 1) A per-service function F Operate in the coordinate Define a set of key coordinates through which packets are routed 2) A policy controls queuing of packets for the service The base routing service need to provide: 1)A component that provide failure information 2)A forwarding engine that forward packets towards a specified key coordinate Part III the extended routing service

1.A service S creates a packet and hands it to the extended routing service (ERS). 2.ERS call the function F associated with S, return a set of possible next key coordinates. 3.The packet is routed towards a server responsible for one of the coordinates in the set. 4.The base routing service is in charge of failure.

Part III the extended routing service Examples: The function F for the aggregation service defines a deterministic path through the key coordinate space. The function F used by the TCP/IP service at the source returns the key coordinates of all one-hop neighbors.

Part IV: conclusion Individual services can obtain better performance by utilizing their own protocols. The network load is reduced for all services uses their own optimized protocols. An extended routing service is feasible to make other services implement their own routing protocols easily.

Thank you!