SDN + Storage

Outline Measurement of storage traffic Network aware placement Control of resources SDN + Resource allocation Predicting Resources utilization Bring it all together

HDFS Storage Patters Maps reads from HDFS Local read versus Non-local read Rack locality or not Locality!!! 80%

HDFS Storage Patters Maps reads from HDFS Local read versus Non-local read Rack locality or not Cross-rack Traffic 80%

HDFS Storage Patters Reducers writes to HDFS 3 copies of file written to HDFS 2 rack local and 1 non-rack local Fault tolerance and good performance THERE MUST BE CROSS RACK TRAFFIC Ideal Goal: Minimize Congestion

Real Life Traces Analyze Facebook traces: 33% of time spent in network Network links are highly utilized; why? Determine cause of network traffic Job output Job input Pre-processing

Current Ways To Improve HDFS Transfers Change Network Paths Hedera, MicroTE, C-thru, Helios Change Network Rates Orchestra, D3 Increase Network Capacity VL2, Portland (Fat-Tree)

The case for Flexible Endpoints Traffic Matrix limits benefits of techniques that change paths of network rates Ability to Change Matrix is important 90% 20% 90% 80%

Flexible Endpoints in HDFS Recall: Constraint placed by HDFS 3 replicas 2 fault domains Doesn’t matter where as long as constraints are met The source of transfer is fixed! However destination, location of 3 replicas is not fixed

Sinbad Determine placement for block replica Benefits Place replicas to avoid hotspots Constraints: 3 copies Spread across 2 fault domains Benefits Faster writes: Faster transfers

Sinbad: Ideal Algorithm Input: Blocks of diff size Links of diff capacity Objective: Minimize write time (transfer time) Challenges: Lack of future knowledge Location & duration of hotspots Size and arrival times of new replicas

Sinbad Heuristic Assumptions Heuristic: Link utilizations are stable True for 5-10 seconds All block have same size Fixed-size large blocks Heuristic: Pick least-loaded link/path Send block from file with least amount to send

Sinbad Architecture Recall: original DFS is master-slave architecture Sinbad has similar

Sinbad Determine placement for block replica Benefits Place replicas to avoid hotspots Constraints: 3 copies Spread across 2 fault domains Benefits Faster writes: Faster transfers

Orchestrating the Entire Cluster How to control Compute, Network, Storage? Challenges from SinBAD How to determine future replica demands? You can’t control job arrival You can control task scheduling If you predict job characteristics you can determine future How to determines future hot spots? Control all network traffic (SDN) Use future

Ideal Centralized Entity Controls: Storage, CPU, N/W Determines: Which task to run Where to run the task When to start Network transfer What rate to transfer at Which network path

Predicting Job Characteristics To predict resources that a job needs to complete, what do you need?

Predicting Job Characteristics Job’s DAG (job’s traces history) Computations time for each node Data transfer size between nodes Transfer time between nodes

Things you absolutely know! Input data Size of input data Location of all replicas Split of input data Job’s D.A.G # of Map # of Reduce 200GB HDFS 3 Mappers Map Map Map Reduce Reduce 2 Reducers HDFS

Approaches to Prediction: Input/intermediate/Output Data Assumption: Map & Reduce run same code over and over Code gives the same ratio of reduction E.g. 50% reduction from Map to intermediate E.g. 90% reduction from intermediate to output Implications: Given size of input, you can determine size of future transfers Problems: Not always true!!! 200GB HDFS Map Map Map 100GB Reduce Reduce 10 GB HDFS

Approaches to Prediction: Task Run Time Assumption: Task is dominated by reading input Time to run a task is essentially time to read input If Local: Time to read from Disk If non-local: Time to read across Network Implication: If you can model read time you can determine task run time Problems: How do you model disk I/O? How do you model I/O interrupt contention? 200GB HDFS Map Map Map 100GB Reduce Reduce 10 GB HDFS

Predict Job Runs Given: Can you predict job completion time? Prediction of tasks, transfers, and of Dag Can you predict job completion time? How do you account for interleaving between jobs? How do you determine optimal # of slots? How do you determine optimal network bandwidth?

Really easy right? But what happens if the network only has 2 slots 100GB 200GB 10 GB Map Reduce 2 1 23 3 Map HDFS 8 HDFS 2 2 Map Reduce 23 1 30 3 0 sec 10 sec 40 sec Really easy right? But what happens if the network only has 2 slots You can’t run map in parallel

Which tasks to run in which order? How many slots to assign? 100GB 200GB 10 GB Map Reduce 2 1 23 3 HDFS 8 Map HDFS 2 2 Map Reduce 23 1 30 3 0 sec 3 sec 13 sec 33 sec Which tasks to run in which order? How many slots to assign?

Approaches to Prediction Job Run Times Assumption: Job Runtime  Function (# slots) Implication: Given N slots, I can predict completion time Jockey Approach [EuroSys’10] Track job progress: fraction of completed tasks Build a map of [{% done + # of slots}  time to complete] Use simulator to build map Iterate through all possible combination of # of slots and %done. Problems: Ignores network transfers: Network congestion Cross job contention on server can impact completion time Not all tasks are equal: # of tasks done isn’t a good representation of progress

Open Questions What about background traffic? Control messages Other bulk transfer What about unexpected events? Failures? Loss of data? What about protocol inefficiencies? Hadoop scheduling TCP inefficiencies Server scheduling