1. A Row-Permutated Data Reorganization Algorithm for Growing Server-less VoD Systems Presented by Ho Tsz Kin

2. Agenda Background Existing solutions Row-Permutated (RP) Algorithm Multi-RP Algorithm Performance Evaluation Conclusion

3. Background Each node keeps balance video data blocks Nodes join the system Data must be reorganized to utilize storage and streaming capacity 0 4 8 12 16 1 5 9 13 17 2 6 10 14 18 3 7 11 15 19 n0n0 n1n1 n3n3 n2n2 node n 4 joins 0 5 10 15 1 6 11 16 2 7 12 17 3 8 13 18 n0n0 n1n1 n3n3 n2n2 4 9 14 19 n3n3

4. Background Data reorganization Require data block movement Consume bandwidth Should not disrupt services Achieve storage and streaming balance

5. Existing Solutions Round-robin Reorganization Round-robin placement policy Advantages: Perfect storage and streaming balance Drawbacks: Nearly all the data blocks must be reorganized 0 4 8 12 16 1 5 9 13 17 2 6 10 14 18 3 7 11 15 19 n0n0 n1n1 n3n3 n2n2 node n 4 joins 0 5 10 15 1 6 11 16 2 7 12 17 3 8 13 18 n0n0 n1n1 n3n3 n2n2 4 9 14 19 n3n3

6. Existing Solutions Randomized Reorganization Randomized placement policy Blocks are distributed to nodes randomly n0n0 n1n1 n3n3 n2n2 0 Assign to each node with equal probability 3 8 9 15 16 1 2 4 13 17 0 6 11 12 19 5 7 10 14 18 n0n0 n1n1 n3n3 n2n2

7. Existing Solutions Reorganization Algorithm Number of nodes, N Probability of residing in same node = Probability of moving to new node = 3 8 9 15 16 1 2 4 13 17 0 6 11 12 19 5 7 10 14 18 n0n0 n1n1 n3n3 n2n2 n4n4 P =

8. Existing Solutions 3 8 9 15 16 1 2 4 13 17 0 6 11 12 19 5 7 10 14 18 n0n0 n1n1 n3n3 n2n2 node n 4 joins Randomized Reorganization Advantages: Block movement is minimized, achieve reasonable storage balance Drawbacks: Streaming load is imbalance 3 8 9 15 16 1 2 4 13 17 0 6 11 12 19 5 7 1014 18 n0n0 n1n1 n3n3 n2n2 n4n4 imbalance row

9. Goal Two extreme cases Round-robin Reorganization Overhead is maximum, balance streaming load Randomized Reorganization Overhead is minimum, imbalance streaming load Two Goals: Maintain balance streaming load but lower the overhead of round-robin reorganization Allow controllable tradeoff between overhead and streaming load balance

10. Row-Permutated (RP) Algorithm Idea: the sequence of blocks within each row is not important in streaming load Row-permutated placement policy Streaming load is still balanced 1032 n0n0 n1n1 n3n3 n2n2 0123 n0n0 n1n1 n3n3 n2n2 Both maintain balanced streaming load Round-robin PlacementRow-Permutated Placement

11. Row-Permutated (RP) Algorithm Reorganization Algorithm Reorganize one row per iteration Identify overflow and underflow nodes Overflow if more than 1 block Underflow if no block Move excess block from overflow nodes to underflow nodes 0 7 8 13 16 1 4 10 12 17 3 5 9 14 19 2 6 11 15 18 n0n0 n1n1 n3n3 n2n2 n4n4 Overflow Node Underflow Node Excess block Target row in this iteration

12. Row-Permutated (RP) Algorithm Perfect streaming and storage balance Significantly lower down number of block movement during reorganization n0n0 n1n1 n3n3 n2n2 0 7 8 13 16 1 4 10 12 17 3 5 9 14 19 2 6 11 15 18 node n 4 joins n0n0 n1n1 n3n3 n2n2 0 7 8 13 16 1 4 1012 17 3 5 9 14 19 2 6 11 1518 n4n4

13. Multi-RP Algorithm Tradeoff between overhead and streaming balance Control streaming balance by window size, w n0n0 n1n1 n3n3 n2n2 0 4 11 12 16 1 5 9 13 20 2 6 10 14 18 3 7 8 15 22 172119 23 n0n0 n1n1 n3n3 n2n2 11 12 16 1310 14 18 15 1719 n4n4 w =2 Consider 2 rows

14. Multi-RP Algorithm Reorganization Algorithm Reorganize w rows per iteration Identify overflow and underflow nodes Overflow if more than w blocks Underflow if fewer than w blocks n0n0 n1n1 n3n3 n2n2 11 12 16 1310 14 18 15 1719 n4n4 w =2 Overflow Nodes Underflow Nodes

15. Multi-RP Algorithm In each overflow node Choose row with largest number of block Take blocks in this row as excess blocks Move to underflow nodes Contains smallest number of blocks in this row n0n0 n1n1 n3n3 n2n2 11 12 16 1310 14 18 15 1719 n4n4 n0n0 n1n1 n3n3 n2n2 11 12 16 1310 14 18 15 17 19 n4n4 randomly

16. Multi-RP Algorithm Idea: Spread out blocks within row n0n0 n1n1 n3n3 n2n2 11 12 16 1310 14 18 15 17 19 n4n4 row with largest number of blocks n0n0 n1n1 n3n3 n2n2 11 1216 131014 18 15 1719 n4n4 n0n0 n1n1 n3n3 n2n2 11 12 16 1310 14 18 15 17 19 n4n4

17. Performance Evaluation Experiment Details Number of data blocks = 4000 Grow from 1 node to 200 nodes Metrics Data Reorganization Overhead Number of block movement Streaming Load Balance Proportion of missing data block within one row, given that each node can only send out one block each round

18. Data Reorganization Overhead

19. Streaming load balance

20. Conclusion Identify the shortcomings of round-robin and randomized reorganization RP and multi-RP reorganization are proposed Perfect streaming load balance with lower overhead Controllable tradeoff between overhead and streaming load balance