22/07/ The MDS Scaling Problem for Cloud Storage Yu-chong Hu Institute of Network Coding

Background 2 A webmaster wants to upload his website for Michael Jackson to Internet

Background Dedicated Storage: purchase storage devices (servers) for his demand. A webmaster wants to upload his website for Michael Jackson to Internet

Background 4 Dedicated Storage: purchase storage devices (servers) for his demand. Cloud Storage: purchase storage services for his demand. A webmaster wants to upload his website for Michael Jackson to Internet

Background  The demand is dynamic Normal case During the 3 months after MJ died (BURST!) $50,000 $100,000 Return to normal 3 months later $100,000

Background  The demand is dynamic Normal case During the 3 months after MJ died (BURST!) $50,000 $100,000 Return to normal 3 months later $100,000 waste

Background  The demand is dynamic 7 Normal case During the 3 months after MJ died (BURST!) $50,000 Standard : $1 per hour $100,000 High Performance : $2 per hour Return to normal 3 months later $100,000 Standard : $1 per hour waste

Background  The demand is dynamic 8 Normal case During the 3 months after MJ died (BURST!) $50,000 Standard : $1 per hour $100,000 High Performance : $2 per hour Return to normal 3 months later $100,000 Standard : $1 per hour waste Always match the demand: No waste

Motivation 9 Network Applications Network Applications

Motivation 10 Network Applications Network Applications Dynamic Demand Dynamic Demand

Motivation 11 Network Applications Network Applications Dynamic Demand Dynamic Demand Cloud Storage Cloud Storage

Motivation 12 Network Applications Network Applications Dynamic Demand Dynamic Demand Cloud Storage Cloud Storage Scale the capacity Up Scale the capacity Up Scale the capacity down Scale the capacity down

Motivation 13 Network Applications Network Applications Dynamic Demand Dynamic Demand Cloud Storage Cloud Storage How to do the scaling as fast as possible? How to do the scaling as fast as possible? Scale the capacity Up Scale the capacity Up Scale the capacity down Scale the capacity down

Motivation 14 Network Applications Network Applications Dynamic Demand Dynamic Demand Cloud Storage Cloud Storage minimizing data migration scaling traffic minimizing data migration scaling traffic How to do the scaling as fast as possible? How to do the scaling as fast as possible? Scale the capacity Up Scale the capacity Up Scale the capacity down Scale the capacity down

Motivation  Replication-based data can be scaled very easily

Motivation  Replication-based data can be scaled very easily  Erasure coding-based data are more reliable. E.g:

Motivation  Replication-based data can be scaled very easily  Erasure coding-based data are more reliable. E.g: vs Replication A A B B (4,2) MDS erasure code A B A+B A+2B Node 1 Node 2 Node 3 Node 4 Node 1 Node 2 Node 3 Node 4 Source A B File Source Node

Motivation  Replication-based data can be scaled very easily  Erasure coding-based data are more reliable. E.g: vs Replication A A B B (4,2) MDS erasure code A B A+B A+2B Tolerate one failure Node 1 Node 2 Node 3 Node 4 Node 1 Node 2 Node 3 Node 4 Source A B File Source Node

Motivation  Replication-based data can be scaled very easily  Erasure coding-based data are more reliable. E.g: vs Replication A A B B (4,2) MDS erasure code A B A+B A+2B Tolerate one failure Tolerate two failures Node 1 Node 2 Node 3 Node 4 Node 1 Node 2 Node 3 Node 4 Source A B File Source Node

MDS Scaling Problem  Problem Statement: How to minimize the scaling traffic in storage system based on MDS codes.

MDS Scaling Problem  Problem Statement: How to minimize the scaling traffic in storage system based on MDS codes.  Network Coding + MDS Scaling Problem 21

MDS Scaling Problem  Problem Statement: How to minimize the scaling traffic in storage system based on MDS codes.  Network Coding + MDS Scaling Problem Source Node 1 n MDS (n,k)

MDS Scaling Problem  Problem Statement: How to minimize the scaling traffic in storage system based on MDS codes.  Network Coding + MDS Scaling Problem Source Node 1 n n’ Scaling MDS (n,k)MDS (n’,k’)

MDS Scaling Problem  Problem Statement: How to minimize the scaling traffic in storage system based on MDS codes.  Network Coding + MDS Scaling Problem 24 Source Node 1 n n’ Data collector any k’ of n’ nodes can rebuild file Data collector Data collector MDS (n,k)MDS (n’,k’) Scaling

MDS Scaling Problem  Problem Statement: How to minimize the scaling traffic in storage system based on MDS codes.  Network Coding + MDS Scaling Problem 25 Source Node 1 n n’ Scale traffic Data collector any k’ of n’ nodes can rebuild file Data collector Data collector MDS scaling problem = multicasting on information flow graph. MDS scaling problem = multicasting on information flow graph. MDS (n,k)MDS (n’,k’)

MDS Scaling Problem  Related work 1.Dress Codes[1]: The scaling problem is considered, but dress codes are especially designed for optimal repairing at first, and the value of k is fixed; MDS scaling problem needs to consider optimal scaling at first, and k can be scaled to k’ Dress Codes for the Storage Cloud: Simple Randomized Constructions, Sameer Pawar et al. 2.FastScale: Accelerate RAID Scaling by Minimizing Data Migration, W Zheng et al., FAST 2011’ 3.ALV: A new data redistribution approach to RAID-5 scaling, G Zhang et al. Transactions on computers

MDS Scaling Problem  Related work 1.Dress Codes[1]: The scaling problem is considered, but dress codes are especially designed for optimal repairing at first, and the value of k is fixed; MDS scaling problem needs to consider optimal scaling at first, and k can be scaled to k’. 2.RAID Scaling [2, 3]: The authors use a technique “sliding window” to reduce the moved packets for scaling of RAID codes, but they do not give a theoretical bound for scaling traffic Dress Codes for the Storage Cloud: Simple Randomized Constructions, Sameer Pawar et al. 2.FastScale: Accelerate RAID Scaling by Minimizing Data Migration, W Zheng et al., FAST 2011’ 3.ALV: A new data redistribution approach to RAID-5 scaling, G Zhang et al. Transactions on computers

MDS Scaling Problem  There are very few theoretical results about the scaling problem.  No bounds obtained  No optimal code and scaling schemes presented  Outline of my talk:  (n,k)MDS → (n+m,k) MDS  Example 1: (3,2) MDS → (4,2) MDS  Example 2: (3,2) MDS → (4,3) MDS  (n,k)MDS → (n+m,k+m) MDS  Example 3: (4,3) MDS → (5,4) MDS 28

Example 1: (3,2)MDS → (4,2)MDS 29 (3,2)MDS (4,2)MDS Data collector any 2 of 3 nodes can rebuild file The scaling (n,k)MDS → (n+m,k) MDS increases more data reliability because more failures can be tolerated. Data collector any 2 of 4 nodes can rebuild file Tolerate one failureTolerate two failures

Example 1: (3,2)MDS → (4,2)MDS 30 Data collector Source M: any 2 of 3 nodes can rebuild file (3,2)MDS Source Node A B C D A⊕CA⊕C B⊕DB⊕D A B C D

Example 1: (3,2)MDS → (4,2)MDS 31 Data collector Source M: any 2 of 3 nodes can rebuild file Source Node A B C D A⊕CA⊕C B⊕DB⊕D A B C D A (3,2)MDS

Example 1: (3,2)MDS → (4,2)MDS 32 Data collector Source M: any 2 of 3 nodes can rebuild file Source Node A B C D A⊕CA⊕C B⊕DB⊕D A B C D A ⊕ (3,2)MDS

Example 1: (3,2)MDS → (4,2)MDS 33 Data collector Source M: any 2 of 3 nodes can rebuild file Source Node A B C D A⊕CA⊕C B⊕DB⊕D A B C D A C⊕DC⊕D ⊕ (3,2)MDS

Example 1: (3,2)MDS → (4,2)MDS 34 Data collector Source M: any 2 of 3 nodes can rebuild file Source Node A B C D A⊕CA⊕C B⊕DB⊕D A B C D A C⊕DC⊕D B⊕DB⊕D ⊕ (3,2)MDS

Example 1: (3,2)MDS → (4,2)MDS 35 Data collector Source M: any 2 of 3 nodes can rebuild file Source Node A B C D A⊕CA⊕C B⊕DB⊕D A B C D A C⊕DC⊕D B⊕DB⊕D ⊕ (3,2)MDS

Example 1: (3,2)MDS → (4,2)MDS 36 Data collector Source M: any 2 of 3 nodes can rebuild file Source Node A B C D A⊕CA⊕C B⊕DB⊕D A B C D A C⊕DC⊕D B⊕DB⊕D B⊕CB⊕C ⊕ (3,2)MDS

Example 1: (3,2)MDS → (4,2)MDS 37 Data collector Source M: any 2 of 3 nodes can rebuild file Source Node A B C D A⊕CA⊕C B⊕DB⊕D A B C D A C⊕DC⊕D B⊕DB⊕D B⊕CB⊕C ⊕ (3,2)MDS

Example 1: (3,2)MDS → (4,2)MDS 38 Source M: Source Node A B C D A⊕CA⊕C B⊕DB⊕D A B C D B⊕CB⊕C A⊕B⊕DA⊕B⊕D A C⊕DC⊕D B⊕DB⊕D ⊕ (4,2)MDS

Example 1: (3,2)MDS → (4,2)MDS 39 Data collector Source M: any 2 of 4 nodes can rebuild file Source Node A B C D A⊕CA⊕C B⊕DB⊕D A B C D B⊕CB⊕C A⊕B⊕DA⊕B⊕D A C⊕DC⊕D B⊕DB⊕D ⊕ (4,2)MDS

Flow graph: 1-node Scaling = 1-loss Repairing 40 Data collector Source M: any 2 of 4 nodes can rebuild file Source Node A B C D A⊕CA⊕C B⊕DB⊕D A B C D 1 loss repair of (4,2) RC code B⊕CB⊕C A⊕B⊕DA⊕B⊕D B⊕CB⊕C A⊕B⊕DA⊕B⊕D A C⊕DC⊕D B⊕DB⊕D ⊕

Flow graph: 1-node Scaling = 1-loss Repairing 41 (3,2)MDS → (4,2)MDS 1-loss repair of (4,2) RC 2 2

Results  Conclusion 1: The optimal scaling problem from (n,k)MDS to (n+1,k)MDS is equivalent to the 1-loss repair problem based on (n+1,k)RC. 42

Results  Conclusion 1: The optimal scaling problem from (n,k)MDS to (n+1,k)MDS is equivalent to the 1-loss repair problem based on (n+1,k)RC.  Conclusion 2: The opitmal scaling problem from (n,k)MDS to (n+r,k)MDS is equivalent to the r-loss repair problem based on (n+r,k)RC for multiple failures. 43

Open Problems How to design the optimal scaling algorithms in which (n,k)MDS can grow to different MDS? 44

Open Problems How to design the optimal scaling algorithms in which (n,k)MDS can grow to different MDS? Motivation: In cloud storage, a consumer, at first, may select a default level of storage service with availability of 99.99%. Later, the consumer wants to increase the availability, so the cloud storage service should provide different higher level of storage redundancy for user’s selection. 45

Open Problems How to design the optimal scaling algorithms in which (n,k)MDS can grow to different MDS? Motivation: In cloud storage, a consumer, at first, may select a default level of storage service with availability of 99.99%. Later, the consumer wants to increase the availability, so the cloud storage service should provide different higher level of storage redundancy for user’s selection. Example: (3,2)MDS:99.99% 46 (4,2)MDS: % (5,2)MDS: %

Open Problems How to design the optimal scaling algorithms in which (n,k)MDS can grow to different MDS? Motivation: In cloud storage, a consumer, at first, may select a default level of storage service with availability of 99.99%. Later, the consumer wants to increase the availability, so the cloud storage service should provide different higher level of storage redundancy for user’s selection. Example: (3,2)MDS:99.99% Difficulty: This optimal scaling problem may not be equivalent to the optimal repair problem, because there exist at least two optional scaling size. 47 (4,2)MDS: % (5,2)MDS: %

Example 2: (3,2)MDS → (4,3)MDS 48 (3,2)MDS (4,3)MDS Data collector any 2 of 3 nodes can rebuild file (n,k)MDS → (n+m,k+m) MDS scaling increases higher I/O performance because more I/O bandwidth for data collector can be obtained. Data collector any 3 of 4 nodes can rebuild file 10MB/s

Example 2: (3,2)MDS → (4,3)MDS 49 Data collector Source M: any 2 of 3 nodes can rebuild file A B Source Node C D A B C E F D E F A⊕DA⊕D B⊕EB⊕E C⊕FC⊕F (3,2)MDS

Example 2: (3,2)MDS → (4,3)MDS 50 Source M: A B (4,3)MDS Source Node C D A B C E F D E F A⊕DA⊕D B⊕EB⊕E C⊕FC⊕F

Example 2: (3,2)MDS → (4,3)MDS 51 Source M: A B (4,3)MDS Source Node C D A B C E F D E F A⊕DA⊕D B⊕EB⊕E C⊕FC⊕F C

Example 2: (3,2)MDS → (4,3)MDS 52 Source M: A B (4,3)MDS Source Node C D A B E F D E F A⊕DA⊕D B⊕EB⊕E C⊕FC⊕F C

Example 2: (3,2)MDS → (4,3)MDS 53 Source M: A B (4,3)MDS Source Node C D A B E F D E F A⊕DA⊕D B⊕EB⊕E C⊕FC⊕F C D

Example 2: (3,2)MDS → (4,3)MDS 54 Source M: A B (4,3)MDS Source Node C D A B E F E F A⊕DA⊕D B⊕EB⊕E C⊕FC⊕F C D

Example 2: (3,2)MDS → (4,3)MDS 55 (4,3)MDS A⊕DA⊕D B⊕EB⊕E C⊕FC⊕F

Example 2: (3,2)MDS → (4,3)MDS 56 (4,3)MDS A⊕DA⊕D B⊕EB⊕E C⊕FC⊕F A⊕D⊕B⊕EA⊕D⊕B⊕E B⊕E⊕C⊕FB⊕E⊕C⊕F

Example 2: (3,2)MDS → (4,3)MDS 57 Source M: A B (4,3)MDS Source Node C D A B E F E F A⊕D⊕B⊕EA⊕D⊕B⊕E B⊕E⊕C⊕FB⊕E⊕C⊕F (3,2)MDS C D

Example 2: (3,2)MDS → (4,3)MDS 58 Data collector Source M: any 3 of 4 nodes can rebuild file A B (4,3)MDS Source Node C D A B E F E F A⊕D⊕B⊕EA⊕D⊕B⊕E B⊕E⊕C⊕FB⊕E⊕C⊕F (3,2)MDS C D

Example 2: (3,2)MDS → (4,3)MDS 59 Data collector Source M: any 3 of 4 nodes can rebuild file A B (4,3)MDS Source Node C D A B E F E F A⊕D⊕B⊕EA⊕D⊕B⊕E B⊕E⊕C⊕FB⊕E⊕C⊕F (3,2)MDS C D Scaling traffic = M/3

Example 2: (3,2)MDS → (4,3)MDS 60 Data collector Source M: any 3 of 4 nodes can rebuild file A B (4,3)MDS Source Node C D A B E F E F A⊕D⊕B⊕EA⊕D⊕B⊕E B⊕E⊕C⊕FB⊕E⊕C⊕F (3,2)MDS C D Scaling traffic = M/3 The size of new node is M/3, so the scaling traffic is minimal

Example 3: (4,3)MDS → (5,4)MDS (n+m,k+m) MDS→(n+m+l,k+m+l)MDS scaling (iterative scaling) increases higher and higher I/O performance. (4,3)MDS (5,4)MDS Data collector any 3 of 4 nodes can rebuild file 10MB/s Data collector any 4 of 5 nodes can rebuild file 10MB/s

Example 3: (4,3)MDS → (5,4)MDS In real storage systems, a data object is divided into a block stream for some practical reasons: A1 B1 C1 D1 E1 F1 A2 B2 C2 D2 E2 F2 …… Data Object:

Example 3: (4,3)MDS → (5,4)MDS (4,3)MDS C1 D1 A1 B1 E1 F1 A1 ⊕ D1 ⊕ B1 ⊕ E1 B1 ⊕ E1 ⊕ C1 ⊕ F1 A2 ⊕ D2 ⊕ B2 ⊕ E2 B2 ⊕ E2 ⊕ C2 ⊕ F2 A2 B2 E2 F2 C2 D2 Segment 1Segment 2 In real storage systems, a data object is divided into a block stream for some practical reasons: A1 B1 C1 D1 E1 F1 A2 B2 C2 D2 E2 F2 …… Data Object:

Example 3: (4,3)MDS → (5,4)MDS C1 D1 A1 B1 E1 F1 A1 ⊕ D1 ⊕ B1 ⊕ E1 B1 ⊕ E1 ⊕ C1 ⊕ F1 A2 ⊕ D2 ⊕ B2 ⊕ E2 B2 ⊕ E2 ⊕ C2 ⊕ F2 A2 B2 E2 F2 C2 D2 Segment 1Segment 2

Example 3: (4,3)MDS → (5,4)MDS C1 D1 A1 B1 E1 F1 A1 ⊕ D1 ⊕ B1 ⊕ E1 B1 ⊕ E1 ⊕ C1 ⊕ F1 A2 ⊕ D2 ⊕ B2 ⊕ E2 B2 ⊕ E2 ⊕ C2 ⊕ F2 A2 B2 E2 F2 C2 D2 Segment 1Segment 2 Consider two adjacent segments

Example 3: (4,3)MDS → (5,4)MDS 66 (4,3)MDS C1 D1 E1 F1 A1 ⊕ D1 ⊕ B1 ⊕ E1 B1 ⊕ E1 ⊕ C1 ⊕ F1 A2 ⊕ D2 ⊕ B2 ⊕ E2 B2 ⊕ E2 ⊕ C2 ⊕ F2 E2 F2 C2 D2 A1 B1 A2 B2

Example 3: (4,3)MDS → (5,4)MDS 67 C1 D1 E1 F1 A1 ⊕ D1 ⊕ B1 ⊕ E1 B1 ⊕ E1 ⊕ C1 ⊕ F1 A2 ⊕ D2 ⊕ B2 ⊕ E2 B2 ⊕ E2 ⊕ C2 ⊕ F2 E2 F2 C2 D2 (5,4)MDS A1 B1 A2 B2

Example 3: (4,3)MDS → (5,4)MDS 68 C1 D1 E1 F1 A1 ⊕ D1 ⊕ B1 ⊕ E1 B1 ⊕ E1 ⊕ C1 ⊕ F1 A2 ⊕ D2 ⊕ B2 ⊕ E2 B2 ⊕ E2 ⊕ C2 ⊕ F2 E2 F2 C2 D2 (5,4)MDS A1 B1 A2 B2

Example 3: (4,3)MDS → (5,4)MDS 69 C1 D1 E1 F1 A1 ⊕ D1 ⊕ B1 ⊕ E1 B1 ⊕ E1 ⊕ C1 ⊕ F1 A2 ⊕ D2 ⊕ B2 ⊕ E2 B2 ⊕ E2 ⊕ C2 ⊕ F2 E2 F2 C2 D2 (5,4)MDS A1 B1 A2 B2 E1

Example 3: (4,3)MDS → (5,4)MDS 70 C1 D1 F1 A1 ⊕ D1 ⊕ B1 ⊕ E1 B1 ⊕ E1 ⊕ C1 ⊕ F1 A2 ⊕ D2 ⊕ B2 ⊕ E2 B2 ⊕ E2 ⊕ C2 ⊕ F2 E2 F2 C2 D2 (5,4)MDS A1 B1 A2 B2 E1

Example 3: (4,3)MDS → (5,4)MDS 71 C1 D1 F1 A1 ⊕ D1 ⊕ B1 ⊕ E1 B1 ⊕ E1 ⊕ C1 ⊕ F1 A2 ⊕ D2 ⊕ B2 ⊕ E2 B2 ⊕ E2 ⊕ C2 ⊕ F2 E2 F2 C2 D2 (5,4)MDS A1 B1 A2 B2 E1 D1

Example 3: (4,3)MDS → (5,4)MDS 72 C1 F1 A1 ⊕ D1 ⊕ B1 ⊕ E1 B1 ⊕ E1 ⊕ C1 ⊕ F1 A2 ⊕ D2 ⊕ B2 ⊕ E2 B2 ⊕ E2 ⊕ C2 ⊕ F2 E2 F2 C2 D2 (5,4)MDS A1 B1 A2 B2 E1 D1

Example 3: (4,3)MDS → (5,4)MDS 73 A1 ⊕ D1 ⊕ B1 ⊕ E1 B1 ⊕ E1 ⊕ C1 ⊕ F1 A2 ⊕ D2 ⊕ B2 ⊕ E2 B2 ⊕ E2 ⊕ C2 ⊕ F2 (5,4)MDS

Example 3: (4,3)MDS → (5,4)MDS 74 A1 ⊕ D1 ⊕ B1 ⊕ E1 B1 ⊕ E1 ⊕ C1 ⊕ F1 A2 ⊕ D2 ⊕ B2 ⊕ E2 B2 ⊕ E2 ⊕ C2 ⊕ F2 (5,4)MDS A1 ⊕ D1 ⊕ B1 ⊕ E1B1 ⊕ E1 ⊕ C1 ⊕ F1 ⊕ A2 ⊕ D2 ⊕ B2 ⊕ E2 B1 ⊕ E1 ⊕ C1 ⊕ F1 B2 ⊕ E2 ⊕ C2 ⊕ F2 ⊕

Example 3: (4,3)MDS → (5,4)MDS 75 C1 F1 E2 F2 C2 D2 (5,4)MDS A1 B1 A2 B2 E1 D1 A1 ⊕ D1 ⊕ C1 ⊕ F1 B1 ⊕ E1 ⊕ C1 ⊕ F1 ⊕ B2 ⊕ E2 ⊕ C2 ⊕ F2 A2 ⊕ D2 ⊕ B2 ⊕ E2

Example 3: (4,3)MDS → (5,4)MDS 76 C1 F1 E2 F2 C2 D2 (5,4)MDS A1 B1 A2 B2 E1 D1 A1 ⊕ D1 ⊕ C1 ⊕ F1 B1 ⊕ E1 ⊕ C1 ⊕ F1 ⊕ B2 ⊕ E2 ⊕ C2 ⊕ F2 A2 ⊕ D2 ⊕ B2 ⊕ E2 Data collector any 4 of 5 nodes can rebuild file

Example 3: (4,3)MDS → (5,4)MDS 77 C1 F1 E2 F2 C2 D2 (5,4)MDS A1 B1 A2 B2 E1 D1 A1 ⊕ D1 ⊕ C1 ⊕ F1 B1 ⊕ E1 ⊕ C1 ⊕ F1 ⊕ B2 ⊕ E2 ⊕ C2 ⊕ F2 A2 ⊕ D2 ⊕ B2 ⊕ E2 Data collector any 4 of 5 nodes can rebuild file Scaling traffic = 3 blocks

Example 3: (4,3)MDS → (5,4)MDS 78 C1 F1 E2 F2 C2 D2 (5,4)MDS A1 B1 A2 B2 E1 D1 A1 ⊕ D1 ⊕ C1 ⊕ F1 B1 ⊕ E1 ⊕ C1 ⊕ F1 ⊕ B2 ⊕ E2 ⊕ C2 ⊕ F2 A2 ⊕ D2 ⊕ B2 ⊕ E2 Data collector any 4 of 5 nodes can rebuild file Scaling traffic = 3 blocks

Example 3: (4,3)MDS → (5,4)MDS 79 C1 F1 E2 F2 C2 D2 (5,4)MDS A1 B1 A2 B2 E1 D1 A1 ⊕ D1 ⊕ C1 ⊕ F1 B1 ⊕ E1 ⊕ C1 ⊕ F1 ⊕ B2 ⊕ E2 ⊕ C2 ⊕ F2 A2 ⊕ D2 ⊕ B2 ⊕ E2 Data collector any 4 of 5 nodes can rebuild file Scaling traffic = 3 blocks Optimal code

Open Problems 1.What is the theoretical bound of scaling traffic from (n,k)MDS to (n+m,k+m)MDS? 2.How to design the generalized optimal scaling algorithms to match the bound? 3.How to deal with more complicated cases, such as iterative scaling? 4.What about (n,k)MDS → (n-m,k-m)MDS? 80 Generalize form: How to minimize the scaling traffic in storage system from (n,k) to (n’,k’) MDS codes. Generalize form: How to minimize the scaling traffic in storage system from (n,k) to (n’,k’) MDS codes.

Fin 81