1. Parity Lost and Parity Regained Andrew Krioukov, Lakshmi N. Bairavasundaram, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau University of Wisconsin - Madison Garth R. Goodson, Kiran Srinivasan, Randy Thelen

2. Bare-bones RAID Stripe data across multiple drives Store redundant parity data Can reconstruct data with any single disk failure Will RAID protect data in all single failure cases? ABC Data 1Data 2Data 3Parity P(ABC) 2

3. Bare-bones RAID Problems Stripe contains file ABC consisting of 3 blocks RAID has redundancy to recover data RAID does not detect corruption Data 1Data 2Data 3Parity P(ABC) Corruption Read file ABC Return Corrupt File AABB@#$%C RAID Stripe 3

4. Bare-bones RAID Problems RAID cannot detect partial disk failures: – Corruptions – Torn writes – Lost writes – Misdirected writes RAID only protects against – Complete disk failures – Errors reported by the disk (e.g. Latent Sector Errors) 4

5. Data Protection Techniques Need improvements to bare-bones RAID – Techniques needed to help detect errors Checksums are common – Many kinds: block, sector, parent checksums Which type of checksums are used? We examined real systems to determine protection schemes 5

6. Enterprise RAID Systems Mixed bag of protections ScrubSector Cksum Block Cksum Parent Cksum Write Verify Phys Iden t Logical Ident Write Stamp Dell Power- vault √√√ Hitachi Thunder √√√ NetApp ONTAP √√√√√ Sun ZFS √√ 6

7. Question Which errors do these systems protect against? How can we ensure complete data protection? Need method to identify all corruption & data loss scenarios in a design 7

8. Model Checking Solution Create a model of storage system design using primitives Checker exhaustively searches space of all possible states – Start with clean RAID stripe – Apply single disk error – Apply any number of disk operations (e.g. write) Identifies all possible data loss scenarios 8

9. Results Summary Applied model checking on enterprise RAID system designs For all designs, a single error can cause data loss Identified a common problem, parity pollution – Partial disk failure goes undetected – The erroneous data is used to compute parity – Recovery is no longer possible Presented a design that protects against all single failures 9

10. Outline Introduction Background: Storage Errors Model Checking Approach Data Protection Design & Analysis Conclusion 10

11. Storage Errors Latent Sector Errors – Data is inaccessible – Explicit error code returned – Affect 19% of nearline, 2% of enterprise disks in 2 years [Bairavasundaram et al. SIGMETRICS’07] Corruptions – Data is silently corrupted – Affect 0.6% of nearline and 0.06% of enterprise disks in 17 months [Bairavasundaram et al. FAST’08] Reality: Partial disk failures happen 11

12. Storage Errors (Cont’d) Torn Write – Only part of a block is written – Some sectors are lost – Write returns success code Lost Writes – Write returns success code – Data not reflected on disk A Write B Success A Write B Success B 12

13. Storage Errors (Cont’d) Misdirected Writes – Write goes to wrong location (either wrong block or wrong disk) – Combination of lost write and corruption 13 A Overwrite A  A’ Success BA’

14. Outline Introduction Background: Storage Errors Model Checking Approach Data Protection Design & Analysis Conclusion 14

15. Modeling Storage System Use primitives to describe: – On disk layout in terms of sectors – Data protections Checker uses built-in models: – Storage errors – Disk operations (e.g. Read/Write) – Basic RAID functionality 15

16. Model Checking Assumptions – Single RAID stripe – Single storage error – Single parity protection – Data disks are interchangeable Apply error followed by any number of disk operations Generate state diagram with all data loss states 16

17. State Diagram Example Bare-bones RAID state diagram Clean Parity Error Corrupt(p), Torn(p), Lost(p), Misdir(p) Wadd(x+) Disk x Error Corrupt(x), Torn(x), Lost(x), Misdir(x) Wsub(x+) Corrupt Data Polluted Parity R(x) Wadd() W(x+) Wadd(!x) 17

18. Outline Introduction Background: Storage Errors Model Checking Approach Data Protection Design & Analysis Conclusion 18

19. Data Protection Design Need fault tolerance for all partial failures Bare-bones RAID handles latent sector errors and complete disk failures Corruption is next most common failure Add protections cumulatively until design has complete protection 19

20. Protections Protections in red will be discussed in the talk Scrubbing Sector checksums Block checksums Parental checksums Write verify Physical identity Logical identity Version mirroring 20

21. Checksums Checksum per data block Checksum per sector Parent checksum – Checksum stored in parent inode 21 A cksum(A) ck(a1) ck(a2) a2 a1 … A

22. Corruption scenario is now fixed Data 1Data 2Data 3Parity B cksum(B) A cksum(A) C cksum(C) P(ABC) cksum(P) Checksum Example 22 Corruption Read file ABC @#$% @#$% cksum(C) Perform reconstruction File is valid ABP(ABC) CC

23. Checksum Problems Great for protecting against corruption errors Fails to protect when data and checksum are lost together: – Lost write (with any type of checksums) – Torn write (only with sector checksums) Parity pollution can occur 23

24. Data 1Data 2Data 3Parity B cksum(B) A cksum(A) C cksum(C) P(ABC) cksum(P) Checksum Problems – Lost Write Block checksums Overwrite C→C’ P(ABC’) Lost Write Read file ABC’ C cksum(C) Return data (ABC) Return Corrupt Data (C instead of C’)

25. Write Verify Attempt to solve lost write problem Costly solution, expect good protection Procedure: 1.Write data to disk 2.Read back to verify 3.If lost write detected, write again or remap to new location Overwrite C→C’ Lost Write C cksum(C) Read back (C) Lost write detected, write C’ again C’ Success cksum(C’) 25

26. Write Verify Problems Protects against lost writes Susceptible to misdirected writes – Cannot detect/recover the overwritten data 26

27. Write Verify – Misdirected Write Overwrite X → X’ Misdirected Write A cksum(A) Read back X Lost, Re-write X X’B cksum(B) Parity P(ABC) cksum(P) cksum(X’) Read file ABC X’ Return Corrupt Data (A has been corrupted) Data 1Data 2 Initially… Later… Data 3 C cksum(C) BC 27 XYZP(XYZ)X’P(X’YZ)

28. Physical Identity Protection against misdirected writes Store disk & block number of destination in each block 28 A 1 Overwrite Block 1: A  A’ B 2A’ 1 Read Block 2 Returned (A’, 1) Block num does not match (1≠2) Misdirected Write Detected Misdirected Write Data, Block Number

29. Problem Solved? Write verify with block checksums and physical identity offers complete protection But… twice the I/O cost! Need a more efficient solution 29

30. Logical Identity Less expensive protection against lost writes Store file identifier (e.g. inode number) in each data block Test that file identifier matches on a read 30 A cksum(A) Lost Write Overwrite File 0 with File 1 (X) File 0 Read File 1 Logical ID does not match. Lost Write Detected A File 0

31. Logical Identity Problem Cannot be verified when re-computing parity – Not reading a file Parity pollution may occur 31

32. Parity Pollution Example Data 1Data 2Data 3Parity B cksum(B) A cksum(A) C cksum(C) P(ABC) cksum(P) C→C’, P(ABC’) Lost Write Overwrite AB →A’B’ Parity: A’ cksum(A’) B’ cksum(B’) A’B’ C Read Data 3 P(A’B’C) Write File 1 Later… Write File 2 P(A’B’C) Parity consistent with invalid data File 0 File 2 New Parity Later… Read File 1 Logical ID mismatch (File 0 ≠ File 1) Reconstruct… Data is consistent! C File 0 Report Data Loss A File0B File0C File0P(ABC)P(ABC’) A’ File2B’ File2C’ File1 P(A’B’C’) What should be on the disk

33. Version Mirroring Lost write protection Verifiable at RAID level Store a version number in each data block Mirror the version numbers on parity disk Versions numbers verified on read 33 B cksum(B) A cksum(A) C cksum(C) P(ABC) cksum(P) Ver0 0,0,0

34. Parity Pollution Solved Data 1Data 2Data 3Parity B cksum(B) A cksum(A) C cksum(C) P(ABC) cksum(P) C→C’, P(ABC’) Lost Write Overwrite AB →A’B’ Parity: A’ cksum(A’) B’ cksum(B’) A’B’ Read Data 3 P(A’B’C’) Write File 1 Later… Write File 2 P(A’B’C’) Ver0 Ver1 New Parity 0,0,00,0,1 Ver0 0,0,1 Version mismatch Reconstruct Data 3 Ver1 C C’ AB P(ABC’) C’ cksum(C’) 1,1,1 A Ver0B Ver0C Ver0P(ABC)P(ABC’) A’ Ver1B’ Ver1 C’ Ver1 P(A’B’C’) What should be on the disk C’

35. Problem Solved… Efficiently Version mirroring with block checksums and physical identity provides complete protection Use with logical identity for efficiency More efficient than write verify 35

36. Conclusion Applied model checking on real system designs – For all designs, a single error can cause data loss – Parity pollution is a common problem – Version mirroring is a key technique to offering complete and efficient data protection Partial failures are complex, no obvious data protection solution – Model checking is useful 36

37. 37 ADvanced Systems Laboratory www.cs.wisc.edu/adsl Advanced Technology Group http://www.netapp.com/company/research/