Deconstructing Commodity Storage Clusters Haryadi S. Gunawi, Nitin Agrawal, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau Univ. of Wisconsin - Madison Jiri Schindler Corporation

2 Storage system  Storage system –Important components of large-scale systems –Multi-billion dollar industry  Often comprised of high-end storage servers –A big box with lots of disks inside  The simple question –How does storage server work? –Simple but hard – closed storage subsystem design

3 Why need to know?  Better modeling –How system behaves under different workload –Example in storage industry: capacity model for capacity planning –Model is limited if the information is limited  Product validation –Validate what product specs say –Performance numbers cannot confirm  Critical evaluation of design and implementation choices –Control what is occurring inside

4 Traditionally black box  Highly customized and proprietary hardware and OS –Hitachi Lightning, NetApp Filers, EMC Symmetrix –EMC Symmetrix: disk/cache manager, proprietary OS  Internal information is hidden behind standard interfaces Client Storage System ? Acks

5 Modern graybox storage system  Cluster of commodity PCs running commodity OS –Google FS cluster, HP FAB, EMC Centera  Advantages of commodity storage clusters –Direct internal observation – visible probe points –Leverage existing standardized tools Client Storage System Update DB

6 Intra-box Techniques  Two “Intra-box” techniques –Observation –System perturbation  Two components of analysis –Deduce structure of main communication protocol Object Read and Write protocol –Internal policy decisions Caching, prefetching, write buffering, load balancing, etc.

7 Goal and EMC Author  Objectives –Feasibility of deconstructing commodity storage clusters, no source code –Results achieved without EMC assistance  EMC Author –Evaluate correctness of our findings –Give insights behind their design decisions

8 Outline  Introduction  EMC Centera Overview –Intra-box tools  Deducing Protocol –Observation and Delay Pertubation  Inferring Policies –System Perturbation  Conclusion

9 WAN LAN Centera Topology Client AN 1 AN 2 SN 1 Access Nodes SN 2 SN 3 SN 4 SN 5 SN 6 Storage Nodes

10 Storage Node Commodity OS Reiserfs IDE driver TCP/UDP Linux Access Node Linux Client Client SDK TCP Centera Software Reiserfs IDE driver WAN LAN TCP/UDP

11 Probe Points – Observation  Internal probe points –Trace traffic using standardized tools tcpdump: trace network traffic Pseudo Device Driver: trace disk traffic Storage Node Reiserfs IDE drives TCP/UDP Centera Software Access Node Centera SW. TCP/UDP Client Client SDK TCP tcpdump Pseudo Dev. Driver

12 Probe Points – Perturbation  Perturbing system at probe points –Modified NistNet: delay particular messages –Pseudo Dev. Driver: delay disk I/O traffic –Additional Load CPU Load: High priority while loop Disk Load: File copy Storage Node Reiserfs IDE drives TCP/UDP Centera Software Access Node Centera SW TCP/UDP Client TCP tcpdump Pseudo Dev. Mod. NistNet Add CPU Load: while(1) {..} Add Disk Load: cp fX fY Client SDK User-level Process + Delay

13 Outline  Introduction  EMC Centera Overview  Deducing Protocol –Observation and Delay Perturbation  Inferring Policies –System Perturbation  Conclusion

14 Understanding the protocol  Understanding Read/Write protocol –Read and Write implementations in big distributed storage systems are not simple –Deconstruct the protocol structure Which pieces are involved? Where data is sent to? Data reliably stored, mirrored, striped?

15 Observing Write Protocol  Deconstruct protocol using passive observation –Run a series of write workload –Observe network and disk traffic –Correlation tools: convert traces into protocol structure Client EMC Centera Access Nodes Storage Nodes write( )

16 Observation Results  Object Write Protocol findings –Phase 1: Write request establishment –Phase 2: Data transfer –Phase 3: Disk write, notify other SNs, commit –Phase 4: Series of acknowledgement  Determine general properties –Primary SN handles generation of 2 nd copy –Two new TCP connections / object write time R R R Write Req. Request Ack. Client Access Node Primary SN Secondary SN Data Transfer Write-Commit Transfer Ack. Write Complete Write-Commit Software ACKs Software ACKs TCP Setup Write Req Request Ack. SNv SNw SNx SNy TCP Setup

17 Resolving Dependencies  Cannot conclude dependencies from observation only –B after A != B depends on A Must delay A, and see if B is delayed AN Primary SN Secondary SN Secondary Commit (sc) Primary commit (pc) Conclude causality by delaying: -disk write traffic and -secondary commit From observation only: Primary commit depends on secondary commit and sync. disk write

18 Delaying a Particular Message  Need to delay a particular message –Leverage packet sizes –Modify NistNet Delay specific message, not link –Ex: delay sc (90 bytes) 299 bytes Client Access Node Primary SN Secondary SN sc prim. commit Primary SN CentraStar Linux TCP/UDP if size= 90 Mod. NistNet incoming packet no yes delay queue 90 bytes

19 Delaying secondary-commit AN Primary SN Secondary SN Secondary commit Primary commit  Resolving first dependency –Delay secondary commit  primary commit also gets delayed  Primary commit depends on the receipt of secondary commit + delay

20 Delaying disk I/O traffic  Delay disk writes at primary storage node Primary SN Secondary-commit Primary-commit + Delay Disk Write Primary SN CentraStar ReiserFS if WRITE Pseudo-Dev disk req no yes delay queue IDE Driver From observation and delay: Primary commit depends on secondary commit message and sync. disk write

21 Ability to analyze internal designs  Intra-box techniques: Observation and perturbation by delay –Able to deduce Object Write protocol –Give ability to analyze internal design decisions  Serial vs. Parallel –Primary SN handles the generation of 2 nd copy (Serial) vs. AN handles both 1 st and 2 nd (Parallel) –EMC Centera: write throughput is more important –Decrease load on access nodes – increase write throughput  New TCP connections (internally) / object write –vs. using persistent connection to remove TCP setup cost –Prefer simplicity – no need to manage persistent conn. for all requests Client AN SN1 SN2 Client AN SN1 SN

22 Outline  Introduction  EMC Centera Overview  Deducing Protocol  Inferring Policies –Various system perturbation  Conclusion

23 Inferring internal policies  Write policies –Level of replication, Load balancing, Caching/buffering  Read policies –Caching, Prefetching, Load balancing  Try to infer –Is particular policy implemented? –At which level it is being implemented? Ex: Read Caching at Client, Access Node, Storage Node?

24 System Pertubation  Perturb the system –Delay and extra load  4 common load- balancing factors: –CPU load High priority while loop –Disk load Background file copy –Active TCP connection –Network delay Access Node Client write() SN 1 SN 2 ?? ? SN 3 SN … ? CPU Active TCP + net delay

25 Write Load Balancing  What factors determined which storage nodes are selected?  Experiment : –Observe which primary storage nodes selected –Without load: writes are balanced –With load: writes skew toward unloaded nodes ? AN sn#1 Unloaded sn#2 Unloaded ? sn#2 Loaded

26 Write Load Balancing Results Normal No Perturb Additional CPU Load Disk LoadNetwork Load Incoming Net. Delay sn#1 sn#2 sn#1 +CPU sn#1 +Disk sn#1 +TCP sn#1 +Delay

27 Summary of findings Write Policies ReplicationTwo copies in two nodes attached to different power (reliability) Load balancingCPU usage (locally observable status) Network status is not incorporated Write bufferingStorage nodes write synchronously Read Policies CachingStorage node only (commodity filesystem) Access node and client does not cache. PrefetchingStorage node only (commodity filesystem) Access node and client does not prefetch Load BalancingNot implemented in earlier version Still reads from busy nodes EMC Centera: Simplicity and Reliability

28 Conclusion  Intra-box: –Observe and perturb –Deconstruct protocol and infer policies –No access to source code  Power of probe points –More observation places –Ability to control the system  Systems built with more externally visible probe points –Systems more readily understood, analyzed, and debugged –Higher-performing, more robust and reliable computer systems

29 Questions?