Speculative Execution in a Distributed File System Ed Nightingale Peter Chen Jason Flinn University of Michigan

2 Motivation Why are distributed file systems slow(er)? –Sync n/w messages provide consistency –Sync disk writes provide safety Sacrifice guarantees for speed Can DFS can be safe, consistent and fast? –Yes! With OS support for speculative execution

3 Big Idea: Slow Way RPC Req Client RPC Resp Guarantees without blocking I/O! Server Block!2) Speculate! 1) Checkpoint Big Idea: Speculator 3) Correct? Yes: discard ckpt.No: restore process & re-execute RPC Req RPC Resp RPC Req RPC Resp

4 Conditions for Success Operations are highly predictable –Conflicts are rare Checkpoints are cheaper than network I/O –52 µs for small process Computers have resources to spare –Need memory and CPU cycles for speculation

5 Outline Motivation Implementing speculation Multi-process speculation Using Speculator Evaluation

6 Undo log Implementing Speculation Process Checkpoint Spec 1) System call2) Create speculation Time

7 Speculation Success Undo log Checkpoint 1) System call2) Create speculation Process 3) Commit speculation Time Spec

8 Speculation Failure Undo log Checkpoint 1) System call 2) Create speculation Process 3) Fail speculation Process Time Spec

9 Ensuring Correctness Spec processes often affect external state Three ways to ensure correct execution –Block –Buffer –Propagate speculations (dependencies)

10 Systems Calls Block calls that externalize state –Allow read-only calls (e.g. getpid) –Allow calls that modify only task state (e.g. dup2) File system calls -- need to dig deeper –Mark file systems that support Speculator getpid reboot mkdir Call sys_getpid() Block until specs resolved Allow only if fs supports Speculator

11 Output Commits “stat worked” “mkdir worked” Undo log Checkpoint Spec (stat) Spec (mkdir) 1) sys_stat2) sys_mkdir Process Time 3) Commit speculation

12 Multi-Process Speculation Processes often cooperate –Example: “make” forks children to compile, link, etc. –Would block if speculation limited to one task Allow kernel objects to have speculative state –Examples: inodes, signals, pipes, Unix sockets, etc. –Propagate dependencies among objects –Objects rolled back to prior states when specs fail

13 Spec 1 Multi-Process Speculation Spec 2 pid 8001 Checkpoint inode 3456 Chown -1 Write -1 pid 8000 Checkpoint Chown -1 Write -1

14 Multi-Process Speculation What we handle: –DFS objects, RAMFS, Ext3, Pipes & FIFOs –Unix Sockets, Signals, Fork & Exit What we don’t (i.e. we block) –System V IPC –Multi-process write-shared memory

15 Outline Motivation Implementing speculation Multi-process speculation Using Speculator Evaluation

16 Example: NFSv3 Linux Client 1Client 2Server Open B Getattr Modify B Write Commit

17 Example: SpecNFS Modify B speculate Getattr Open B speculate Open B Getattr speculate Write+Commit Client 1Client 2Server

18 Problem: Mutating Operations bar depends on cat foo What does client 2 view in bar? Client 1 1. cat foo > bar Client 2 2. cat bar

19 Solution: Mutating Operations Server determines speculation success/failure –State at server never speculative Send server hypothesis speculation based on –List of speculations an operation depends on Requires server to track failed speculations Requires in-order processing of messages

20 Group Commit Previously sequential ops now concurrent Sync ops usually committed to disk Speculator makes group commit possible write commit Client Server

21 Putting it all Together: SpecNFS Apply Speculator to an existing file system Modified NFSv3 in Linux 2.4 kernel –Same RPCs issued (but many now asynchronous) –SpecNFS has same consistency, safety as NFS –Getattr, lookup, access speculate if data in cache –Create, mkdir, commit, etc. always speculate

22 Putting it all Together: BlueFS Design a new file system for Speculator –Single copy semantics –Synchronous I/O Each file, directory, etc. has version number –Incremented on each mutating op (e.g. on write) –Checked prior to all operations. –Many ops speculate and check version async

23 Outline Motivation Implementing speculation Multi-process speculation Using Speculator Evaluation

24 Apache Benchmark SpecNFS up to 14 times faster

25 The Cost of Rollback All files out of date SpecNFS up to 11x faster

26 Conclusion Speculator greatly improves performance of existing distributed file systems Speculator enables new file systems to be safe, consistent and fast

27 Group Commit & Sharing State

28 Apache Benchmark

29 Related Work Chang & Gibson, Fraser & Chang –Speculative pre-fetching Time Warp –Virtual Time: distributed simulations Hardware branch prediction Transactional file systems