Speculative Execution in a Distributed File System Ed Nightingale Peter Chen Jason Flinn University of Michigan Best Paper at SOSP 2005 Modified for CS739

2 Motivation Why are distributed file systems slow(er)? –Sync network messages provide consistency –Sync disk writes provide safety Sacrifice guarantees for speed Can DFS can be safe, consistent and fast? –Yes! With OS support for speculative execution

3 Big Idea: Slow Way RPC Req Client RPC Resp Guarantees without blocking I/O! Server Block!2) Speculate! 1) Checkpoint Big Idea: Speculator 3) Correct? Yes: discard ckpt.No: restore process & re-execute RPC Req RPC Resp RPC Req RPC Resp

4 Conditions for Success Operations are highly predictable –Conflicts are rare Checkpoints are cheaper than network I/O –52 µs for small process (6.3 ms for 64 MB process) Computers have resources to spare –Need memory and CPU cycles for speculation

5 Assumptions --> Experiments Operations are highly predictable –What happens if not (lots of sharing)? –Will cost of checkpointing and rolling back ever harm performance? Checkpoints are cheaper than network I/O –What happens with large memory processes? Computers have resources to spare (Clients) –What happens when running multiple processes? –Can network become bottleneck? –Can server become bottleneck?

6 Outline Motivation Implementing speculation Multi-process speculation Using Speculator Evaluation

7 Implementing Speculation Implementation –All within OS –No changes needed to applications –Three new interfaces Create_speculation() Commit_speculation() Fail_speculation() Goal: Ensure speculative state is Never externalized (sent to terminal, disk, network) Never directly observed by non-speculative processes

8 Undo log For each kernel object Implementing Speculation Process 1) System call2) Create speculation Time Spec Tracks objects that depend on speculation Checkpoint Copy-on-write fork

9 Speculation Success Undo log Checkpoint 1) System call2) Create speculation Process 3) Commit speculation Time Spec

10 Speculation Failure Undo log Checkpoint 1) System call 2) Create speculation Process 3) Fail speculation Process Time Spec

11 Replay after Failure Does code after checkpoint need to be deterministic for correct results?

12 Ensuring Correctness Spec processes often affect external state –Speculative state should never be visible to user or any external device –Process should never view speculative state unless it is already speculatively dependent on that state Three ways to ensure correct execution –Block –Buffer –Propagate speculations (dependencies)

13 Systems Calls Modify system call jump table Block calls that externalize state –Allow read-only calls (e.g. getpid) –Allow calls that modify only task state (e.g. dup2) File system calls -- need to dig deeper –Mark file systems that support Speculator getpid reboot mkdir Call sys_getpid() Block until specs resolved Allow only if fs supports Speculator

14 Output Commits: Buffer “stat worked” “mkdir worked” Undo log Checkpoint Spec (stat) Spec (mkdir) 1) sys_stat2) sys_mkdir Process Time 3) Commit spec 1

15 Multi-Process Speculation Processes often cooperate –Example: “make” forks children to compile, link, etc. –Would block if speculation limited to one task Allow kernel objects to have speculative state –Examples: inodes, signals, pipes, Unix sockets, etc. –Propagate dependencies among objects –Objects rolled back to prior states when specs fail

16 Spec 1 Multi-Process Speculation Spec 2 pid 8001 Checkpoint inode 3456 Chown -1 Write -1 pid 8000 Checkpoint Chown -1 Write -1 Stat AStat B

17 Multi-Process Speculation What we handle: –DFS objects, RAMFS, Ext3, Pipes & FIFOs –Unix Sockets, Signals, Fork & Exit What we don’t (i.e. we block) –System V IPC –Multi-process write-shared memory

18 Outline Motivation Implementing speculation Multi-process speculation Using Speculator Evaluation

19 Example: NFSv3 Linux Client 1Client 2Server Open B Getattr Modify B Write Commit Why were asynchronous writes added to NFSv3? Is this situation of no overlap expected to be the common case? When can the commit return from the server?

20 Example: SpecNFS Modify B speculate Getattr Open B speculate Open B Getattr speculate Write+Commit Client 1Client 2Server When can the commit return from the server?

21 Problem: Mutating Operations bar depends on cat foo What does client 2 view in bar? Client 1 1. cat foo > bar Client 2 2. cat bar

22 Solution: Mutating Operations Server determines speculation success/failure –State at server never speculative Send server hypothesis speculation based on –List of speculations an operation depends on Requires server to track failed speculations –Would like to be convinced OK when server crashes! Requires in-order processing of messages

23 Group Commit Previously sequential ops now concurrent Sync ops usually committed to disk Speculator makes group commit possible write commit Client Server Updating different files… Can significantly improve disk throughput

24 Putting it all Together: SpecNFS Apply Speculator to an existing file system Modified NFSv3 in Linux 2.4 kernel –Same RPCs issued (but many now asynchronous) –SpecNFS has same consistency, safety as NFS –Getattr, lookup, access speculate if data in cache –Create, mkdir, commit, etc. always speculate Choose aliases for unknown file handles

25 Putting it all Together: BlueFS Design a new file system for Speculator –Single copy semantics –Synchronous I/O Each file, directory, etc. has version number –Incremented on each mutating op (e.g. on write) –Checked prior to all operations. –Many ops speculate and check version async

26 Outline Motivation Implementing speculation Multi-process speculation Using Speculator Evaluation

27 Apache Benchmark SpecNFS up to 14 times faster

28 Apache Benchmark

29 The Cost of Rollback (Q1) All files out of date SpecNFS up to 11x faster

30 Evaluation Other experiments you would like to see?

31 Evaluation Other experiments you would like to see? Impact of checkpoint size Resource constraints: Scalability –How many clients can server handle? –Is network a constraint? –Server must still handle poll requests, even if in bg Fault tolerance –Server now maintains state: per-client list of failed speculations

32 Conclusion Speculator greatly improves performance of existing distributed file systems –Especially in wide-area Speculator enables new file systems to be safe, consistent and fast How general do you think Speculator is? –Speculation clearly improves poll-based consistency (NFSv3, BlueFS) –How much improvement with callbacks and leases? (AFS, Coda, NFSv4)?

33 Group Commit & Sharing State

34 Related Work Chang & Gibson, Fraser & Chang –Speculative pre-fetching Time Warp –Virtual Time: distributed simulations Hardware branch prediction Transactional file systems