1. University of Wisconsin-Madison Presented by: Nick Kirchem
SafetyNet: Improving the Availability of Shared Memory Multiprocessors with Global Checkpoint/Recovery
Daniel J. Sorin, et al.
University of Wisconsin-Madison
Presented by: Nick Kirchem
March 26, 2004

2. Motivation and Goals Availability is crucial Goals for paper
Internet services and database management systems highly relied upon
Unless architecture is changed, availability will decrease as # of components increases
Goals for paper
Lightweight mechanism providing E2E recovery for transient and permanent faults
Decouple recovery from detection (use traditional detection techniques: RAID, ECC, duplicate ALUs, etc)

3. Solution SafetyNet A global checkpoint/recovery scheme
Creates periodic system-wide logical checkpoints
Log all changes to the architected state

4. Recovery Scheme Challenges
(1) Saving previous values before every register/ cache update or coherence message would require too much storage space
(2) All processors and components must recover to a consistent point
(3) SafetyNet must determine when it is safe to roll-back to a recovery checkpoint

5. (1) Checkpointing Via Logging
Checkpoints contain a complete copy of system’s architectural state
Taken at coarse granularity (e.g. 100,000 cycles)
Log is only taken on first altering action per checkpoint interval.

6. (2) Consistent Checkpoints
All components coordinate local checkpoints through logical time
Coherence transaction appears logically atomic once it completes
Point of atomicity
When previous owner processes request
Response includes CN of this PoA
Requestor does not advance recovery point until all outstanding transactions are complete

7. (2) PoA Example

8. (3) Validating Checkpoints
States: current state, checkpoints waiting to be validated, recovery point
Validation: determining which checkpoint is the recovery point
All prior execution must be fault-free
Coordination is pipelined and performed in background (off critical path)

9. (3) Validation continued
Validation latency depends on fault detection latency
Output commit problem
Delay all output events until checkpoint is validated
Depends on validation latency
Input Commit problem
Log incoming messages

10. Recovery Processors restore their register checkpoints
Caches and memories unroll local logs
State from coherence transactions in progress is discarded
Reconfiguration if necessary

11. Implementation

12. Implementation (Cont’d)
Checkpoint Log Buffers (CLBs)
Associated with each cache and memory component
Store log state
Shadow registers
2D torus
MOSI Directory protocol

13. Logical Time Base
Loosely synchronous checkpoint clock distributed redundantly
Ensures no single point of failure
Edge of clock increments current checkpoint number (CCN)
Works as long as skew < minimum communication time between nodes
Assigning transaction  checkpoint interval is protocol-dependent

14. Logging
Memory block written to CLB whenever action might have to be undone
CLBs are write-only (except during recovery) and off critical path
CN added to each block in cache
Steps taken for update-action
CCN compared with block’s CN
Block is logged if CCN >= CN
Updates block’s CN to CCN+1
Performs the update action
Updated CN sent with coherence response

15. Checkpoint Creation/Validation
Choose suitable checkpoint clock freq
Detection latency tolerance
Total CLB storage
Lost messages (timeout) trigger recovery
Recovery point checkpoint number (RPCN) broadcasted when recovery point is advanced
After fault, recovery msg sent (includes RPCN)
Interconnection network is drained
Processors, cache, memories recover to RPCN

16. Implementation Summary
Processor/Cache Changes Required
Processor must be able to checkpoint its register state
Must be able to copy old versions out of cache before transferring them
CNs added to L1 cache block
Directory Protocol Changes
CNs added to data response messages
Coherence requests can be nack’ed
Final ack required from requestor to directory

17. Evaluation Parameters
16-processor target system
Simics + memory hierarchy simulator
4 commercial workloads, 1 scientific
In order processor, 4 billion instr/sec
MOSI directory protocol with 2D torus
Checkpoint interval = 100,000 cycles

18. Experiments (1) Fault-free performance (2) Dropped messages
Overhead determined to be negligible
(2) Dropped messages
Periodic transient faults injected (10/sec)
Recovery latency << crash + reboot
(3) Lost Switch
Hard fault – kill half-switch
Crash avoided – performance suffers due to restricted bandwidth

19. Sensitivity Analysis Cache bandwidth Storage cost
Depends on frequency of stores requiring logging (additional b/w consumed reading old copy of the block)
Cache ownership transfer: no additional b/w
Storage cost
CLBs sized to avoid performance degradation due to full buffers
Entries per checkpoint corresponds to logging frequency

20. Conclusion
SafetyNet is efficient in common case (error free execution) – little to no latency added
Latency is hidden by pipelining validation of checkpoints
Checkpoints coordinated in logical time (no synch exchanges necessary)

21. Questions What about faults/errors with the saved state itself?
What if you there’s a permanent fault for which you can’t reconfigure (endless loop of recovering to last checkpoint?)