Rebound: Scalable Checkpointing for Coherent Shared Memory Rishi Agarwal, Pranav Garg, and Josep Torrellas Department of Computer Science University of Illinois at Urbana-Champaign http://iacoma.cs.uiuc.edu

Checkpointing in Shared-Memory MPs HW-based schemes for small CMPs use Global checkpointing All procs participate in system-wide checkpoints Global checkpointing is not scalable Synchronization, bursty movement of data, loss in rollback… rollback save chkpt Fault checkpoint P1 P2 P3 P4

Alternative: Coordinated Local Checkpointing Idea: threads coordinate their checkpointing in groups Rationale: Faults propagate only through communication Interleaving between non-comm. threads is irrelevant P1 P2 P3 P4 P5 P1 P2 P3 P4 P5 Local Chkpt GlobalChkpt Local Chkpt + Scalable: Checkpoint and rollback in processor groups Complexity: Record inter-thread dependences dynamically.

Contributions Rebound: First HW-based scheme for scalable, coordinated local checkpointing in coherent shared-memory Leverages directory protocol to track inter-thread deps. Opts to boost checkpointing efficiency: Delaying write-back of data to safe memory at checkpoints Supporting multiple checkpoints Optimizing checkpointing at barrier synchronization Avg. performance overhead for 64 procs: 2% Compared to 15% for global checkpointing

Background: In-Memory Checkpt with ReVive [Prvulovic-02] Execution Register Dump P1 P2 P3 CHK Displacement Caches Dirty Cache lines Writebacks W W W W WB Checkpoint Application Stalls Writeback old old Logging old Log Memory

Background: In-Memory Checkpt with ReVive [Pvrulovic-02] Old Register restored P1 P2 P3 CHK W WB Fault Caches Cache Invalidated Memory Lines Reverted Log Memory Global Broadcast protocol Local Coordinated Scalable protocol

Coordinated Local Checkpointing Rules wr x rd x P1 P2 Producer rollback Consumer P1 P2 Producer chkpoint Consumer P1 P2 chkpt P checkpoints  P’s producers checkpoint P rolls back  P’s consumers rollback Banatre et al. used Coordinated Local checkpointing for bus-based machines [Banatre96]

Rebound Fault Model Log (in SW) Main Memory Chip Multiprocessor Any part of the chip can suffer transient or permanent faults. A fault can occur even during checkpointing Off-chip memory and logs suffer no fault on their own (e.g. NVM) Fault detection outside our scope: Fault detection latency has upper-bound of L cycles Redo fig

Rebound Architecture Main Memory Chip Multiprocessor LW-ID MyProducer Directory Cache LW-ID MyProducer MyConsumer Dep Register P+L1 Redo fig

Rebound Architecture Main Memory Chip Multiprocessor L2 Directory Cache LW-ID MyProducer MyConsumer Dep Register P+L1 Dependence (Dep) registers in the L2 cache controller: MyProducers : bitmap of proc. that produced data consumed by the local proc. MyConsumers : bitmap of proc. that consumed data produced by the local proc. Redo fig

Rebound Architecture Main Memory Chip Multiprocessor L2 Directory Cache LW-ID MyProducer MyConsumer Dep Register P+L1 Dependence (Dep) registers in the L2 cache controller: MyProducers : bitmap of proc. that produced data consumed by the local proc. MyConsumers : bitmap of proc. that consumed data produced by the local proc. Processor ID in each directory entry: LW-ID : last writer to the line in the current checkpoint interval. Redo fig

Recording Inter-Thread Dependences P1 writes MyProducers MyProducers MyConsumers MyConsumers Write LW-ID P1 D Log Memory Assume MESI protocol

Recording Inter-Thread Dependences MyConsumers  P2 P2 MyProducers P2 reads MyProducers P1 P2 MyConsumers MyConsumers MyProducers  P1 LW-ID P1 D S Write back Logging Log Memory Assume MESI protocol

Recording Inter-Thread Dependences P1 writes MyProducers MyProducers P1 P2 MyConsumers MyConsumers LW-ID P1 S P1 D Log Memory Assume MESI protocol

Recording Inter-Thread Dependences Clear Dep registers P2 P1 checkpoints MyProducers MyProducers P1 P2 MyConsumers MyConsumers Clear LW-ID LW-ID LW-ID should remain set till the line is checkpointed P1 S Writebacks P1 D Logging Log Memory Assume MESI protocol

Lazily clearing Last Writers Clear LW-IDs  Expensive process ! Write Signature encodes all line addresses that the processor has written to (or read exclusively) in the current interval. At checkpoint, the processors clear their Write Signature Potentially stale LW-ID

Lazily clearing Last Writers P1 P2 NO ! MyProducers P2 reads MyProducers WSig MyConsumers MyConsumers Addr ? Clear LW-ID Stale LW-ID P1 S Log Memory

Distributed Checkpointing Protocol in SW Interaction Set [Pi]: set of producer processors (transitively) for Pi Built using MyProducers P1 P2 P3 P4 chk InteractionSet : P1 P1 initiate checkpoint

Distributed Checkpointing Protocol in SW Interaction Set [Pi]: set of producer processors (transitively) for Pi Built using MyProducers P1 P2 P3 P4 chk InteractionSet : P1 P1 Ck? Ck? P2 P3 initiate checkpoint

Distributed Checkpointing Protocol in SW Interaction Set [Pi]: set of producer processors (transitively) for Pi Built using MyProducers P1 P2 P3 P4 chk InteractionSet : P1 , P2, P3 P1 Ck? Ck? P2 P3 initiate checkpoint

Distributed Checkpointing Protocol in SW Interaction Set [Pi]: set of producer processors (transitively) for Pi Built using MyProducers P1 P2 P3 P4 chk InteractionSet : P1 , P2, P3 P1 Accept Accept Ck? Ck? P2 P3 Ck? initiate checkpoint P4

Distributed Checkpointing Protocol in SW Interaction Set [Pi]: set of producer processors (transitively) for Pi Built using MyProducers P1 P2 P3 P4 chk InteractionSet : P1 , P2, P3 P1 Accept Accept Ck? Ck? P2 P3 Ack Ck? Decline initiate checkpoint P4

Distributed Checkpointing Protocol in SW Interaction Set [Pi]: set of producer processors (transitively) for Pi Built using MyProducers P1 P2 P3 P4 chk InteractionSet : P1 , P2, P3 P1 Accept Accept Ck? Ck? P2 P3 Ack Ck? Decline initiate checkpoint P4 Checkpointing is a 2-phase commit protocol.

Distributed Rollback Protocol in SW Rollback handled similar to the Checkpointing protocol: - Interaction set is built transitively using MyConsumers Rollback involves Clearing the Dep. Registers and Write Signature Invalidating the processor caches Restoring the data and register context from the logs up to the latest checkpoint. No Domino Effect

Optimization1 : Delayed Writebacks Time sync Checkpoint Interval I2 sync WB dirty lines Checkpoint Interval I1 Interval I2 Stall Interval I1 Stall WB dirty lines Checkpointing overhead dominated by data writebacks Delayed Writeback optimization Processors synchronize and resume execution Hardware automatically writes back dirty lines in background Checkpoint only completed when all delayed data written back Still need to record inter-thread dependences on delayed data

Delayed Writeback Pros/Cons + Significant reduction in checkpoint overhead - Additional support: Each processor has two sets of Dep. Registers and Write Signature Each cache line has a delayed bit Increased vulnerability A rollback event forces both intervals to roll back

Delayed Writeback protocol MyConsumers0  P2 P1 P2 YES ! MyProducers0 MyProducers0 WSig0 xxx P2 MyConsumers0 MyConsumers0 Addr ? MyProducers1 P2 reads MyProducers1 P1 WSig1 NO ! MyConsumers1 MyConsumers1 Addr ? MyProducers1  P1 LW-ID P1 D S Write back Logging Log Memory

Optimization2 : Multiple Checkpoints Problem: Fault detection is not instantaneous Checkpoint is safe only after max fault-detection latency (L) Fault Detection Latency Dep registers 1 Dep registers 2 Rollback Ckpt 1 Ckpt 2 tf Solution: Keep multiple checkpoints On fault, roll back interacting processors to safe checkpoints No Domino Effect

Multiple Checkpoints: Pros/Cons + Realistic system: supports non-instantaneous fault detection - Additional support: Each checkpoint has Dep registers Dep registers can be recycled only after fault detection latency - Need to track communication across checkpoints - Combination with Delayed Writebacks: one more Dep register set

Optimization3 : Hiding Chkpt behind Global Barrier Global barriers require that all processors communicate Leads to global checkpoints Optimization: Proactively trigger a global checkpoint at a global barrier Hide checkpoint overhead behind barrier imbalance spins

Hiding Checkpoint behind Global Barrier Lock count++ if(count == numProc) Iam_last = TRUE /*local var*/ Unlock If(I am_last) { count = 0 flag = TRUE … } else while(!flag) {} Update

Hiding Checkpoint behind Global Barrier Lock count++ if(count == numProc) Iam_last = TRUE /*local var*/ Unlock If(I am_last) { count = 0 flag = TRUE … } else while(!flag) {} Update Processor P1 Processor P2 Processor P3 BarCK? Notify flag = TRUE ICHK = {P3} while(!flag) ICHK = {P2, P3} ICHK = {P1, P3} Update First arriving processor initiates the checkpoint Others: HW writes back data as execution proceeds to barrier Commit checkpoint as last processor arrives After the barrier: few interacting processors

Evaluation Setup Analysis tool using Pin + SESC cycle-acc. simulator + DRAMsim Applications: SPLASH-2 , some PARSEC, Apache Simulated CMP architecture with up to 64 threads Checkpoint interval : 5 – 8 ms Modeled several environments: Global: baseline global checkpointing Rebound: Local checkpointing scheme with delayed writeback. Rebound_NoDWB: Rebound without the delayed writebacks.

Avg. Interaction Set: Set of Producer Processors Most apps: interaction set is a small set Justifies coordinated local checkpointing Averages brought up by global barriers 64 38

Checkpoint Execution Overhead Rebound’s avg checkpoint execution overhead is 2% Compared to 15% for Global 15 2

Checkpoint Execution Overhead Rebound’s avg checkpoint execution overhead is 2% Compared to 15% for Global Delayed Writebacks complement local checkpointing

Rebound Scalability Constant problem size Rebound is scalable in checkpoint overhead Delayed Writebacks help scalability

Also in the Paper Delayed write backs also useful in Global Barrier optimization is effective but not universally applicable Power increase due to hardware additions < 2% Rebound leads to only 4% increase in coherence traffic

Conclusions Rebound: First HW-based scheme for scalable, coordinated local checkpointing in coherent shared-memory Leverages directory protocol Boosts checkpointing efficiency: Delayed write-backs Multiple checkpoints Barrier optimization Avg. execution overhead for 64 procs: 2% Future work: Apply Rebound to non-hardware coherent machines Scalability to hierarchical directories

Rebound: Scalable Checkpointing for Coherent Shared Memory Rishi Agarwal, Pranav Garg, and Josep Torrellas Department of Computer Science University of Illinois at Urbana-Champaign http://iacoma.cs.uiuc.edu