An Integrated Framework for Dependable and Revivable Architecture Using Multicore Processors Weidong ShiMotorola Labs Hsien-Hsin “Sean” LeeGeorgia Tech Laura FalkUniversity of Michigan Mrinmoy GhoshGeorgia Tech

2 Problem Statement Highly Available, Reliable, and Revivable networked services. Explore new programming and usage models for Multi- core processors Provide “architectural support” for network services to be –Autonomic –Remote-exploits revivable –Self-recoverable Achieve high performance

3 Problem Statement Highly Available, Reliable, and Revivable networked services. Explore new programming and usage models for Multi-core processors Provide “architectural support” for network services to be –Autonomic –Remote-exploits revivable –Self-recoverable Achieve high performance

4 Toward Self-recovery Network Services Causes of Network Service Loss Accidental TransientHeisenbugsDamage Aging Intentional DoS Buffer Overflow Solutions Replication Rejuvenation Checkpoint Remote Exploit Self- recovery

5 Multicore: An ideal platform Exploit insulation: Each core of a multicore can be programmed to run at different privilege levels with different OS. Dual Core (Merome) Server Core Monitor Core Shared L2 Tight coupling of cores comparing with SMP Fine-grained processor state monitoring Concurrent monitoring, efficient state backup and recovery Massive multi-core will have many idle cores

6 INDRA: A Dependable and Revivable Architecture Monitor Core L2 Cache IL1 Cache IL1 Cache DL1 Cache DL1 Cache Monitor Insulation Issue Recovery Control Memory Interface Watch Dog Memory Interface Watch Dog Physical Memory Space (used by service OS and applications) Protected Memory Space (monitor BIOS, OS, and SW) Server Core (Network Apps) Server Core (Network Apps) IL1 Cache IL1 Cache DL1 Cache DL1 Cache Trace Filter Trace Filter Trace FIFO Trace FIFO Code origin check CFG check Control signals

7 Data Page Code Page Monitor Core: Insulated Parallel Inspection [Kiriansky et al., USENIX 2002] Vuln_func() { // Attack!! // Return address changed } FunctionA() { Vuln_func(); A =3; } Malicious_func() { } Code Page Code Origin Check Control Flow Graph Check Exception Handling

8 Server Core: Request Based Recovery Issue state backup request Issue state backup request Read network request (Request for page arch.ece.gatech.edu) Read network request (Request for page arch.ece.gatech.edu) Process network request Monitor Signalled Error? NoYes Restore Checkpointed State Restore Checkpointed State

9 Comparison of Backup and Recovery BackupRecovery Approach Software checkpointing Slow Fast, modify page translation Memory Update Log Fast Log based undo slow Virtual Checkpointing Copy dirty page on demand, slow Fast, modify TLB entry INDRA Fast, no page copy

10 INDRA Backup Page Record Active Page Modified TLB Global Timestamp Register (GT) GT=4 Backup Page TLB Extension for Backup and Rollback Dirty Block Bitvector Backup Page (Physical Address) Rollback Bitvector Rollback Valid Local Timestamp Active Page (Physical Address) Tag Dirty Block Bitvector Backup Page (Physical Address) Local Timestamp Rollback Bitvector Rollback Valid 3 Processor Memory

11 INDRA Backup Page Record Active Page Modified TLB Global Timestamp Register (GT) GT=4 Backup Page TLB Extension for Backup and Rollback Backup Page Record Processor Memory Dirty Block Bitvector Backup Page (Physical Address) Local Timestamp Rollback Bitvector 3 Dirty Block Bitvector Backup Page (Physical Address) Rollback Bitvector Backp Record Rollback Valid Local Timestamp Active Page (Physical Address) Tag Rollback Valid 3

12 INDRA Recovery Example Active Page Global Timestamp Register (GT) GT=5 Backup Page Modified TLB TLB Extension for Backup and Rollback 3 Dirty Block Bitvector Backup Page (Physical Address) Rollback Bitvector Backup Record Rollback Valid Local Timestamp Active Page (Physical Address) Tag Current Operation Wr memory line 7 REQUEST n 5

13 INDRA Recovery Example Active Page Global Timestamp Register (GT) GT=5 Backup Page Modified TLB TLB Extension for Backup and Rollback 3 Dirty Block Bitvector Backup Page (Physical Address) Rollback Bitvector Backup Record Rollback Valid Local Timestamp Active Page (Physical Address) Tag Current Operation REQUEST n 5 Wr memory line 2

14 INDRA Recovery Example Active Page Global Timestamp Register (GT) GT=5 Backup Page Modified TLB TLB Extension for Backup and Rollback 3 Dirty Block Bitvector Backup Page (Physical Address) Rollback Bitvector Backup Record Rollback Valid Local Timestamp Active Page (Physical Address) Tag REQUEST n 5 Failure Signal Restore system resource allocation Restore process context 1

15 INDRA Recovery Example Active Page Global Timestamp Register (GT) GT=5 Backup Page Modified TLB TLB Extension for Backup and Rollback 3 Dirty Block Bitvector Backup Page (Physical Address) Rollback Bitvector Backup Record Rollback Valid 1 Local Timestamp Active Page (Physical Address) Tag REQUEST n+1 5 Current Operation Rd memory line 7

16 INDRA Recovery Example Active Page Global Timestamp Register (GT) GT=5 Backup Page Modified TLB TLB Extension for Backup and Rollback 3 Dirty Block Bitvector Backup Page (Physical Address) Rollback Bitvector Backup Record Rollback Valid 1 Local Timestamp Active Page (Physical Address) Tag REQUEST n+1 5 Current Operation Wr memory line 1

17 INDRA Recovery Example Active Page Global Timestamp Register (GT) GT=5 Backup Page Modified TLB TLB Extension for Backup and Rollback 3 Dirty Block Bitvector Backup Page (Physical Address) Rollback Bitvector Backup Record Rollback Valid 1 Local Timestamp Active Page (Physical Address) Tag REQUEST n+1 5 Current Operation Handle Next Request Global Timestamp Register (GT) GT=6 Record system resource allocation Record process context

18 INDRA Recovery Example Active Page Global Timestamp Register (GT) GT=5 Backup Page Modified TLB TLB Extension for Backup and Rollback 3 Dirty Block Bitvector Backup Page (Physical Address) Rollback Bitvector Backup Record Rollback Valid 1 Local Timestamp Active Page (Physical Address) Tag REQUEST n+2 5 Current Operation Global Timestamp Register (GT) GT=6 Wr memory line 4 6

19 Test Bed (Bochs + TAXI [Vlaovic & Davidson, ICCD’02] ) Monitor (Stripped Down OS, Security SW, 10MB) Monitor (Stripped Down OS, Security SW, 10MB) Linux Network Server Linux Network Server Bochs + TAXI Host OS Network Requests Server Response Run production OS with real service applications, httpd, ftpd, bind, sendmail, etc. Recoverability evaluated by applying real x86 remote exploits from security websites. Experiment with documented exploits

20 Inter-Request Interval (# of Instructions)

21 I-Cache Miss Rate Code Origin Check reads traces of code read from L2 Cache Number of Instructions in the Trace is Proportional to L1 I Cache Miss Rate Overhead of monitoring code origin depends on L1 I Cache Miss Rate

22 Monitoring Overhead

23 Sensitivity of Monitoring Queue Size Queue Size Queue Size vs. Performance Slowdown

24 Backup Overhead of Modified Lines

25 Performance of Recovery + Monitoring

26 Conclusions Real time exploit monitoring with autonomic recovery increases revivability and availability. Multicore architectures are an ideal candidate for new type of revivable system. INDRA-based Multicore system can provide improved reliability and availability. More research is required to explore the trade-off between availability, performance, architecture design, and cost.

27 Questions and Answers Thank you !