Reading Group, July Based on slides retrieved from SHARK: Architectural Support for Autonomic Protection Against Stealth by Rootkit Exploits Vikas R. Vasisht, Hsien-Hsin S. Lee School of Electrical and Computer Engineering Georgia Tech Computer Architecture Lab at

Rootkits Programs used to conceal malware presence – E.g. hide keylogger or network activity Hard to detect – Anti-rootkit tool executes inside compromised system that tries to hide the intrusion 2

Simple Rootkit 3 API Function User Program Library System Administrator (e.g., “ps”, “top”) Choose Interrupt Handler from IDT Choose Syscall from SSDT Syscall Function Return Import Address Table Interrupt Descriptor TableSystem Service Descriptor Table User Space Kernel Space Malware removed from list of running processes

Existing Anti-Rootkit Tools Signature-Based Only works with known rootkits Behavior-Based – Calculate time and instructions spent on system queries Unusually high count indicates the O/S is manipulating response Too many false positives Cross-level system view – Query system in high and low level E.g., compare ps response against scheduler lists The compromised O/S can manipulate all levels 4

Sophisticated (Anti-)Rootkits Integrity-based detection – Compare snapshot of memory against trusted baseline Can be fooled Shadow-Walker – Mark all pages as unavailable – On TLB-i (execute) page fault: present malicious code – On TLB-d (snapshot) page fault: return ‘correct’ page 5 App Page Table TLB-i View TLB-d View AUDIT OK!

Even More Sophisticated Rootkits 6 M2 M1 Mal O/S App2 App1 Host O/S Hardware App2 App1 Host O/S Virtual Machine Monitor Hardware SubVirt – Install virtual machine monitor – Downgrade host O/S to guest O/S – Run Malicious O/S on the side Blue Pill – Install tiny malicious hypervisor on-the-fly

Outline Motivation – Rootkits SHARK – High-level View – Page Table Encryption, Update, Decryption – Address Space Protection – Stealth Checker Evaluation Security Vulnerabilities Conclusion 7

Problem Nearly impossible to detect all rootkits – Especially the virtualization-based ones: guest O/S is clean – All tools depend on the malicious O/S Proposed solution – Have the processor securely report every application that enjoys hardware resources – Malicious apps running in different VMM will be exposed 8

SHARK: Bird’s Eye View Hardware should record running processes – Must force O/S to give real PID to CPU Trusted third party audits system – Compare H/W and O/S reports – Discrepancy raises alarm O/S cannot hide a malicious app Tamper-evident address space – O/S blocked from manipulating normal apps Malware hijacking an app will leave a trail Idea: Page Table Encryption 9

Page Table Encryption: Key App’s page table is encrypted using PID – Decrypted on-the-fly during hardware walk using same PID O/S assigned PIDs are vulnerable – Frequent reuse problematic for security SHARK provides hardware PID (HPID) – 64-bit counter: won’t overflow, nonce Only care for memory-resident rootkits, reset will eliminate them During app creation – O/S request PID and provides an initialized page table – H/W encrypts page table and returns HPID++ 10

Page Table Encryption (cont) Encrypt last level page table entries (counter mode) Encrypt first & last level valid bits (counter mode) – Force same PID to be used during walk every time All updates through special H/W instruction – MODPT (modify page table) – Must provide correct PID Otherwise meaningless 1 st level decryption 11

V V Translation Array Valid Bit Array Encryption 12 V V 1 1 Translation Array 0 0 … … … … … … … … 0 0 Page Directory 1 Page Frame 1 st 128-bit Valid Bit Block Counter = HPID||HPID 2 nd 128-bit Valid Bit Block Counter = (HPID||HPID)+1 N 128-bit Valid Bit Block Counter = (HPID||HPID)+N-1 Counter Base (128 bit) HPID||HPID Pipelined Counter Mode Encryption Engine AES-128 Hardware Secret Key: Burnt-in, unexposable Offset Actual Frame stored in memory

Page Table: Initial Encryption 13 V V Translation Array … … … … Last Level Page Directory 1 Page Frame ADDR NULL ADDR Counter (128 bit) HPID||HPID Pipelined Counter Mode Encryption Engine AES-128 Hardware Secret Key: Burnt-in, unexposable 4 entries concatenated 128-bit V V Translation Array ADDR Actual Frame stored in memory

Page Table: Entry Update 14 If other valid entries, decrypt and reencrypt, otherwise just encrypt

Counter (128 bit) (HPID||HPID)+k V V Page Table Decryption – TLB update 15 V V PDE Counter Mode Decryption AES-128 Page Directory HW Key … … … … Normal Page Table Walk V V PDE ADDR Last Level Counter (128 bit) HPID||HPID Counter Mode Decryption AES-128 HW Key V V … … … … PDE ADDR 4 entries encrypted 128-bit Counter (128 bit) HPID||HPID Counter Mode Decryption AES-128 HW Key 4 entries decrypted 128-bit TLB update

Address Space Protection O/S should not tamper with address space of apps – Protect applications from swap-hijacking – Expose malware during auditing Shouldn’t mark as unavailable and return convenient page H/W signs pages and checks signatures 16 Tamper-evident swapping Caution: No check during first page mapping Mem Page 4KB 32B Disk Counter (128 bit) HPID||HPID Counter Mode Encryption AES-128 HW Key SHA-256 checksum Mem Page 4KB 32B Disk SHA-256 checksum Counter Mode Encryption AES-128 =?=? Valid/Invalid update

Stealth Checking Firmware called on context switch – Record incoming PID of new process – Every 10 switches, timestamp, encrypt and call O/S to ship 17 Tamper-evident timestamped packet. Lost packets or O/S tampering trigger alarm. OS Context Switch Hardware HPID PID1 PID2 PID3 Master PID List Other SHARK H/W P P I I D D s s … … Alarm

Outline Motivation – Rootkits SHARK – High-level View – Page Table Encryption, Update, Decryption – Address Space Protection – Stealth Checker Evaluation Security Vulnerabilities Conclusion 18

Evaluation Emulation using Bochs – All installed rootkits detected Timing results using Simics – Close match to Core 2 processors – In-order simulation – First 2 billion instructions of SPEC apps Most page faults (and overhead) during initialization 19

Expected: More TLB entries, less overhead Evaluation (cont) 20

Outline Motivation – Rootkits SHARK – High-level View – Page Table Encryption, Update, Decryption – Address Space Protection – Stealth Checker Evaluation Security Vulnerabilities Conclusion 21

Security Hole #1 22 1st page map cannot be checked – Possible exploit by cleaner-app? Mali- cious App’s Page Table Name: xcalc Mali- cious App’s Page Table Name: xcalc Audit app launched: Pagefault Write Access Note: cleaner-app will leave a PID trace in the stealth checker All pages marked as invalid, app works through TLBs Present malicious signature & restore mapping, but freeze auditing

Security Hole #2 All page tables but the last (completely) and first (valid bits only) are plaintext – O/S can manipulate internal mappings 23 VALID View during execution View during auditing Level 1 Level 2 Level 3Data Pages

Security Hole #3 Security 101 – Encryption: to keep something secret, private – Signing: to make something unmodifiable, tamper-evident and bind message pieces together BAD idea to use one for the other – Creates illusion of protection Why encrypt the page table? – O/S already knows it, so why keep it a secret? – Signing is necessary, encryption is useless – Paper based on assumption that any tampering will result in meaningless page table 24

Security Hole #3: Exploit 25 Counter H/W Key 128-bit plaintext AES 128 ⊕ 128-bit Ciphertext Counter H/W Key 128-bit plaintext AES 128 ⊕ 128-bit Ciphertext Counter Mode EncryptionCounter Mode Decryption D(E(A)) = E(A) ⊕ Cloud = A ⊕ Cloud ⊕ Cloud = A O/S can overwrite E(A) with: (E(A), A, B known to O/S) E(A) ⊕ A ⊕ B = A ⊕ Cloud ⊕ A ⊕ B = B ⊕ Cloud = E(B) H/W will normally decrypt to B E(A) = A ⊕ Cloud

Security Hole #3 (cont) What they intended to say – The entry for virtual address V, for process PID, at level lvl, is physical P (next level pointer or data location) How it should have been written – P, σ Κ (P, V, PID, lvl) (σ = signature) – Bind each page table entry with where it should be located What was actually written – Some entries are secret; I won’t tell you Though you can ask me …And you already know 26

Security Hole #4 The virtual memory system is used as a fence – (theoretically) tamper-evident page table O/S can simply bypass the VMM – Issue writes to physical address directly – O/S can purge malicious pages prior to auditing Similar to cleaner-app, but without any trace 27

Security Vulnerability #1 Stealth checker employs upgradeable firmware Rootkit that is stored in disk can upgrade firmware – System reboot will not clean system – (Beyond the scope of this paper) Compromised O/S may compromise firmware – Enjoy the CPU spying on you – Impossible to detect 28

Security Vulnerability #2 BAD idea to reuse counter of counter-mode encryption 29 V V Translation Array … … … … Last Level Page Directory 1 Page Frame ADDR NULL ADDR Counter (128 bit) HPID||HPID Pipelined Counter Mode Encryption Engine AES-128 Hardware Secret Key: Burnt-in, unexposable 4 entries concatenated 128-bit V V Translation Array ADDR Actual Frame stored in memory Same for every last-level encryption

Conclusion Rootkits are a growing concern Hardware monitoring is a good idea for protection – Malware cannot cheat hardware that easily – Good idea for rootkits also: BIOS rootkit in the works Paper’s implementation does more harm than good – Broken Algorithm: signing of each page table level is needed But O/S can bypass virtual memory manager Cleaner-app is also a problem LBA should target O/S protection 30

Questions? 31 SYSTEM PROTECTED