1. Ahmed M. Azab, Peng Ning, Zhi Wang, Xuxian Jiang Department of Computer Science, North Carolina State University Xiaolan Zhang IBM T.J. Watson Research Center Nathan C. Skalsky IBM Systems & Technology Group 2011/3/81ADL Meeting

2. Outline About SMM Introduction and Background Assumptions, Threat Model, and Security Requirements The HyperSentry Framework Verifying the Integrity of the Xen Hypervisor – a Case Study Implementation and Experimental Evaluation Conclusion 2011/3/82ADL Meeting

3. About SMM - Reference Phrack Magazine: Issue #65: System Management Mode Hack: Using SMM for “Other Purposes”System Management Mode Hack: Using SMM for “Other Purposes” Issue #66: A Real SMM Rootkit: Reversing and Hooking BIOS SMI HandlersA Real SMM Rootkit: Reversing and Hooking BIOS SMI Handlers Duflot, Using CPU System Management Mode to Circumvent Operating System Security FunctionsUsing CPU System Management Mode to Circumvent Operating System Security Functions Intel Architecture Software Developers Manuals, Volume 3: System Programming 2011/3/83ADL Meeting

4. About SMM SMM: System Management Mode [wiki][wiki] Intel manuals: “The Intel System Management Mode (SMM) is typically used to execute specific routines for power management. … SMM operates independently of other system software, and can be used for other purposes too.” 2011/3/84ADL Meeting

5. About SMM Real Address Mode Protected Mode Virtual 8086 Mode SMM Mode PE = 1 PE=0 or reset VM = 0 VM = 1 reset -> SMI (interrupt) <- rsm or reset <- rsm instruction 2011/3/85ADL Meeting

6. About SMM Instruction RSM can be used just inside the SMM Paging is disabled 16-bit mode of operation But all physical memory can be addressed (4GB) The same privileges as in Ring 0 Interrupts are blocked in SMM (So are NMI Interrupts) SMI: System Management Interrupt SMRAM: System Management RAM 2011/3/86ADL Meeting

7. About SMM SMRAM SMRAM region is at 0xA0000-0xBFFFF Video card memory-mapped base address SMRAM Control Register Bit 6 – D_OPEN SMBASE is redirected to SMRAM if D_OPEN is set Bit 4 – D_LCK To protect SMRAM 2011/3/87ADL Meeting

8. About SMM SMI_EN register: Control which devices can generate an SMI The least significant bit specifies whether SMIs are enable or not SMI_STS register: Keep track of which device last caused an SMI 2011/3/88ADL Meeting

9. Introduction Hypervisors did not turn out to be completely secure Xen is used in Amazon’s EC2 2008 Black Hat: Xen 0wning trilogy (Blue Pill) [part1] [part2] [part3] [code_and_demo] [part1][part2][part3][code_and_demo] Attack Xen There are at least 17 vulnerabilities reported for Xen 3.x There are at least 165 vulnerabilities reported in Vmware ESX 3.x The true challenge lies in the measurement of hypervisor integrity at runtime 2011/3/89ADL Meeting

10. Introducion Copilot uses PCI device to provide integrity measurement Cannot access CPU state (e.g., CR3 register) HyperGuard and HyperCheck Rely on SMM None of them provide a way to trigger the integrity measurement without alerting hypervisor Scrubbing Attacks 2011/3/810ADL Meeting

11. Introduction Challenges Stealthy integrity measurement Verifiable measurement agent Deterministic and non-interruptible execution In-context measurement Attestation to the authenticity of the measurement output 2011/3/811ADL Meeting

12. Introduction HyperSentry relies on a Trust Computing Base (TCB) composed of hardware, firmware and software HyperSentry is triggered by an out-of-band communication channel SMI: System Management Interrupt IPMI: Intelligent Platform Management Interface [wiki][wiki] May use AMT (Intel Active Management Technology) to trigger SMI HyperSentry resides in the SMM 2011/3/812ADL Meeting

13. Introduction SMM does not offer all the necessary contextual info Set the CPU to the required context Provide a verifiable and protected environment to run a measurement agent in the hypervisor context IBM BladeCenter H chassis with HS21 XM blade servers Xen 2011/3/813ADL Meeting

14. Background IPMI: Server-oriented platform management interface Hardware + Firmware Baseboard Management Controller (BMC) on motherboard SMM 2011/3/814ADL Meeting

15. Assumptions, Threat Model, and Security Requirements Assumptions: Equipped with an out-of-band channel Physically secured TCG’s trusted boot hardware and Trusted Platform Module 2011/3/815ADL Meeting

16. Assumptions, Threat Model, and Security Requirements Threat Model: The adversary, once compromising the hypervisor, will attempt to attack the measurement software and/or forge measurement output Periodic integrity measurement Do not handle attacks that do not cause a persistent change 2011/3/816ADL Meeting

17. Assumptions, Threat Model, and Security Requirements Security Requirements: (SR1) Stealthy Invocation (SR2) Verifiable Behave (SR3) Deterministic Execution (SR4) In-context Privileged Measurement (SR5) Attestable Output 2011/3/817ADL Meeting

18. The HyperSentry Framework 2011/3/818ADL Meeting Host (root) Mode Guest (non-root) Mode VM Hardware Hypervisor Virtualized Platform System Management Mode Remote Verifier IPMI/BMC SMI Handler Measurement Agent Trusted Components are Shaded in Green

19. The HyperSentry Framework When interrupted by the SMI… CPU may run in either the hypervisor (VMX root operation) or one of the guest VM (VMX non-root operation) In order to measure the integrity of hypervisor, the measurement agent needs to access the hypervisor’s code, data and CPU state When CPU runs in VMX non-root operation at SMI, all pointers to VMX data structures are saved internally to the CPU and cannot be retrieved via software 2011/3/8ADL Meeting19

20. The HyperSentry Framework Challenges How to detect the intercepted CPU operation mode? Hypervisor or guest VM? How to retrieve the context needed for measurement? E.g., CR3 and page table Solution Inject a privileged instruction to force the CPU to fall back to the hypervisor mode Run the measurement agent in the same context as the hypervisor Agent runs in a protected execution environment 2011/3/820ADL Meeting

21. The HyperSentry Framework 2011/3/821ADL Meeting Host (root) Mode Guest (non-root) Mode System Management Mode Hardware Prepare SMM fallback Hypervisor Guest VM SMI RSM Execution Path Privileged instruction PC (cache misses = 1)APIC (SMI on PC overflow) Inject privileged instruction and flush cache PC (cache misses = 0) Verify the measurement agent SMI The measurement agent RSM Store measurement output SMI VM exit

22. The HyperSentry Framework If an interrupt is received during handling ths SMI… Inject another copy of the instruction at each interrupt handler Using LIDT instruction to locate interrupt handler 2011/3/8ADL Meeting22

23. The HyperSentry Framework In-context Integrity Measurement Some Intel Trusted eXecution Technology late launch registers are hidden from the SMM SMM is relatively slow 2 orders of magnitude slower than protected mode So, HyperSentry’s measurement agent runs in protected mode 2011/3/8ADL Meeting23

24. The HyperSentry Framework Measurement Agent Verification SMI handler calculates the hash of the measurement agent’s code SMI handler disables all maskable interrupts by clearing the corresponding bit in EFLAGS register SMI handler modifies the IDT (physical memory) Malicious DMA write operation threat is handled by verifying that the agent is included in the DMA protected ranges provided by Intel VT-d 2011/3/8ADL Meeting24

25. The HyperSentry Framework Handling Multi-core Platforms Freeze all cores Only Boot Strap Processor (BSP, core 0) will execute HyperSentry 2011/3/8ADL Meeting25

26. The HyperSentry Framework Is out-of-band invocation sufficient to achieve stealthy invocation? Unfortunately … 2011/3/826ADL Meeting

27. The HyperSentry Framework 2011/3/827ADL Meeting Host (root) Mode Guest (non-root) Mode VM Hardware Hypervisor System Management Mode Remote Verifier SMI Handler Typical Scenario BMC/IMM

28. The HyperSentry Framework 2011/3/8ADL Meeting28 Host (root) Mode Guest (non-root) Mode VM Hardware Hypervisor System Management Mode Remote Verifier SMI Handler Attack Scenario BMC/IMM Compromised hypervisor cannot intercept SMIs. But what if it tries to block real SMIs and generate fake ones?

29. The HyperSentry Framework Can we prevent the hypervisor from blocking SMIs? Not possible with existing hardware Solution Detecting fake SMIs generated by the (compromised) hypervisor Verifying status registers to ensure that the measurement is invoked by the out-of-band channel Key reason: HW SMI and SW SMI are distinguishable 2011/3/829ADL Meeting

30. The HyperSentry Framework 2011/3/830ADL Meeting BMC AMM IPMI CPU Core 0 Target Platform (IBM HS21XM Blade Server) Remote Verifier IO Control Hub (South Bridge) Memory Control Hub (North Bridge) GPI 0 SSH SMI_EN GPI_ROUT 0 …..0 0…….0 SMI_STS 0 ……………….0 ALT_GPI_SMI_STS ALT_GPI_SMI_EN 1 10 1 CPU Core 1 CPU Core n 1 1 - All status register are non writable - Measurement is invoked only if all other bits are 0 - A fake SMI is easily detectable 0910 SMI

31. The HyperSentry Framework HyperSentry requires that GPI_ROUT is configured so that only GPI 0 can generate SMIs If a compromised hypervisor disable SMI by overwriting GPI_ROUT Easily detected due to lack of response 2011/3/8ADL Meeting31

32. The HyperSentry Framework Attesting to the Measurement Output Challenge Absence of a dedicated hardware for attestation The hypervisor controls the hardware most of time Solution Providing the SMRAM with a private key Using this key to attest to the measurement results 2011/3/832ADL Meeting

33. The HyperSentry Framework 2011/3/833ADL Meeting Host Mode Guest Mode System Management Mode Hardware Guest VM TPM SMI handler Initialization code SMM private key SMM public key K smm K smm -1 Hypervisor Bootstrapping Remote Verifier Integrity measurement output Attestation request K smm -1 { Output|Nonce} K AIK -1 {K smm |Handler|Nonce}

34. The HyperSentry Framework Stealthy Invocation If configurations are not changed  guaranteed by hardware If configurations change  fake SMIs are detectable Verifiable Behavior The measurement agent is measured every time before it executes Deterministic Execution The measurement agent possesses full control over the system In-context privileged measurement Guarantee falling back to the hypervisor mode The measurement agent runs in the same context as the hypervisor Attestable output The measurement output is signed by a verifiable and protected key 2011/3/834ADL Meeting

35. Implementation and Experimental Evaluation IBM HS21XM blade server Measuring the Xen hypervisor End-to-end execution time: 35 ms 2011/3/835ADL Meeting

36. Implementation and Experimental Evaluation Periodical measurement: Every 8 seconds: 2.4% overhead; every 16 seconds: 1.3% overhead 2011/3/8ADL Meeting36

37. Conclusion HyperSentry A novel framework for measuring the integrity of the most privileged system software A measurement agent for the Xen hypervisor Low overhead Next step Measurement agent for Linux/KVM Verifying the hypervisor’s dynamic integrity 2011/3/8ADL Meeting37

38. Reference https://researcher.ibm.com/researcher/files/us- xmeng/Azab.pptx 2011/3/8ADL Meeting38