1. Computer Science Cloud Computing Infrastructure Security Peng Ning With Ahmed Azab, Xiaolan Zhang, Wu Zhou, Xuxian Jiang, and Zhi Wang. June 29, 20121ACNS 2012 Supported by the US NSF under grant # 0910767, the US ARO under the grant # W911NF-08-1-0105, and IBM under Open Collaboration Research (OCR) Awards.

2. Computer Science Outline Background Security threats to cloud computing Security of cloud computing infrastructure –Driven by a new security architecture for cloud computing –Hypervisor-based security services –Offline VM image security services –Hypervisor integrity services –Isolation that can bypass hypervisor control Conclusion June 29, 2012ACNS 20122

3. Computer Science What is Cloud Computing Wikipedia –Cloud computing is a paradigm of computing in which dynamically scalable and often virtualized resources are provided as a service over the Internet –Users need not have knowledge of, expertise in, or control over the technology infrastructure in the "cloud" that supports them –Reduction in hardware, operational and administrative costs Virtualization is a key to cloud computing –Scalability –Ease of use –Affordable pricing June 29, 2012ACNS 20123

4. Computer Science Example: Amazon Elastic Compute Cloud (EC2) June 29, 2012ACNS 20124 EC 2 Management Console Start an Instance Set up the Instance Launch the Instance Verify the Instance

5. Computer Science Security Threats in Cloud Computing External threats Guest-to-guest threats Guest-to-cloud threats Cloud-to-guest threats June 29, 20125ACNS 2012

6. Computer Science Cloud Computing Infrastructure Security Our proposal –A security architecture for compute clouds Focus on Infrastructure as a Service (IaaS) –Addition of security architecture components Hypervisor-based security services Offline VM image security services Hypervisor integrity services Isolation mechanisms that can bypass the hypervisor June 29, 2012ACNS 20126

7. Computer Science A Typical Compute Cloud June 29, 20127ACNS 2012

8. Computer Science Virtualization-based Runtime Security Services June 29, 20128ACNS 2012 HIMA [ACSAC ’09] HookSafe [CCS ’09]

9. Computer Science Example Service: HIMA HIMA: Hypervisor based Integrity Measurement Agent Validation of VMs with runtime guarantees –Measure the VM OS and applications loaded into guest VMs –Actively monitor all guest events that could change measured applications Time of Check to Time of Use (TOCTTOU) consistency Prototypes –Initial implementation works for Xen (para-virtualization) –Ported to support KVM (hardware assisted virtualization) June 29, 20129 Ahmed M. Azab, Peng Ning, Emre C. Sezer, and Xiaolan Zhang, "HIMA: A Hypervisor Based Integrity Measurement Agent," in Proceedings of ACSAC 2009, December 2009. Ahmed M. Azab, Peng Ning, Emre C. Sezer, and Xiaolan Zhang, "HIMA: A Hypervisor Based Integrity Measurement Agent," in Proceedings of ACSAC 2009, December 2009. ACNS 2012

10. Computer Science VM Image Security Services June 29, 201210ACNS 2012 Nüwa—offline patching [ACSAC ’10] Offline virus scanning [CCSW ’09]

11. Computer Science Example Service: Nüwa （女娲） Nüwa – Offline Patching of VM Images Motivation –Dormant VM images usually contain vulnerabilities –Offline patching service A desirable security service in compute clouds Technical challenge –Current patching system: Designed for running systems –Pre- and post-processing scripts –Examples: Stop/start daemons; Conditional updates June 29, 201211ACNS 2012

12. Computer Science Nüwa Approach Two phases –Phase 1: Automatic script rewriting –Phase 2 (Leftovers): Resort to online updates –Our research focus is on script rewriting Variations –Standalone Nüwa: Offline patching of individual VM images in emulated environments –Mirage-based Nüwa: Batched offline patching using the Mirage VM image library June 29, 2012ACNS 201212 Wu Zhou, Peng Ning, Xiaolan Zhang, Glenn Ammons, Vasanth Bala, Ruowen Wang, "Always Up-to-date -- Scalable Offline Patching of VM Images in a Compute Cloud," in Proceedings of ACSAC 2010, December 2010. Wu Zhou, Peng Ning, Xiaolan Zhang, Glenn Ammons, Vasanth Bala, Ruowen Wang, "Always Up-to-date -- Scalable Offline Patching of VM Images in a Compute Cloud," in Proceedings of ACSAC 2010, December 2010.

13. Computer Science Some Evaluation Results Standalone Nüwa –Base VM image 64-bit Ubuntu 8.04; 406 patches (collected on 10/26/2009) 402 patches can be applied offline (99%) –Failure cases Mono-gac package and three others that depend on it Mirage-based Nüwa –100 VM images based on 32-bit Ubuntu 8.04 Using 100 randomly selected subsets of basic Ubuntu tasks –Top 8 security updates from Ubuntu Security Notices Ranked by Ubuntu popularity contest All data collected on 01/18/2010 June 29, 201213ACNS 2012

14. Computer Science Some Evaluation Results (Cont’d) Performance gain by standalone Nüwa –About 4 times speedup June 29, 201214 * “Average” refers to the average of all 402 packages. ACNS 2012

15. Computer Science Some Evaluation Results (Cont’d) Additional speedup by Mirage-based Nüwa –Another 2 – 10 times June 29, 201215ACNS 2012

16. Computer Science Hypervisor Integrity Services June 29, 2012ACNS 201216 HyperSentry [CCS ’10]

17. Computer Science Example Service: HyperSentry Why HyperSentry? –Hypervisor is the highest privileged software –Compromise of hypervisor  compromise of the system –Hypervisors cannot be blindly trusted Example #1: Xen owning trilogy [BlackHat 2008] Example #2: VM Ware ESX 3.x –6/18/12: 67 Secunia advisories; 562 vulnerabilities; 7% Secunia advisories not patched –Hypervisor's code base is growing  More vulnerabilities? June 29, 2012ACNS 201217

18. Computer Science HyperSentry –Stealthy and in-context measurement of hypervisor integrity Challenges –A fundamental problem How to measure the integrity of the highest privileged software? –Hypervisor has full control of the software system (most of the time) Scrubbing attacks Tampering with the measurement agent Tampering with the measurement results –Relying on a higher privileged software goes back to the same problem June 29, 2012ACNS 201218

19. Computer Science The HyperSentry Approach HyperSentry –A generic framework to stealthily measure the integrity of a hypervisor in its context Key ideas –Allow the measurement software to gain the highest privilege temporarily –Measurement is triggered stealthily Scrubbing attacks –Isolate measurement results from the hypervisor June 29, 2012ACNS 201219 Ahmed M. Azab, Peng Ning, Zhi Wang, Xuxian Jiang, Xiaolan Zhang, Nathan C. Skalsky, "HyperSentry: Enabling Stealthy In-context Measurement of Hypervisor Integrity,” in ACM CCS 2010, October 2010.

20. Computer Science Case Study: Verifying the Integrity of Xen Integrity measurement –Code: SHA-1 hash of Xen's code; Control flow verification –Date: Detect unauthorized sharing of physical pages across guest VMs Performance (on IBM HS21 XM blade server) –End-to-end execution time: 35ms –Periodical measurement: Every 8s: 2.4%; every 16s: 1.3% June 29, 2012ACNS 201220

21. Computer Science Isolated Execution Bypassing Hypervisor Control June 29, 2012ACNS 201221 SICE [CCS ’11]

22. Computer Science VM SICE: Strongly Isolated Computing Environment June 29, 2012ACNS 201222 Legacy Host (Hypervisor/OS) Legacy Host (Hypervisor/OS) Hardware Virtualized Platform VM VM/ Workload VM/ Workload Legacy Host (Hypervisor/OS) Legacy Host (Hypervisor/OS) Hardware Virtualized Platform with SICE SICE Ahmed Azab, Peng Ning, Xiaolan Zhang, “SICE: A Hardware-Level Strongly Isolated Computing Environment for x86 Multi-core Platforms,” in Proceedings of ACM CCS 2011, October 2011.

23. Computer Science Foundation of SICE System Management Mode (SMM) –x86 operating mode for system management functions –Single entry point: System Management Interrupt (SMI) –SMRAM: Isolated from the rest of system Not accessible by the system software (e.g., hypervisor) –AMD processors implementation Resizing the SMRAM at runtime Separate SMRAM range for each CPU core –Main challenges SMM has escalated privileges The CPU runs slowly and has limited functionality June 29, 2012ACNS 201223

24. Computer Science Foundation of SICE (Cont’d) Trusted/Secure boot –Building the trust chain during system initialization C-RTM  BIOS  Boot loader  Initial hypervisor/OS image –Secure hardware extensions (e.g., the TPM) Seals and authenticates the measurement output –Main challenge Trust cannot be sustained due to potential runtime attacks June 29, 2012ACNS 201224

25. Computer Science Implementation requirements 1.Hardware: AMD processors 2.BIOS: Load the SMI handler 3.Legacy host: Provide the communication channel Hardware Legacy Host SICE Architecture 25 June 29, 2012ACNS 2012 VM Virtualized Platform with SICE SMRAM CPU, TPM, BIOS, etc… DMA Devices Hypervisor/OS APP Isolated Environment Security Manager (hypervisor) Isolated Workload (VM) SMI Handler (SICE) SMI Handler (SICE) Trusted Untrusted Isolated Secure boot Consists of 300 SLOC to: 1.Maintain the isolation 2.Initialize and attest to the isolated environment Provided by the user Special hypervisor: Confines the isolated workload Trusted by the host Communication channel: Providing hardware services

26. Computer Science SICE Operations Operating modes –Time-sharing Mode (an intermediate step) –Multi-core Mode Remote attestation –The initial image of the isolated environment –Secure communication with remote users June 29, 2012ACNS 201226

27. Computer Science Legacy Host Time-sharing Mode I nitialization –Secure boot –SMI to load the isolated workload Execution environment switching –SMI to trigger the isolated environment –Changing the saved CPU state –Changing the SMRAM memory range –Fresh CPU start in the new environment –SMI to return to the legacy host Termination ACNS 201227 Hardware Legacy Host SMRAM Isolated Env. Workload Image Security Manager Image BIOS/TPM SMRAM SMI Handler (SICE) SMI Handler (SICE) Running Workload Running Workload Security Manager (Hypervisor) SMI June 29, 2012

28. Computer Science Multi-core Mode Concurrent sharing of the hardware –Good utilization –One or more CPU cores are assigned to either The isolated environments The legacy host Main challenges –Event isolation –Memory isolation June 29, 2012ACNS 201228

29. Computer Science General multi-core processor architecture Core n Multi-core Mode (Cont’d) June 29, 2012ACNS 201229 Memory Control Hub (North Bridge): Configuration registers L2 Cache I/O registers Memory Control Hub (North Bridge): Configuration registers L2 Cache I/O registers Core 1 Core 0: Registers MSRs L-APIC L1 Caches Core 0: Registers MSRs L-APIC L1 Caches Core n Memory Control Hub (North Bridge): Configuration registers L2 Cache I/O registers Memory Control Hub (North Bridge): Configuration registers L2 Cache I/O registers Core 1 Core 0: Registers MSRs L-APIC L1 Caches Core 0: Registers MSRs L-APIC L1 Caches AMD processors: Define the SMRAM SMM_BASE SMM_MASK

30. Computer Science Hardware I nitialization –Secure boot –Loading the isolated workload Multi-core Mode Operations June 29, 2012 ACNS 201230 Legacy Host SMRAM Security Manager Workload Image SMI SMI Handler (SICE) SMI Handler (SICE)

31. Computer Science Isolated CPU Core SMRAM Legacy Host Running the isolated environment: The isolated core –Changing saved CPU state E.g., page tables, interrupt descriptor, instruction & stack pointers –Changing the SMRAM memory range (password stored in SMRAM) –Fresh CPU start in the isolated environment Hardware Multi-core Mode Operations June 29, 2012 ACNS 201231 Isolated Env. Security Manager Workload Image SMRAM SMI Handler (SICE) SMI Handler (SICE) Host CPU Core Legacy Host Security Manager Isolated Workload BIOS/TPM SMI Handler (SICE) Security Manager (Hypervisor) Workload Image Running Isolated Workload (VM) Running Isolated Workload (VM)

32. Computer Science Isolated CPU Core SMRAM Legacy Host Running the isolated environment: The host core –Return to the legacy host No environment switching necessary Hardware Multi-core Mode Operations June 29, 2012 ACNS 201232 Isolated Env. SMRAM SMI Handler (SICE) SMI Handler (SICE) Host CPU Core Legacy Host Security Manager Isolated Workload BIOS/TPM SMI Handler (SICE) Security Manager (Hypervisor) Workload Image Running Isolated Workload (VM) Running Isolated Workload (VM)

33. Computer Science Multi-core Mode Event Isolation Event isolation –Prevent the legacy host and the isolated workloads send events to each other –Events between cores: Inter-Processor Interrupts (IPI) Two types of IPIs –Maskable IPIs Can be blocked by recipient core’s APIC –Non-maskable IPIs Can be controlled by Global Interrupt Flag (GIF) Clear GIF to ignore or hold all IPIs June 29, 2012ACNS 201233

34. Computer Science Multi-core Mode Event Isolation (Cont’d) Protecting the host core –The security manager runs as a thin hypervisor Prevents the isolated workload from privileged hardware access Protecting the isolated core –The security manager Clear GIF All IPIs are disabled –The isolated workload Set the GIF and re-enable maskable IPIs IPIs will cause a VM exit, which are examined by the security manager for processing June 29, 2012ACNS 201234

35. Computer Science Registers: cr3 MSRs: SMM_Mask SMM_Address Registers: cr3 MSRs: SMM_Mask SMM_Address Multi-core Mode Memory Isolation June 29, 2012ACNS 201235 Host Core(s) Physical Memory Memory Isolation –Assigning different memory ranges to different CPU cores Memory double-view technique Each CPU core has its own SMRAM Registers: Nested cr3 MSRs: SMM_Mask SMM_Address Registers: Nested cr3 MSRs: SMM_Mask SMM_Address Isolated Core(s) Security Manager Isolated Workload Set by the SMI handler Set by the security manager Shared Memory SMI Handler

36. Computer Science The SICE TCB The isolated environment –Hardware + BIOS + SMI handler (300 SLOC) The legacy host –Hardware + BIOS + SMI handler + The security manager –Similar to micro-hypervisor approaches June 29, 201236ACNS 2012

37. Computer Science SICE Prototype & Evaluation IBM LS 22 blade server Modifying Qemu/KVM to support a SICE isolated Linux guest –No disk emulation: RAM disk image Time needed for context switching: 46 µs Evaluation results 37June 29, 2012ACNS 2012

38. Computer Science Conclusion Infrastructure security of cloud computing –Necessary for new research –Better security protection for cloud workloads Security architecture for cloud computing –Hypervisor-based runtime security services –VM image security services –Hypervisor integrity services –Isolated execution bypassing hypervisor control –Not necessarily complete Hopefully a guidance/framework for innovative ideas Stay relevant!!! June 29, 2012ACNS 201238

39. Computer Science Questions? Thank You! June 29, 2012ACNS 201239