1. 1 DIFFERENTIATED NETWORK QoS in XEN Dual Degree Final Stage Presentation by Saransh Mittal (04d05016) under the supervision of Prof. Varsha Apte and co-supervision of Prof. Purushottam Kulkarni

2. 2 Outline Server Virtualization using Xen Need for Network Service Differentiation across VMs in Xen Xen I/O Model Network Scheduler Design Network Scheduler Validation Conclusion & Future Work

3. 3 Server Virtualization Using Xen DomUs Dom 0 VMM Physical Devices

4. 4 Xen CPU Scheduler CPU is shared between domains Credit Scheduler Assigns credits to each domains according to their cap or weight specification cap spec is a limit in % of CPU weight spec is a guarantee in % of CPU ◦ Promise to allocate specified % if load consumes ◦ If more demand more % CPU can further be allocated

5. 5 Xen Network I/O Scheduler Simple FIFO Scheduler Does not provide network I/O performance isolation Essential because different domains may have different service requirements. For example: domain 1 running video traffic and domain 2 running web traffic. Overall Goal of this work: Design and implement I/O Scheduler that provides differentiated network I/O to each domain

6. 6 Steps towards the new I/O Scheduler Understand and characterize current Xen network I/O scheduler performance Define I/O QoS specification Understand Xen I/O architecture and implemention Design Scheduler to support such a specification Implement Scheduler in Xen VMM Experimentally validate implementation

7. 7 Experimental study and characterization of current Xen network I/O performance and QoS capabilities

8. 8 Scheduling network I/O using the CPU scheduler Load using 25% CPU, 2MBps bandwidth started on Domain 1 Load using 35% CPU, 2MBps bandwidth started on Domain 2 Experiment 1 : CPU weights are set as 1:1 Experiment 2 : CPU weights are set as 1:4 No correlation between CPU weights and bandwidth consumption

9. Scheduling network I/O using CPU scheduler 9 CPU limits are set as 20% for Domain 1 and 80% for Domain 2 Experiment 3: Load using 25% CPU, 2MBps bandwidth started on Domain 1 Load using 35% CPU, 2MBps bandwidth started on Domain 2 Experiment 4: Load using 35% CPU, 5MBps bandwidth started on Domain 2 No correlation between CPU limits and bandwidth consumption,

10. 10 Experiments under other scenarios All experiments show that network I/O differentiation cannot be achieved using existing mechanisms in Xen

11. 11 Experiments confirm the need for a new I/O scheduler that provides differentiated network I/O to different domains as per their specified quantitative requirement !!

12. 12 Network I/O QoS Specification A user should be able to specify network bandwidth specification in terms of quantitative requirements. ◦ Bandwidth weights: guarantees promised to a domain in MBps ◦ Bandwidth caps: limits promised to a domain in MBps Example: 2MBps weight and 5MBps limit to domain 1 & 3MBps weight and 7 MBps limit to domain 2. Analogous to CPU credit scheduler.

13. 13 Xen I/O Model Guest Domains Front End Interface Control Transfer Back End Interface Dom0 (device drivers) Data Transfer Data Transfer

14. 14 Network I/O Model : dom-1 having 2 virtual interfaces and dom-0 having 1 virtual interface External ethernet interface Network Bridge Backend Interface Data /Control Transfer Frontend Interface

15. 15 New I/O Scheduler Design : Domain network interface structure

16. 16 Design contd.. : FIFO scheduler modifications

17. 17 Scheduler Design : Credit Allocator

18. 18 Scheduler Design : Credit Allocator contd..

19. Network Scheduler Working : Step 1 19 CPU Credit Scheduler Shared Memory Dom3 gets scheduled and sends a packet via netfront driver

20. 20 Scheduler Working : Step 2 dom0 runs netback driver to schedule packet of dom3

21. Scheduler Working : Step 3 21 Internal Ack being sent back from physical device to dom3

22. 22 Scheduler Validation (static loads) Load using 100% CPU, 10MBps bandwidth started on Domain 1 & 2 Experiment 1 : Bandwidth weights are set as 2MBps for domain 1 and 8MBps for domain 2 Experiment 2: Bandwidth limits are set as 2MBps for domain 1 and 8MBps for domain 2

23. Scheduler Validation (static loads) 23 Load using 100% CPU, 10MBps bandwidth started on Domain 1. Load using 50% CPU, 5MBps bandwidth started on Domain 2. Experiment 3 : Bandwidth weights are set as 2MBps for domain 1 and 8MBps for domain 2 Experiment 4: Bandwidth limits are set as 2MBps for domain 1 and 8MBps for domain 2

24. 24 Validation (DYNAMIC LOAD 1)

25. 25 Validation (DYNAMIC LOAD 2)

26. 26 Validation (DYNAMIC LOAD 3)

27. Summarizing Validation Experiments 27 BenefitsDescription 1 Provides a way to do differentiated network I/O across domains on quantitative specifications. 2 Does guarantee and limit based network I/O scheduling across different domains 3 Responds quickly to load changes in the environment.

28. 28 Conclusion & Future Work We designed and implemented a network I/O credit scheduler which ◦ Provides guarantee and limit specification in terms of bandwidth ◦ Specification independent of load dynamics in a domain. ◦ Responds quickly to load changes ◦ Flexibility to accommodate other network differentiated services like latency – time based scheduler. Future work ◦ Remove CPU scheduler interference when CPU is in contention ◦ Extend to support service time based service differentiation

29. Summary of Contribution Compilation, Installation & Debugging of Xen and domains Benchmark & Load Development ◦ 3 configurable micro benchmarks created ◦ Different loads generated from the combination of these benchmarks Experiments to demonstrate the absence of differentiated network I/O Exhaustive set of cases were considered: ◦ CPU is not in contention, I/O is not in contention ◦ CPU is not in contention, I/O is in contention ◦ CPU is in contention, I/O is not in contention ◦ CPU is in contention, I/O is in contention 29

30. Code Exploration to understand Xen network I/O model. ◦ Not much documentation available ◦ netback folder explored in drivers/xen/ Network Scheduler Design ◦ CPU Credit Scheduler in Xen ◦ DWRR Scheduler in Routers Network Scheduler Implementation ◦ Modifications in the domain interface structure ◦ Changes in network FIFO scheduler ◦ Addition of Credit Allocator Experiments to validate the network scheduler. Scenarios with ◦ Static loads ◦ Dynamic loads 30

31. 31 Acknowledgement I would like to thank Prof. Varsha Apte and Prof. Purushottam Kulkarni for their consistent help and support in my work.

32. 32

33. 33

34. Introduction Virtualization : abstraction of services. Examples: ◦ Virtual Memory ◦ Desktop Virtualization ◦ Storage Virtualization ◦ High level language Virtualization ◦ Server Virtualization Server Virtualization: Virtualization applied to servers containing virtual infrastructure like CPU, memory, I/O, disk, network, etc. ◦ Advantages:  Workload Consolidation  Workload Isolation  Workload Migration  Application Heterogeneity

35. Server Virtualization Overview Architecture ◦ Hosted ◦ Bare Metal Techniques ◦ Full Virtualization ◦ Para Virtualization ◦ H/W Assist Virtualization

36. Experimental Setup, Microbenchmarks & Terminologies Category I load Category II load Category III load burns CPU cycles configurable periodic sleeps Max 100% CPU udp load configurable periodic sleeps 100% CPU, 10MBps network scp load 40 % CPU, 10MBps network (x,y)CPU_weights(a,b)CPU_limits(a,b)((a,b)(c,d) x% CPU, y MBpsa % CPU guarantee to dom-1 and b% CPU guarantee to dom-2 a% CPU limit to dom-1 and b% CPU limit to dom-2 (a,b) load running on dom-1, (c,d) load running on dom-2

37. Scheduling network I/O using CPU scheduler 37 40% CPU, 10Mbps bandwidth on domain 1. 40% CPU, 4MBps bandwidth on domain 2. Exp 5: CPU weights 1:1 Exp 6: CPU weights 1:4 CPU limits 20:80 Exp 7: 40% CPU, 10Mbps bandwidth on domain 1. 40% CPU, 4MBps bandwidth on domain 2. Exp 8: 50% CPU, 10Mbps bandwidth on domain 1.

38. Scheduling network I/O using CPU Scheduler 38 CPU limits 20:80 Exp 15: 60% CPU, 10Mbps bandwidth on domain 1 & 2 Exp 16: 100% CPU, 3Mbps bandwidth on domain 2. CPU limits 20:80 Exp 11: 100% CPU, 3Mbps bandwidth on domain 1 & 2. Exp 12: 80% CPU, 3Mbps bandwidth on domain 2.

39. Observations Possible solutions : Adhoc method to schedule network I/O via changing control knobs of CPU credit scheduler. Traffic Control scripts at bridge level by creating filters at ethernet interfaces of domains. Above solutions have problems !!! ProblemDescription 1Different VMs have different service requirements 2CPU scheduling to allocate n/w bandwidth across different VMs is adhoc & incomplete 3CPU scheduling to provide latency based service differentiation across different VMs is adhoc & incomplete

40. 40 Working (Issues) CPU weights do not change the network I/O allocation as CPU not in contention given bandwidth configuration. CPU weights do change the network I/O allocation as CPU in contention given bandwidth configuration.

41. 41 Possible Solutions Shutdown I/O scheduler, perform feedback based adhoc based CPU scheduling. Give equal weights to domains and perform network scheduling. It does best effort service considering all domains. Average CPU utilization over the entire 30-day period is 53%, with the best-practice participant averaging 75% utilization. The best-practice participants are defined as those organizations that have the highest utilization while maintaining acceptable service levels. During prime shift (Monday through Friday, 7:00 a.m. to 6:00 p.m.), survey participants achieve average utilization of 68%, with the best-practice participant reaching a high of 83%.. Average CPU utilization over the entire 30-day period is 53%, with the best-practice participant averaging 75% utilization. The best-practice participants are defined as those organizations that have the highest utilization while maintaining acceptable service levels. During prime shift (Monday through Friday, 7:00 a.m. to 6:00 p.m.), survey participants achieve average utilization of 68%, with the best-practice participant reaching a high of 83%..

42. 42 Scheduler Validation Conclusion Service time has no direct correlation with our network I/O credit scheduler

43. References Virtualization Overview. Technical report, VMWare. Understanding Full Virtualization, Paravirtualization and Hardware Assist. Technical report, VMWare, 2007. Xen Architecture Overview. Technical report, XenSource, 2008. Paul Barham, Boris Dragovic, Keir Fraser, Steven hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, and Andrew Warfied. Xen and the Art of Virtualization. SOSP, October 2003. Ludmila Cherkasova, Diwaker Gupta, and Amin Vahdat. Comparison of the Three CPU Schedulers in Xen. ACM, September 2007. Kenneth J. Duda and David R. Cheriton. Borrowed-virtual-time Scheduling: supporting latency-sensitive threads in a general purpose scheduler. ACM, December 1999. Chuck Semeria. Supporting Differentiated Service Classes: Queue Scheduling Disciplines. Technical report, Juniper Networks, Part Number:: 200020-001, December 2001. Jeremy Sugerman, Ganesh Venkitachalam, and Beng hong Lim. Virtualizing I/O devices on VMwareWorkstation’s hosted Virtual Machine Monitor. Technical report, VMWare Inc., 2002. [20] Carl A. Waldspurget. Memory Resource Management in VMware ESX Server. Technical report, VMWare Inc., 2002.

44. Links http://lists.xensource.com/archives/html/xen-devel/2006-10/msg00305.html. http://www.itcomparison.com/virtualization/xenvsvi3/xenvsvmware3esx.htm. http://www.xen-tools.org/software/xen-tools/. Ian Campbell http://xenbits.xensource.com/linux-2.6. 1 8- xen.hg?rev/7fe1c6d02a2b. Mathew Donovan. http://wiki.xensource.com/xenwiki/xennetworking. Computer Economics. http://www.computereconomics.com/. Microsoft. http://technet.microsoft.com/en-us/library/cc728211(ws.10).aspx. George Ou. http://articles.techrepublic.com.com/510010878116074941.html. Ramirsoe. http://wiki.xensource.com/xenwiki/xenintro.