Presentation is loading. Please wait.

Presentation is loading. Please wait.

Nova Scheduler Shane Wang (王庆), Intel Open Source Technology Center 微信号: qq559382.

Similar presentations


Presentation on theme: "Nova Scheduler Shane Wang (王庆), Intel Open Source Technology Center 微信号: qq559382."— Presentation transcript:

1 Nova Scheduler Shane Wang (王庆), Intel Open Source Technology Center 微信号: qq559382

2 Agenda What is current situation? How scheduler works in Juno and Kilo Resource Tracking Filters and Weight Utilization Based Scheduling (UBS) What is next plan? Gantt Dynamic Resource Scheduling (DRS)

3 How scheduler works in Juno and Kilo Conductor API Scheduler Compute 1.User request and with scheduler hints to include scheduling policy 2. Submit new task 3. Request host that match the request_spec and filter_properties 4. Returns selected hosts 5. Call the selected compute 6. Rescheduling after claim resource failed or other failure

4 Resource usage Tracking Conductor Scheduler Compute 2. Submit new task 3. Request host that match the request_spec and filter_properties 4. Returns selected hosts 5. Call the selected compute 6. Rescheduling after claim resource failed or other failure DB Hypervisor Resource Claiming 1) Validate the resource usage 2) Update the resource Usage 3) Update to DB 1) Fetch newest compute node stats for each call 2) Filter and weight the host 3) Consuming the resource for selected host Periodically update the node resource with 60 seconds interval 1) Get hypervisor resource 2) Consuming the resource 3) Update to DB

5 Filters and weight hosts Request Spec: Image Instance_properties Instance_type Filter_properties Scheduler-hints Assist parameter: retry Nova boot –flavor 1 –image …… --hint group=‘sg1’ --hint Send arbitrary key/value pairs to the scheduler for custom use. scheduler_host_subset_size=1 scheduler_available_filters='nova.scheduler.filters.all_filters‘ scheduler_default_filters= [……] scheduler_weight_classes=nova.scheduler.weights.all_weighers

6 Filters Resource: CoreFilter AggregateCoreFilter: cpu_allocation_ratio=16.0 RamFilter AggregateRamFilter: ram_allocation_ratio=1.5 DiskFilter AggregateDiskFilter: disk_allocation_ratio=1.0 IoOpsFilter AggregateIoOpsFilter: max_io_ops_per_host=8. IoOps means resize, building, image snaphsot. Migration, rescues, unshelve, backup PciPassthroughFilter: Generic PCI device or SRIOV assignment NUMATopologyFilter: NUMA in J, CPUPinning, Hugepage in K

7 Filters Affinity: DifferentHostFilter, SameHostFilter: scheduler_hints: different_host/ same_host =[‘instance uuid’…] ServerGroupAffnityFilter, ServerGroupAntiAffinityFilter: nova server-group-create Create a new server group with the specified details. nova server-group-delete Delete specific server group(s). nova server-group-get Get a specific server group. nova server-group-list Print a list of all server groups. boot with scheduler-hints: group=uuid Boot new instance into server group SimpleCIDRAffinityFilter: scheduler_hints: cidr, build_near_host_ip TypeAffinityFilter, AggregateTypeAffinityFilter: instance_type

8 Filters Topology: AggregateImagePropertiesIsolation: image properties matchs aggregate metadata IsolatedHostsFilter: isolated_hosts, isolated_images, restrict_isolated_hosts_to_isolated_images AggregateInstanceExtraSpecsFilter: Flavor’s extra spec match aggregate metadata AggregateMultiTenancyIsolation: filter_tenant_id AvailabilityZoneFilter

9 Filters Others: ComputeCapabilitiesFilter: work with instance type extra_spec: ‘capabilities:’ ComputeFilter: The compute node is live or disabled ImagePropertiesFilter: architecture, hypervisor type, vm_mode, hypervisor_version_requires JsonFilter: scheduler_hints:query NumInstancesFilter, AggregateNumInstancesFilter, max_instances_per_host RetryFilter TrustedFilter

10 Weight IoOpsWeigher MetricsWeigher RAMWeigher

11 Utilization Based Scheduling CPU Utilization data Memory Utilization data Network Bandwidth data etc

12 Utilization Based Scheduling Conductor Scheduler Compute 2. Submit new task 3. Request host that match the request_spec and filter_properties 4. Returns selected hosts 5. Call the selected compute 6. Rescheduling after claim resource failed or other failure DB Hypervisor 1) Fetch newest compute node stats for each call 2) Filter and weight the host 3) Consuming the resource for selected host CPU Monitor NetworkBand Width MemoryCache Monitor Update 60 seconds interval Notification Bus AMQP

13 Utilization Based Scheduling MetricsWeigher: weight_multiplier: Multiplier used for weighing metrics. weight_setting: How the metrics are going to be weighed. Required: If true, use the MetricsFilter weight_of_unavailable

14 How scheduler strategy affects performance?

15 Benchmark Accuracy

16 Smart Scheduling Efficiency QoS meet SLA contract

17 What is monitored now? Not Enough Not Enough CPU usage stats of host Network usage stats of host Intel Node Manager Power data Cache Qos Monitoring(CQM) data …… Ceilometer no hardware pollsters Nova not easy to add how to use?

18 What are missing? Policy management Break policy into QoS parameter Mapping QoS parameter to metrics Actions Live migration Resource reallocation Enforcement … … Knowledge model to evaluate complex policy situations(e.g. predict future VM workload)

19 Dynamic Resource Scheduling Policy Ceilometer Nova collector Ceilometer collector Other agents Other collectors Pluggable Executors Logging resource reallocation Alarming Evaluating Enforcement Live migration De-virtualizing Benchmarking Evaluator Parser Analyzer Historic metrics data Pluggable Collectors Other actions admins API Existing components To be implemented Knowledge model Nova set alarm trigger

20 Next: Gantt Scheduler-as-a-Service project Split from Nova first, then for other projects Plan to split begin from L

21 Gantt in Kilo: Refactor, Refactor, Refactor…. The Scheduler before Juno API SchedulerCompute The scheduler in Kilo Conductor API Scheduler Compute 1.User request and with scheduler hints to include scheduling policy 2. Submit new task 3. Request host that match the request_spec and filter_properties 4. Returns selected hosts 5. Call the selected compute 6. Rescheduling after claim resource failed or other failure Scheduler API: select_destinations update_resource_stats

22 Refactor https://blueprints.launchpad.net/nova/+spec/make-resource-tracker-use-objects https://blueprints.launchpad.net/nova/+spec/detach-service-from-computenode https://blueprints.launchpad.net/nova/+spec/resource-objects https://blueprints.launchpad.net/nova/+spec/request-spec-object https://blueprints.launchpad.net/nova/+spec/sched-select-destinations-use-request-spec- object https://blueprints.launchpad.net/nova/+spec/sched-select-destinations-use-request-spec- object https://blueprints.launchpad.net/nova/+spec/isolate-scheduler-db

23 Thanks

24 Backup

25 The problem of current Nova scheduler Server Group Can’t add/remove active server to/from server-group https://review.openstack.org/ https://review.openstack.org/ With affinity policy means you can’t evacuate Ignore down host when populate the instance: https://review.openstack.org/#/c/135607/https://review.openstack.org/#/c/135607/ Remove the instance from server group: https://review.openstack.org/136487, but won’t land in K, maybe L. It also won’t work for something automatic HAhttps://review.openstack.org/ https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/soft-affinity-for-server-group,n,z Anti-affinity policy race problem, may trigger extra rescheduling Race for migration Support unshelve, rebuild, live-migration, migration, resize in K….but not resolve the anti-affinity policy problem. Unshelve: https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bug/ ,n,z Rebuild: https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:rebuild_schedule,n,zhttps://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:rebuild_schedule,n,z Migration/live-migration on going…

26 The problem of current Nova scheduler Missing resource claiming and retry for migration Unshelve: https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bug/ ,n,z Rebuild: https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:rebuild_schedule,n,zhttps://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:rebuild_schedule,n,z Migration/live-migration on going… Scheduling-hints can’t persist You only can specific your scheduling policy at the beginning Violate the policy after migration https://review.openstack.org/88983 block in K, maybe L https://review.openstack.org/88983 Race Problem the bug link https://bugs.launchpad.net/nova/+bug/ https://bugs.launchpad.net/nova/+bug/ scheduler_host_subset_size=N Ironic integration https://bugs.launchpad.net/nova/+bug/

27 Any more problem for scheduler? Only do initial placement! Each project have own scheduler

28 DRS in Openstack Gantt Tetris https://docs.google.com/document/d/1DMsnGxQ3P- OwZCF3uxaUeEFaKX8LqUqmmgQ_7EVK7Y8/edithttps://docs.google.com/document/d/1DMsnGxQ3P- OwZCF3uxaUeEFaKX8LqUqmmgQ_7EVK7Y8/edit Purview(Tetris) will provide framework to quickly implement and enforce different kinds of policies. Policies can be different types. Here are a few examples of policies in clouds: Availability Policies, Performance Policies, Load balancing Policy, User Defined Policy. Congress https://wiki.openstack.org/wiki/Congresshttps://wiki.openstack.org/wiki/Congress Congress is a policy-based management framework for the cloud. It is designed to work with any cloud software that reasonably fits within the relational data model. It automatically prevents policy violations when possible and corrects them when not, and it enables administrators to control the extent to which enforcement is automatic Tetris is domain-specific policy system Congress is domain-independent policy system domain-independent and domain-specific policy systems are highly complementary


Download ppt "Nova Scheduler Shane Wang (王庆), Intel Open Source Technology Center 微信号: qq559382."

Similar presentations


Ads by Google