Presentation is loading. Please wait.

Presentation is loading. Please wait.

Nova Scheduler Shane Wang(王庆), Intel Open Source Technology Center

Similar presentations


Presentation on theme: "Nova Scheduler Shane Wang(王庆), Intel Open Source Technology Center"— Presentation transcript:

1 Nova Scheduler Shane Wang(王庆), Intel Open Source Technology Center
微信号:qq559382

2 Agenda What is current situation? What is next plan?
How scheduler works in Juno and Kilo Resource Tracking Filters and Weight Utilization Based Scheduling (UBS) What is next plan? Gantt Dynamic Resource Scheduling (DRS)

3 How scheduler works in Juno and Kilo
3. Request host that match the request_spec and filter_properties 4. Returns selected hosts User request and with scheduler hints to include scheduling policy API Conductor Compute 6. Rescheduling after claim resource failed or other failure 2. Submit new task 5. Call the selected compute

4 Resource usage Tracking
Scheduler 1) Fetch newest compute node stats for each call 2) Filter and weight the host 3) Consuming the resource for selected host 3. Request host that match the request_spec and filter_properties Resource Claiming 1) Validate the resource usage 2) Update the resource Usage 3) Update to DB 4. Returns selected hosts Conductor Compute 6. Rescheduling after claim resource failed or other failure 2. Submit new task Hypervisor Hypervisor Hypervisor 5. Call the selected compute Periodically update the node resource with 60 seconds interval 1) Get hypervisor resource 2) Consuming the resource 3) Update to DB DB

5 Filters and weight hosts
scheduler_available_filters='nova.scheduler.filters.all_filters‘ scheduler_default_filters= [……] scheduler_weight_classes=nova.scheduler.weights.all_weighers Request Spec: Image Instance_properties Instance_type scheduler_host_subset_size=1 Filter_properties Scheduler-hints Assist parameter: retry Nova boot –flavor 1 –image …… --hint group=‘sg1’ --hint <key=value> Send arbitrary key/value pairs to the scheduler for custom use.

6 Filters Resource: CoreFilter AggregateCoreFilter: cpu_allocation_ratio=16.0 RamFilter AggregateRamFilter: ram_allocation_ratio=1.5 DiskFilter AggregateDiskFilter: disk_allocation_ratio=1.0 IoOpsFilter AggregateIoOpsFilter: max_io_ops_per_host=8. IoOps means resize, building, image snaphsot. Migration, rescues, unshelve, backup PciPassthroughFilter: Generic PCI device or SRIOV assignment NUMATopologyFilter: NUMA in J, CPUPinning, Hugepage in K

7 Filters Affinity: DifferentHostFilter, SameHostFilter: scheduler_hints: different_host/ same_host =[‘instance uuid’…] ServerGroupAffnityFilter, ServerGroupAntiAffinityFilter: nova server-group-create Create a new server group with the specified details. nova server-group-delete Delete specific server group(s). nova server-group-get Get a specific server group. nova server-group-list Print a list of all server groups. boot with scheduler-hints: group=uuid Boot new instance into server group SimpleCIDRAffinityFilter: scheduler_hints: cidr, build_near_host_ip TypeAffinityFilter, AggregateTypeAffinityFilter: instance_type

8 Filters Topology: AggregateImagePropertiesIsolation: image properties matchs aggregate metadata IsolatedHostsFilter: isolated_hosts, isolated_images, restrict_isolated_hosts_to_isolated_images AggregateInstanceExtraSpecsFilter: Flavor’s extra spec match aggregate metadata AggregateMultiTenancyIsolation: filter_tenant_id AvailabilityZoneFilter

9 Filters Others: ComputeCapabilitiesFilter: work with instance type extra_spec: ‘capabilities:’ ComputeFilter: The compute node is live or disabled ImagePropertiesFilter: architecture, hypervisor type, vm_mode, hypervisor_version_requires JsonFilter: scheduler_hints:query NumInstancesFilter, AggregateNumInstancesFilter, max_instances_per_host RetryFilter TrustedFilter

10 Weight IoOpsWeigher MetricsWeigher RAMWeigher

11 Utilization Based Scheduling
CPU Utilization data Memory Utilization data Network Bandwidth data etc

12 Utilization Based Scheduling
Scheduler 1) Fetch newest compute node stats for each call 2) Filter and weight the host 3) Consuming the resource for selected host 3. Request host that match the request_spec and filter_properties Update 60 seconds interval 4. Returns selected hosts Conductor CPU Monitor Compute 6. Rescheduling after claim resource failed or other failure 2. Submit new task Hypervisor NetworkBandWidth Hypervisor Hypervisor 5. Call the selected compute MemoryCache Monitor DB Notification Bus AMQP

13 Utilization Based Scheduling
MetricsWeigher: weight_multiplier: Multiplier used for weighing metrics. weight_setting: How the metrics are going to be weighed. Required: If true, use the MetricsFilter weight_of_unavailable

14 How scheduler strategy affects performance?

15 Benchmark Accuracy

16 Smart Scheduling Efficiency QoS meet SLA contract

17 What is monitored now? Nova Not Enough Ceilometer OpenStack Service
not easy to add how to use? OpenStack Service Type Metrics (e.g.) Nova Static capabilities CPU features hypervisor version Dynamic Resources free memory/disk vCPU # PCI devices # of NIC virtual functions Ceilometer Resources creation/deletion VM network/subnet/port image …… Resources usage data CPU usage in VM memory usage in VM network usage in VM storage usage stats CPU usage stats of host Network usage stats of host Intel Node Manager Power data Cache Qos Monitoring(CQM) data …… Not Enough Ceilometer no hardware pollsters

18 What are missing? Policy management Actions
Break policy into QoS parameter Mapping QoS parameter to metrics Actions Live migration Resource reallocation Enforcement … … Knowledge model to evaluate complex policy situations(e.g. predict future VM workload)

19 Dynamic Resource Scheduling
Existing components To be implemented Knowledge model Policy API admins Pluggable Executors Logging Alarming Evaluator Evaluating Parser Historic metrics data Enforcement Pluggable Collectors Analyzer resource reallocation Other collectors alarm trigger Live migration set alarm De-virtualizing Ceilometer collector Nova collector Other agents Benchmarking API API API Other actions Ceilometer Nova

20 Next: Gantt Scheduler-as-a-Service project
Split from Nova first, then for other projects Plan to split begin from L

21 Gantt in Kilo: Refactor, Refactor, Refactor….
The Scheduler before Juno API Scheduler Compute Scheduler The scheduler in Kilo Scheduler API: select_destinations update_resource_stats 3. Request host that match the request_spec and filter_properties 4. Returns selected hosts User request and with scheduler hints to include scheduling policy API Conductor Compute 6. Rescheduling after claim resource failed or other failure 2. Submit new task 5. Call the selected compute

22 Refactor https://blueprints.launchpad.net/nova/+spec/make-resource-tracker-use-objects https://blueprints.launchpad.net/nova/+spec/detach-service-from-computenode https://blueprints.launchpad.net/nova/+spec/resource-objects https://blueprints.launchpad.net/nova/+spec/request-spec-object https://blueprints.launchpad.net/nova/+spec/sched-select-destinations-use-request-spec- object https://blueprints.launchpad.net/nova/+spec/isolate-scheduler-db

23 Thanks

24 Backup

25 The problem of current Nova scheduler
Server Group Can’t add/remove active server to/from server-group https://review.openstack.org/136487 https://review.openstack.org/139272 With affinity policy means you can’t evacuate Ignore down host when populate the instance: https://review.openstack.org/#/c/135607/ Remove the instance from server group: https://review.openstack.org/136487, but won’t land in K, maybe L. It also won’t work for something automatic HA https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/soft-affinity-for-server-group,n,z Anti-affinity policy race problem, may trigger extra rescheduling Race for migration Support unshelve, rebuild, live-migration, migration, resize in K….but not resolve the anti-affinity policy problem. Unshelve: https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bug/ ,n,z Rebuild: https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:rebuild_schedule,n,z Migration/live-migration on going…

26 The problem of current Nova scheduler
Missing resource claiming and retry for migration Unshelve: https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bug/ ,n,z Rebuild: https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:rebuild_schedule,n,z Migration/live-migration on going… Scheduling-hints can’t persist You only can specific your scheduling policy at the beginning Violate the policy after migration https://review.openstack.org/88983 block in K, maybe L Race Problem the bug link https://bugs.launchpad.net/nova/+bug/ scheduler_host_subset_size=N Ironic integration https://bugs.launchpad.net/nova/+bug/

27 Any more problem for scheduler?
Only do initial placement! Each project have own scheduler

28 DRS in Openstack Gantt Tetris https://docs.google.com/document/d/1DMsnGxQ3P- OwZCF3uxaUeEFaKX8LqUqmmgQ_7EVK7Y8/edit Purview(Tetris) will provide framework to quickly implement and enforce different kinds of policies. Policies can be different types. Here are a few examples of policies in clouds: Availability Policies, Performance Policies, Load balancing Policy, User Defined Policy. Congress https://wiki.openstack.org/wiki/Congress Congress is a policy-based management framework for the cloud. It is designed to work with any cloud software that reasonably fits within the relational data model. It automatically prevents policy violations when possible and corrects them when not, and it enables administrators to control the extent to which enforcement is automatic Tetris is domain-specific policy system Congress is domain-independent policy system domain-independent and domain-specific policy systems are highly complementary


Download ppt "Nova Scheduler Shane Wang(王庆), Intel Open Source Technology Center"

Similar presentations


Ads by Google