1. Doctor Tech Deep Dive Tomi Juvonen, Nokia Ryota Mibu, NEC

2. https://wiki.opnfv.org/display/doctor/

3. Fault Management Flow Detection Reaction Virtualized Infrastructure Applications Application Manager (VIM User) Virtualized Infrastructure Manager (VIM) Virtual Compute Virtual Storage Virtual Network Virtualization Layer Hardware Resources App Our Focus

4. Fault Management Architecture 4 Monitor Notifier Manager Virtualized Infrastructure Alarm Conf. 3. Update State 2. Find Affected Applications Controller Resource Map 1. Raw Fault Inspector 4. Notify all 4. (alt) Notify 5. Notify Error 0. Set Alarm 6-. Action Failure Policy Monitor Cinder Neutron Nova Ceilometer+Aodh Vitrage Congress Designed by Doctor Project

5. Challenges in OpenStack Introducing Fault Management sequence across multiple OpenStack projects Letting OpenStack users know corresponding resource state properly/immediately Building correlation mechanism to support various OpenStack deployment flavors and operator policies

6. Blueprints in Liberty/Mitaka Cycles ProjectBlueprintSpec DrafterDeveloperStatus Ceilometer /Aodh Event Alarm Evaluator Ryota Mibu (NEC) Completed (Liberty) Nova New nova API call to mark nova- compute down Tomi Juvonen (Nokia) Roman Dobosz (Intel) Completed (Liberty) Support forcing service down Tomi Juvonen (Nokia) Carlos Goncalves (NEC) Completed (Liberty) Get valid server state Tomi Juvonen (Nokia) Completed (Mitaka) Add notification for service status change Balazs Gibizer (Ericsson) Completed (Mitaka) CongressPush Type DataSource Driver Masahito Muroi (NTT) Completed (Mitaka)

7. Nova – Objects Server (VM) Service / Host Nova Compute Process on Host Nova API/DB User Enable/Disable Force Down Reset State Report Server(Instance): vm_state, … Service: status, enable, force_down, … External Monitoring Service Admin Boot/Shutdown Fencing Exec (e.g. Evacuate)

8. Nova – Force-Down & Exposing Service State Host / Machine Hypervisor VM nova compute nova api nova conductor nova scheduler nova DB queue External Monitoring Service vSwitch BMC EXISTING (periodic update) Force-down API [Mark Nova-Compute Down] Notifying that the nova service is no longer available Client Service disable API Evacuation API Reset Server State API Allows Nova to integrate with External Monitoring Services, and make sure Nova handles requests for the host properly service state service.update notification

9. Nova – Force-Down API [centos@controller ~]$ curl controller -g -i -X PUT http://localhost:8774/v2.1/$tenant_id/os-services/force-down -H "X-OpenStack- Nova-API-Version: 2.11" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: $OS_AUTH_TOKEN" -d '{"binary": "nova- compute", "host": "compute2", "forced_down": true}' HTTP/1.1 200 OK Content-Length: 80 Content-Type: application/json X-Openstack-Nova-Api-Version: 2.11 Vary: X-OpenStack-Nova-API-Version X-Compute-Request-Id: req-a92108ee-0cd4-487e-8beb-c6b5b55a819c Date: Thu, 09 Jun 2016 10:52:02 GMT} {"service": {"binary": "nova-compute", "host": "compute2", "forced_down": true}}

10. Nova – Force-Down Command [centos@controller ~]$nova help service-force-down usage: nova service-force-down [--unset] Force service to down. (Supported by API versions '2.11' - '2.latest') [hint: use '--os-compute-api-version' flag to show help message for proper version] Positional arguments: Name of host. Service binary. Optional arguments: --unset Unset the force state down of service. [centos@controller ~]$nova service-force-down --unset compute2 nova-compute +----------+--------------+-------------+ | Host | Binary | Forced down | +----------+--------------+-------------+ | compute2 | nova-compute | False | +----------+--------------+-------------+

11. Ceilometer/Adoh – Event Alarm sample Notification-driven alarm evaluator NEW Shortcut (notification-based) EXISTING (polling-based) Manager Audit Service stats notification event Cinder Neutron Nova Aodh

12. Ceilometer/Aodh – Create Event Alarm POST http:// :8777/v2/alarms/ { "type": "event“, "name": "Standard test2: InstanceStatusAlarm", "description": "Standard test: an event alarm", "enabled": true, “alarm_actions”: [ "http:// :8000/alarm" ], "repeat_actions": false, "severity": "moderate", "event_rule": { "event_type": "compute.instance.update", "query" : [ { "field" : "traits.instance_id", "type" : "string", "value" : "c1b55d43-9cd9-49ba-ba41-1dc5628ddfe7", "op" : "eq“ }, { "field" : "traits.state", "type" : "string", "value" : "stopped", "op" : "eq“ } ] }

13. Aodh – Alarm Notification (1/3) POST http:// :8000/alarm { "alarm_id": "718a79b5-f066-4f60-aa37-df0061536e02", "alarm_name": "Standard test: InstanceStatusAlarm", "current": "alarm", "previous": "insufficient data", "reason": "Event (message_id=cbe38285-0c16-490a-bc18-12a1292f03ab) hit the query of alarm (id=718a79b5- f066-4f60-aa37-df0061536e02)", "reason_data": { "event": { "event_type": "compute.instance.update", "generated": "2015-11-17T06:22:32.977348", "message_id": "cbe38285-0c16-490a-bc18-12a1292f03ab", "message_signature": "938198b933f640560a0d88a0f72739d95a21bb6a362b12fe026ba847c46303f7",

14. Aodh – Alarm Notification (2/3) "raw": {}, "traits": [ [ "state", 1, "stopped" ], [ "user_id“, 1, "47de9baced284660b63a7ee6c218f058" ], [ "service", 1, "compute" ], [ "disk_gb", 2, 20 ], [ "instance_type", 1, "m1.small" ], [ "tenant_id", 1, "abf09656727c4154abcdc23e033d1f3b" ], [ "root_gb", 2, 20 ], [ "ephemeral_gb", 2, 0 ], [ "instance_type_id“, 2, 5 ], [ "vcpus", 2, 1 ], [ "memory_mb", 2, 2048 ], [ "instance_id", 1, "415f624b-9a4e-4448-bce0-05ee44581a81“ ],

15. Aodh – Alarm Notification (3/3) [ "host“, 1, "opnfv-demo-controller" ], [ "request_id", 1, "req-410449bc-c9a3-4fa8-9879-807d9479d15a" ], [ "project_id", 1, "abf09656727c4154abcdc23e033d1f3b" ], [ "launched_at", 4, "2015-11-10T00:09:07" ] ] }, "type": "event" }, "severity": "moderate" }

16. Nova – Get valid host state Host / Machine ‘host_status’ can have values: UP if nova-compute is up. UNKNOWN if nova-compute not reported by service group driver. DOWN if nova-compute is forced down. MAINTENANCE if nova-compute is disabled. Empty string indicates there is no host for server. nova API nova DB Force-down API Server API Server APIs to have ‘host_status’: GET /v2.1/ ​ {tenant_id} ​ /servers/detail GET /v2.1/ ​ {tenant_id} ​ /servers/ ​ {server_id} ​ nova compute nova conductor nova scheduler queue EXISTING (periodic update) Service disable API service state

17. Nova – Servers API with host_status [centos@controller ~]$ nova show vm1 +--------------------------------------+----------------------------------------------------------------+ | Property | Value | +--------------------------------------+----------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | compute2 | | OS-EXT-SRV-ATTR:hostname | vm1 | | OS-EXT-SRV-ATTR:hypervisor_hostname | compute2 |... | | hostId | 2ac4c517ae75cba421755a6af321a4910c83d763da85b23aa619039b | | host_status | UP | | id | 8c184ded-9d88-4f8f-9d2f-626596feede8 | | image | cirros-0.3.4-x86_64-uec (8edd0cdb-4c00-4373-94f6-93cb025d5a9c) |... +--------------------------------------+----------------------------------------------------------------+ Note! ’host_status’ is supported from API micro-version '2.16‘. The policy file needs to be configured to show ‘host_status’ to the admin and the server owner: "os_compute_api:servers:show:host_status": "rule:admin_or_owner"

18. Blueprints Under Work ProjectBlueprintSpec DrafterDeveloperStatus NovaMaintenance Reason To Server Tomi Juvonen (Nokia) Drafting (Ocata) This blueprint spec will continue review in OpenStack Ocata. Basic idea is that Nova project do not want more host specific functionality like maintenance, but it can have a link to external tool hosting more information. There will be new API to set this link and it will be available trough different APIs for user and admin. Also notification will be sent when link changes, that can be further more consumed externally and alarm can be then risen to tenant for example about coming maintenance effecting to his VMs. As spec deals with exposing new field to several APIs, it at the same time exposes ‘forced_down’ also to hypervisor API. This blueprint will be an enabler to implement maintenance part of Doctor requirements.

19. Neutron – Objects Port Agent Agent Process on Compute Host Neutron Server User Enable/Disable Force Down Update status Report External Monitoring Service Admin Up/Down Port: status, admin_state, … Agent: alive, admin_state, force_down?, Notification Fencing Note: Under Discussion

20. Doctor Wiki: https://wiki.opnfv.org/display/doctorhttps://wiki.opnfv.org/display/doctor Visit Doctor Booth! OPNFV Doctor OpenStack Congress