2. Doctor + OPenStack Congress
NTT DOCOMO, NEC, Intel

3. PoC Demo Description
Doctor: fast and dynamic fault management in OpenStack (DOCOMO, NTT, NEC, Nokia, Intel)
Doctor is an OPNFV project implementing fault management framework for high service availability in OpenStack. In this framework, there are two choices for handling fault events which will be notified to users immediately, by OpenStack Congress and OpenStack Vitrage.
Congress is the Policy Evaluation Engine for enforcing flexible and dynamic failure identification policy defined by an Operators demands.
Vitrage is a new Root Cause Analysis Engine for organizing, analyzing and expanding OpenStack alarms & events, yielding insights regarding the root cause of the problems, and deducing the existence of faults before they are directly detected.
This PoC shows how fast fault recovery is performed by using these two options, thereby ensuring the required service availability of telecom nodes.

4. API extension for Port state update
Demo Scenario
Application
Manager
1. Set Alarm on Port event
7. Notify Error
Virtualized Infrastructure
(Resource Pool)
Neutron
6. Notify all
Controller
Aodh Ceilometer
Controller
Resource Map
Alarm
Conf.
API extension for Port state update
VM0
VM1
VM2
5. Update State
4. Find Affected
Link Monitor Script
Port0
Port1
Port2
Monitor
Doctor Driver
Congress
Monitor
Failure Policy
Bridge
Bridge
2. Monitor
3. Notify Raw Failure
NIC0
NIC1
NIC2
Bonding

5. Notification Logic for Demonstration
Setup
Make Congress fetches Neutron Port info (vif_type and hostname) periodically
Create Ceilometer/Aodh Alarm definition, specifying notification URI of the App manager, Neutron Port ID and context of port update event
(Trigger Error)
The monitor gets NIC status from the compute host
If an fault observed, The monitor report the fault event (as nic down) to Congress
Congress (Policy engine) evaluates policy with received event and find effected ports
Congress sends request to update status of effected ports to down (using Neutron driver)
Neutron updates status of the ports and notify these port updates to Ceilometer/Aodh
Ceilometer/Aodh fires the alarm notifications of these port failures to Manager

6. Difference from the PoCs so far
Alarming “Single Point of failure”
Failure mapping in logical resource view
@ OPNVF Summit 2015, OpenStack Summit Austin
Host
(Hypervisor)
Server
(VM)
Port
(vNIC)
Network
@ OPNVF Summit 2016

7. Configurations Policies in Congress Alarm Definition in Aodh
nic_down(host, physnet) : doctor:events(hostname=host, physical_network=physnet, type="host.nic1.down")
nic_down(host, physnet) : doctor:events(hostname=host, physical_network=physnet, type="host.nic2.down")
execute[neutronv2:force_down_port(port)] : neutron:ports(id=port, host_id=host, network_id=net), neutronv2:networks(id=net, physical_network="default"), nic_down(hostname=host, physical_network=physnet)
Alarm Definition in Aodh
aodh alarm create -t event --name "NICFailureAlarm" --event-type port.update.end \ --query "traits.forced_down=string::True;traits.resource_id=string::<Neutron Port ID>” \ --alarm-action <URI to notify> (…)