Doctor + OPenStack Congress NTT DOCOMO, NEC, Intel

PoC Demo Description Doctor: fast and dynamic fault management in OpenStack (DOCOMO, NTT, NEC, Nokia, Intel) Doctor is an OPNFV project implementing fault management framework for high service availability in OpenStack. In this framework, there are two choices for handling fault events which will be notified to users immediately, by OpenStack Congress and OpenStack Vitrage. Congress is the Policy Evaluation Engine for enforcing flexible and dynamic failure identification policy defined by an Operators demands. Vitrage is a new Root Cause Analysis Engine for organizing, analyzing and expanding OpenStack alarms & events, yielding insights regarding the root cause of the problems, and deducing the existence of faults before they are directly detected. This PoC shows how fast fault recovery is performed by using these two options, thereby ensuring the required service availability of telecom nodes. http://events.linuxfoundation.org/events/opnfv-summit/extend-the-experience/opnfv-poc-zone

API extension for Port state update Demo Scenario Application Manager 1. Set Alarm on Port event 7. Notify Error Virtualized Infrastructure (Resource Pool) Neutron 6. Notify all Controller Aodh Ceilometer Controller Resource Map Alarm Conf. API extension for Port state update VM0 VM1 VM2 5. Update State 4. Find Affected Link Monitor Script Port0 Port1 Port2 Monitor Doctor Driver Congress Monitor Failure Policy Bridge Bridge 2. Monitor 3. Notify Raw Failure NIC0 NIC1 NIC2 Bonding

Notification Logic for Demonstration Setup Make Congress fetches Neutron Port info (vif_type and hostname) periodically Create Ceilometer/Aodh Alarm definition, specifying notification URI of the App manager, Neutron Port ID and context of port update event (Trigger Error) The monitor gets NIC status from the compute host If an fault observed, The monitor report the fault event (as nic down) to Congress Congress (Policy engine) evaluates policy with received event and find effected ports Congress sends request to update status of effected ports to down (using Neutron driver) Neutron updates status of the ports and notify these port updates to Ceilometer/Aodh Ceilometer/Aodh fires the alarm notifications of these port failures to Manager

Difference from the PoCs so far Alarming “Single Point of failure” Failure mapping in logical resource view @ OPNVF Summit 2015, OpenStack Summit Austin Host (Hypervisor) Server (VM) Port (vNIC) Network @ OPNVF Summit 2016

Configurations Policies in Congress Alarm Definition in Aodh nic_down(host, physnet) :- doctor:events(hostname=host, physical_network=physnet, type="host.nic1.down") nic_down(host, physnet) :- doctor:events(hostname=host, physical_network=physnet, type="host.nic2.down") execute[neutronv2:force_down_port(port)] :- neutron:ports(id=port, host_id=host, network_id=net), neutronv2:networks(id=net, physical_network="default"), nic_down(hostname=host, physical_network=physnet) Alarm Definition in Aodh aodh alarm create -t event --name "NICFailureAlarm" --event-type port.update.end \ --query "traits.forced_down=string::True;traits.resource_id=string::<Neutron Port ID>” \ --alarm-action <URI to notify> (…)