24 February 2015 Ryota Mibu, NEC Fault Management (Doctor) 24 February 2015 Ryota Mibu, NEC

Questions How big are the telco systems? How many applications and VMs will be run on NFV platform? How many physical machines will be deployed and managed? How could you manage faults to keep all critical services up? 24 February 2015 OPNFV Prague Hackfest

Use Case 24 February 2015 OPNFV Prague Hackfest

High Level Architecture Applications VIM User and Administrator App App App Virtualized Infrastructure Virtualized Infrastructure Manager (VIM) = OpenStack Virtual Compute Virtual Storage Virtual Network Virtualization Layer Hardware Resources 24 February 2015 OPNFV Prague Hackfest

Fault Management Sequence Detection Reaction Applications VIM User and Administrator App App App Virtualized Infrastructure Virtualized Infrastructure Manager (VIM) = OpenStack Virtual Compute Virtual Storage Virtual Network Virtualization Layer Hardware Resources Doctor Initial Focus 24 February 2015 OPNFV Prague Hackfest

Key Requirements as VIM Immediate notification to VIM user and administrator Fault notification of affected virtual resources Configurable notification by VIM admin and user Catch all faults in NFVI 24 February 2015 OPNFV Prague Hackfest

TO-BE: Functional Blocks Applications VIM User and Administrator App App App Virtualized Infrastructure VIM Virtual Compute Virtual Storage Virtual Network Controller Notifier Virtualization Layer Monitor Inspector Hardware Resources 24 February 2015 OPNFV Prague Hackfest

Fault Management Scenarios (1/2) Applications 0. Set Alarm User-side Manager Admin-side Manager 6-. Action 5. Notify Error Virtualized Infrastructure 4. Notify all Controller Controller Notifier Controller Resource Map Alarm Conf. 3. Update State 2. Find Affected 4. (alt) Notify Monitor Monitor Inspector Monitor Failure Policy 1. Raw Failure 24 February 2015 OPNFV Prague Hackfest

Fault Management Scenarios (2/2) Applications User-side Manager Admin-side Manager 0. Set Alarm 6-. Action 5. Notify Error Virtualized Infrastructure 4. Notify all Controller Controller Notifier Controller Resource Map Alarm Conf. 3. Update State 2. Find Affected 4. (alt) Notify Monitor Monitor Inspector Monitor Failure Policy 1. Raw Failure 24 February 2015 OPNFV Prague Hackfest

AS-IS: OpenStack Juno (1/3) How can you find faults as a tenant user? Keep-a-live check to each VM Polling VM state to Nova API Set alarm on metering service 24 February 2015 OPNFV Prague Hackfest

AS-IS: OpenStack Juno (2/3) How does the metering service work? Resource controller such as Nova monitors usage of resource [Periodically] Get samples from resource controller and register them to DB [Periodically] Evaluate alarm definition on samples [Periodically] Raise alarm depend on result of the evaluation VM Hypervisor Nova Ceilometer (Heat) 1. 2. 4. 3. Machine Samples 24 February 2015 OPNFV Prague Hackfest

AS-IS: OpenStack Juno (3/3) Notification OpenStack components post events to messaging queue Ceilometer collects, transform and publish those events which can be used for billing NFVI Nova Queue Neutron Ceilometer (Billing) Cinder Samples 24 February 2015 OPNFV Prague Hackfest

Implementation Plan in OpenStack Applications VIM User and Administrator Virtualized Infrastructure Queue Alarm Set Notifier Nova Immediate Notificatioin Error Injection Zabbix Inspector Plugin ? 24 February 2015 OPNFV Prague Hackfest

Demo (1/3) User Scenario Public Net Private Net Web Server HTTP Clients HTTP Clients Load Balancer Web Server HTTP Clients Web Server Launch New VM Public Net Private Net 24 February 2015 OPNFV Prague Hackfest

Demo (2/3) Demo 1 Demo 2 3. Create New Web Server VM Ceilometer Hypervisor Nova Agent Alarm (Heat) Machine 1. Collect CPU time samples 2. Alarm Heat if CPU runtime = 0 Samples VM Ceilometer Hypervisor Nova Agent Alarm (Heat) 2. Notify as Event Machine 1. Hook 3. Alarm Heat 24 February 2015 OPNFV Prague Hackfest

Scenario 1: Ceilometer Normal Demo 1 Demo 2 90 sec 26 sec 24 February 2015 OPNFV Prague Hackfest

Backup 24 February 2015 OPNFV Prague Hackfest

Fault Management Sequence (Optional) Detection Reaction Applications VIM User and Administrator App App App Virtualized Infrastructure Virtualized Infrastructure Manager (VIM) = OpenStack Auto Reaction Virtual Compute Virtual Storage Virtual Network Virtualization Layer Hardware Resources 24 February 2015 OPNFV Prague Hackfest

Fault Management Scenarios (Optional) Applications 0. Set Alarm User-side Manager Admin-side Manager 5. Notify Error 6-. Action Auto Reaction Virtualized Infrastructure 4. Notify all Controller Controller Notifier Controller Resource Map Alarm Conf. 3. Update State 2. Find Affected 4. (alt) Notify Monitor Monitor Inspector Monitor Failure Policy 1. Raw Failure 24 February 2015 OPNFV Prague Hackfest