1. Gerald Kunzmann, DOCOMO Carlos Goncalves, NEC Ryota Mibu, NEC
OPNFV Summit 2015
Doctor: Failure Detection and Notification for NFV
Gerald Kunzmann, DOCOMO
Carlos Goncalves, NEC
Ryota Mibu, NEC

2. Doctor Overview Goal Approach Status
Build fault management and maintenance framework
Approach
Identify requirement
Gap Analysis
Implementation work in Upstream (OpenStack)
Integration and testing
Status
Initial Requirement study, architecture design, Gap analysis : Done
Collaborative Development: On-going (3 merged Blueprints in OpenStack Liberty)
Standardization Sync: On-going (by NFV member efforts, joint meeting)

3. Key Requirements as VIM
Consistent Resource State Awareness
Immediate Notification
Extensible Monitoring
Fault Correlation

4. Doctor Demo Overview Quick Recovery Switch Act-Sby Application Manager
Streaming Server
Video Player
ACT
SBY
Virtualized Infrastructure
Virtualized Infrastructure Manager (VIM)
= OpenStack
VM-1 Down
Virtual Compute
Virtual Storage
Virtual Network
Reaction
Detection without Doctor (few minutes)
Detection with Doctor (1 second)
Virtualization Layer
Hardware Resources
Host-A Down

5. Fault Management Sequence
App Manager
+ Viewer
Application
0. Set Alarm
Streaming Server
Manager
6-. Action
5. Notify Error
Liberty
Virtualized Infrastructure (Resource Pool)
4. Notify all
Controller
Controller
Notifier
Controller
Resource Map
Alarm
Conf.
Nova
Ceilometer
3. Update State
2. Find Affected
Monitor
Monitor
Inspector
Monitor
Failure Policy
1. Raw Failure
Log Monitor
State Reflector

6. Service Healing Process
Alarm Notification
Host A
Host B
Control
VM9
VM0
VM1
App Manager
Streaming Server
Streaming Server
vNIC
vNIC
vSwitch
vSwitch
NIC
NIC
Data Flow (Before)
Video Player
Data Flow (After)
Switch

7. Demo Operation Console
Doctor Demo Screen
App Manager
Service Control
VM List
(Horizon)
Demo Operation Console
Video Player
(with Doctor)
App Manager
Event/Action Log
VM Egress Stats
(Zabbix)
Video Player
(without Doctor)

8. Doctor Demo

9. Doctor Blueprints in OpenStack Liberty Cycle
✓ Using in This Demo
Project
Blueprint
Spec Drafter
Developer
Status
Ceilometer
Event Alarm Evaluator
Ryota Mibu (NEC)
Completed (Liberty)
Nova
New nova API call to mark nova-compute down
Tomi Juvonen (Nokia)
Roman Dobosz (Intel)
Support forcing service down
Carlos Goncalves (NEC)
Get valid server state
Spec approved (Mitaka)
Add notification for service status change
Balazs Gibizer (Ericsson)
Waiting for spec approval (Mitaka) ✓ ✓ ✓

10. Doctor BP Detail: Nova – Mark Nova-Compute Down
External Monitoring Service
Client
Monitoring
NEW API
to update nova-compute
service state
Force-down API
Host / Machine
nova api VM
service state
nova compute
queue
nova conductor
nova DB
Hypervisor
EXISTING (periodic update)
vSwitch
nova scheduler
BMC

11. Doctor BP Detail: Ceilometer - Event Alarm
Nova
Neutron
Cinder
Manager
event
stats
sample
Notification-driven alarm evaluator
EXISTING (polling-based)
notification
NEW Shortcut (notification-based)
Audit Service

12. Who made this demo? Upstream OSS Community & Developer
OpenStack Contributors including Doctor Developers
OPNFV Doctor Team
Doctor contributors who worked on requirement study, gap analysis and implementation design
Doctor PoC Demo Team
NTT DOCOMO
NEC: Toshiaki Takahashi, Takahiro Suzuki, Ryuji Ishikawa, ...

13. Visit DOCOMO Booth, PoC Demo Zone