1. Tomi Juvonen SW Architect, Nokia
Doctor
OPNFV Finland meetup
Tomi Juvonen SW Architect, Nokia

2. Doctor Project Fault management and maintenance
One of the first projects
Started as requirement project
Feature project
Perfect example of how to work to upstream trough OPNFV
Virtualized Infrastructure
Applications
VIM User and Administrator
Virtualized Infrastructure Manager (VIM)
= OpenStack
Virtual Compute
Virtual Storage
Virtual Network
Virtualization Layer
Hardware Resources
App
Doctor Scope

3. Doctor Achievements 2014 2015 2016 2017 2018 Design
Requirement Document
Figured out basic use cases and minimal requirements in fault management
To-be-architecture
Implementation
Open Source Project Mapping
Key features are available in OpenStack
Gap Analysis
work in Upstream (OpenStack)
Good example of how to upstream
Integration and testing
Functest / Installer
OPNFV users can test fault management scenario quickly with performance profile
Doctor CI
Performance test support
Profiler
Project graduation
2015
2016
2017
2018
One of the first projects

4. Doctor typical flow
One of the first projects

5. Demo @ OpenStack Summit Barcelona (October 2016) https://youtu
vEPC Failover keeping phone call session online

6. Assumption of VNF (NFV Application)
Telco Applications are generally deployed in active-standby or active-active fashion
App state needs to be switched when failure occurs
App (Active)
App (Standby)
App and App Manager (VNFM) cannot detect HW failures directly VM VM
Machine
Machine

7. X Use Case 1: Fault management V Consumer C1 Consumer C2 Consumer C3
4. Switch to SBY configuration
Consumer C1
Consumer C2
Consumer C3
3. Fault Notification
(VM ID, Fault ID)
5. Instruction
(VM ID)
OpenStack Northbound Interface
2. Inform the Consumer?
If YES, find owner of affected VMs from database
Virtualized Infrastructure Manager (VIM), e.g. OpenStack
VM-1
VM-2
VM-7
VM-4
Resource Map
Server – VM mapping
Server S1
VM-1, VM-2
Server S2
VM-7
Server S3
VM-4
6. Execute Instruction
- e.g. migrate VM
Ownership information
VM-1, VM-7
Consumer C1
VM-2
Consumer C2
VM-4
Consumer C3
Resource Pool
Hypervisor
Hypervisor
Hypervisor
Hardware Server S1
Hardware Server S2
Hardware Server S3 X
1. Fault Monitoring
- Hardware fault
- Hypervisor fault
- Host OS fault

8. Use Case 2: Maintenance V Administrator Consumer C1 Consumer C2
4. Switch to SBY configuration V
Administrator
Consumer C1
Consumer C2
Consumer C3
3. Maintenance Notification
(VM ID)
1. Maintenance Request (Server S3)
5. Instruction
(VM ID)
OpenStack Northbound Interface
Virtualized Infrastructure Manager (VIM), e.g. OpenStack
VM-1
VM-2
VM-7
VM-4
6. Execute Instruction
- e.g. migrate VM
Resource Map
Server – VM mapping
Server S1
VM-1, VM-2
Server S2
VM-7
Server S3
VM-4
Resource Pool
Hypervisor
Hypervisor
Hypervisor
Ownership information
VM-1, VM-7
Consumer C1
VM-2
Consumer C2
VM-4
Consumer C3
Hardware Server S1
Hardware Server S2
Hardware Server S3
2. Which VMs are affected?
Find Consumer owning the VM(s) from the database.

9. Key Requirements as VIM
Consistent Resource State Awareness
Immediate Notification
Extensible Monitoring
Fault Correlation

10. Doctor Architecture and Typical Scenario
Application
0. Set Alarm
Manager
6-. Action
5. Notify Error
Virtualized Infrastructure (Resource Pool)
4. Notify all
Controller
Controller
Notifier
Controller
Resource Map
Alarm
Conf.
Consistent Resource State Awareness
Immediate Notification
3. Update State
2. Find Affected
4. Notify all (new faster way)
Monitor
Monitor
Inspector
Monitor
Failure Policy
Extensible Monitoring
1. Raw Failure
Fault Correlation

11. Virtualized Infrastructure (Resource Pool)
Doctor OSS Map
Application
0. Set Alarm
Manager
6-. Action
5. Notify Error
Virtualized Infrastructure (Resource Pool)
4. Notify all
Controller
Controller
Notifier
Nova
Controller
Resource Map
Alarm
Conf.
Neutron
Ceilometer /Aodh
Cinder
3. Update State
2. Find Affected
4. Notify all (new faster way)
Monitor
Monitor
Inspector
Monitor
Failure Policy
Zabbix
1. Raw Failure
Collectd
Vitrage
Congress

12. Analyzed Gaps and Development Items
Application
0. Set Alarm
Manager
6-. Action
5. Notify Error
State Correction
Event Alarm
Virtualized Infrastructure (Resource Pool)
4. Notify all
Controller
Controller
Notifier
Nova
Controller
Resource Map
Alarm
Conf.
Neutron
Ceilometer /Aodh
Cinder
3. Update State
2. Find Affected
4. Notify all (new faster way)
Monitor
Monitor
Inspector
Monitor
Failure Policy
Zabbix
Vitrage
Collectd
1. Raw Failure
Congress
Event-driven RCA

13. Doctor Blueprints in OpenStack
Project
Blueprint
Spec Drafter
Developer
Status
Ceilometer/Aodh
Event Alarm Evaluator
Ryota Mibu (NEC)
Completed (Liberty)
Nova
New nova API call to mark nova-compute down
Tomi Juvonen (Nokia)
Roman Dobosz (Intel)
Support forcing service down
Carlos Goncalves (NEC)
Get valid server state
Completed (Mitaka)
Add notification for service status change
Balazs Gibizer (Ericsson)
Congress
Push Type DataSource Driver
Masahito Muroi (NTT)
Adds Doctor Driver
Completed (Newton)
Neutron
Port data plane status
Completed (Pike)

14. Timeline Q4 Q1 2015 Q2 Q3 Q1 2016 Q1 2017 OPNFV launch Doctor creation
30 Sep, 2014
Doctor creation
2 Dec, 2014
Arno release
4 May, 2015
OPNFV Summit 2015
9 Nov, 2015
Brahmaputra release
1 Mar, 2016
OPNFV Summit 2016
12 Jun, 2016
Colorado release
16 Sep, 2016
OpenStack Summit Barcelona 2017
25 Oct, 2016
Danube release
4 Apr, 2017
Euphrates release
October 2017
ARNO
Requirement document
BRAHMAPUTRA
Ceilometer “Immediate Notification”
Nova “Mark Host Down”
Functional test cases
PoC demo at OPNFV Summit
Documentation updates
COLORADO
Nova: “Get valid server state” and “Add notification for service status change”
Integration of Congress as Doctor Inspector
Extended functional tests
PoC demo at OPNFV Summit and OpenStack Summit
Documentation updates
OPNFV Plugfest
DANUBE
Neutron “Port Status update”
Inspector design guidelines
Performance profiler
Documentation updates
OPNFV Plugfest
EUPHRATES
Congress: parallel policy action execution for faster fencing, notification and recovery
Maintenance specs
Code refactoring to Python
Collectd as Doctor Monitor

15. Next: NFVI Maintenance http://docs. opnfv
Admin
Application
Ack + Action
App Manager
Switch over
Schedule maintenance
Set Alarm
Cloud configuration DB
VM1
APP1
act
VM2
APP2
VM3
APP1
stdby
VM4
APP2
VM1
APP1
stdby
VM2
APP2
Maintenance alarm
Fleet management tool
Action
Craton
Virtualized Infrastructure (Resource Pool)
Physical host
Physical host
Physical host
Notifier
Workflow engine
???
Maintenance state notification
Alarm
Conf.
Ceilometer /Aodh
Maintenance
Empty
Inspector
Failure Policy
Controller
Resource Map
Migrate
Controller
Controller
Vitrage
Nova
Congress
Audit and maintenance workflow actions

16. Thanks!!!