1. ABACUS & COSMOS Monitoring review
Frank Locci on behalf of ABACUS and COSMOS team BE-ICS,BE-CO - Technical Meeting:

2. Agenda ABACUS review outcomes COSMOS project overview and plan
Finding on existing solutions
Mandate and investigations
Review proposal
COSMOS project overview and plan
Project mandate
Approach and scope
Some technical aspects
Conclusions

3. ABACUS Proposal
Plug our systems to CollectD and ITMON services CO sysadmin needs to be aligned with IT solution Set up an Icinga2 monitoring solution for all other use cases Icinga2 covers our major requirements, collectD support, has CERN users community (ATLAS+CMS)
New paradigm: Each user/expert responsible for some monitoring will add the specific part over the monitoring core infrastructure (collect, visualization)
Let me start with the ABACUS conclusions
It propose to ...
... benefit from developements and central services
Finally It propose a new paradigm to delegate responsabilities of the specifc part to the user (...)

4. Our finding on the existing solution 1/2
Diamon
Moon
None of the tools satisfy all the requirements
Lot of feature duplication in different solutions
Actual situation it is not adequate for our infrastructure today
Too many tools with no integration
Not easy to be configured for new systems
Scaling up and support are bottleneck points
Some missing features
Auto-discovery, Downtime notif., self-monitoring, etc.
Lemon
Custom viewers & tools
Xymon
Kibana/ES
So what id we find out from …. ….
To display, organize, filter, compute the data
...
... even no longer tenable in term of maintenance
Meter (IT)
Spectrum MIB
RT metrics
rsyslog/ atop
Proprietary
tools (HP)
FESA/RDA

5. Our finding on the existing solution 2/2
Stacking of house-made solutions lead to complexity and specialization  difficult to operate and to maintain for single and centralized team
Need more modular and interconnected solutions  to decentralize services
...
… allow to redistribute the responsabilities depending on the domain and expertise level
In most of the cases the user himself can take resp. to implement the collection and visualization of the data over the big-data core services maintained by the monitoring team.
BIG-DATA CORE
Data processing
Aggregation
Reduction
History logs
Storage
SOURCES
VIZUALISATION
Data collection
Formatting
Ingestion Transport
Alerting
Searching
Tables, Tree, Graphs,
Dashboards
user
user
monitoring teams
Open Source market extremely rich in the domain of monitoring

6. Abacus Mandate (Jun 2016) Gather and document the user requirements.
Identify and document the existing solutions
Organize the monitoring review meeting with external reviewers
Perform a market survey to understand the available solutions
Deliver a sound organizational proposal to setup the realization project
This proposal has then to be approved and staffed by the SLM.
So, here is the ABACUS mandate quickly:
...
considering the major components of the current monitoring system
BE-ICS is particular
and potential candidates

7. People Core members Reviewers Collaborators
. Felix Nikolaus Ehm . Luigi Gallerani . Joel Lauener . Frank Locci
Reviewers
. Marc Buttner . Brice Copy . Stephane Deghaye . Chris Roderick . Marc Vanden Eynden
Collaborators
. Marine Gourber-Pace . Fernando Varela Rodriguez
Here is the list of people involved in the review
. A core members team with CO people
. Reviewers and collaborators from both CO and ICS groups

8. Phases of the review
Requirements collection and analysis - July-Aug   High priority for users, common and specific requirements, Survey/Interviews with lot of experts Existing solutions comparison, feature matrix - Sept Diamon, Moon, Lemon, Xymon, log tracing, specific (i.e. timing, FESA) Matrix  Available, missing or duplicated features
Unified Monitoring Infrastructure Proposal - Jan layers architecture data sources, core, visualization as new paradigm with delegation of metric responsibility
Evaluation of Icinga2, Zabbix, Cern ITMON, Prometheus - Feb 
. Setup and configuration for basic monitoring, alerts and dashboard
. No time to test advanced features, performance and scalability in real conditions. . Golden use cases like White Rabbit Switches monitoring successfully implemented.
The review was conducted in 4 steps:
. The first step to analyse and ...
. Second step to compare major solutions deployed at CERN (especially in BE and IT).
We had technical meetings with the different experts to present their product, their showstoppers and potential Improvements
.In the 3rd step, We prepared a first proposal based on IT UMI solution but it was discussed because of limited time and resources, and probably oversized proposal. So we agreed to evaluate other alternatives and lower the scope of the product to more integrated solution.
.This led us to the 4th step to evaluate some candiate solutions … selected 4 well-established product of the market.
..for few 100s of hosts

9. People contacted - Interviews
Stephen Page and Alastair Bland as sysadmin part
Jakub Wozniak as Logging expert
Jean-Claude Bau, Andrzej Dworak and Luca Molinari as Timing expert
Julien Palluel as White Rabbit and WFIP expert,
Lukasz Burdzanowski as Database expert.
Kajetan Fuchsberger as reference from OP group,
Alberto Aimar and Pedro Andrade from IT Monitoring team.
A dedicated interdepartmental CNIC meeting has been proposed by Luigi and Stephan Lueders (
Timo Hakulinen for Access and Safety monitoring;
Franco Brasolin ATLAS TDAQ sysadmin monitoring with Icinga2 ;
Lavnia Darlea and Marc Dobson for CMS monitoring, Service,
Hirsto Mohamed for LHCb online monitoring,
Véronique Lefébure for IT-ICS network monitoring,
Sebastian Bukowiec for the IT Windows monitoring.
Thanks to Javier Serrano that actively helped with ICALEPCS contacts, we also have received inputs from:
Prados Boda Cesar at GSI, Jean-Michel Chaize from ESRF, Alain Buteau from Synchrotron Soleil, Enzo Carrone from SLAC, Artur Barczyk and Vincent Hardion from MAX IV.

10. Key common requirements
Reduce solutions multiplicity and services overlap
Easy : Plug & Play or auto-discovery of new systems
Self definition / tuning of metrics, alerts, parameters
Multitenant, Multi-view, custom visualization, integrated/coherent GUIs
Multi Platform access, web, mobile devices, control room
System relations and dependencies, hierarchy view
Reliable notifications, Acknowledge/Maintenance
History (long one)
Fast History, Fast Search, Easy Filters, Easy Query
No real surprise with the outcome of that requirements study:

11. Investigation of open solutions
We have investigated the 4 major candidates
ITMON/ CollectD
Icinga2
Zabbix
Prometheus
Full stack OSS solution - ++ +
Learning curve
Aligned with CO sysadmin
+ (Ansible, file-based)
Aligned with IT solutions
+ (Collectd, DBOD)
Setup, configuration, ease of use
Extent of customization
++ (checks, config files, advanced templating,...)
+ (browser-based Gui)
Data processing (filtering, aggreg./reduct., analysis, …)
+ (built-in procedures)
++ (powerful language)
Security (comm., authentication, multi-user, …)
++ (SSL/TLS)
Scalability and network distribution
High-availability (load balancing, redundancy, ..)
++ (fully indep. zones)
The results of these comparative tests are summarized in this table which contents only significant topics.
Clearly, Icinga2 and Zabbix can do 80% of the job but we have selected Icinga2 for different reasons:
1. We consider it is better aligned with CO sysadmin who has a long history and preference for file-based systems
2. It provides support for collectD and it can works with IT DB on-demand service for configuration and metrics databases
3. Concept of check process fits very well with the new delegation of responsability paradigm
4. It is also very well designed for distributed layout and high-availability (secure load balancing and redundant features)

12. ABACUS Proposal (reminder)
Plug our systems to CollectD and ITMON services CO sysadmin needs to be aligned with IT solution Set up an Icinga2 monitoring solution for all other use cases Icinga2 covers our major requirements, collectD support, has CERN users community (ATLAS+CMS)
New paradigm: Each user/expert responsible for some monitoring will add the specific part over the monitoring core infrastructure (collect, visualization)
So to conclude on the ABACUS review, let me show you again the outcomes of the review that led us to the COMOS project approved in March this year.

13. COSMOS Mandate (March 2017) Controls Open Source MOnitoring System
Study and implement a new CO monitoring system and concentrate efforts on our core business  use de-facto standard technologies and open-source solutions
Start LS2 (end 2018): CO infrastructure monitoring focusing on our first priorities - All CO hosts+hw-modules and services base checks (ping, system metrics, process/services state) - Golden use-cases: QuadServer, JBOD/Raid, crates and field-buses (WR-Switch, Pulse Rep., WFIP)
End LS2: System integration and extension - Architecture consolidation (scalability, HA, performance) - Configuration management tools - CO and partners services integration (alarms, logging, GUIs, custom monitoring, ...) - Advanced features (actions/feedback, dependencies tree, …) - Gradual phase-out of DIAMON
6 people from different CO sections and different expertise have been partially allocated to COSMOS (~1.5FTE so far)
In that project, we propose to :
Study and implement …
The project is splitted in 2 parts: . First, setup the monitoring of the all CO infrast. before start LS2, including: …
In the second step, plan for end LS2 (and possibly before) we will focus on the integration and extension of the system
Frank
Julien
Luigi
Laura
Felix
Sergey

14. COSMOS monitoring definition
It does
Constantly track the system resources
Detect fluctuations, slow or failing components
Provide health reports for each host and service
Collect metrics
Notify (by , SMS)
It is not
A database
A control command system
A (smart) diagnostic tool
A Logging tool
So we must first define what we mean by COSMOS monitoring
[----] It can be a diagnostic tool somehow but not an artificial intelligent system
… but don’t worry, it still has many other things to do!
[----]
And now, the main question is: where to put the cursor?
=================
And we have an answer for that: monitoring is not tracing or profiling and requires static configuration … we should know what we are looking for!
. In this picture you can see the data complexity growing from the left to the right and where is the limit of the COSMOS monitoring service.
. It includes:
. count and check hosts and services state (up/down)
. … to check not only the service is running but it is running properly (we can perform get/set to RDA for instance)
. usually it requires introspection to provide internal metrics and expert data of SW and HW components.
. COSMOS monitoring should not cover: logging, … or even more sophisticated application like: analysis, …
Service metrics
introspection
Host state
Process state
Service state
Acc. equipment check
COSMOS Monitoring
Diagnostic
ping
up/down
JMS,
RDA, SQL,
JMX, CMX,
FESA, …
Logging, profiling, search, processing, …
Offl. analysis, AFT, DocFIP,
CANDI, …
Static configuration
Dynamic configuration
We know what we are looking for!

15. COSMOS core scope COSMOS features and support included
Linux and windows OS/network metrics (cpu, mem, disk, netw, …)  collectD
Single and common interface to collect health report and perf. data  Nagios CHECK
Host, Process and Service state, Apps. status/metrics ingestion
Metrics and performance data storage, query support  influxDB (time-series DB)
Multiview and Multitenant support for visualization  Grafana
Core services: Config. Managment, Alerts, Interactive troubleshooting
Out of the COSMOS support
Custom checks, Data analysis, Tracing/Searching, Op. Alarms, Diagnostic
Icinga2 checks and API can be used to plug external services (Elastic, Logstash, Prometheus, etc.)

16. Solution: Icinga2 check
Active check
Also called plugin (Nagios): standalone extension to Icinga2 core
A script or binary: >check_http –H cwe-513-vol510 –w 10 –c 20 –f follow
Returning a status code
Returning an output message HTTP OK: HTTP/ OK bytes in second response time
Returning some perf. Data (optional) |time= s; ; ; size=85997B;;;0
>3000 plugins developed by the community
Icinga2 process
Initiate on periodic basis
status+ metrics
Status code
Host status
Service status 1 2 3 UP
UNSTABLE
DOWN/UNREACHABLE OK
WARNING
CRITICAL
UKNOWN
plugin
Passive check
Icinga2 process
status+ metrics
asynchronous
plugin

17. Solution: COSMOS layout
GPN
IT firewall rules TN
Grafana service IT
agent
Collectd server
Sys. metrics
agent
agent
agent IT
time series
CO routing
agent
perf.data
check
agent
Icinga2 server
p-check
check
check
events
check
Icinga2 satellite
check IT
Icinga2web server
check

18. Current status: November 2017
Icinga2: master and web servers (x2 Pro, Dev) available for development and tests
IT Services: Large effort done to make the whole chain integrated
Databases: COSMOS relies on DB on-demand service with TN/GPN routing (IDO: MySQL, TS: influxDB)
Visualisation: COSMOS relies on IT Openshift service for Grafana
Security: LDAP/SSO support for ICINGA2 servers, Grafana setup as multi-tenant service
Collectd: Provide metrics for CO Linux infrast. with data accessible from Grafana!
In progress: Monitoring of diskless platforms (FEC)
CO checks:
Host alive check: ping
SNMP services: WR-Switch, ELMA & Wiener crates, Pulse repeater
In the pipeline: IPMI check for Quad servers and enclosures (JBOD/RAID systems)

19. Conclusions
COSMOS is progressing well, with respect to the review recommendations
Initial developments already addressed many technical queries
Full-stack monitoring test successful: Icinga2+Collectd  IT DBs (status/time-series)  IT Grafana
Still some points to investigate to reassure us definitely before LS2:
Define the optimal network distribution over GPN/TN (scalability/ availability)
Evaluate the system performances in operational conditions
Cohabitation of COSMOS and DIAMON until end LS2

20. Questions?