1. Smart Data for Industrial Control Systems
CERN Technical Workshop
Filippo Tilaro, Fernando Varela (BE/ICS)
in collaboration with Siemens AG CT Munich, St. Petersburg, Brasov
09/01/2018

2. Data Analytics for Industrial Controls
Take advantage of control data
A multitude of Industrial Control Systems
Control Data analytics
Design analytics algorithms
Expert systems
Machine learning to deal with heterogeneous control systems
Big Data platform to improve:
Control system stability and efficiency
Reduce maintenance cost
Large scale performances (even physic data quality)
Reliability and Safety
Cooling &
Ventilation
VACUUM
Cryogenics
GAS
Electric Grid
LHC Circuit, QPS, WIC, PIC, …
50% data produced last year
Storing +100 TB/year

3. Use cases and Algorithms
LHC Circuit Monitoring
Tank leak detection in Cooling and Ventilation

4. Online monitoring of the LHC control systems
Condition monitoring use-cases
LHC Circuit Monitoring
Online analyse the power converter circuits signals and the system status in order to assess their health and detect anomalies
Condition Monitoring for Cryogenics
Online analysis of the LHC superconducting magnets pressure, temperature measurements and the openings of their related valves
Analysis of control systems alarms based on custom indexes

5. LHC circuit monitoring
Condition monitoring analysis (in collaboration with TE-MPE)
Main Goal: evaluation of the superconducting circuits health
Degradation after 20 years of operations
Monitoring conditions: anomalous change of current flows, impedance, circuit functioning …
Different analyses:
Scheduled analysis depending on the machine cycles, e.g. during the energy ramp, during the energy extraction, etc.
Continuous Analysis to check the circuits health
Replay of historical data for qualification and validation
Electrical circuits:
magnets, power converters, switches …
Different working phases:
anomaly detection
‘energy’ unload and
‘fail-safe

6. LHC circuit monitoring control infrastructure
Supervision layer:
WinCC OA
16 servers
Control layer:
44 industrial FECs
Readout (from 10KHz to 1Hz)
RBAC access
Time sync
Post-Mortem buffer
CMW
Field layer:
2800 radiation-hard devices
WorldFIP fieldbus
Fast readout and protection for magnets and powering circuits
Signals number: 60 * 4000 QPS * 2000 FGC systems

7. LHC circuit monitoring analysis
Condition monitoring analysis
Expert system
Translate experts’ knowledge into formulation sets / rules
Asset-based description
Rules central storage
Rule template to be reused, parametrized, validated
Signal Processing Language (SPL):
Domain specific language (DSL)
Simple formulation
Time reasoning and temporal expression
Mathematical and logical functions
Event emitter/ handler
Alarms
Trigger of rules chain
Rule definition:
Truth(sma(I_Meas, 1m30s)> I_Min)): duration(>=1h)
Rules
List of similar assets
New DSL under implementation: SEPL!

8. Cooling and ventilation control system
Leak detection in Tanks (in collaboration with BE-ICS-AP)
Anomaly detection based on historical data
Dual factors analysis:
Valve opening time
Time between valve openings
Dynamic control
Different alarms thresholds
Changing filling conditions
Detection of “large” leaks:
Anomalous valve opening time
Detection of “small” leaks:
Anomalous time interval between valve openings
Outlier identification based on time distribution of and between valves opening! VM
Driver
ELVis cluster
Client

9. DCEP and ELVis integration Industrial IoT support MindSphere
Analytical platforms
DCEP and ELVis integration
Industrial IoT support
MindSphere

10. Smart Data for Industrial Control Systems
Combining cloud and edge computing into a single framework
ELVis cluster
DCEP Cluster
Analytics master
DCEP Master
Expert systems:
Manual rules
ML rules
Cloud computing
DCEP Cluster
Data Source
Broker
Data Source UI
DCEP Cluster
Rules VM
Driver
Middleware
Client
Cloud & Edge Link
The analytical framework involves all the 3 control layers because it combines Cloud computing with Edge computing through the integration of 2 Siemens internal projects which are called DCEP and ELVis.
Cloud computing:
DCEP (Distributed Complex Event Processing engine) multiple clusters of nodes with a single master as a central rules deployment (the user configure only the master that spreads the computational load across the different clusters)
ELVis is a cloud computing framework for online analysis (data streams) which provides an advanced UI and based on Spark framework and dockers.
ELVis is a pure cloud system and DCEP allows the integration with edge computing
Edge computing:
IoT scenario with analytical capabilities next to the sensors
Advantages:
Reduce the amount of data sent to the upper layer
Faster analysis
Component analysis (vibration/ oscillation analysis requires high frequency data in order to increase the precision in the spectrum analysis. Edge computing avoids sending data through all the control layers but they can be analysed close to the sensors )
Machine learning analysis is done on the cloud infrastructure and its model is used for anomaly detection in the edge infrastructure
Quality of Service
Set different analytical priorities and service levels
Modularity and Isolation
Add/replace IoT devices
No single point of failure
Detect and isolate fault/crash
Multi-platform support
Reliability
Check the service status
Maintain the network topology during the transmission of anomalies
Do not disrupt the control process
Analytical optimization
Workload allocation based on devices resources, topology, special HW, data sources proximity, latency, network usage …
Fieldbus
DCEP Cluster
DCEP Cluster
Data Source
Broker
Analytics worker
DCEP Cluster
Edge computing

11. Industrial IoT islands
Fully autonomous DCEP cluster that can interoperate with other IIoT islands
Industrial IoT analytical benefits:
Shorter latency for online analysis
Reduced network load
Industrial modelling/segmentation
Increased reliability
Siemens involvement:
New collaboration with an IIoT group
Existing collaboration with the cloud computing group
Multi-platform support
Raspberry Pi, Siemens IoT 2020/2040, …

12. DCEP Industrial IoT islands
IOT support in the analytical framework
Dynamic device registration
Compatible with the control system structure
Sensors description
Available resources
Dynamic configuration
Service discovery
Services endpoints
Permitted operations
Event subscription
Dynamic update due to device mobility
Data ingestion:
From field devices and SCADA
Multi-protocol support
Time synchronization
Reduced impact on Industrial Controls
Network load across the nodes/SCADA
Not invasive fieldbus communication
Analytics
master
Rules
Rule deployment
IoT island S1 S2 …
IoT device
Analytic worker
Data
Source
IoT island
IoT device
Analytic worker
Data Sources S1 S2 …

13. MindSphere Evaluating analytical modules:
Cloud-based Platform as a Service (PaaS) provided by Siemens
Evaluating analytical modules:
Sensors validation
Anomaly detection in sensors measurements
Control process assessment and availability
Condition monitoring system based on KPIs
Integration with CERN control systems
Web services to upload and run analytical modules
Open API
Migration of CERN analytical algorithm to MindSphere
Implementation of the PID evaluation in the Siemens web service domain

14. Summary Different CERN use-cases
Expert systems for LHC circuit monitoring and anomaly detection in CV
New analytical platform for both cloud and edge computing
Deployment and integration with CERN control systems
SEPL to support for numerical and text analysis
MindSphere
Cloud-based Platform as a Service (PaaS) provided by Siemens
2017 data analytics publications in international conferences:
An expert knowledge based methodology for online detection of signal oscillations – CIVEMSA 2017, F. Tilaro, M. Gonzalez, B. Bradu, M. Roshchin
Model Learning Algorithms for Faulty Sensors Detection in CERN Control Systems - ICALEPCS 2017, F. Tilaro, B. Bradu, F. Varela, M. Roshchin
Automatic PID Performance Monitoring Applied to LHC Cryogenics - ICALEPCS 2017, B. Bradu, E. Blanco, F. Tilaro, R. Marti
Data Analytics Reporting Tool for CERN SCADA Systems - ICALEPCS 2017, P. J. Seweryn, M. Gonzalez-Berges, J. B. Schofield, F. M. Tilaro

15. https://be-dep-ics.web.cern.ch/
Thank you!
CERN BE-ICS