1. BE/ICS Group Meeting: Scada & Distributed Systems Section Highlights of November presented on behalf of the Section by Piotr Golonka
People
Services
Applications
Innovation
Upgrades

3. Some people come, some people go...
Left
Paul Burkimsher (STAFF)
Carlos Garcia (TECH, Openlab)
Valentin Comte (TECH)
Josef Hofer (FELLOW)
Victor Lampreia (TECH)
Axel Voitier (STAFF)
Next Departures
Manuel Gonzalez (STAFF)
Stefan Klikovits (DOCT)
Joined
Pedro Agostinho (PJAS)
Jakub Guzik (FELLOW, Openlab)
Matej Marecek (FELLOW)
Petra Mazdin (VIA)
Ivan Prieto (STAFF)
Brad Schofield (STAFF)
Piotr Seweryn (TECH, Openlab)
Riku-Pekka Silvola (STAFF)
Summer:
Eleni and Francesca
BE-ICS Group Meeting, 24 November 2016

4. BE-ICS Group Meeting, 24 November 2016
... but some people stay
Alejandro Herrero (TECH, Openlab)
Rafal Kułaga (STAFF)
Giulio Morpurgo (STAFF)
Mariusz Suder (PJAS)
Jean-Charles Tournier (STAFF)
Jonas Arroyo (STAFF)
Adrian Crestar (FTEC Trainee)
Piotr Golonka (STAFF)
Lukasz Góralczyk (STAFF)
James Hamilton (FELLOW)
High turnover of people: loss of experience is recovering slowly...
BE-ICS Group Meeting, 24 November 2016

5. Disclaimer
Many of the activities presented have been done in collaboration
with other sections of the group (PCS, CIC)
and with other CERN Groups.
They are not explicitly mentioned all the time.
BE-ICS Group Meeting, 24 November 2016

6. BE-ICS Group Meeting, 24 November 2016
Services
BE-ICS Group Meeting, 24 November 2016

7. BE-ICS Group Meeting, 24 November 2016
Keep services running
WinCC OA Service
WinCC OA 3.11SP1 in production
WinCC OA 3.14, 3.15
validated on new platforms (Centos7, Windows 10/2012 Server)
repackaged and made available to users (CMF, RPM, Downloads page)
Support
630 WinCC OA License requests
220 Support requests
Notable issue of UI crashes in VAC project (due to non-standard architecture)
90 issues delegated to ETM
3 Training Courses for WinCC OA/JCOP
take-over from Paul
ENS Support issues
Nov’15 - Nov’16
BE-ICS Group Meeting, 24 November 2016

8. BE-ICS Group Meeting, 24 November 2016
Keep services running
JCOP and UNICOS Framework for WinCC OA 3.11SP1
JCOP Framework
5.2.0 (January)
5.2.1 (February)
UNICOS Framework
6.4.0 (February)
6.4.1 (August)
6.4.2 (September)
Individual component releases with new features and bugfixes:
fwDeviceComment, fwSynopticLocator, unMessageText, unLoggingDB, fwAccessControl, fwRPC, fwTrending, fwCaen
BE-ICS Group Meeting, 24 November 2016

9. BE-ICS Group Meeting, 24 November 2016
Keep services running
SCADA Application Service
~200 applications managed
Application-health monitoring, Oracle archiving database
New applications and servers
Coordination of TechnicalStops with SUWG, IT-DB, users,
Component upgrades
BE-ICS Group Meeting, 24 November 2016

10. BE-ICS Group Meeting, 24 November 2016
Keep services running
SCADA Application Service
~200 applications managed
Application-health monitoring, Oracle archiving database
New applications and servers
Coordination of TechnicalStops with SUWG, IT-DB, users,
Component upgrades
Disaster recovery for PSEN: collaboration with IT-DB and BE-CO
Geographical redundancy of WinCC OA servers and database
CCC
(513)
CCR
(874)
Fiber link
BE-ICS Group Meeting, 24 November 2016

11. BE-ICS Group Meeting, 24 November 2016
Keep services running
SCADA Application Service
~200 applications managed
Application-health monitoring, Oracle archiving database
New applications and servers
Coordination of TechnicalStops with SUWG, IT-DB, users,
Component upgrades
Disaster recovery for PSEN: collaboration with IT-DB and BE-CO
Geographical redundancy of WinCC OA servers and database
Incidents
Database incidents and TS upgrades
LHC Logging data loses
Application configuration issues (e.g. Access Control)
BE-ICS Group Meeting, 24 November 2016

12. BE-ICS Group Meeting, 24 November 2016
Applications
BE-ICS Group Meeting, 24 November 2016

13. Applications: Machine Protection
Power Interlock Controller (PIC)
Consolidation work
Improvements for LHC H/W commissioning
Better visualization and sector-access logic
Quench Protection Systems (QPS)
Integration of new device type: DQAMCSD
Warm Interlock Controller (WIC)
New systems for ELENA, LINAC4
Preparation for new projects
SM18 QPS
FAIR
Strings
TE-MPE Testbed
BE-ICS Group Meeting, 24 November 2016

14. BE-ICS Group Meeting, 24 November 2016
Applications: DSS
LHC Experiments Detector Safety Systems
7 production installations
Alarm/Action matrix: improved reporting
Preparation for upgrade to WinCC OA 3.15
BE-ICS Group Meeting, 24 November 2016

15. BE-ICS Group Meeting, 24 November 2016
Applications
PSEN: Electrical Network Supervision
2 major releases, numerous performance improvements
1st version of Event Replay
Handover of autonomous engineering
NA62DCS: Detector Control System for NA62
Stable run, with continuous development/integration
NEWCHOD detector
BE-ICS Group Meeting, 24 November 2016

16. BE-ICS Group Meeting, 24 November 2016
Innovations
BE-ICS Group Meeting, 24 November 2016

17. BE-ICS Group Meeting, 24 November 2016
Innovation: WebUI
Access WinCC OA applications from a web browser
Displays native panels, with no need for modification, no plugins needed
In preparation for deployment in EYETS
BE-ICS Group Meeting, 24 November 2016

18. Innovation: Quality Assurance
Code analysis
Continued effort: automated syntax-checking, unit- tests, GUI tests, documentation coverage
New dashboard (SonarQube)
Code repository and release process
Restructuring of SVN: catalogue of all WinCC OA components
Unified release process with Jenkins
Paves the way for GIT migraton
Redundancy test setup
Powered by docker
Research: automatic test-case generation
Translation of Ctrl to C#, MS PEX tool, mocking
PhD topic of Stefan
BE-ICS Group Meeting, 24 November 2016

19. BE-ICS Group Meeting, 24 November 2016
Reporting
Statistics on value-changes and alarms from Oracle Archive
WinCC OA log analysis
Dashboards
Summary statistics
PSEN
CIET Early Warning System
MOON statistics
Based on the The Open Source Elastic Stack:
lasticsearch ogstash/ eats ibana (= )
Run on the IT-managed service
BE-ICS Group Meeting, 24 November 2016

20. Openlab: „NextGen Archiver” for WinCC OA
Successor of Oracle RDB Archiver, enabler for Data Analytics
Requirements: High-performance, cost-effective
WinCC OA
Project ?
Archiver EM
DRV
BE-ICS Group Meeting, 24 November 2016

21. Openlab: „NextGen Archiver” for WinCC OA
Successor of Oracle RDB Archiver, enabler for Data Analytics
Multiple, pluggable „backends” working in parallel
WinCC OA
Project
ORACLE
Backend
NextGen
Archiver
SQLite
Backend
Hadoop
Backend EM
...
Custom
Backend
DRV +
BE-ICS Group Meeting, 24 November 2016

22. Openlab: „NextGen Archiver” for WinCC OA
Successor of Oracle RDB Archiver, enabler for Data Analytics
Multiple, pluggable „backends” working in parallel, streaming option
WinCC OA
Project
ORACLE
Backend
NextGen
Archiver
SQLite
Backend
Hadoop
Backend EM
...
Custom
Backend
DRV +
Data Analytics e.g.
Streaming
Backend
BE-ICS Group Meeting, 24 November 2016

23. Openlab: „NextGen Archiver” for WinCC OA
Successor of Oracle RDB Archiver, enabler for Data Analytics
Multiple, pluggable „backends” working in parallel, streaming option
Development team at First release expected in 2018.
ETM
CERN
WinCC OA
Project
ORACLE
Backend
NextGen
Archiver
SQLite
Backend
Hadoop
Backend EM
...
Custom
Backend
DRV +
Data Analytics e.g.
Streaming
Backend
BE-ICS Group Meeting, 24 November 2016

25. Preparation for WinCC OA upgrades
BE-ICS Group Meeting, 24 November 2016

26. Preparation for WinCC OA upgrades
Motivation for upgrade
Ensure support, avoid divergence from the mainline version (danger for future upgrades)
WinCC OA version lifetime is 3 years
Enable new features
Object-oriented programming, improvements in User-Interface design and drivers
Price to pay: porting/validation efforts:
limit new features delivered in the frameworks
BE-ICS Group Meeting, 24 November 2016

29. Preparation for WinCC OA upgrades
Initial plan: upgrade to WinCC OA 3.14 CERN-wide during EYETS;
Extensive testing; frameworks ready in August
Current goal: upgrade to WinCC OA 3.15
Important risk factor: Release 20 December 2016
Early engagement in testing and debugging with ETM (beta versions), agreed with users in September
Development and validation is on track
First WinCC OA 3.15 beta release: 20 October
JCOP Framework-8.0.0, UNICOS-Framework 8.0.0: 15 November
Rollback strategies prepared in case of unexpected problems
Challenging planning & execution
Application-specific constraints need to be taken into account for over 200 BE-ICS projects
Coordination with BE/CO for OS upgrades on more than 100 servers
Support for developers responsible for ~500 projects at the LHC experiments
BE-ICS Group Meeting, 24 November 2016

30. Most memorable moment of 2016
„Weasel incident” 29 April 2016
BE-ICS Group Meeting, 24 November 2016

31. Weasel incident: behind the scenes
A chain of events...
Deployment of new version of PSEN (Electicity SCADA) the day before
Weasel went for a walk, which led to...
... EL network perturbation led to overload of the PSEN
... which uncovered a rare weakness in WinCC OA (only PSEN affected)
... which led to a collapse of the supervisory system (out-of-memory)
Handled as critical event with ETM
Complex analysis by experts led to discovery of the weakness mechanism
Immediate „workaround” recommendation
Patch made available for WinCC OA 3.11SP1 (even though out-of-lifetime)
Warning displayed to a programmer about possibly dangerous (yet valid) construct
Updated documentation
BE-ICS Group Meeting, 24 November 2016

32. BE-ICS Group Meeting, 24 November 2016
... and that was not the end...
BE-ICS Group Meeting, 24 November 2016

33. BE-ICS Group Meeting, 24 November 2016
Summary
A year of change...
Particularly high turnover of personel (5 staff gone during last 2 years)
Preparations for WinCC OA upgrades during EYETS
First things first
Reliable work of core services contributed to the excellent performance of the LHC during 2016 Run
Innovation
Prepare for challenges of LS2, and beyond, already now!
BE-ICS Group Meeting, 24 November 2016

34. BE-ICS Group Meeting, 24 November 2016