BE/ICS Group Meeting: Scada & Distributed Systems Section Highlights of 2016 24 November 2016 presented on behalf of the Section by Piotr Golonka People Services Applications Innovation Upgrades

Some people come, some people go... Left Paul Burkimsher (STAFF) Carlos Garcia (TECH, Openlab) Valentin Comte (TECH) Josef Hofer (FELLOW) Victor Lampreia (TECH) Axel Voitier (STAFF) Next Departures Manuel Gonzalez (STAFF) Stefan Klikovits (DOCT) Joined Pedro Agostinho (PJAS) Jakub Guzik (FELLOW, Openlab) Matej Marecek (FELLOW) Petra Mazdin (VIA) Ivan Prieto (STAFF) Brad Schofield (STAFF) Piotr Seweryn (TECH, Openlab) Riku-Pekka Silvola (STAFF) Summer: Eleni and Francesca BE-ICS Group Meeting, 24 November 2016

BE-ICS Group Meeting, 24 November 2016 ... but some people stay Alejandro Herrero (TECH, Openlab) Rafal Kułaga (STAFF) Giulio Morpurgo (STAFF) Mariusz Suder (PJAS) Jean-Charles Tournier (STAFF) Jonas Arroyo (STAFF) Adrian Crestar (FTEC Trainee) Piotr Golonka (STAFF) Lukasz Góralczyk (STAFF) James Hamilton (FELLOW) High turnover of people: loss of experience is recovering slowly... BE-ICS Group Meeting, 24 November 2016

Disclaimer Many of the activities presented have been done in collaboration with other sections of the group (PCS, CIC) and with other CERN Groups. They are not explicitly mentioned all the time. BE-ICS Group Meeting, 24 November 2016

BE-ICS Group Meeting, 24 November 2016 Services BE-ICS Group Meeting, 24 November 2016

BE-ICS Group Meeting, 24 November 2016 Keep services running WinCC OA Service WinCC OA 3.11SP1 in production WinCC OA 3.14, 3.15 validated on new platforms (Centos7, Windows 10/2012 Server) repackaged and made available to users (CMF, RPM, Downloads page) Support 630 WinCC OA License requests 220 Support requests Notable issue of UI crashes in VAC project (due to non-standard architecture) 90 issues delegated to ETM 3 Training Courses for WinCC OA/JCOP take-over from Paul ENS Support issues Nov’15 - Nov’16 BE-ICS Group Meeting, 24 November 2016

BE-ICS Group Meeting, 24 November 2016 Keep services running JCOP and UNICOS Framework for WinCC OA 3.11SP1 JCOP Framework 5.2.0 (January) 5.2.1 (February) UNICOS Framework 6.4.0 (February) 6.4.1 (August) 6.4.2 (September) Individual component releases with new features and bugfixes: fwDeviceComment, fwSynopticLocator, unMessageText, unLoggingDB, fwAccessControl, fwRPC, fwTrending, fwCaen BE-ICS Group Meeting, 24 November 2016

BE-ICS Group Meeting, 24 November 2016 Keep services running SCADA Application Service ~200 applications managed Application-health monitoring, Oracle archiving database New applications and servers Coordination of TechnicalStops with SUWG, IT-DB, users, Component upgrades BE-ICS Group Meeting, 24 November 2016

BE-ICS Group Meeting, 24 November 2016 Keep services running SCADA Application Service ~200 applications managed Application-health monitoring, Oracle archiving database New applications and servers Coordination of TechnicalStops with SUWG, IT-DB, users, Component upgrades Disaster recovery for PSEN: collaboration with IT-DB and BE-CO Geographical redundancy of WinCC OA servers and database CCC (513) CCR (874) Fiber link BE-ICS Group Meeting, 24 November 2016

BE-ICS Group Meeting, 24 November 2016 Keep services running SCADA Application Service ~200 applications managed Application-health monitoring, Oracle archiving database New applications and servers Coordination of TechnicalStops with SUWG, IT-DB, users, Component upgrades Disaster recovery for PSEN: collaboration with IT-DB and BE-CO Geographical redundancy of WinCC OA servers and database Incidents Database incidents and TS upgrades LHC Logging data loses Application configuration issues (e.g. Access Control) BE-ICS Group Meeting, 24 November 2016

BE-ICS Group Meeting, 24 November 2016 Applications BE-ICS Group Meeting, 24 November 2016

Applications: Machine Protection Power Interlock Controller (PIC) Consolidation work Improvements for LHC H/W commissioning Better visualization and sector-access logic Quench Protection Systems (QPS) Integration of new device type: DQAMCSD Warm Interlock Controller (WIC) New systems for ELENA, LINAC4 Preparation for new projects SM18 QPS FAIR Strings TE-MPE Testbed BE-ICS Group Meeting, 24 November 2016

BE-ICS Group Meeting, 24 November 2016 Applications: DSS LHC Experiments Detector Safety Systems 7 production installations Alarm/Action matrix: improved reporting Preparation for upgrade to WinCC OA 3.15 BE-ICS Group Meeting, 24 November 2016

BE-ICS Group Meeting, 24 November 2016 Applications PSEN: Electrical Network Supervision 2 major releases, numerous performance improvements 1st version of Event Replay Handover of autonomous engineering NA62DCS: Detector Control System for NA62 Stable run, with continuous development/integration NEWCHOD detector BE-ICS Group Meeting, 24 November 2016

BE-ICS Group Meeting, 24 November 2016 Innovations BE-ICS Group Meeting, 24 November 2016

BE-ICS Group Meeting, 24 November 2016 Innovation: WebUI Access WinCC OA applications from a web browser Displays native panels, with no need for modification, no plugins needed In preparation for deployment in EYETS BE-ICS Group Meeting, 24 November 2016

Innovation: Quality Assurance Code analysis Continued effort: automated syntax-checking, unit- tests, GUI tests, documentation coverage New dashboard (SonarQube) Code repository and release process Restructuring of SVN: catalogue of all WinCC OA components Unified release process with Jenkins Paves the way for GIT migraton Redundancy test setup Powered by docker Research: automatic test-case generation Translation of Ctrl to C#, MS PEX tool, mocking PhD topic of Stefan BE-ICS Group Meeting, 24 November 2016

BE-ICS Group Meeting, 24 November 2016 Reporting Statistics on value-changes and alarms from Oracle Archive WinCC OA log analysis Dashboards Summary statistics PSEN CIET Early Warning System MOON statistics Based on the The Open Source Elastic Stack: lasticsearch + ogstash/ eats + ibana (= ) Run on the IT-managed service BE-ICS Group Meeting, 24 November 2016

Openlab: „NextGen Archiver” for WinCC OA Successor of Oracle RDB Archiver, enabler for Data Analytics Requirements: High-performance, cost-effective WinCC OA Project ? Archiver EM DRV BE-ICS Group Meeting, 24 November 2016

Openlab: „NextGen Archiver” for WinCC OA Successor of Oracle RDB Archiver, enabler for Data Analytics Multiple, pluggable „backends” working in parallel WinCC OA Project ORACLE Backend NextGen Archiver SQLite Backend Hadoop Backend EM ... Custom Backend DRV + BE-ICS Group Meeting, 24 November 2016

Openlab: „NextGen Archiver” for WinCC OA Successor of Oracle RDB Archiver, enabler for Data Analytics Multiple, pluggable „backends” working in parallel, streaming option WinCC OA Project ORACLE Backend NextGen Archiver SQLite Backend Hadoop Backend EM ... Custom Backend DRV + Data Analytics e.g. Streaming Backend BE-ICS Group Meeting, 24 November 2016

Openlab: „NextGen Archiver” for WinCC OA Successor of Oracle RDB Archiver, enabler for Data Analytics Multiple, pluggable „backends” working in parallel, streaming option Development team at + . First release expected in 2018. ETM CERN WinCC OA Project ORACLE Backend NextGen Archiver SQLite Backend Hadoop Backend EM ... Custom Backend DRV + Data Analytics e.g. Streaming Backend BE-ICS Group Meeting, 24 November 2016

Preparation for WinCC OA upgrades BE-ICS Group Meeting, 24 November 2016

Preparation for WinCC OA upgrades Motivation for upgrade Ensure support, avoid divergence from the mainline version (danger for future upgrades) WinCC OA version lifetime is 3 years Enable new features Object-oriented programming, improvements in User-Interface design and drivers Price to pay: porting/validation efforts: limit new features delivered in the frameworks BE-ICS Group Meeting, 24 November 2016

Preparation for WinCC OA upgrades Initial plan: upgrade to WinCC OA 3.14 CERN-wide during EYETS; Extensive testing; frameworks ready in August Current goal: upgrade to WinCC OA 3.15 Important risk factor: Release 20 December 2016 Early engagement in testing and debugging with ETM (beta versions), agreed with users in September Development and validation is on track First WinCC OA 3.15 beta release: 20 October JCOP Framework-8.0.0, UNICOS-Framework 8.0.0: 15 November Rollback strategies prepared in case of unexpected problems Challenging planning & execution Application-specific constraints need to be taken into account for over 200 BE-ICS projects Coordination with BE/CO for OS upgrades on more than 100 servers Support for developers responsible for ~500 projects at the LHC experiments BE-ICS Group Meeting, 24 November 2016

Most memorable moment of 2016 „Weasel incident” 29 April 2016 BE-ICS Group Meeting, 24 November 2016

Weasel incident: behind the scenes A chain of events... Deployment of new version of PSEN (Electicity SCADA) the day before Weasel went for a walk, which led to... ... EL network perturbation led to overload of the PSEN ... which uncovered a rare weakness in WinCC OA (only PSEN affected) ... which led to a collapse of the supervisory system (out-of-memory) Handled as critical event with ETM Complex analysis by experts led to discovery of the weakness mechanism Immediate „workaround” recommendation Patch made available for WinCC OA 3.11SP1 (even though out-of-lifetime) Warning displayed to a programmer about possibly dangerous (yet valid) construct Updated documentation BE-ICS Group Meeting, 24 November 2016

BE-ICS Group Meeting, 24 November 2016 ... and that was not the end... BE-ICS Group Meeting, 24 November 2016

BE-ICS Group Meeting, 24 November 2016 Summary A year of change... Particularly high turnover of personel (5 staff gone during last 2 years) Preparations for WinCC OA upgrades during EYETS First things first Reliable work of core services contributed to the excellent performance of the LHC during 2016 Run Innovation Prepare for challenges of LS2, and beyond, already now! BE-ICS Group Meeting, 24 November 2016

BE-ICS Group Meeting, 24 November 2016