1. Control Status Readiness Report
Eugenia Hatziangeli on behalf of the CERN Accelerator and Beams Controls Group
Collimation project
The choice of the FE electronics was PXI (an extension of the Compact Pci). This is not part of the standard CO FE catalogue, so the responsibility of the FE software fell on the side of the equipment responsible.
On top of this a fesa server is written which access the collimators like a black box (from the point of view of Controls). So from the CO point of view there is no difference between collimators or BI (access via CMW and FESA).
The "Labview" disease is less of a problem. We have a JAVA based application program and a FESA based front-end infrastructure. There is a possibility to bypass all this and to control the collimatorsfrom a specific labview (test) application. This door will be closed by our security team.
12 June 2008
LHC – MAC

2. Outline LHC controls infrastructure – overview
Status Report on Core Controls
Front ends Hardware and Software
Databases
Industrial Controls
Machine Interlocks
The LHC timing system
Core Services and Applications
Post mortem
Sequencer for HWC
See next talk for Beam Sequencer
Logging
Software Interlock System
LSA (see next talk)
Controls Security
Role Based Access
Management of Critical Settings
LHC Controls Security Panel
Controls Infrastructure Tests
Deployment on LEIR, SPS, LHC TL
Dry runs - Commissioning
Scalability Tests
LHC Timing Crash Tests
RBAC tests
Monitor and Diagnostics
LASER
DIAMON
Injector Renovation
Summary
12 June 2008
E. Hatziangeli AB/CO

3. Outline LHC controls infrastructure – overview
Status Report on Core Controls
Front ends Hardware and Software
Databases
Industrial Controls
Machine Interlocks
The LHC timing system
Core Services and Applications
Post mortem
Sequencer for HWC
See next talk for Beam Sequencer
Logging
Software Interlock System
LSA (see next talk)
Controls Security
Role Based Access
Management of Critical Settings
LHC Controls Security Panel
Controls Infrastructure Tests
Deployment on LEIR, SPS, LHC TL
Dry runs - Commissioning
Scalability Tests
LHC Timing Crash Tests
RBAC tests
Monitor and Diagnostics
LASER
DIAMON
Injector Renovation
Summary
12 June 2008
E. Hatziangeli AB/CO

4. LHC controls infrastructure – Overview
The 3-tier architecture
Hardware Infrastructure
Software layers
Resource Tier
VME crates, PC GW & PLC dealing with high performance acquisitions and real-time processing
Database where all the setting and configuration of all LHC device exist
Server Tier
Application servers
Data Servers
File Servers
Central Timing
Client Tier
Interactive Consoles
Fixed Displays
GUI applications
Communication to the equipment goes through Controls MiddleWare CMW
Applications Layer
Client tier
Business Layer
Server tier
CMW
Hardware
CTRL DB
Resource tier
12 June 2008
E. Hatziangeli AB/CO

5. Outline LHC controls infrastructure – overview
Status Report on Core Controls
Front ends Hardware and Software
Databases
Industrial Controls
Machine Interlocks
The LHC timing system
Core Services and Applications
Post mortem
Sequencer for HWC
See next talk for Beam Sequencer
Logging
Software Interlock System
LSA (see next talk)
Controls Security
Role Based Access
Management of Critical Settings
LHC Controls Security Panel
Controls Infrastructure Tests
Deployment on LEIR, SPS, LHC TL
Dry runs - Commissioning
Scalability Tests
LHC Timing Crash Tests
RBAC tests
Monitor and Diagnostics
LASER
DIAMON
Injector Renovation
Summary
12 June 2008
E. Hatziangeli AB/CO

6. Front-Ends Hardware and Software
Hardware Installations
All Front-end controls equipment in place (> 200 VMEBus systems and > 250 industrial PCs)
WorldFIP infrastructure operational for PO, QPS, Cryogenics and BI (400 km network, nodes)
General Machine Timing (GMT) network operational, including transmissions for LHC Collimators and for LHC Experiments
Front End Software Architecture (FESA)
FESA V2.10 Framework operational, including support for machine critical settings, transactional commands and in-depth diagnostics of RT behavior (via Alarms system LASER)
Front-End FESA classes developed by AB equipment groups (> 250 classes deployed on > 400 front-ends)
Deployment process =>AB-CO supports 3 last FESA releases
All industrial PCs running now Linux O/S
On-going Actions
Two major tendering exercise for the procurement of AB front-end hardware (adjudication during CERN FC in September 2008)
LHC Alarm Service
12 June 2008
E. Hatziangeli AB/CO

7. Accelerator Databases Readiness
Off-line Databases
Online Databases
Layout
Racks & electronics incorporated up to a high level of detail
Layout data is now used as foundation for the controls system
Still to do
More data is being captured relating layout and assets information
Tools for data maintenance still to be put in place
Operational Settings
Data model enhanced to cover functional extensions for Role Based Access (RBAC) , XPOC, Sequencer
PS Controls Renovation requirements
Logging Service
New database hosting since Mar. 08
Common logging infrastructure for the complete accelerator chain
Sustained increasing logging requirements for HWC& beam data
Improved data retrieval tool
12 June 2008
E. Hatziangeli AB/CO

8. Service Availability and Data Security
Additional server for testing: Standby database for LSA
Controls Configuration LSA Settings E-Logbook CESAR
HWC Measurements Measurements
Logging
2 x quad-core 2.8GHz CPU 8GB RAM
CTRL
CTRL
11.4TB usable
Clustered NAS shelf 14x146GB FC disks
Clustered NAS shelf 14x300GB SATA disks
Service Availability
New infrastructure has high-redundancy for high-availability
Deploy each service on a dedicated Oracle Real Application Cluster
The use of a standby database will be investigated
objective of reaching 100% uptime for LSA
Secure database account granting specific privileges to dedicated db accounts
DIAMON agent on Oracle Application Servers
For all CO databases
CO puts the UR, and pays for the hardware
IT chooses the hardware, hosts, supports and maintains
Database: CO requirement IT chooses the material,CO pays, IT hosts and maintains and supports without extra cost SLA of IT to the suppliers, Dell 24x7 support included in the purchase cost, after 3 years we need to decide disks working hours
12 June 2008
E. Hatziangeli AB/CO

9. Industrial Controls
Industrial Controls for LHC have reached a high level of maturity
All systems, fully deployed for HWC in 2007, are presently in their operational version
Machine protection (PIC, WIC, QPS, Circuits)
Collimator Environment Monitoring Package (temperature, water cooling)
Survey
Cryogenic controls
Cryogenics Instrumentation Expert Tools (CIET)
Most of the SCADA applications have been ported to Linux
The front-end FESA software has been ported to Linux
Migration to last version of FESA (v2.10) to be done for next shutdown
The interface toward logging database has been consolidated
DIAMON is used for diagnostics
PLC agents are available, tested and ready to be deployed
PVSS diagnostics will be soon available
Most of the SCADA applications have been ported to Linux (and so usable from all operator consoles)
CIET provides part of the data to the main Cryo controls (temperature)
SCADA system, on top of FESA, on top of FIP GW (CIET, QPS, Survey)
Rest of industrial controls are the classical PLS and the SCADA on top to control the system
Maturity : 1 year of operations encountered several problems and solved many bug
Linux: one uniform platform for OP
12 June 2008
E. Hatziangeli AB/CO

10. Cryogenics Control System
Local & Central
Control Rooms
SCADA Data Servers
LHCA
LHCCA
LHCB
LHCCB RM PA
Profibus DP
WorldFIP
Return Module S78 & S81
Main Dryer
Comp 4.5K
Comp 1.8K
LN2 Buffer
Comp 1.8K
Comp 4.5K
Main Dryer
Surface
QSAA
QSCA
QSCCA
QSCCB
QSCB
QSDN
QSAB
UCB 4.5K
Cold Box 4.5K
QSKA
QSRB
QSRA
Shaft
QURA
Several system distributed around the machine that need to constantly interchange information via Ethernet
Same color indicate same component
CB 1.8K
Cavern
CB 1.8K
QURCA
QURCB
QUI
Connection
Box
Alcoves
RM78
Sector 78 (3.3 Km)
Sector 81 (3.3 Km)
RM81
12 June 2008
E. Hatziangeli AB/CO
Tunnel

11. Cryogenic Controls Reliability
The operation of cryogenics sectors has revealed a high risk dependency of the cryo control system on the reliability of the Technical Network
Steps taken to reduce the dependencies
PLC architecture was rationalized  no dependency on Ethernet of the cryogenics control loops for production equipments
Architecture of the network components was optimized  minimum dependency on communications equipment (switches)
Powering of network component was checked homogenization where possible with the cryo powering
Work ongoing
Identify the weak network components and improve (fiber–copper)
Consolidate the restart of communication after a network failure
Ensure interventions on Technical Network (hardware & software) are carefully planned and agreed with Operation
All components are put in the same switched when they were in several
All powered with proper power and all switched from the same
Consolidate the restart of communication after a network failure (connection between GUIs and server on loss of communication is not correct they block
Tunnel exception
WorldFip based architecture imposed the use of FEs and then a Ethernet connection between those FEs and the CRYO PLCs.
This communication is unavoidable with this architecture and conditions strongly the reliability of the controls.
The Ethernet dependence is also influencing
FEs: boot server
CIET, QPS comms: CMW name server
12 June 2008
E. Hatziangeli AB/CO

12. Machine Interlocks Beam Interlock System Powering Interlock System +
for Protecting the Equipments
for Beam Operation
for Protecting Supra-ConductingMagnets
and Normal Conducting Magnets
Beam Interlock System
(VME based)
Powering Interlock System
(PLC based) +
Safe Machine Parameters system
(VME based)
Warm Magnet Interlock System
(PLC based)
Fast Magnet current Change Monitors (FMCM)
12 June 2008
E. Hatziangeli AB/CO

13. Powering & Warm Magnets Interlocks
Powering Interlock Controllers
36 units of PLC based system protecting ~800 LHC electrical circuits
monitored via PVSS Supervision
Operational and daily used during HWC
Warm magnet Interlock Controllers
8 units of PLC based system protecting ~150 LHC normal conducting magnets
monitored via PVSS Supervision
Operational and daily used during HWC
12 June 2008
E. Hatziangeli AB/CO

14. Monitored by Operational Application
Beam Interlock System
Beam Interlock Controllers
19 VME systems and
~200 connections with most of the LHC systems
Monitored by Operational Application
User Permit is received by all systems which give an input to BIS
Everything is installed and tested (System tests)
Integration tests in progress. Test are done point by point
No RBAC for the moment, BIS is self protected and the safe beam flag will unmask the signals automatically at the hw level if the condition change and we have high intensity or energy.
Individual System Tests successfully performed 
on going BIS Commissioning (involving all User systems)
done in // with HWC
3/8 points already performed
BIS will be ready for the machine checkout…
12 June 2008
E. Hatziangeli AB/CO

15. (TT40 incident in 2004) 12 June 2008 E. Hatziangeli AB/CO
No marks or damage
on magnet flanges
Beam
Vacuum chamber cut
(outside view)
~110 cm
(inside view)
Ejected material opposite cut
(inside view)
12 June 2008
E. Hatziangeli AB/CO

16. Fast Magnet current Change Monitors - FMCM
Successful collaboration with DESY
DESYdevelopment + CERN adaptation
First units successfully used during the SPS Extraction tests and CNGS runs in 2007
currently being re-commissioned for 2008 runs
Installation and commissioning in progress
12 monitors deployed in the LHC (+ 14 in Transfer Lines), including ALL septa families
LHC installations to be completed next month (12 devices)
1st version of FESA class and Java supervision available since June 2007
minor consolidation work in progress
View of
FMCM board
I (A)
Must talk to Brunofor the GUI shots *********
The FMCM measures difference in current in a very short timescale. Min deviation that FMCM triggers is ~ 2x10-4
FMCM will be one of the systems to send PM data to the PM server in case of an event.
FMCM: reaction time and % current change detection level => values were set out of simulation failure of supplies by simulating exponential current decay and what happens to the beam in those cases
So different values were for different types of magnets. In general is ~ 3x10-4 of Nominal current (0.1per mille of nominal ) up to 5x10-3 for other types Reaction time starting from 50 microsecond up to 1milli second, but it will trigger as soon as it see the change of current.
I (A)
10 ms
500 ms
FMCM <103
PC current
FMCM trigger
 0.1% drop !
12 June 2008
E. Hatziangeli AB/CO
time (ms)
time (ms)

17. Timing System major components
The LHC central timing
Master, Slave, Gateway using reflective memory, and hot standby switch
The LHC Injector chain timing (CBCM)
Master, Slave and Gateway using reflective memory, and hot standby switch
Timing is distributed over dedicated network to timing receivers CTRx in front ends
LHC and SPS safe machine parameter distribution
12 June 2008
E. Hatziangeli AB/CO

18. Safe Machine Parameters
The SPS and LHC safe beam flags and beam energy are distributed on the LHC timing network
Work needs to be done for the final system to be ready
The CTR timing receiver modules are able to distribute the beam energy and Safe Beam flags without any software to ensures higher reliability
All timing receivers are monitored by DIAMON
Powerful diagnostics for receivers
simulated beam energy is distributed on the LHC timing network
12 June 2008
E. Hatziangeli AB/CO

19. Outline LHC controls infrastructure – overview
Status Report on Core Controls
Front ends Hardware and Software
Databases
Industrial Controls
Machine Interlocks
The LHC timing system
Core Services and Applications
Post mortem
Sequencer for HWC
See next talk for Beam Sequencer
Logging
Software Interlock System
LSA (see next talk)
Controls Security
Role Based Access
Management of Critical Settings
LHC Controls Security Panel
Controls Infrastructure Tests
Deployment on LEIR, SPS, LHC TL
Dry runs - Commissioning
Scalability Tests
LHC Timing Crash Tests
RBAC tests
Monitor and Diagnostics
LASER
DIAMON
Injector Renovation
Summary
12 June 2008
E. Hatziangeli AB/CO

20. Post Mortem - Towards Beam Commissioning
Global PM Analysis:Global Event sequence, summaries, advised actions, event DB,…
Global event sequence
Machine Protection OK
Advised Actions
Validation of machine protection features
Pre-analysis of PM buffers into result files
Flagging of interesting systems/data reduction
Database catalogue
Individual System Analysis & Checks
IPOC-BIS
Event Sequence
BLM, BPM > threshold
Circuit events
I/XPOC
Data completeness and consistencycheck at system and global level (minimum data, configurable)
0) Specification are written for data completeness and consistency defining the amount of data in case of a global post mortem event.
This means did we get everything ensuring all appropriate data is there in a common place.
This is the most important thigh to have place for the beginning with some viewers to view the data.
With BI the work is advanced as the use the same template to send data to the PM (50MB data fpr the BLMs)
1) DATA: BLM, BPM tests data, Circuits events, XPOC
2) First analysis mechanism to be in place
3) Global analysis is for later and Advised action (ready to take beam again or not)-> SIS to allow or block injection
Concerns
PM data could be lost if there is a power cut in the FE but maybe possible to reconstruct of having the other devices of the same type
Even BIC is not protected against this
Upon beam dump / self triggering, systems start pushing data to PM system, Logging, Alarms, etc…
BLM
BPM
FGC
QPS
PIC/WIC
BIS
XPOC
FMCM
12 June 2008 …
E. Hatziangeli AB/CO … 20

21. Post Mortem Readiness
Many tools are ready for HWC and tested on sectors 5-6 and 7-8
Implemented since December 2007
Automatic Test Analysis on three applications (expert override possible)
Calculated result parameters sent to Sequencer for MTF upload.
Well defined GUI for each test step.
Test results electronically signed by role using RBAC
Event recognition with Event Builder
Redundant services for PM collection and data
Scalability tests with first beam clients started (BLM, BPM)
To do for 2008
Parallel sector commissioning still to be tested
For the 600A circuits many steps are still to be automated.
Further validation tests with beam clients to be done (BLM, BPM, RF, etc…)
Extend framework from HWC to beam operation
Implement higher level of Automated Test Analysis
Data completeness checks & Individual system tests
Automatic tests analysis (PNO)
Test results electronically signed by role using RBAC (5 different roles defined)
Event recognition with Event Builder (Quench, Fast Power Abort, …).
Redundant service (both receiving the PM data at the same time from the equipment) Analysis tools get data from master. (2 HW services in Meyrin and CCR)
Up to now only 2 fronts in ll was Ok but several sectors in the near future might bring surprises but we do not anticipate any issues from experience. Will there be bottlenecks?
Implement 1st and 2nd level of automated analysis (Data completeness checks and Individual system tests for main clients)
12 June 2008
E. Hatziangeli AB/CO 21

22. Readiness of HWC sequencer
First version of the sequencer deployed in early 2007
Many new versions with improvements deployed since
HWC Sector 7-8 May-Jul Winter 08, 5-6Spring 08
~ 35 sequences written and maintained by 3 HWC experts
Sector 4-5: over 1700 sequences executed, in 5-6 over 600
Essential tool for HWC
Overall it works well and satisfies the requirements
Sequencer (the tool) is complete, no important new features needed
Sequences (the tests) are maintained by HWC experts
Ready for multi-sector / multi-front HWC
Used in multi-front operations for over a year
Recent experience in multi-sector operations
“normal” HWC is done in sector 78
training quenches are done in sector 56
No scalability issues are anticipated
Experience in multi-sector is more recent: for the last couple of weeks, “normal” HWC is done in sector 78 while training quenches are done in sector 56 on the Main Bend magnet. However, I don’t forsee any scalability problem for the sequencer at all.
HWC Sequencer Overview
12 June 2008
E. Hatziangeli AB/CO

23. Logging Service Readiness
Logging for Operation
Data logged from PS Complex, SPS, CNGS, LHC HWC, LHC , any type of equipment
Processes run continuously on dedicated machines
Monitored through Alarms system LASER & DIAMON & diagnostic application
Logging for equipment commissioning
Dedicated service, running in the environment of the specialist
Aim: validation of the equipment behavior before operational deployment
No interference with Operational logging
Requirement for a watchdog system (coming weeks)
For critical data (INB, CNGS neutrino events, ...) continuous monitoring of data logged in DB, generation of a specific alarm
12 June 2008
E. Hatziangeli AB/CO

24. Software Interlock System - SIS Overview
Very useful system to anticipate failures and gives early alarms
Accommodates complex interlock logic
Complements BIS (hardware) as protection system
Proved to be reliable tool for operations
Excellent experience in SPS (900 parameters monitored)
12 June 2008
E. Hatziangeli AB/CO

25. SIS for LHC Gives 2 Permits for Injection BICs(Beam1 & Beam2)
All PCs not HW interlocked (~ 800, orbit correctors, warm magnets)
Current of separation dipoles and MCBX orbit correctors
Ring & injection screens (only IN when mode inject-dump)
Extraction screens
Circulating beam intensity limit
Gives Permit for the LHC ring (dumps the beam - initially alarms)
Integrated field of orbit correctors (beam dump energy tracking)
Extraction screens combined with intensity + energy
Orbit at TCDQ
Future work
RBAC integration
Critical settings monitoring (MCS) from LSA
Refinement of the configuration as we progress with LHC
Current of separation dipoles and MCBX orbit correctors (IR1, 2, 5, 8 exp. protection)
MCBX – orbit correctors
MCS - management of critical setting
3 Permit trees (2 inj and 1 ring)
12 June 2008
E. Hatziangeli AB/CO

26. Outline LHC controls infrastructure – overview
Status Report on Core Controls
Front ends Hardware and Software
Databases
Industrial Controls
Machine Interlocks
The LHC timing system
Core Services and Applications
Post mortem
Sequencer for HWC
See next talk for Beam Sequencer
Logging
Software Interlock System
LSA (see next talk)
Controls Security
Role Based Access
Management of Critical Settings
LHC Controls Security Panel
Controls Infrastructure Tests
Deployment on LEIR, SPS, LHC TL
Dry runs - Commissioning
Scalability Tests
LHC Timing Crash Tests
RBAC tests
Monitor and Diagnostics
LASER
DIAMON
Injector Renovation
Summary
12 June 2008
E. Hatziangeli AB/CO

27. Role Based Access (RBAC) Overview
Need to prevent
Well meaning person from doing the wrong thing at the wrong moment
Ignorant person from doing anything at any moment
Application
RBAC
RBAC Token:
Application name
User name
IP address/location
Time of authentication
Time of expiry
Roles[ ]
Digital signature (RBA private key)
CMW client
FESA
CMW server
Access
MAP T
Server
Configuration DB
Authentication:
User requests to be authenticated
RBAC authenticates user via NICE user name and password
RBA returns Tokento Application
Authorization:
Application sends token to Application Server (3-tier env.)
CMW client sends token to CMW server
CMW server (on front-end) verifies token
CMW server checks AccessMapfor role, location, application, mode
12 June 2008
E. Hatziangeli AB/CO

28. Management of Critical Settings - MCS
Need to ensure
Critical parameters, which can compromise the safety of the machine
can only be changed by an authorized person and nobody else
are what they are supposed to be
MCS ensures
Critical parameters are only changed by authorized person
RBAC for Authentication & Authorization
It signs the data with a unique signature to ensure critical parameters have not changed since the authorized person has updated it
Public-private key digital signatures
12 June 2008
E. Hatziangeli AB/CO

29. LHC Controls Security Panel - LCSP
The LHC Controls Security Panel is mandated to address all the technical and non-technical issues concerning AB security for Controls
Take responsibility for the RBAC data (ROLES and RULES)
Ensure all critical parts of the machine are protected
Take responsibility for the CNIC actions
reduction of Trusted list, change of operational account passwords,..
CNIC: Computing and Network Infrastructure for Controls
12 June 2008
E. Hatziangeli AB/CO

30. Outline LHC controls infrastructure – overview
Status Report on Core Controls
Front ends Hardware and Software
Databases
Industrial Controls
Machine Interlocks
The LHC timing system
Core Services and Applications
Post mortem
Sequencer for HWC
See next talk for Beam Sequencer
Logging
Software Interlock System
LSA (see next talk)
Controls Security
Role Based Access
Management of Critical Settings
LHC Controls Security Panel
Controls Infrastructure Tests
Deployment on LEIR, SPS, LHC TL
Dry runs - Commissioning
Scalability Tests
LHC Timing Crash Tests
RBAC tests
Monitor and Diagnostics
LASER
DIAMON
Injector Renovation
Summary
See next talk
12 June 2008
E. Hatziangeli AB/CO

31. LHC controls infrastructure Scalability
Tests =>Preliminary results, issues & foreseen solutions
Systems which scale to LHC full load
Software Interlock System SIS
Data Concentrators (BLMs, BPMs)
Alarm system LASER (new architecture)
Controls Middle Ware (CMW) /Java API Parameter Control (JAPC)
Diagnostic & Monitoring tool DIAMON
Systems potentially critical (tests ongoing- results mid June)
Post Mortem
Logging service
Scalable to LHC load from all clients except LHC BLM
Preliminary limit : ~ 5000 parameters/second
Bottleneck : SQL calls management by Oracle Server
12 June 2008
E. Hatziangeli AB/CO

32. Failures in Central Timing
Tests have been performed to validate the behaviour of the Controls Infrastructure when the Central Timing crashes
These “crash” timing tests are on going
Results
The behaviour of the control system with no timing is correct
The application programs, servers and front ends recovered without manual intervention when timing returns
12 June 2008
E. Hatziangeli AB/CO

33. RBAC Dry Runs
The LHC Controls Security Panel (LCSP) is preparing an RBAC dry-run end June/early July
The RBAC default behavior is changed to
“Access with no RBAC token is refused”
Property not protected is not authorized
All Equipment servers will be loaded with RBAC access maps
Typical applications will be tested
LHC 2-tier & 3-tier applications
LHC Core controls (LSA)
Background servers, concentrators
Fixed Displays
12 June 2008
E. Hatziangeli AB/CO

34. Outline LHC controls infrastructure – overview
Status Report on Core Controls
Front ends Hardware and Software
Databases
Industrial Controls
Machine Interlocks
The LHC timing system
Core Services and Applications
Post mortem
Sequencer for HWC
See next talk for Beam Sequencer
Logging
Software Interlock System
LSA (see next talk)
Controls Security
Role Based Access
Management of Critical Settings
LHC Controls Security Panel
Controls Infrastructure Tests
Deployment on LEIR, SPS, LHC TL
Dry runs - Commissioning
Scalability Tests
LHC Timing Crash Tests
RBAC tests
Monitor and Diagnostics
LASER
DIAMON
Injector Renovation
Summary
12 June 2008
E. Hatziangeli AB/CO

35. Alarms (LASER) for LHC An important increase in expected alarm events
Required availability 365 days/24h
Alarm console extended
Allows for dynamic grouping of alarms
New alarm definition database schema
Ensures the data quality by reducing redundancy and protecting against incomplete data
Alarm server modified fundamentally to allow
Fast response to increase in load
Increased resilience to external failures and improved diagnostics tools
LASER console
LASER CORE
LASER source
LASER DB
12 June 2008
E. Hatziangeli AB/CO

36. DIAgnostic & MONitoring System - DIAMON
DIAMON provides
Software infrastructure for monitoring the AB Controls Infrastructure
Navigation Tree
Group View
Monitoring Tests Details Repair tools
Easy to use first line diagnostics and tool to solve problems or help to decide about responsibilities for first line intervention
On the left via the navigation tree the OP have access to the various groups of monitored items (grouped by Acce, location, function, …anything allowed)
In the group view the summary state of each item in the group is indicated
The detailed panel provided information for the selected item – Op can make the 1st line diagnostic through this and proceed to repair
12 June 2008
E. Hatziangeli AB/CO

37. Outline LHC controls infrastructure – overview
Status Report on Core Controls
Front ends Hardware and Software
Databases
Industrial Controls
Machine Interlocks
The LHC timing system
Core Services and Applications
Post mortem
Sequencer for HWC
See next talk for Beam Sequencer
Logging
Software Interlock System
LSA (see next talk)
Controls Security
Role Based Access
Management of Critical Settings
LHC Controls Security Panel
Controls Infrastructure Tests
Deployment on LEIR, SPS, LHC TL
Dry runs - Commissioning
Scalability Tests
LHC Timing Crash Tests
RBAC tests
Monitor and Diagnostics
LASER
DIAMON
Injector Renovation
Summary
12 June 2008
E. Hatziangeli AB/CO

38. Injector Controls Renovation - Status Report
Injector Controls Architecture - InCA
Architecture validation with critical Use Cases
Check interfacing of the various components
LSA core
Standard CO components for Acquisition
Standard PS and LSA Applications interfaced to the core
Check data flow
Low-level trim and monitoring values of correctors
Orbit correction using LHC Beam steering application
High-level trim + drive a front end
Results
Whole data flow validated
Architecture closer to the final one
Injector Complex FE Renovation – 2nd half of 2008
A "strategic" plan for the renovation of the FE controls infrastructure is due by mid-2008
Development and validation of new Front-end solutions in view of their first deployment in 2009
don’t check the data validity
12 June 2008
E. Hatziangeli AB/CO

39. Summary
The LHC controls infrastructure had been targeted for readiness for an engineering run at 450 GeV in November This goal has been met.
The ongoing hardware commissioning and the extensive use of programs and databases (“learn by doing”) have significantly changed the specifications and the resulting follow-up and work has been done.
Additional functionality has been prepared in network security (RBAC) - diagnostic tools (DIAMON)
12 June 2008
E. Hatziangeli AB/CO

40. End
12 June 2008
E. Hatziangeli AB/CO

41. Accelerator Databases
Off-line Databases
Data not needed directly to interact with the accelerator
Database serves as source for on-line databases
Synchronization mechanisms propagate the data for consistency
Many database services, e.g.
Layout database
Online Databases
Data needed instantaneously to interact with the accelerator
Database is between the accelerator equipment and the client (operator, equipment specialist, software developer)
Many database services, including APIs, and applications
LSA – Accelerator Settings database
MDB – Measurement database
LDB – Logging database
CCDB – Controls Configuration
LASER – Alarms database
Part of the offline db
Assets (MTF)
Documentation (EDMS)
Drawings (CDD)
Part of the online database
TIM – Technical Infrastructure Monitoring database
E-Logbook – Electronic Logbooks
CESAR – SPS-EA Controls
12 June 2008
E. Hatziangeli AB/CO

42. Offline Database Readiness Status
Layout database
Scope includes accelerator equipment (ring, injection & dump lines) and related electronics (tunnel and surface buildings) and field bus connections
Racks & electronics incorporated up to the level of detail of the equipment group requirements
Layout data is now used as foundation for the controls system
Still to do
More data is being captured relating layout and assets information
Tools for data maintenance still to be put in place – data needs to be kept up-to-date by the equipment owners
Layout DB:
(Cryoinstr, BLM, BPM level of details very high (navigation down to the lowest level of details down to the card installed)
More data is being captured relating layout and assets information=> important as machine goes to exploitation
12 June 2008
E. Hatziangeli AB/CO

43. Database Readiness Status
Operational Settings
Highest-availability, Fast response
Data model enhanced to cover functional extensions for Role Based Access (RBAC) , XPOC, Sequencer
PS Controls Renovation requirements (InCA)
Logging Service
High-availability
Currently >109 records/day ~30GB/day ~10TB per year
New database hosting in place since March 2008
Common logging infrastructure for the complete accelerator chain
Sustained increasing logging requirements for HWC and beam related data
Improved data retrieval tool (TIMBER) in line with end-used needs
Now we have ~5 TB of logged data HWC will go down and beam operations will increase in data.
Operational Settings: last year model stabilized and this year enhanced to cover InCA requirements, + RBAC, XPOC => confidence in the model)
Logging
Sustained increasing logging requirements for HWC and beam related data – currently +109 records/day ~30GB/day (BLMs tests with beam related data….) today above 1 billion records a day with its timestamp or 30 GB => 10 Tera bytes per year…) The main issue here is to indentify correctly the important information that needs to be in the database, level of confidence increase data should decrease and stabilize
I need a picture to relate the part
What happened if db is off line and not
Operational settings is the most critical because ether LS operators will not be able to controls and sett the devices
Statistics (23 Aug 2007) LHC LOG scheduled outages time took and impact on operations – 1 occasion per year from min to some hours there were no settings.
If no setting in LHC the OP will dump depending in which part of the cycle exists and HW interlocks will kick in if there is
We do not need the DB to stop the beam – promise to look in the stand by databases (< 10 Gbytes data for LSA) double the infrastructure and do not use just a failover in catastrophe case.
Everything is now clustered since march , (see the slide in 8th May TC) mother board exchanged, things happened without the end users noticing, disk are hot pluggable, double everything,..
Scheduled interventions (quarterly security patches) by IT are agreed for convenient time for OP.
Logging is buffered before they send it to the logging DB to the measurement DB
If all DB goes down this indicated a serious
12 June 2008
E. Hatziangeli AB/CO

44. SMP: Safe Machine Parameters
Generation and distribution of critical parameters over the Timing network
Interfaces with:
← BETS (Beam Energy Tracker system)
← BCT system
BLM system
Kicker system
Experiments
Beam Interlock System
Beam Energy,
Safe Beam Flags,
Stable Beam Flag,
etc…
BLM receive the energy to adapt the thresholds accordingly
Kickers to adapt the strength of the kick to the energy of the beam
Experiments to judge the right time to put their movable devices in the beam
BIS using the safe beam flag to know if some of the BIS signals can be masked by OP (low energy and low intensity)
For SPS generated flag will indicate safe extraction.
Timing is sending the safe Beam Flags , 10HZ aand there is a timeout in the BIS system and when it does not receive the flag it makes the flags unsafe.
SPS TI8 is ready and installed and tested during the dry run
SMP is also critical with high intensity – basic version 
- for the SPS: First version already in operation
for the LHC: First version will be installed in June 2008
Final version for both machines are planned for 2009
Two SMP variants:
one VME controller for LHC
another one for SPS
View of
generic SMP board
12 June 2008
E. Hatziangeli AB/CO

45. Post Mortem for HWC MTF LSA
Event Analyser
PM events waiting
To be signed by experts
HWC sequencer
Result of test
Result
Current cycle
MTF
LSA
The sequencer is currently used in 3 different places: HWC, SPS and LHC dry runs.
This slide shows its usage for HWC, laters slides will show the SPS/LHC usage.
The light blue parts are the same as on the slide before, The brown parts are specific to the HWC usage, and are going to be explained next.
Let’s first explain how a typical HWC test works, The purpose of HWC is to check if electrical circuits work well.
A circuit is composed of one or more power converters (FGC), the PIC and the QPS.
A typical HWC test sequence executes a series of test steps and checks if everything works fine.
1) The sequencer starts the HWC tests sequence by loading information about the tests to be executed and parameters of the electrical circuit from the LSA DB.
2) Then it sends commands to this equipment to drive it through a series of tests steps.
It may take some measurements and provokes a PM event.
If a PM event is triggered, the equipment sends data to the PM analysis system, where experts check if everything went as expected.
When the sequence has finished, the sequencer collects all results of the test (amongst others from the PM analysis).
This data is stored into the LSA DB for real-time follow-up and sent to the MTF system for long-term archival.
The HWC sequencer is sending the current cycle to be executed by the equipment
Current cycle produces a PM event (+ logging data) which are stored in their corresponding DB
Sequencer sends the current cycle to event analyser.
Sequence tells the EA which tests are run on which circuits.
The list of the PM event waiting to be analysed are displayed.
The OP will start the test analysis application and pass or fail it (and sign it)
The selected test shows two experts having signed the test and one still has to sign.
RBAC is used for the signature
RBAC protects that the right people sign for the corresponding tests – no cross signatures allowed
One of the analysis application is thePiNO (Power To Nominal) for the current to nominal
The red curve shows the current ramp to nominal.
V_Meas is the voltage measured on the PC (the jump seen at the beginning of the ramp is the inductance seen by the circuit seen at the beginning of the ramp
U_LEAD voltage on the leads. Based on these we can measure the resistance of the circuit.The lead voltage sees only the resistive part of the circuit (increase a bit on the plateau due to heat)
Notice that testing of the limits are done during the analysis of the data and the green colour indicates specific tests are passed but the OP will define the end result (OP assisted analysis)
Expert sign by clicking pass or fail
The second screen shot shows the electronic sign window: On any
Analysis application, clicking on the Pass or Fail button, this
window opens and the expert enters his NICE login name and using RBAC
we check if this person has the correct role to sign for the test.
This protects against errors and gives traceability.
Viewing & Test analysis applications
Data collection, pre-processing and storage
Equipment
Post Mortem
Logging DB
12 June 2008
E. Hatziangeli AB/CO

46. The Logging Service LOGGING MEAS DB DB LOGGING DB
Contain relatively slow time series data Store data online for at least 20 years
MEASUREMENT DB
Raw time series data, up to 2Hz rate Store data for 7 days Send filtered data of interest to Logging DB each 15 minutes
LOGGING DB
Filter each 15 min
MEAS DB
LOGGING
process
LOGGING
process
LOGGING
process
60 106records / day
DEVICE
CMW
METER browsing & data extraction
TIMBER browsing & data extraction
LOGGING :
resp for acquiring data from the devices of the machine
buffering and writing them to the DB
Eventually perform data processing on data
Objective: by extracting data from the Loggting DB, end users are able to do a post mortem analysis of a specific event which occurred at a given time in the machine. For ex. to be able to reconstruct an accident by correlating a beam loss to a glitch of the current in a power converter.
Configuration of JAPC MM: 2 ways of configuring (list of devices to be monitored) JapcMMdepdg on the client
LHC HWC from a pre-defined list of devices in the LSA DB for LHC HWC (necessary as the device set changes from one test to the next)
Xml file (for other clients where the device set is static)
Config of logging: from a config file: to map the value received from the japcMM to the corresponding variable in the DB
Databases:
Very close collaboration with ABCODM specially with Chris.
2 DBs + JAVA API between logging and Measurement DB: under DM responsibility.
MEAS DB: designed for short term storage of data and allows high frequency loading.
LOGGING DB: data flushed every 15 min from MEAS DB to LOGGING DB.Transfer of long term marked variables to Logging DB for permanent storage
Based on JapcMMresp for subscription mechanism
JAPC
Intensive usage of factories by Jakub
use of japc multiple factories by Roman:
TGM
Intermediate servers
SPS sems
DIM
Connection to Ilya
DEVICE
DEVICE
12 June 2008
E. Hatziangeli AB/CO

47. LCSP – RBAC actions Simplification of the definition of RBAC Rules
Introduction of device and property grouping
Reduce number of rules and enhance maintainability of RBAC data
Equipment State definition
Equipment will only have 2 RBAC states : OPERATIONAL or NON-OPERATIONAL
This state is derived from the LHC Machine Mode distribution
Standardize the ROLES definition :
OP group -> responsible for OP roles
Roles used for get/set/monitor when STATE is OPERATIONAL
Equipment groups -> responsible for expert roles
Roles used for get/set/monitor when STATE is NON OPERATIONAL
A PIQUET role per equipment group, OP managed => all rights all time
Roles are empty
Filled only by OP when an intervention is required
Equipment State definition : first version is foreseen for mid-June
12 June 2008
E. Hatziangeli AB/CO

48. LCSP – CNIC actions
The generic accounts (more than 500) are reviewed with the aim of reducing its number and forcing more frequent password changes
Enforce the CNIC policies by reducing the number of Trusted hosts
Replace their functionalities with either
Clusters of Windows Terminal Servers (WTS)
Locally managed consoles where AB/CO manages the installation
Trusted hosts that have access to the Technical Network from the General Purpose Network
Software packages and patches by AB/CO on Locally managed console
12 June 2008
E. Hatziangeli AB/CO