CERN Dependable Design Example ITER – Machine ProtectionB. ToddMay 2010 Thanks to : TE/MPE/MI, CERN Machine Protection Panel, et al 1v0 Beam Related Machine Protection

CERN ITER – Machine Protection CERN CERN, the LHC and Machine Protection CERN 3 of 23 CERN Accelerator Complex Lake Geneva Geneva Airport CERN LAB 1 (Switzerland) CERN LAB 2 (France)

CERN ITER – Machine Protection CERN CERN, the LHC and Machine Protection CERN 4 of 23 CERN Accelerator Complex Lake Geneva Geneva Airport CERN LAB 1 (Switzerland) CERN LAB 2 (France) Proton Synchrotron (PS) Super Proton Synchrotron (SPS) Large Hadron Collider (LHC)

CERN ITER – Machine Protection CERN CERN, the LHC and Machine Protection CERN 5 of 23 CERN Accelerator Complex Large Hadron Collider (LHC) Beam-1 Transfer Line (TI2) Beam-2 Transfer Line (TI8) CERN Neutrinos to Gran-Sasso (CNGS) Beam Dumping Systems ~ 9 km ~ 5.5 miles Super Proton Synchrotron (SPS) 150m underground, 100us for one turn, 1e12 protons / injection

CERN ITER – Machine Protection The Large Hadron Collider 6 … to get 7 TeV operation… LHC needs 8.3 Tesla dipole fields with circumference of 27 kms (16.5 miles) … to get 8.3 Tesla … LHC needs super-conducting magnets <2°K (-271°C) with an operational current of ~13kA cooled in super fluid helium maintained in a vacuum Stored energy in the magnet circuits is about 9GJ …To see the rarest events… LHC needs high luminosity of [cm -2 s -1 ] Which gives a stored beam energy of 360 MJ per beam Overall consideration for machine protection: an accidental release of beam or magnet energy can lead to massive damage 1 ppm Collisions generate PetaBytes of data Per year two orders of magnitude higher than others A magnet will QUENCH with milliJoule deposited energy World’s largest machine 10x less pressure than on moon surface [11]

CERN ITER – Machine Protection Stored Magnetic Energy 7 Kinetic Energy of Aircraft Carrier at 50 km/h ≈ 9 GJoule

CERN ITER – Machine Protection Stored Beam Energy 8 E proton ≈ 1.1 x N p_bunch ≈ 1.15 x N bunch ≈ 2808 Kinetic Energy of a 200m train at 155 km/h ≈ 360 MJoule

CERN ITER – Machine Protection Disposing of the Energy 9 CERN, the LHC and Machine Protection 9 of 23 1.Magnet Energy Powering Interlock Controllers + Quench Protection System Emergency Discharge 2.Beam Energy Many Systems + Beam Interlock System + LHC Beam Dumping System Emergency Dump … during a 10 hour mission… …if anything goes wrong… 8m long absorber Graphite = 800°C Concrete Shielding Beam is ‘painted’ diameter 35cm

CERN ITER – Machine Protection Beam Related Machine Protection 10 …Injector chain : left to right… Beam becomes dangerous in SPS! Injection (450 GeV) … % beam loss = QUENCH magnet … 0.5% beam loss = DAMAGE machine Collision (7 TeV) … % beam loss = QUENCH magnet … 0.005% beam loss = DAMAGE machine [15] PS = Proton-Synchrotron 1-25GeV SPS = Super-Proton-Synchrotron GeV LHC = Large Hadron Collider GeV

CERN ITER – Machine Protection SPS experiment at 450 GeV 11 Controlled SPS experiment to qualify simulations At 450GeV … 8x10 12 protons causes damage beam size σ x/y = 1.1mm/0.6mm Plate 2mm thick 6 cm 8x x x x % LHC Full Beam Energy! Beam in LHC is 10x smaller!! [14]

CERN ITER – Machine Protection Machine Protection System 12 best failure detection time = 40 us = half turn

CERN ITER – Machine Protection Machine Protection System 13

CERN ITER – Machine Protection Beam Interlock System Function 14 BIS Both-Beam Beam-1 Beam-2 ~200 User Systems distributed over 27 kms LHC has 2 Beams Some User Systems give simultaneous permit Others give independent permit Designed to protect all CERN complex = SPS / LHC / INJECTION / EXTRACTION

CERN ITER – Machine Protection Signals 15 of 25

CERN ITER – Machine Protection Signals 16 of 25

CERN ITER – Machine Protection Signals 17 of 25 NON-CRITICAL / Monitoring = DIFFERENT device Hardware MATRIX = 9500 Complex Programmable Logic Device (CPLD) Hardware Description Language (VHDL)

CERN ITER – Machine Protection Reaction Time 18

CERN ITER – Machine Protection MPS Dependability Requirements 19 MPS safety based on IEC losses = downtime and repair cost Safety === protection investment in LHC Availability === get data to experiments Therefore needs equivalent to SIL3( As Low As Reasonable Possible) Only a SUB-SET of the system - beam losses before beam damage!

CERN ITER – Machine Protection Safe Sub-Set 20 Considering that beam loss is needed before beam damage is possible…

CERN ITER – Machine Protection Reliability Sub-Working Group 21 of 29 Operational Scenario: 200 days = 400 x 10h missions + 2h checks Work here thanks J.Uythoven & many others [16] Diagnostics Effectiveness: LHC Beam Dump System As Good As New after checks Beam Interlock System As Good As New after checks Beam Loss Monitors partially regenerated Quench Protection System regenerated periodically Power Interlock Controllers regenerated periodically Redundancy: Beam Loss Monitors have no redundancy Dump Request Apportionment: 60% are planned dumps (end physics) 15% fast beam losses 15% slow beam losses 10% other types of failure Reliability Sub-Working Group established to study the sub-set… Assumptions made:

CERN ITER – Machine Protection Failure Types and Apportionment 22 of 29 Dump Event Planned Unforeseen Beam Loss Other BIS LBDS Beam Dumped Slow Fast BLM QPS PIC 60% 15% 10% Work here thanks J.Uythoven & many others [16]

CERN ITER – Machine Protection Failure Types and Apportionment 23 of 29 Dump Event Planned Unforeseen Beam Loss Other BIS LBDS Beam Dumped Slow Fast BLM QPS PIC 60% 15% 10% Work here thanks J.Uythoven & many others [16] SYSTEM UNSAFETY P(yr -1 ) UNAVAILABILITY Mean (yr -1 ) & S.D. LBDS2.4 x (x2)4 (x2) +/-1.9 BIS1.4 x /- 0.5 BLM 1.44 x x /- 4.0 PIC0.5 x /- 1.2 QPS2.3 x /- 3.9 MPS5.75 x /- 6.0 SIL310% Newer figures in next slides

CERN ITER – Machine Protection BIS Dependable Design 24 CERN, the LHC and Machine Protection High Dependability High Safety High Reliability High Availability Maintainable “…[BIS] must react to a single change in USER PERMIT by correctly actioning the relevant BEAM PERMIT with equivalent safety better than or equal to Safety Integrity Level 3. Less than 1% of missions must be aborted due to failures in the Beam Interlock System...” BIS has a dependability specification

CERN ITER – Machine Protection So…BIS === SIL3 or better == FMECA 25 Failure Modes, Effects and Criticality Analysis In what way can something go wrong?… …when it does go wrong, what happens to the system?… …and just how much of a problem does this cause?

CERN ITER – Machine Protection FMECA 26 FMECA starts at the Component Level of a system get subsystem schematics, component list, and understand what it does Break a large system into blocks, defining smaller, manageable sub-systems get MTBF of each component on the list, derive P FAIL (mission) derive failure modes and failure mode ratios for each component explain the effect of each failure mode on both the subsystem and system determine the probability of each failure mode happening. Draw conclusions! MIL-STD-1629 FMD-97 MIL-HDBK-338 MIL-HDBK-217

CERN ITER – Machine Protection FMECA 27 MIL-HDBK-217F or manufacturer FMD-97 MIL-HDBK-338 Bill of Materials

CERN ITER – Machine Protection FMECA 28 Designer Knowledge MIL-HDBK-338 Schematic multiply through

CERN ITER – Machine Protection Full Redundancy FMECA Results 29 NE = No EffectM = Maintenance False Dump = unavailabilityBlind Failure = unsafety ~1% of all fills are lost due to a failure of the BIS better than SIL 3 FD = False DumpBF = Blind Failure

CERN ITER – Machine Protection Dependability vs. Configuration 30 Hourly rate is based on MIL, Manufacturer etc. Extrapolation is difficult, whole MPS FMECA approach being verified by another PhD

CERN ITER – Machine Protection Analysed Components 31 of 29 Non-critical = DIFFERENT device and circuits ≈90000 components in BIS Critical = small & simple as possible FMECA = GOOD for discrete NO GOOD for FIRMWARE! 

CERN ITER – Machine Protection Dependable Design Flow 32 Specification – including safety requirements Design – to meet specification FMECA … Signal Integrity Analysis – slew rate, impedance, connections Design for Testing – test coverage, test benching Design for Manufacture – layout, sizes, procurement Over sizing / Thermal considerations – layout, heating, packages Electro-Magnetic Compatibility Testing – shields, grounds, supplies, noise Radiation / Single Event Testing – Single Event Effects, Total Ionising Dose Build Test bench for each board – supplier contract depends on passed tests Power Soak – weeks in lab, switch on, fail? - return to manufacturer Controller Testing – Assemble complete controller 100% testbench Installation & Commissioning Operational Experience … Audited by internal / external reviewers … Finally have hardware system adhering to ALL requirements Should be constant failure rate – flat part of bathtub curve What about VHDL? How does that ‘fail’? Is our design complete?

CERN ITER – Machine Protection Signals 33 of 29 NON-CRITICAL / Monitoring = DIFFERENT device Hardware MATRIX = 9500 Complex Programmable Logic Device (CPLD) Hardware Description Language (VHDL)

CERN ITER – Machine Protection CRITICAL Matrix Verification 1/2 34 of 29 Complete, exhaustive VHDL simulation Two different engineers wrote code (A vs B) BEAM_PERMIT_INFO – One Impossible combination – Not critical Code coverage 100% on critical signals

CERN ITER – Machine Protection CRITICAL Matrix Verification 2/2 35 of 29 Complete, exhaustive Hardware test-bench - 100% of critical signals 4 hours to test everything at ~100k combinations per second Top-Down verification of the Matrix function Front View Rear View

CERN ITER – Machine Protection CRITICAL Matrix Verification 2/2 36 of 29 Complete, exhaustive Hardware test-bench - 100% of critical signals 4 hours to test everything at ~100k combinations per second After installation – we test critical paths online … Front View Rear View

CERN ITER – Machine Protection Online Testing and Checking 37 of 29 1.BIS Pre Operational Testing a) Static Checks Hardware ID numbers Enabled / Disabled Channels Power Supply Redundancy Software Servers b) Dynamic Checks 100% Coverage Internal Test Mode External Test with Users History Buffer Time alignment Safe Beam Flag Reception Post-Mortem Trigger Check Hardware Statuses All OK? Check Hardware Configuration Check Time Alignment Stand Alone Tests Global Tests Rearm System Yes No Intervention Next Slide

CERN ITER – Machine Protection Online Testing and Checking 38 of Diagnosis and Monitoring Hardware ID numbers Enabled / Disabled Channels Power Supply Redundancy Software Servers Glitch Counters Frequency Measurements 3. Post Operational Checks who started the dump – which user? Or was the BIC responsible? Internal Fault? Redundancy compromised? Time delays respected? Beam Dump Online Monitoring Diagnosis Non-critical failures = schedule maintenance Critical failures = BIS hardware forces False dump No Yes Post Operational Check Post-Mortem Validation All OK? 4. Post Mortem Checks the whole MPS works correctly No Intervention Back to pre-op checks Previous Slide Yes

CERN ITER – Machine Protection Operational Figures so far… 39 of 29 LHC system used throughout CERN for > 3 years Feedback into production and upgrades already TRACOPOWER Pessimistic figure 217F Some failures due to Non-conforming installation Monitor weakness Identified in 2007 = New PCB design 2008 A lot of time spent bedding-in the system with operations One double blind failure during commissioning several revised specifications as a result

CERN ITER – Machine Protection In Conclusion 40 From the start LHC needed a Dependable Protection System FMECA important tool for verifying our designs injector chain serves as a useful guinea pig online tools and tests we can verify our installation to 100% as good as new VHDL is an unknown! BUT.. Split critical and non-critical Make critical as small as possible Then TEST, TEST and more TEST accept the remaining risk dependable design starts from the first draft of the specification Dedicated teams should work on dependable design Frameworks and Tools are needed CERN has set things in motion for dependable design to be a core competency

CERN ITER – Machine Protection 41 ITER – Machine Protection CERN Fin Thank you for your attention