2. CERN Dependable Design Example ITER – Machine ProtectionB. ToddMay 2010 Thanks to : TE/MPE/MI, CERN Machine Protection Panel, et al 1v0 Beam Related Machine Protection

3. CERN benjamin.todd@cern.ch ITER – Machine Protection CERN CERN, the LHC and Machine Protection CERN 3 of 23 CERN Accelerator Complex Lake Geneva Geneva Airport CERN LAB 1 (Switzerland) CERN LAB 2 (France)

4. CERN benjamin.todd@cern.ch ITER – Machine Protection CERN CERN, the LHC and Machine Protection CERN 4 of 23 CERN Accelerator Complex Lake Geneva Geneva Airport CERN LAB 1 (Switzerland) CERN LAB 2 (France) Proton Synchrotron (PS) Super Proton Synchrotron (SPS) Large Hadron Collider (LHC)

5. CERN benjamin.todd@cern.ch ITER – Machine Protection CERN CERN, the LHC and Machine Protection CERN 5 of 23 CERN Accelerator Complex Large Hadron Collider (LHC) Beam-1 Transfer Line (TI2) Beam-2 Transfer Line (TI8) CERN Neutrinos to Gran-Sasso (CNGS) Beam Dumping Systems ~ 9 km ~ 5.5 miles Super Proton Synchrotron (SPS) 150m underground, 100us for one turn, 1e12 protons / injection

6. CERN benjamin.todd@cern.ch ITER – Machine Protection The Large Hadron Collider 6 … to get 7 TeV operation… LHC needs 8.3 Tesla dipole fields with circumference of 27 kms (16.5 miles) … to get 8.3 Tesla … LHC needs super-conducting magnets <2°K (-271°C) with an operational current of ~13kA cooled in super fluid helium maintained in a vacuum Stored energy in the magnet circuits is about 9GJ …To see the rarest events… LHC needs high luminosity of 10 34 [cm -2 s -1 ] Which gives a stored beam energy of 360 MJ per beam Overall consideration for machine protection: an accidental release of beam or magnet energy can lead to massive damage 1 ppm Collisions generate PetaBytes of data Per year two orders of magnitude higher than others A magnet will QUENCH with milliJoule deposited energy World’s largest machine 10x less pressure than on moon surface [11]

7. CERN benjamin.todd@cern.ch ITER – Machine Protection Stored Magnetic Energy 7 Kinetic Energy of Aircraft Carrier at 50 km/h ≈ 9 GJoule

8. CERN benjamin.todd@cern.ch ITER – Machine Protection Stored Beam Energy 8 E proton ≈ 1.1 x 10 -6 N p_bunch ≈ 1.15 x 10 11 N bunch ≈ 2808 Kinetic Energy of a 200m train at 155 km/h ≈ 360 MJoule

9. CERN benjamin.todd@cern.ch ITER – Machine Protection Disposing of the Energy 9 CERN, the LHC and Machine Protection 9 of 23 1.Magnet Energy Powering Interlock Controllers + Quench Protection System Emergency Discharge 2.Beam Energy Many Systems + Beam Interlock System + LHC Beam Dumping System Emergency Dump … during a 10 hour mission… …if anything goes wrong… 8m long absorber Graphite = 800°C Concrete Shielding Beam is ‘painted’ diameter 35cm

10. CERN benjamin.todd@cern.ch ITER – Machine Protection Beam Related Machine Protection 10 …Injector chain : left to right… Beam becomes dangerous in SPS! Injection (450 GeV) … 0.0008% beam loss = QUENCH magnet … 0.5% beam loss = DAMAGE machine Collision (7 TeV) … 0.0000005% beam loss = QUENCH magnet … 0.005% beam loss = DAMAGE machine [15] PS = Proton-Synchrotron 1-25GeV SPS = Super-Proton-Synchrotron 25-450 GeV LHC = Large Hadron Collider 450-7000 GeV

11. CERN benjamin.todd@cern.ch ITER – Machine Protection SPS experiment at 450 GeV 11 Controlled SPS experiment to qualify simulations At 450GeV … 8x10 12 protons causes damage beam size σ x/y = 1.1mm/0.6mm Plate 2mm thick 6 cm 8x10 12 6x10 12 4x10 12 2x10 12 0.1% LHC Full Beam Energy! Beam in LHC is 10x smaller!! [14]

12. CERN benjamin.todd@cern.ch ITER – Machine Protection Machine Protection System 12 best failure detection time = 40 us = half turn

13. CERN benjamin.todd@cern.ch ITER – Machine Protection Machine Protection System 13

14. CERN benjamin.todd@cern.ch ITER – Machine Protection Beam Interlock System Function 14 BIS Both-Beam Beam-1 Beam-2 ~200 User Systems distributed over 27 kms LHC has 2 Beams Some User Systems give simultaneous permit Others give independent permit Designed to protect all CERN complex = SPS / LHC / INJECTION / EXTRACTION

15. CERN benjamin.todd@cern.ch ITER – Machine Protection Signals 15 of 25

16. CERN benjamin.todd@cern.ch ITER – Machine Protection Signals 16 of 25

17. CERN benjamin.todd@cern.ch ITER – Machine Protection Signals 17 of 25 NON-CRITICAL / Monitoring = DIFFERENT device Hardware MATRIX = 9500 Complex Programmable Logic Device (CPLD) Hardware Description Language (VHDL)

18. CERN benjamin.todd@cern.ch ITER – Machine Protection Reaction Time 18

19. CERN benjamin.todd@cern.ch ITER – Machine Protection MPS Dependability Requirements 19 MPS safety based on IEC-61508 - losses = downtime and repair cost Safety === protection investment in LHC Availability === get data to experiments Therefore needs equivalent to SIL3( As Low As Reasonable Possible) Only a SUB-SET of the system - beam losses before beam damage!

20. CERN benjamin.todd@cern.ch ITER – Machine Protection Safe Sub-Set 20 Considering that beam loss is needed before beam damage is possible…

21. CERN benjamin.todd@cern.ch ITER – Machine Protection Reliability Sub-Working Group 21 of 29 Operational Scenario: 200 days = 400 x 10h missions + 2h checks Work here thanks J.Uythoven & many others [16] Diagnostics Effectiveness: LHC Beam Dump System As Good As New after checks Beam Interlock System As Good As New after checks Beam Loss Monitors partially regenerated Quench Protection System regenerated periodically Power Interlock Controllers regenerated periodically Redundancy: Beam Loss Monitors have no redundancy Dump Request Apportionment: 60% are planned dumps (end physics) 15% fast beam losses 15% slow beam losses 10% other types of failure Reliability Sub-Working Group established to study the sub-set… Assumptions made:

22. CERN benjamin.todd@cern.ch ITER – Machine Protection Failure Types and Apportionment 22 of 29 Dump Event Planned Unforeseen Beam Loss Other BIS LBDS Beam Dumped Slow Fast BLM QPS PIC 60% 15% 10% Work here thanks J.Uythoven & many others [16]

23. CERN benjamin.todd@cern.ch ITER – Machine Protection Failure Types and Apportionment 23 of 29 Dump Event Planned Unforeseen Beam Loss Other BIS LBDS Beam Dumped Slow Fast BLM QPS PIC 60% 15% 10% Work here thanks J.Uythoven & many others [16] SYSTEM UNSAFETY P(yr -1 ) UNAVAILABILITY Mean (yr -1 ) & S.D. LBDS2.4 x 10 -7 (x2)4 (x2) +/-1.9 BIS1.4 x 10 -8 0.5 +/- 0.5 BLM 1.44 x 10 -3 0.06 x 10 -3 17 +/- 4.0 PIC0.5 x 10 -3 1.5 +/- 1.2 QPS2.3 x 10 -4 15.8 +/- 3.9 MPS5.75 x 10 -8 41 +/- 6.0 SIL310% Newer figures in next slides

24. CERN benjamin.todd@cern.ch ITER – Machine Protection BIS Dependable Design 24 CERN, the LHC and Machine Protection High Dependability High Safety High Reliability High Availability Maintainable “…[BIS] must react to a single change in USER PERMIT by correctly actioning the relevant BEAM PERMIT with equivalent safety better than or equal to Safety Integrity Level 3. Less than 1% of missions must be aborted due to failures in the Beam Interlock System...” BIS has a dependability specification

25. CERN benjamin.todd@cern.ch ITER – Machine Protection So…BIS === SIL3 or better == FMECA 25 Failure Modes, Effects and Criticality Analysis In what way can something go wrong?… …when it does go wrong, what happens to the system?… …and just how much of a problem does this cause?

26. CERN benjamin.todd@cern.ch ITER – Machine Protection FMECA 26 FMECA starts at the Component Level of a system get subsystem schematics, component list, and understand what it does Break a large system into blocks, defining smaller, manageable sub-systems get MTBF of each component on the list, derive P FAIL (mission) derive failure modes and failure mode ratios for each component explain the effect of each failure mode on both the subsystem and system determine the probability of each failure mode happening. Draw conclusions! MIL-STD-1629 FMD-97 MIL-HDBK-338 MIL-HDBK-217

27. CERN benjamin.todd@cern.ch ITER – Machine Protection FMECA 27 MIL-HDBK-217F or manufacturer FMD-97 MIL-HDBK-338 Bill of Materials

28. CERN benjamin.todd@cern.ch ITER – Machine Protection FMECA 28 Designer Knowledge MIL-HDBK-338 Schematic multiply through

29. CERN benjamin.todd@cern.ch ITER – Machine Protection Full Redundancy FMECA Results 29 NE = No EffectM = Maintenance False Dump = unavailabilityBlind Failure = unsafety ~1% of all fills are lost due to a failure of the BIS better than SIL 3 FD = False DumpBF = Blind Failure

30. CERN benjamin.todd@cern.ch ITER – Machine Protection Dependability vs. Configuration 30 Hourly rate is based on MIL, Manufacturer etc. Extrapolation is difficult, whole MPS FMECA approach being verified by another PhD

31. CERN benjamin.todd@cern.ch ITER – Machine Protection Analysed Components 31 of 29 Non-critical = DIFFERENT device and circuits ≈90000 components in BIS Critical = small & simple as possible FMECA = GOOD for discrete NO GOOD for FIRMWARE! 

32. CERN benjamin.todd@cern.ch ITER – Machine Protection Dependable Design Flow 32 Specification – including safety requirements Design – to meet specification FMECA … Signal Integrity Analysis – slew rate, impedance, connections Design for Testing – test coverage, test benching Design for Manufacture – layout, sizes, procurement Over sizing / Thermal considerations – layout, heating, packages Electro-Magnetic Compatibility Testing – shields, grounds, supplies, noise Radiation / Single Event Testing – Single Event Effects, Total Ionising Dose Build Test bench for each board – supplier contract depends on passed tests Power Soak – weeks in lab, switch on, fail? - return to manufacturer Controller Testing – Assemble complete controller 100% testbench Installation & Commissioning Operational Experience … Audited by internal / external reviewers … Finally have hardware system adhering to ALL requirements Should be constant failure rate – flat part of bathtub curve What about VHDL? How does that ‘fail’? Is our design complete?

33. CERN benjamin.todd@cern.ch ITER – Machine Protection Signals 33 of 29 NON-CRITICAL / Monitoring = DIFFERENT device Hardware MATRIX = 9500 Complex Programmable Logic Device (CPLD) Hardware Description Language (VHDL)

34. CERN benjamin.todd@cern.ch ITER – Machine Protection CRITICAL Matrix Verification 1/2 34 of 29 Complete, exhaustive VHDL simulation Two different engineers wrote code (A vs B) BEAM_PERMIT_INFO – One Impossible combination – Not critical Code coverage 100% on critical signals

35. CERN benjamin.todd@cern.ch ITER – Machine Protection CRITICAL Matrix Verification 2/2 35 of 29 Complete, exhaustive Hardware test-bench - 100% of critical signals 4 hours to test everything at ~100k combinations per second Top-Down verification of the Matrix function Front View Rear View

36. CERN benjamin.todd@cern.ch ITER – Machine Protection CRITICAL Matrix Verification 2/2 36 of 29 Complete, exhaustive Hardware test-bench - 100% of critical signals 4 hours to test everything at ~100k combinations per second After installation – we test critical paths online … Front View Rear View

37. CERN benjamin.todd@cern.ch ITER – Machine Protection Online Testing and Checking 37 of 29 1.BIS Pre Operational Testing a) Static Checks Hardware ID numbers Enabled / Disabled Channels Power Supply Redundancy Software Servers b) Dynamic Checks 100% Coverage Internal Test Mode External Test with Users History Buffer Time alignment Safe Beam Flag Reception Post-Mortem Trigger Check Hardware Statuses All OK? Check Hardware Configuration Check Time Alignment Stand Alone Tests Global Tests Rearm System Yes No Intervention Next Slide

38. CERN benjamin.todd@cern.ch ITER – Machine Protection Online Testing and Checking 38 of 29 2. Diagnosis and Monitoring Hardware ID numbers Enabled / Disabled Channels Power Supply Redundancy Software Servers Glitch Counters Frequency Measurements 3. Post Operational Checks who started the dump – which user? Or was the BIC responsible? Internal Fault? Redundancy compromised? Time delays respected? Beam Dump Online Monitoring Diagnosis Non-critical failures = schedule maintenance Critical failures = BIS hardware forces False dump No Yes Post Operational Check Post-Mortem Validation All OK? 4. Post Mortem Checks the whole MPS works correctly No Intervention Back to pre-op checks Previous Slide Yes

39. CERN benjamin.todd@cern.ch ITER – Machine Protection Operational Figures so far… 39 of 29 LHC system used throughout CERN for > 3 years Feedback into production and upgrades already TRACOPOWER Pessimistic figure 217F Some failures due to Non-conforming installation Monitor weakness Identified in 2007 = New PCB design 2008 A lot of time spent bedding-in the system with operations One double blind failure during commissioning several revised specifications as a result

40. CERN benjamin.todd@cern.ch ITER – Machine Protection In Conclusion 40 From the start LHC needed a Dependable Protection System FMECA important tool for verifying our designs injector chain serves as a useful guinea pig online tools and tests we can verify our installation to 100% as good as new VHDL is an unknown! BUT.. Split critical and non-critical Make critical as small as possible Then TEST, TEST and more TEST accept the remaining risk dependable design starts from the first draft of the specification Dedicated teams should work on dependable design Frameworks and Tools are needed CERN has set things in motion for dependable design to be a core competency

41. CERN benjamin.todd@cern.ch ITER – Machine Protection benjamin.todd@cern.ch 41 ITER – Machine Protection CERN Fin Thank you for your attention