TT40 incident at 23:46 on 25th October 2004 J. Wenninger AB-OP The incident : cause and consequences. Direct ‘follow up’ actions. Lessons for the future. With input from many colleagues of the AB department, in particular B. Balhan, E. Carlier, B. Goddard, M. Jonker, V. Mertens, R. Schmidt, J. Uythoven 25.11.2004 J. Wenninger

What happened ? During a high intensity extraction test, a LHC beam with 3.2 ×1013 protons (nominal LHC injection) impacted in the second quadrupole of the TT40 tranfer line to LHC and CNGS following a PC fault on the extraction septum magnet. The magnet had to be replaced 1 week later : ~ 24 hours downtime for SPS and obviously some dose to the personnel (mainly vacuum and magnet group). The vaccum chamber of the quadrupole was ripped open. The quadrupole coil may be damaged (tbc). We learned a lesson and many people woke up and realized what high intensity beams are ! 25.11.2004 J. Wenninger

(Future) SPS extractions North Area (Slow Extr.) LHC Ring 2 + CNGS (Fast Extr.) LHC Ring 1 (Fast Extr.) 25.11.2004 J. Wenninger

in LSS4 (Long Straight Section) Extraction channel Extraction bumpers (= strong & fast orbit correctors, 4 / plane) :  35 mm amplitude horizontal bump @ beam position monitor. Extraction kicker MKE (5 magnets, 0.53 mrad). Magnetic septum MSE (6 magnets, 22000 A, 12 mrad) : This magnet has a very short time constant of 23 ms ! Extracted beam Circulating beam in LSS4 (Long Straight Section) 25.11.2004 J. Wenninger

(Very) few words on interlocks The TT40/TI8 lines are equipped with a beam interlock system that is essentially identical to the future LHC beam interlock system. The interlock system was fully operational during the test. For the power converter surveillance : The current of the PCs was surveyed a few ms before extraction. No extraction permit was given if the current fell outside a tolerance. The tolerance ranges (TT40 & TI8) : 3 ×10-4 on main dipoles and quadrupoles (2 PCs). 1-2 ×10-3 on other magnets (26 PCs). The average current over 10 ms was used for the interlocks : The ‘dead zone’ where a problem (PC fault) could not be detected anymore was in the range 6 ms + delay from averaging. For the extraction septum this interlock is not sufficient to ensure full safety because the time constant is too short. This fact was KNOWN. A solution to this problem is/was under development (also for the LHC). 25.11.2004 J. Wenninger

Extraction septum in the SPS tunnel 25.11.2004 J. Wenninger

Temperature sensor cables Circulating beam Extracted beam Temperature sensor cables 25.11.2004 J. Wenninger

First part of TT40 in the SPS tunnel QTRF4002 beam impact 25.11.2004 J. Wenninger

The damage on the vacuum chamber Signs of heating over ~ 1 m Chamber is cut over ~ 20 cm 25.11.2004 J. Wenninger

Incident sequence Before the incident we observed PC faults on the MSE extraction septum correlated to unphysical temperature interlocks from the magnet. The magnet expert detected spurious beam induced interlocks due to Electro-Magnetic Coupling (EMC) of beam signals on temperature sensor cables (used for magnet protection). Since the interlocks were FAKE, the expert decided to disconnect the temperature sensors (there is a redundant protection over water T). The beam tests continued, and we were struck by another magnet interlock that was not understood at the time. This interlock fell exactly into our interlock system ‘dead zone’ ! Further tests performed 2 weeks later showed that there was also EMC between the temperature sensors cables and an interlock signal cable on water valves that most likely caused the interlock  ‘coupled’ interlock. 25.11.2004 J. Wenninger

Incident timing Magnetic septum current change The BLUE curve is obtained from a PC simulation (PC off) by AB/PO. The timing of PC current survey (0.1 % tolerance) and of the precise extraction time is obtained from the Beam Interlock System logging. This reconstrction is consistent (within ~ 0.5 ms) with the beam impact point. (reconstructed) (logging) Magnetic septum current change time within SPS super-cycle 25.11.2004 J. Wenninger

What is the probability for such an event ? Naively : The dead-time of the surveillance is ~ 10 ms. The SPS cycle is 28.8 seconds long.  random fault probability ~ 3 ×10-4 More realistic – with our test conditions : The faults were correlated to a high intensity beam with very short bunch length close to the MSE  faults occur mostly close to the extraction time ! Therefore the fault occurs mostly in a time window of 100-1000 ms near extraction.  fault probability ~ 1-10%  rather ‘likely’ ! The lesson : beware of correlated ‘faults’ ! 25.11.2004 J. Wenninger

Follow up / 1 A new interlock cable was installed in the week after the incident between the magnet interlock PLC and the Beam Interlock system. The interlock logic in the MSE PLC was modified : First an interlock signal is sent to the beam interlock system. 10 ms LATER the interlock signal is sent to the PC. the beam is inhibited well in advance. this is also the logic we try to apply as much as possible in the LHC : First cut the beam, then the equipment ! 25.11.2004 J. Wenninger

>> the tests were repeated sucessfully 2 weeks later Follow up / 2 2 shifts were devoted to the setting up the high intensity extraction under stable conditions with a correctly operating MSE magnet (note that most T signal cables had to be disconnected in the tunnel). An analysis of the data from the first TI8 test led us to fine tune the interlock levels on the current surveillance : ‘Dead time’ could be reduced to ~ 2 ms with a single current sample. Interlock tolerances were tightened on many magnets (not the septum). >> the tests were repeated sucessfully 2 weeks later 25.11.2004 J. Wenninger

General situation At the SPS the high intensity beam projects (LHC and CNGS) never incorporated machine protection and interlock aspects into their project. The interlock system for the SPS extraction emerged eventually from an excellent collaboration among AB-Controls (B. Puccio, R. Schmidt) and AB-Operation (R. Giachino, J. Wenninger) with the help of many AB colleagues that devoted some time for the generation of interlocks. But the resources had to be found outside the SPS LHC/CNGS projects. We propose now to FORMALLY incorporate the SPS interlocks into the LHC Machine Protection Working. 90% of the persons involved in SPS interlocks are also members of the MPWG. 25.11.2004 J. Wenninger

What when wrong… The MSE magnet interlock system was ‘swamped’ by beam induced EMC. The presence of EMC was KNOWN to BT experts – effects underestimated. We missed an important interlock. People in charge of interlocks were not informed about the PLC controlling the MSE magnet. Not enough BEAM time was devoted to interlock testing  might have tightened current surveillance and prevented incident. The high intensity extraction setting up was ‘mixed’ with the actual high intensity beam tests : Led to time pressure, in particular because of other delays. The TI8 commissioning and TT40 high intensity were grouped into a continous 72 hour period (high intensity at the end) : No time to analyse ‘quietly’ the interlock system and its performance. Many persons were need throughout the period. Not ideal even if absence of rest was not in itself the triggering problem. There was no single person responsible for the tests and for safe operation. 25.11.2004 J. Wenninger

In the future The MSE magnet interlock system must be made EMC-free. Resources must be made available to develop an operational system for the fast surveillance of critical magnets, also for the LHC – a pre-prototype exists. This system should be able to detect a fault on the MSE within 1 ms. We must schedule DEDICATED (beam) time for interlock testing. At the same time we will formalize the interlock test procedures – also for the LHC. We must leave sufficient time between the first operation of the interlock system with beam and high intensity running to fully analyse the interlock system performance (and short-comings). This will be very important when/if we startup with CNGS in 2006 : equivalent of 3 times the beam intensity lost during the incident every ~ 30 seconds…. 25.11.2004 J. Wenninger

Septum PLC timing test Current history of the MSE PC Interlock recording by the BIC MSE current (A) MSE PC off 23 ms Interlock received by BIC Cycle time (ms) 25.11.2004 J. Wenninger