1. TE-MPE-CP, RD, 23-Nov-2010 1 Summary of Radiation Induced QPS Events in LHC 2010 R. Denz TE-MPE-CP

2. TE-MPE-CP, RD, 23-Nov-2010 2 Radiation induced QPS events – general remarks  LHC is not a radiation test area but an environment where installed equipment is exposed to different kind of stress –EMC, ionizing radiation, tunnel environment, homo sapiens …  Apparent faults may have a variety of causes –e.g. loss of fieldbus communication can be caused by connection problems, broken power supplies, firmware problems, ionizing radiation …  Each fault must be carefully analyzed to determine the most probable cause –Some cases require additional tests in the lab to reproduce symptoms  Fault statistics is very poor ;-))  Observations in LHC not always in good agreement with results from radiation test campaigns  Most of the faults occurring in the QPS system are related to problems with electro-mechanical components (connectors, circuit breakers, switches …)

3. TE-MPE-CP, RD, 23-Nov-2010 3 QPS events which could have been caused by radiation induced faults Event typeCasesElectronics failurePhysical cause Stalled internal communication on protection crates type DQLPU S (I 2 C TM bus) 12Latched digital I/ORadiation or EMC induced Permanent trigger on DAQ systems type DQAMC 11Latched digital I/ORadiation or EMC induced Lost fieldbus communication DAQ systems type DQAMC 2MicroFip TM chipMost likely radiation induced

4. TE-MPE-CP, RD, 23-Nov-2010 4 QPS events which could have been caused by radiation induced faults II Event typeOccurrenceCorrelation to radiation losses Observation during radiation tests Stalled internal communication on protection crates type DQLPU S LHC operation, proton run NoneNo Permanent trigger on DAQ systems type DQAMC LHC operation, proton and ion run, electrical tests In most casesNo but not explicitly tested Lost fieldbus communication DAQ systems type DQAMC LHC operation, ion run only Clearly linked to beam losses Yes

5. TE-MPE-CP, RD, 23-Nov-2010 5 Fault statistics: stalled internal communication of DQLPU type S crates  Events occurring with and without beam –System availability 99.991% –Remote recovery feasible  No events in dispersion suppressors, matching sections or close to transfer lines  There is no evidence so far that the fault is radiation induced

6. TE-MPE-CP, RD, 23-Nov-2010 6 Fault statistics: permanent trigger on DAQ systems type DQAMC NumberSectorUnitDateRemark 16-7A8L73-AugCollimators point 7 28-1B23L12-SepTechnical stop 6 31-2B9L210-SepTI2 injection tests 48-1A8R814-SepTI8 injection tests 51-2A9L216-SepTI2 injection tests 62-3C14R219-SepQuench test 73-4B9R35-OctCollimators point 3 81-2C22R18-OctPhysics fill 3.5 TeV 98-1A8R817-OctTI8 106-7A9L712-NovPb run losses in Q9.L7 116-7A9L713-NovPb run losses in Q9.L7 System availability 99.997%

7. TE-MPE-CP, RD, 23-Nov-2010 7 Fault statistics: permanent trigger on DAQ systems type DQAMC  Despite the fact that still some events are observed without radiation (e.g. during technical stops), there is a clear correlation to beam losses  Analysis of the hardware layer points to a specific problem with a digital isolator of the ISO150AU type (see presentation to RADWG 28.09.2010) –Device successfully tested in TCC2 by QPS, failures reported recently (CNRAD 2009) by CRYO –Recoverable EMC vulnerability reported by manufacturer –The problem can be partially cured by a firmware upgrade of the DAQ system, which will indicate the fault but allow to continue LHC operation Access still required but can be organized at a more suitable time, e.g. within a normal accelerator stop  Problem became more evident during ion run especially for position A9L7 –2 x firmware updates within one (hot) weekend, second for fieldbus problem –Working correctly now …

8. TE-MPE-CP, RD, 23-Nov-2010 8 Lost field-bus communication DAQ systems type DQAMC  Two events during the ion run observed; both are clearly correlated to beam losses –B9R7 and A9L7 nQPS crate B9R7 installed on top of DQLPU.B9R7 using a similar DAQ system was not affected at all –DQAMC type DAQ systems are equipped with the old MiroFip TM (VLSI/Philips make) nQPS systems use new type (AMI/ON Semiconductor make) –Similar faults have been observed in CNRAD with both versions of the MiroFip TM but not during previous test campaigns in TCC2 and at PSI DQAMC type DAQ system under test at PSI (60 MeV p)

9. TE-MPE-CP, RD, 23-Nov-2010 9 Lost field-bus communication DAQ systems type DQAMC II  The RADMON devices installed close to the concerned QPS systems confirm the increase of SEU during the ion run –SEU counts in the RADMON devices not directly correlated to faults  Firmware of DAQ system has been updated taking into account recent observations in CNRAD –Produced variables of MiroFip TM systematically updated by host controller even if there is no interrupt generated by the MiroFip TM –Further tests on firmware improvements ongoing –Updated version to be deployed for all dispersion suppressors and matching sections during the Xmas break –Apart the firmware upgrade the cleaning efficiency in point 7 has been improved  difficult to quantify the effect of the upgrade

10. TE-MPE-CP, RD, 23-Nov-2010 10 Conclusions  During the 2010 LHC run a number of faults in the QPS system have been observed, which are clearly related to beam losses –Majority of QPS faults however is not at all related to radiation  All radiation induced faults affected one type of DAQ system only; there were no protection systems affected –So far only non-destructive faults recoverable by power cycles –Faults are not stopping a fill but require access prior to refill –Fault occurrence in contradiction to some of the radiation test results (TCC2 and PSI tests)  Problems have been partly cured by firmware updates –Events underline the necessity of a proper hardware solution especially for the fieldbus coupler (  nanoFip) –Intermediate hardware solution allowing automatic or remote power cycles to be envisaged for 2011 –Nevertheless no showstopper for the 2011 LHC run