TE-MPE-EP: REVIEW LS1: “Yellow racks” activities. Protection of 1232 dipoles and 392 quadrupoles TE-MPE-EP, VF, 02-June-2015

TE-MPE-EP TE-MPE-EP, VF, 02-June-2015 Scope and duties concerning the “yellow racks” Ensure the protection of the 1232 dipole magnets and 392 quadrupoles of the LHC machine by deploying a new upgrade of the quench protection. Studied and produced electronics and mechanics for new DQLPU type A Conducted and performed complete tests of new equipment and new functionalities, such as: enhanced quench heater supervision new fully redundant linear power supplies (DYPB) new fuse in DQHDS. Remote commands,… Refurbished fip connection of 392 quadrupoles + updated firmware.

TE-MPE-EP QPS protection for 1232 dipoles DYPB Yellow rack New DQLPU type A QPS protection for 392 quadrupoles DYPQ Yellow rack DQLPU type B Connections between «Yellow racks» DQLIM TE-MPE-EP, VF, 02-June-2015

TE-MPE-EP New DQLPU type A Motherboard DQQDLDQCSU DQHSU DQAMCDQIPF Mechanical enclosures: New DQLPU type A Connections with DQLPU type A (main development) TE-MPE-EP, VF, 02-June-2015

TE-MPE-EP CRATE ENCLOSURES RESSOURCES + - IN (MPE) USED/NEEDED OUT Mechanics (kits delivered) x1 / x2: study, design Design: x1 EM Protos: Atos Contract: purchase services Other company  production + : Price - : No real choice: agreement with an unbalanced country. New product: no experience Had to follow-up every step the company since the beginning, in the end a sample was just copied. Failure: Surface treatment not as required, quality, precision, dimensions (thickness, depth ǂ6mm, poor quality plastic hinges, very late delivery of the last spares. Only after acceptance, we had to provide a plan detailed to explain everything. Assembly 2 people AGH OK, very good progress, efficient and clever work No time to assemble all of the crates Burndy connectors x1: geometry, ordersDirectly soldered on pcb Delay for huge quantities  asked for equivalent, but price! Harting connectors x1: new design Harting x1 EE Good collaboration /interaction: top quality Directly soldered on pcb New product  longer delay New DQLPU type A TE-MPE-EP, VF, 02-June-2015

TE-MPE-EP ELECTRONIC BOARDS RESSOURCES + - IN (MPE) USED/NEEDED OUT Motherboardx1.2 / x1.5EM: fin.+ prod. New design, direct connectors Finalisation: clearance rules changed from MPE to EM: HV withstands problems DQQDL (Local detector) x1: tests / x1.3 repair Re-used from initial crates. New firmware Missing 50 : had to repair old ones. New DQQDLx1: tests x1: rework EM: rework 50 tested200 boards reworked due to a missing wire and wrong resistor values. New version not compatible. DQHSU (heaters supervision unit) x1: design x1 /+x1 for components EM: fin.+ prod. Relays from «Arrowtronic» not able to be soldered: a mess! (broker?) Had to find new supplier in emergency DQCSU (monitoring unit) x1 / x2EM:fin+ prod. Launched under pressure  HV test forgotten  1 bad lot of pcb. DQAMC (communication) x1 / x2 for repair or new design Re-used from old cratesMissing 20  Had to be taken from new crates (No time for new production) DQIPF (WorldFip interface) x1: design / x2 for details EM:fin+ prod. Programming from outside (gain of time) Space between board and front panel had to be adapted. Connectors, screws, etc. New DQLPU type A TE-MPE-EP, VF, 02-June-2015

TE-MPE-EP TESTERS RESSOURCES + - IN (=MPE) USED/NEEDED OUT Connections testerx1 / x2 : study, programming, design x1 / x1.5 : use OKConnections Alignment of connectors Crate functional tester 1p-4 months : study, programming, design 1p-3 months : design, building x1 : use x1p-3 months : design, building OK On time Faster Study: more specifications and more feedback needed. Synchronisation with worldfip Eliminated data from DQAMC Yellow racks functional tester x1 : building x1.5 / x2 : use x1: patches for DYPQ x1 EE : building AVG per shift: at least 11, usually 12 OK Few problems with gateway. Starting tests delayed due to humidity (HV): heating switched off. UPS cables too rigid  fragile. +6 days due to DYPQ back from tunnel. Cscm crate testerx1 : making run, teaching x1 / x2 : use, developing interface, preparation of crates 2 p. no tech. knowled ge Under pressure, timing too tight. New tester to be tuned. New interface needed. DQQDL tester: interlocks and threshold x1: development x1: use OKWire wrap  bad contacts, repaired several times  New pcb New DQLPU type A See report: 234 errors DYPB Yellow rack TE-MPE-EP, VF, 02-June-2015

TE-MPE-EP Report about the 1232 yellow rack functional tester 234 problems: Divers:128: Offsets values X37 DQQDL X4 DQCSU X5 DQHSU X3 DQAMC X30 power cycles (repeated 2 or 3 times and ok) X15 post mortem data X50 interlocks Cables:28 (not connected, wires swapped, pin default, bad connections) DQHDS: 28 changed, 30 other problems with cables DQLIM: 20 problems TE-MPE-EP, VF, 02-June-2015

TE-MPE-EP QUADRUPOLES RESSOURCES + - IN (MPE) USED/NEEDED OUT Taking DYPQ crates from tunnel and returning them. x2 / x3Cables disconnected from the Crawford box: had to be connected back afterwards (not planned) Change Fip connectors on DYPQ (new patches) x1: design2 people AGH OK. Clever work. (Reliability of connections Right screws UNC or Metric) Modification on patches: holes had to be enlarged Programming and test of DQQDL x1: new pcb for tester x1: use (intern) OK. (New firmware, test of interlocks, threshold and discrimination time) DQAMC programming, preparation x1.5 / x2OK. Programmed in advance in 281: saved time. Errors due to shifting of addresses (difference between position and magnet protected but not systematically) 3 missing crates Yellow DYPQ racks very dusty DYPQ Yellow rack DQLPU type B TE-MPE-EP, VF, 02-June-2015

TE-MPE-EP TE-MPE-EP, VF, 23-Nov-2012 COMMISSIONING DYPB+DYPQ QPS protection RESSOURCES + - IN (MPE) USED/NEEDED OUT Interlock testsx1 / x2 Last sector: fully automated Constant manual supervision needed: schedule too tight Timing requested per sector: 8 to 9 hours DYPQ:4x49 interlocks DYPB: 2x154 boards nQPS: 14x52 interlocks Close current loops in tunnel x1 / x2Only few DQQDL to be replaced. Quick check thanks to monitoring Many cables not connected, difficult to reach, remain behind magnet or on the wall behind them. UPS1/UPS2 testsx1 / x2Many cables not properly connected on DQLPUS: many pins bent on Harting Q12, bad insertion of UPS connectors. Interventions after ELQA tests x1 / x1.5Some nQPS patches missing. Not properly marked and placed. DYPQx1 / x2New interlock cables still in plastic packaging with tie wraps and not connected after previous team. A few ripped cables or connectors not seen or not reported.

TE-MPE-EP TE-MPE-EP, VF, 23-Nov-2012 COMMISSIONING (continued) WHAT?RESSOURCES + - IN (MPE) USED/NEEDED OUT Picking up DQCSU and returning x2 / x3 Tracking: two sectors forgotten Rework DQCSUx1 / x2 OK Programming DQAMGS x2 / x3 Change DQQBS with mDQQBS (cscm) X3 / x5 Pressure to change boards from one sector to another due to lack of boards. Programming DQAMC x2 External programming but difficult to download when on (must be switched off two times due to big capacitors) Programming station misplaced (not returned) and then found.

TE-MPE-EP A few illustrations of unexpected events Programming station misplaced and then found after LS1 Patch missing Additional tests Pins bent Lift/mad/pad connectors Cables not connected Security hinges TE-MPE-EP, VF, 02-June-2015

TE-MPE-EP Concrete lessons learned and recommendation from LS1 to increase probability of getting a reliable machine. Better to choose a company that masters a requested product and has demonstrated experience (mechanical enclosures, components suppliers). If not possible, more ressource must be found in advance. Ex: I visited racks made by this company in Cern: looked good, even proposed in EDH. More advanced preparation, more information, clearer specifications must be dispatched (why were all DYPQ cables disconnected?) More motivated, experienced and «well cared for» people needed in the tunnel (Ex: should not lose a good person with two years experience). Real slots of time have to be found to teach external ressources. Not meetings necessarily, but at least a person dedicated to taking note of what has been done is needed (impossible to memorise and remember everything). Ex: forgot two sectors of DQCSU: people were sure that other people already did it before… TE-MPE-EP, VF, 02-June-2015

TE-MPE-EP Concrete lessons learned and recommendation from LS1 to increase probability of getting a reliable machine (continued). Use knowledge and expertise of people more efficiently. Ex: - Too many wrong connections (many pins bent, cables) remained and had to be identify by experts. - With more time or more ressources, we could have had time to investigate more fundamental issues or to do more tests (ex. nqps remote power cycle). Too many cables left without connection, because it was too difficult, too long or due to timing! Should have been reported. Don’t underestimate that. It leads to wasting of time afterwards! = Quality insurance missing. In order to minimize the number of problems, one sector should have been fully tested and declared as "good" before the deployment to the rest of the machine, preferably not s 6-7. Ex: keep only the latest version of firmware in boards. Before disconnecting any cable, systematically put a label on it. TE-MPE-EP, VF, 02-June-2015

TE-MPE-EP Conclusions Generally for LS1, too many tasks were requested to comply with a good organization and to have a fully tested working reliable equipment. Nevertheless, the challenge was overcome. Would have been better to visit, discuss, and give clear instructions to the outsourced company beforehand, by a expert. Take into account our well-noted ressources in order to define and discuss with the whole team the workload, not overestimate these capabilities, we are not supermen. As already started, we have to continue to think about LS2, and prepare right now everything we can in advance. Decisions must be made asap. TE-MPE-EP, VF, 02-June-2015

TE-MPE-EP Conclusions (continued) Really take care of the level of knowledge of the teams involved in our equipment: investing time in teaching would ultimately save time. Too many wrong connections (many pins bent, cables) could have been avoided this way. Especially if one sector was fully commissionned at first and report all the problems encountered would be a real advantage. Priorities: security, quality, work in a hurry cannot be a solution. More time to thoroughly test one device. Ex: Proposal: develop in collaboration a specific «acid tester» to test the limits of a new device (EMC Testing - Transient Immunity, RF Immunity). For LS2 (or sooner), is there really no way to leave wifi ON in the tunnel and turn it off during exploitation? (even G4 is far from optimum). Having another person other than the developer could be an advantage to test a new device. Keep testers operational (maintenance). Having a tracking system for defective boards would be helpful. TE-MPE-EP, VF, 02-June-2015

TE-MPE-EP Thank you ! TE-MPE-EP, VF, 02-June-2015