TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE LS1 The upgrade of 600 A ENERGY EXTRACTION SYSTEMS

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE   600 A EE; what, where, how ?   Reliability   Hardware Failures to be cured   Why upgrade?   Failure causes   Solutions   LS1 actions   Resources   Planning   Showstoppers   Non LHC activities during LS1 Overview

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE Energy Extraction 600 A circuit

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE

600A Energy Extraction Systems   202 systems installed in the LHC tunnel in corrector circuits with stored energy between 2.2 and 150 kJ   In 15 different locations; 8 x UA parallel service tunnel and 6 x RR and 1 x UJ tunnel caverns   Systems developed in close collaboration between CERN and the Budker Institute of Nuclear Physics (BINP), Novosibirsk, Russia.

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE Reliability   Systems reliability from protection point of view: No opening failures during Fast Power Abort events or tests (> openings since 2005) 2 independent opening mechanisms/circuits per breaker; guaranteeing 6-fold redundancy Calculated by Antonio Vergara in December 2003: System Failure probability in 20 years of operation < 0.01% (Preventive maintenance!!)

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE Reliability (2)   Systems reliability from hardware failure point of view: 6 hardware failures in hardware failure in * hardware failures in 2012 For operation: 2 failures per year – not in shadow Mean Time To Recovery : 4 hours (including preparation, access and travel) All failures were “fail-safe” for the s.c. circuits reducing the availability of the system. Changes with respect to 2010 operational year : Preventive maintenance with help of automatic analysis and tracking software tool.

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE Hardware Failures treated during LS1   5-10/year - 95% of all failures: Breaker refuses to close Loosening of holding coil mechanism – LHC re-start delayed   1/year - Breaker is physically open but indicates “CLOSED” micro-switch not actuated / main axle – LHC re-start delayed   1/year - Breaker main contact re-bounces on micro-switch main axle – LHC re-start delayed

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE Why do we upgrade?  Correcting measures are relatively easy  < 100 kCHF  This upgrade will improve the availability of 600 A EE systems for LHC operation by 4-8 hours per year….  Less preventive maintenance

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE Failure causes   1- Loosening of holding coil mechanism Weak design of U-shape   2- Micro-switch not actuated Tolerances in dimensions of housing   3- Main axle fixation Weakness during production – not an increasing fault

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE   1- Loosening of holding coil mechanism Weak design of U-shape and dimensional tolerances Solutions   Needs to achieve Plate Glue Action in all 606 breakers Open all breakers (anyway foreseen for regular maintenance) Plate gluing – Once open: 2-3 minutes per breaker

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE   2- Micro-switch not actuated (temporarily) Tolerances in dimensions of housing Already tracked by automatic analysis software tool Leading to an unscheduled but still preventive maintenance intervention (in the shadow) On-going but to be extended with advice in form of logbook entries (MPE-MS) Solutions cntd.

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE

  3- Main axle fixation A loose main axle means no current in 2 oo 3 phases in the CB leading (> 200A circuit current) to overheating of the 3 rd phase. Weakness during production – no increasing numbers Manufacturer has improved this process for new breakers For existing breakers no solution.. But.. We can track it because of known early symptoms Solutions cntd. Already tracked by the same automatic analysis software tool but with very few cycles left – we can measure the System Resistance! Leading to an unscheduled corrective or preventive maintenance intervention

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE

  3- Main axle fixation tracking improvement - Measurement Solutions cntd.   Needs to achieve Improvement of existing Voltage Measurement Channel Replacement of Voltage Divider by low impedance divider and noise reduction components (270 pcs MPE-EM) Replacement of Measurement Board by low noise separation amplifier (270 pcs: MPE-EM) Replacement of DQAMS by 12 bit to 24 bit ADC (MPE-EP & EM: 220 pcs) Voltage measurement taps to be improved Hi-level SW for on-line analysis (MPE-MS)

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE   3- Main axle fixation (and loose bolts) Save the EE system in case we don’t track (or too late) it with software tools Overcurrent in one or two phases of the breakers will lead to over- temperature on the Equalizing Resistor and eventually destroy all breakers in one system Installation of a Thermostat on each Equalizing Resistor will trip the system early before hardware damage is done A spare internal interlock channel is available and wired up to the power board. Solutions cntd.   Needs to achieve Installation of Thermostats Reprogramming of Firmware in all 404 interface cards (MPE-EP)  Allow trip on over temperature of Equalizing Resistor »Profit to limit cycles of the breakers

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE LS1 Actions Breakers - Regular maintenance and Consolidation Equalizing Resistors – Installation/wiring of Thermostats Voltage measurement-channel improvement Visual inspection of main axles – Endoscope IST SCT HWC

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE Resources MaintenanceIST Installation man-hours T/TEman-hours E/TE per point Even point - 2 UAs10030 Odd point - 2 RRs4010 P7 right 3010 P7 left 3010 UJ Total LHC approx. : 630 h 160 h 130 man-days 6 man-months

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE Planning (not incl. SCT and HWC)   ANYTIME – “ANYWHERE” April 2013 – April 2014 – 6 man-months – Team of 3 persons could come 2 x 1 month  Man-power Calculations based on one team of Specialists from BINP Perfect preparation before start of activity One Engineer or TE as team leader plus two Technicians Working in a “train” Start – to – Finish one point, including IST But..

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE Planning SCT & HWC   SCT & HWC – 2014 SCT was presented as not needed for 600 A EE but … latest tests show.. Yes, if we could we should 1 day per UA or RR – 2 TE or E  15 working-days - NO recourses allocated yet HWC like SOF test PLI3.b1 – remotely from CCC – Automatic Analysis of PM PNO.b1 – 1 day per UA or RR – 2 TE or E  15 working-days - NO recourses allocated yet

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE “Showstoppers”  Preparation for LS1 Planning – Tests – Tooling – Purchasing (thermostat-study still to be finished) A LONG WAY TO GO!!!  No reference contract with Russia If done by non-experts More CERN staff needed for training, the actual job, follow up, debugging of collateral damage etc. Might delay re-start after LS1.  Spare Circuit Breakers: DZNVA – Visit in March Manufacturer’s premises in very bad state and only 20% occupied - might stop delivering (special) breakers Currently: 28 spares (13 operational – 15 to be qualified before TS4) Consumption average: 8-10 per year (including preventive changes)

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE Non LHC activities during LS1  Currently low in spare systems (<1%) and spare breakers (3-4%) spare breakers (3-4%)  In order: 20 spare systems – ATA Autumn 2013 – to be mounted and tested in ISR test building  Urgent: to order spare breakers in order to exclude the regular need of Russian Specialists and cover mid-term needs. Long term strategy for 600 A EE systems:   Switch partially (UA areas) to solid state breakers R&D necessary: 2012 development of Lab version (Bi-polar) 2013/2014 development of prototype – 0.5 day/week 2015 Market Survey/Tender What about the study to prevent Quench-Back on RQTF/D the installation of lower dump resistances in 600A circuits

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE Thank you for your attention.. So far..!

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE Questions before continuing with Cable Activities?

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE Planning activities GJC   Preparation 600 A EE activities to start LS1: 70%   Installation/Maintenance – 2013 Coordination – during the activity 50%   IST – 2013 Coordination – during the activity 50%   SCT – 2014 Coordination/installation/Analysis: 70% during activity   HWC – 2014 PLI3.b1 – remotely from CCC – Automatic Analysis of PM Activities if automatic analysis fails PNO.b1 – 1 day per UA or RR – 2 TE or E Coordination/installation/Analysis: 70% during activity

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE   1- Loosening of holding coil mechanism Weak design of U-shape and dimensional tolerances Vibrations causing the closing failures – if starting increasing with cycles To prevent vibrations the manufacturer proposed decreasing of gaps on the side of the U-Shape by gluing small plates between the U-shape and the housing Several cycling tests have proven the efficiency of this method. Number of closing failures reduces from typically 4% to less than 0.5 % during lifetime of breaker Solutions   Needs to achieve Plate Glue Action in all 606 breakers Open all breakers (anyway foreseen for regular maintenance) Plate gluing – Once open: 3-4 minutes per breaker Glue = Silicon paste (Dow Corning Mil. Standard) approved by breaker manufacturer

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE Solutions

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE   3- Main axle fixation tracking improvement Weakness during production If this occurs there’s an increase from 0.5 mΩ to max 1.25 m Ω of system impedance due to decrease of contact pressure in the breaker. By improving the existing voltage (drop) measurement, correlating it to the circuit current, can lead to an early warning, preventing the circuit to trip An intervention can be planned in the shadow Solutions cntd.   Needs to achieve Improvement of existing Voltage Measurement Channel Replacement of Voltage Divider by low impedance divider and noise reduction components (MPE-EM) Replacement of Measurement Board by low noise separation amplifier (MPE-EM) Replacement of DQAMS by 12 bit to 24 bit ADC (MPE-EP & EM) Voltage measurement taps to be improved Hi-level SW for on-line analysis (MPE-MS)

TE-MPE LS1 workshop – November 22-23, 2012 G.J. Coelingh TE-MPE-EE   3- Main axle fixation (and loose bolts) Overcurrent in one or two phases of the breakers will lead to over- temperature on the Equalizing Resistor and destructing all 3 breakers in one system Installation of a Thermostat on each Equalizing Resistor will trip the system early before hardware damage is done A spare internal interlock channel is available and wired up to the power board. Solutions cntd.   Needs to achieve Installation of Thermostats and Wiring up to the spare channel Dismounting, drilling, threading of all 606 Equalizing Resistors Mounting of pre-cabled Thermostats and a connector PCB Wire installation from connector PCB to Burndy connector SK24 Reprogramming of Firmware in all 404 interface cards (MPE-EP)