Eric Prebys APC/LARP The truth is out there! 2/9/2009

 The primary goal of this workshop was to provide the CERN Directorate with information and recommendations to define the scope of activities and final schedule and plan 2009/2010 running of the LHC.  This planning is obviously dominated by the consequences incident of September 19.  Questions about how this incident occurred were examined to the extent that they impact repair and remediation; however, questions regarding the decision process that allowed to occur were deferred to the external review committee. 2/9/2009 E. Prebys, AEM

 When will the LHC start up?  The DG has made a strong commitment to a physics run of the LHC “in 2009”, provided it does not threaten the long term viability of the accelerator.  Run through the winter?  How much does this cost?  How many sectors to warm up?  There are currently four sectors warm, and a plan to install new pressure relief flanges with dramatically increased capacity.  To install these new relief flanges on the remaining four sectors would require a warm up, which has it’s own risks and scheduling headaches.  The baseline plan is to leave the remaining sectors cold, and effect an intermediate pressure relief enhancement by replacing clamps with springs on a number of existing flanges.  Energy  Quickly became clear the options were 4 or 5 TeV/beam. 2/9/2009 E. Prebys, AEM

 After the close of the meeting, Steve Myers made all the workshop talks public on the CERN Indico site, which you can access with this shortcut:   Of particular interest is Steve’s summary talk at the end  (very) Partial glossary of terms:  “Consolidation” – CERNglish word referring to a set of actions and/or repairs to address known shortcomings of a system.  “MLI” – multi-layer insulation, which was blown all over the tunnel and sucked into the beam pipe  “DN200” – 200 mm pressure relief flange (“bandaid du jour”).  “QPS” – quench protection system  “MCI” – maximum credible incident 2/9/2009 E. Prebys, AEM

 On September 19 th, sector 3-4 was being ramped to 9.3 kA, the equivalent of 5.5 TeV  All other sectors had already been ramped to this level  Sector 3-4 had previously only been ramped to 7 kA (4.1 TeV)  At 11:18AM, a quench developed in the splice between dipole C24 and quadrupole Q24  Not initially detected by quench protection circuit  Power supply tripped at.46 sec  Discharge switches activated at.86 sec  Within the first second, and arc formed at the site of the quench  Helium pressure rose beyond.13 MPa and ruptured into the insulation vacuum.  Vacuum also degraded in the beam pipe  The pressure at the vacuum barrier reached ~10 bar (design value 1.5 bar). The force was transferred to the magnet stands, which broke. 2/9/2009 *see talk by Philippe LeBrun, Monday AM E. Prebys, AEM

 Incident on 19 th of September 2008 => failure of some supports of SSS in sector 3-4 due to longitudinal loads *O. Capatina, Tuesday PM 2/9/2009 E. Prebys, AEM

 Under the rules at the time, there could have been people in adjacent sectors or service areas, who might have been killed or seriously hurt.  It could have occurred in, eg, the dispersion suppression (DS) region, for which spares would have been an issue.  It could have happened in the triplet (which BTW has much better designed joints), and damaged the experiments.  If the arc had cut into both He lines, it would have been 40 kg/s of He, a factor of 20 over design  This MCI drives the new pressure relief specification 2/9/2009 *see talk Jim Strait, Wednesday PM E. Prebys, AEM

 Why did the joint fail?  Quality control problems  Inherent weakness in joint design (no clamps)  Why wasn’t it detected in time?  There was indirect (calorimetric) evidence of an ohmic heat loss, but these data were not routinely monitored  The bus quench protection circuit had a threshold of 1V, a factor of ~1000 too high to detect the quench in time.  Why did it do so much damage?  The pressure relief system was designed around an MCI Helium release of 2 kg/s, a factor of ten below what occurred. 2/9/2009 E. Prebys, AEM

2/9/2009 *see talk by Arjan Verweij, Tuesday PM E. Prebys, AEM

Theory: A resistive joint of about 220 n  with bad electrical and thermal contacts with the stabilizer No electrical contact between wedge and U- profile with the bus on at least 1 side of the joint No bonding at joint with the U-profile and the wedge A. Verweij Loss of clamping pressure on the joint, and between joint and stabilizer Degradation of transverse contact between superconducting cable and stabilizer Interruption of longitudinal electrical continuity in stabilizer Problem: this is where the evidence used to be 2/9/2009 E. Prebys, AEM

 Use calorimetric data to find suspicious joints, then check with precision resistance measurements:  1 sigma = 20 nOhms (design value.6 nOhm)  Thermal ~85 5TeV  Two additional bad joints found, both of them inside magnets, which were removed.  All new joints photographed and check with ultrasound. GOOD BAD 2/9/2009 E. Prebys, AEM

-10 mK S1-2S2-3S3-4S4-5 S5-6S6-7S7-8S8-1 7 kA 9.3 kA 7 kA 8.5 kA7 kA +40 mK -10 mK Relative temp +40 mK Relative temp 1-2 hour flat tops 6 February 2009 Smoking gun? In the future, calorimetric data will be routinely monitored 2/9/2009 *See talk by Nuria Catalan Lasheras, Monday AM E. Prebys, AEM

 Old quench protection circuit triggered at 1V on bus.  New QPS triggers at.3 mV  Factor of 3000  Should be sensitive down to 25 nOhms (thermal runaway at 7 TeV)  Can measure resistences to <1 nOhm  Concurrently installing improved quench protection for “symmetric quenches”  A problem found before September 19 th  Worrisome at >4 TeV 2/9/2009 *See talks by Arjan Verveij and Reiner Denz, Tuesday PM E. Prebys, AEM

New configuration on cold sectors: Turn several existing flanges into pressure reliefs (while cold). Also reinforce stands to hold ~3 bar New configuration on warm sectors: new flanges (12 DN DN100) (DP: Design Pressure) L. Tavian 2/9/2009 *Vittorio Parma and Ofelia Capatina, Tuesday PM E. Prebys, AEM

 15 SSS (MQ)  1 not removed (Q19)  14 removed 8 cold mass revamped (old CM, partial de-cryostating for cleaning and careful inspection of supports and other components) 6 new CMs In this breakdown there is consideration about timing (SSS cryostating tales long time; variants problems).  42 Dipoles (MBs)  3 not removed (A209,B20,C20)  39 removed 9 Re-used (old CM, no decryostating –except one?) 30 new CMs New cold masses are much faster to prepare than rescuing doubtful dipoles) 2/9/2009 *Lucio Rossi (magnets) and Francesco Bertinelli (connections), Tues AM E. Prebys, AEM

 Most dirty beam pipes in removed magnets  Several must be cleaned in tunnel  Soot: successive passes with wet and dry sponges  MLI: little vacuum cleaner 2/9/2009 *Vincent Baglin, Tuesday AM E. Prebys, AEM

 Magnet reinstallation begins this week.  Work on flanges begins in a few weeks.  Decision of Directorate (announced today)  Do NOT warm up remaining four sectors  Start up in late September, collisions in late October  Run through winter until next fall (with short break for Christmas, of course)  Raise energy to 5 TeV as soon as safe (symmetric QPS?)  Collect ~200 pb-1 at TeV ~ Tevatron Run II so far  Possible ion run at the end 2/9/2009 *See Friday talks, particularly Steve Myers’ summary E. Prebys, AEM

 We shouldn’t forget how incredibly well things were going before the incident.  Beam was circulated in both directions within a few hours  Beam was captured within a day  Optics measurements uncovered a few minor wiring problems. Right Wrong 2/9/2009 *Thursday AM E. Prebys, AEM

 Will NOT happen this year  Problem with dipoles “forgetting their training” after the period of storage  Only occurs with one out of the three vendors (Noel)  This was discovered before the turn on  Project LOTS of quenches to go beyond 6 TeV 6.5 TeV: ~10 per sector 7.0 TeV: ~100 per sector  Can only do 2-3 quenches/day 1 to 2 months of quenching to get to 7 TeV Risk of damage? 2/9/2009 *Arjan Verveij, Monday AM and Ezio Todesco, Thursday PM E. Prebys, AEM