Jefferson Lab Reliability Task Force and Initiatives Randy Michaud Deputy Group Leader of Operability Thomas Jefferson National Accelerator Facility Newport News, Virginia USA Accelerator Reliability Workshop Versailles, France October 19, 2017

Jefferson Lab Operated for the US DOE Office of Science-Nuclear Physics Jefferson Lab Research: Experimental, computational and theoretical nuclear physics Accelerator Science, SRF technologies and FEL Radiation detectors and medical imaging Cryogenic technology 1530 users from 236 institutions and 31 countries CEBAF SRF Cryogenics Jefferson Lab is a Department of Energy Office of Science laboratory about 60miles south of here. The labs core mission is to perform experimental nuclear physics and the Continuous Electron Beam Accelerator Facility is operating in support of the nuclear physics program. In addition there is active research in Accelerator science, SRF technologies and cryogenics.

12 GeV CEBAF Overview Polarized electron beam (P>85%) Four 249.5 (or 499) MHz interleaved beams, generating a 1497 MHz CW beam CW SRF linacs Design energy 2.2 GeV/pass: 5 passes, 11 GeV (Halls A,B & C) 5.5 passes, 12 GeV (Hall-D) Flexible extraction options for ABC, 1st…5th pass Hall A & C 1 MW high power dumps

CEBAF 12 GeV Runs 12GeV Commissioning Runs Physics Runs December 2013 – February 2014 (~6 weeks) March 2014 – May 2014 (~10 weeks) October 2014 – December 2014 (~10 weeks) February 2015 – May 2015 (~16 weeks) November 2015 – December 2015 (~6 weeks) Physics Runs February 2016 – April 2016 (~10 weeks) October 2016 – December 2016 (~10 weeks) February 2017 – March 2017 (~8 weeks) Achieved commissioning goals for 12 GeV upgrade project; limited run time to achieve hardened systems for highly reliable Physics running.

Major Failures Spring 2014 – Spring 2017: CEBAF experiences 6 major equipment failures requiring program change/termination. MZAAR03 damaged coil ARC7 Magnet Power Supply melted choke Cold Compressor 4 bearing failure

Reliability Task Force Launch As result of the series of major failures and a 2016 Accelerator Advisory Committee Recommendation, Jefferson Lab Director Hugh Montgomery launched the Jefferson Lab Reliability Task Force. Goal: Reporting to Jefferson Lab Director, recommend actions to improve systems reliability restoring levels achieved during 6 GeV operations (80-90%). Team made up of various divisions: Accelerator Division Engineering Division - Mechanical, Electrical, Cryogenic, and Safety Systems Facilities Operations and Support Division Environmental, Health, Safety and Quality Division

Task Force Recommendations The lab should consider establishing a formal reliability program at Jefferson Lab. For example: Management reviews and discussion of reliability performance metrics Set expectations…system owners are responsible for reliability Establish a graded approach corrective action program. Continuous monitoring by system owners of system outages and failures Support culture of predictive analysis. Implement an integrated planning function for the accelerator schedule. Incorporate non-operations schedule impacts (Facilities, engineering, work for others) Balance operational schedule against labor demands (e.g. SAM1 ground fault)

Accelerator Operations Initiative - CPP Recognizing the importance of improving CEBAF Systems reliability, Accelerator Operations initiated an effort to align with the Reliability Task Force recommendations. Initiated by the Director of Accelerator Operations, a 20 year strategy was developed aimed at meeting requested performance goals from the CEBAF accelerator. The CEBAF Performance Plan (CPP) was written with intent to support the JLab Reliability Task Force initiatives as they become developed lab wide.

Gap – What is missing that prevents achieving desired goal? Gap Analysis CPP was developed using Gap Analysis methods to develop a strategy bridging technical requirements, goals, and resources to achieve the desired level of performance. Gap – What is missing that prevents achieving desired goal? Current Performance Desired Performance How? What is required to close the Gap? Parts, People, Actions, Data, Tools Action Plan = Strategy Each goal (system performance) was analyzed in this manner to understand what is needed to achieve the desired performance level.

Make the case with data “Without data, you’re just another person with an opinion.” –Edward Demming Operational downtime data proved instrumental in conducting the gap analysis, risk assessments, and strategy development.

CEBAF Goals - Vision

Category Discussion Categories requiring most attention to achieve desired goals: Accelerator Systems (reliability, spares) Critical Spare parts, Klystrons, Cryo Plant age Energy Reach (gradient, 12 GeV operations) Cryomodule refurbishment program Operations (staffing and maintenance) 4 Hall operations, Operator staffing, Maintenance scheduling with longer runs Obsolescence (parts, design) Vendor support for parts, new designs

Accelerator Systems - Critical Spares We conducted an accelerator systems wide assessment using Failure Modes Effects and Analysis (FMEA) technique and risk management ranking. “What major failures would bring the machine down hard and do we have a mitigation plan?”

Energy Reach & Operations Gradient team recommendations and assessment; develop a cryomodule refurbishment plan that establishes gradient headroom. “C75” cryomodule design selection Operations: Increase staffing profile to achieve 6 GeV support; managers off shift.

Immediate Needs and Obsolescence “Each system owner reviewed the present capability to support the near term program and developed an immediate needs list.”

Program Management CPP initiatives will be managed as “projects” by Accelerator Operability; they will be funded as projects, separate from traditional operations funds. Program and project managers assigned. Cost profile developed to assist in funding projections.

Summary – Points to Consider Traditional Operations budget cycles failed to address some areas impacting reliability and performance goals; critical spares and obsolescence. Our improvement in data gathering has made using FMEA and risk management methods much easier and accurate – data driven decisions. Combining the technical perspective with the business management perspective of Gap Analysis and Strategy development help make reliability efforts clearer to management; better understanding on what they can expect for their investment. Thank You! Merci! rmichaud@jlab.org

Back Up

Major Failures Spring 2014: ZA magnet coil and vacuum failure; 3 week interruption to replace damaged coil and repair the vacuum chamber. This failure consumed the existing spare coil; recovery from the next failure will take much longer. Spring 2015: Cold compressor 4 failure in 2 K cold-box, SC1; No spare at JLab, consumed the SNS cold compressor spare. Program change required: half design energy after 5 week down. Fall 2015: YR coil on 3-pass extraction generated a spontaneous leak. No 3-pass program for FY16, repaired Summer 2016, consumed a YR coil spare. Required Hall-A DVCS experiment to re-arrange its run plan. Fall 2016: Arc7 box supply failure, no spare. Program change required: to single hall operation until supply repaired. Fall 2016: 5th pass separator vacuum leak. Program change required, could not support 5th pass beam to Hall-A simultaneously with 5.5 pass beam to Hall-D. Spring 2017: Cold compressor 5 failure in 2 K cold-box, SC1; on-going root cause investigation, might be repairable. Scheduled program terminated.