1. Jefferson Lab Reliability Task Force and Initiatives
Randy Michaud
Deputy Group Leader of Operability
Thomas Jefferson National Accelerator Facility
Newport News, Virginia USA
Accelerator Reliability Workshop Versailles, France
October 19, 2017

2. Jefferson Lab
Operated for the US DOE Office of Science-Nuclear Physics
Jefferson Lab Research:
Experimental, computational and theoretical nuclear physics
Accelerator Science, SRF technologies and FEL
Radiation detectors and medical imaging
Cryogenic technology
1530 users from 236 institutions and 31 countries
CEBAF
SRF
Cryogenics
Jefferson Lab is a Department of Energy Office of Science laboratory about 60miles south of here.
The labs core mission is to perform experimental nuclear physics and the Continuous Electron Beam Accelerator Facility is operating in support of the nuclear physics program.
In addition there is active research in Accelerator science, SRF technologies and cryogenics.

3. 12 GeV CEBAF Overview Polarized electron beam (P>85%)
Four (or 499) MHz interleaved beams, generating a 1497 MHz CW beam
CW SRF linacs
Design energy 2.2 GeV/pass:
5 passes, 11 GeV (Halls A,B & C)
5.5 passes, 12 GeV (Hall-D)
Flexible extraction options for ABC, 1st…5th pass
Hall A & C 1 MW high power dumps

4. CEBAF 12 GeV Runs 12GeV Commissioning Runs Physics Runs
December 2013 – February (~6 weeks)
March 2014 – May (~10 weeks)
October 2014 – December (~10 weeks)
February 2015 – May (~16 weeks)
November 2015 – December (~6 weeks)
Physics Runs
February 2016 – April (~10 weeks)
October 2016 – December (~10 weeks)
February 2017 – March (~8 weeks)
Achieved commissioning goals for 12 GeV upgrade project; limited run time to achieve hardened systems for highly reliable Physics running.

5. Major Failures
Spring 2014 – Spring 2017: CEBAF experiences 6 major equipment failures requiring program change/termination.
MZAAR03 damaged coil
ARC7 Magnet Power Supply melted choke
Cold Compressor 4 bearing failure

6. Reliability Task Force Launch
As result of the series of major failures and a 2016 Accelerator Advisory Committee Recommendation, Jefferson Lab Director Hugh Montgomery launched the Jefferson Lab Reliability Task Force.
Goal: Reporting to Jefferson Lab Director, recommend actions to improve systems reliability restoring levels achieved during 6 GeV operations (80-90%).
Team made up of various divisions:
Accelerator Division
Engineering Division - Mechanical, Electrical, Cryogenic, and Safety Systems
Facilities Operations and Support Division
Environmental, Health, Safety and Quality Division

7. Task Force Recommendations
The lab should consider establishing a formal reliability program at Jefferson Lab. For example:
Management reviews and discussion of reliability performance metrics
Set expectations…system owners are responsible for reliability
Establish a graded approach corrective action program.
Continuous monitoring by system owners of system outages and failures
Support culture of predictive analysis.
Implement an integrated planning function for the accelerator schedule.
Incorporate non-operations schedule impacts (Facilities, engineering, work for others)
Balance operational schedule against labor demands (e.g. SAM1 ground fault)

8. Accelerator Operations Initiative - CPP
Recognizing the importance of improving CEBAF Systems reliability, Accelerator Operations initiated an effort to align with the Reliability Task Force recommendations.
Initiated by the Director of Accelerator Operations, a 20 year strategy was developed aimed at meeting requested performance goals from the CEBAF accelerator.
The CEBAF Performance Plan (CPP) was written with intent to support the JLab Reliability Task Force initiatives as they become developed lab wide.

9. Gap – What is missing that prevents achieving desired goal?
Gap Analysis
CPP was developed using Gap Analysis methods to develop a strategy bridging technical requirements, goals, and resources to achieve the desired level of performance.
Gap – What is missing that prevents achieving desired goal?
Current Performance
Desired Performance
How? What is required to close the Gap? Parts, People, Actions, Data, Tools
Action Plan = Strategy
Each goal (system performance) was analyzed in this manner to understand what is needed to achieve the desired performance level.

10. Make the case with data
“Without data, you’re just another person with an opinion.” –Edward Demming Operational downtime data proved instrumental in conducting the gap analysis, risk assessments, and strategy development.

11. CEBAF Goals - Vision

12. Category Discussion
Categories requiring most attention to achieve desired goals:
Accelerator Systems (reliability, spares)
Critical Spare parts, Klystrons, Cryo Plant age
Energy Reach (gradient, 12 GeV operations)
Cryomodule refurbishment program
Operations (staffing and maintenance)
4 Hall operations, Operator staffing, Maintenance scheduling with longer runs
Obsolescence (parts, design)
Vendor support for parts, new designs

13. Accelerator Systems - Critical Spares
We conducted an accelerator systems wide assessment using Failure Modes Effects and Analysis (FMEA) technique and risk management ranking.
“What major failures would bring the machine down hard and do we have a mitigation plan?”

14. Energy Reach & Operations
Gradient team recommendations and assessment; develop a cryomodule refurbishment plan that establishes gradient headroom.
“C75” cryomodule design selection
Operations:
Increase staffing profile to achieve 6 GeV support; managers off shift.

15. Immediate Needs and Obsolescence
“Each system owner reviewed the present capability to support the near term program and developed an immediate needs list.”

16. Program Management
CPP initiatives will be managed as “projects” by Accelerator Operability; they will be funded as projects, separate from traditional operations funds.
Program and project managers assigned.
Cost profile developed to assist in funding projections.

17. Summary – Points to Consider
Traditional Operations budget cycles failed to address some areas impacting reliability and performance goals; critical spares and obsolescence.
Our improvement in data gathering has made using FMEA and risk management methods much easier and accurate – data driven decisions.
Combining the technical perspective with the business management perspective of Gap Analysis and Strategy development help make reliability efforts clearer to management; better understanding on what they can expect for their investment.
Thank You!
Merci!

18. Back Up

19. Major Failures
Spring 2014: ZA magnet coil and vacuum failure; 3 week interruption to replace damaged coil and repair the vacuum chamber. This failure consumed the existing spare coil; recovery from the next failure will take much longer.
Spring 2015: Cold compressor 4 failure in 2 K cold-box, SC1; No spare at JLab, consumed the SNS cold compressor spare. Program change required: half design energy after 5 week down.
Fall 2015: YR coil on 3-pass extraction generated a spontaneous leak. No 3-pass program for FY16, repaired Summer 2016, consumed a YR coil spare. Required Hall-A DVCS experiment to re-arrange its run plan.
Fall 2016: Arc7 box supply failure, no spare. Program change required: to single hall operation until supply repaired.
Fall 2016: 5th pass separator vacuum leak. Program change required, could not support 5th pass beam to Hall-A simultaneously with 5.5 pass beam to Hall-D.
Spring 2017: Cold compressor 5 failure in 2 K cold-box, SC1; on-going root cause investigation, might be repairable. Scheduled program terminated.