1. SEPARATOR SYSTEM STATUS & FUTURE PLANS
Mark Wissmann

2. Talk Outline - System Overview - Separator System Downtime Concerns
Planned Work And Fixes
Tentative Future Plans

3. 12 GeV CEBAF
Two 0.6 GV linacs
CHL-2

4. Separator Cavity Overview

5. Separator Cavity Power Sources
SS AMP PASS 1 in W1
IOT AMPS PASS 2 – 5 Horizontal In W1
SS AMPS For PASS 5 VERTICAL In W2

6. 2019 SEPARATOR DOWNTIME STATISTICS
Data showed rising trip events and rising accelerator down time from January to July
This prompted meetings in July to investigate causes and solutions
I would stress that it was not just one thing that failed 6 times in July. Describe it below the graph…
What failed and why ….

7. SYSTEM DOWNTIME STATISTICS
Stress that availability has been steadily improving…

9. SEPARATOR AVAILABILITY STATISTICS
RF Separator Downtime (Jan1 to Dec 31 by year)
Downtime (Hours) ▼
Number of Incidents
Mean Time to Recover (Hours)
Uptime (Hours)
Mean Time between Failures (Hours)
Hourly Failure Rate
Availability
Loss
Cost at 10k/hr
2016
113.8 33
3.4
4,607.90
139.6
0.007
97.60%
2.40%
$ ,138,000
2017
58.5 28
2.1
1,638.10
0.017
96.60%
3.40%
$ ,000
2018
89.6 48
1.9
4,812.00
100.2
0.01
98.20%
1.80%
$ ,000
2019
28.7 26
1.1
3,847.80
148.3
99.30%
0.70%
$ ,000

10. Vertical Separator Downtime 11/2016 to 8/2019

11. Horizontal Separator Downtime 11/2016 to 8/2019
It would be interesting to normalize this to hours operated. The lower passes have much lower operational hours.
Can you add the 5th pass vertical to the chart too?

12. Separator System Status
The System is fully functional but there are some concerns-
Problems attributed to Separator downtime over the last several years:
• Cavity Vacuum (From tuning bellows rupture)
• Power Supply Current & Voltage Transducer ARC
• IOT Power Supply (HVPS) Wiring – Especially Controls
• LCW And Valve “Choking” Phenomenon
• RF Switch Failure To RESET
• Vertical System Parameters Missing On Operation Screens
• PID Loop Malfunction
- Effects of Un-optimized Cavity Power Couplers
Operation with the tuning capacitor removed from the input cavity
A concern is how long before the other IOT’s tuning caps go bad
- HVPS And It’s Controls - Age
- 5th Pass Horizontal – 750 MHZ System Power And Recovery Time After Trips

13. Current Planned Work And Fixes
2019 FALL DOWN:
• Operate/Run – Especially IOT 1 And 4 This Down
• Fix PID Loop Glitch And Install For All Vertical Passes
• Adjust Tune Of Cavities 3-2 And 5-3
• I&C Group To Install Cabling And So Bring LCW Temp & Flow To Screens For Vertical Separators
• FSD For Pass 5 Vertical
• Re-terminate Control Wiring In The HVPS And Fix Documentation
• Pass 5 Vertical SS Amp Parameters Into EPICS Through VME
• Monitor LCW flow to troubleshoot “Valve Choking”
NEXT DOWN:
• Install New RF Monitor Chassis (Fix REST issue)
• Install New Coupler Loops On Cavities AE00A and AE00B
• Possibly Install New TOMCO SS Amps In Place Of IOT 4
(Will Power Pass 5 Horizontal)

14. In Progress RESET Issue – A new RF Monitor Chassis
LCW: (does not cause many trips but lengthens recovery time)
● Tomasz has limited automatic valve’s minimum flow (Pass 5 Horizontal
restarts)
● HTR PID Code Glitch causing vertical restart problem Complete 2019 Down
● HTR PID Code Needed For Pass 3 & 4 Complete 2019 down
- Vertical Amps require additional parameters brought to EPICS screens
to optimize recovery time. Hardware complete 2019 down
Continued verification of absent Null cavity adjustment on IOT 1
Operated during 2019 down – thumbs up. Also found loose connection inside the IOT.
Still optimizing the new ∆P system from Facilities.

15. THROTTLE VALVE “CHOKING” PROBLEM
We have:
Monitored flow with and without 5 micron filters – No noticeable difference
Have contacted the valve manufacturer – Elbow locations are fine, no other suggestions
We are still scratching our heads

16. Projected Future Plans
Install 750 MHz SS Amps To Power Pass 5 Horizontal
• Benefits:
This would reduce Horizontal Pass 5 Startup times – by half is my estimate.
It would isolate HVPS trips from Pass 5 – Real benefit during Multi-pass ops
Evidence from horizontal vs vertical downtime data indicates SS Amp reliability
• Cost:
Estimate from TOMCO received 11/4/2019 is $364,000 for amps and spares
Other costs?? Not much but lots of work and time.
- Possible Individual & Independent LCW Loops
Speed Up The Reading Of IOT Parameters To Improve Troubleshooting - Involves HVPS Interface Update
This would improve trip source identification
Need to get a report from Joe Matelavich on what he has looked at what he recommends.

17. Conclusion/Summary
We found/fixed several problems during the 2019 fall down:
• Loose wiring in HVPS (10 wires in the HVPS control wiring room)
• Loose connection to IOT 1’s cathode
• PID loop glitch in vertical separator controls
• Active PID LCW controls installed on Pass 1 – 4
and I am anxious to see the effect.
The valve choking issue still puzzles us but I think we have ways of dealing with it
The upgrade/change that would give the most significant improvement in Separator system uptime would be SS Amps for horizontal pass 5.