1. Maintenance and Efficiency
Paul W. Sampson
Maintenance and Accelerator Support
Group Leader:
CAD/RHIC
Brookhaven National Laboratory

2. Short Background CAD at BNL in Upton New York, USA
RHIC Large superconducting collider
500GeV cm for polarized proton running
Species to Au (U this year?)
Dissimilar species
Many “parasitic” users on injectors:
LINAC
Tandem Van de Graff
EBIS
Booster
AGS
April 13th, 2011
Paul W. Sampson

3. April 13th, 2011
Paul W. Sampson

4. April 13th, 2011
Paul W. Sampson

5. Maintenance Support Group Role
Plan and coordinate work throughout the accelerator complex.
Emphasis on operations and uptime.
Integrate Professional Trades, Technical, Engineering and Physics staff activities during shutdown, failure (unscheduled maintenance) and scheduled maintenance periods.
April 13th, 2011
Paul W. Sampson

6. Goals: Maximize uptime and optimize accelerator output
Integrated Luminosity is king?
STAR, PHENIX, Overall integrated luminosity.
What cost for improvement?
Parameters vary for varying species in RHIC
Use and maintenance of different systems.
Differing beam parameters.
Safety/Radiation issues- affects on access and equipment availability.
New systems commissioning.
Internal goals:
Beam parameters.
Available time at store, setup minimization
Calendar time at store.
April 13th, 2011
Paul W. Sampson

7. Reliability formula complicated
End of life.
Installation and commissioning.
Infant mortality
“Roll up period”
Legacy systems.
April 13th, 2011
Paul W. Sampson

8. April 13th, 2011
Paul W. Sampson

9. Shutdown Annual occurrence. Duration factors:
Budget.
Major work schedule.
Experiment upgrades/readiness.
Climate.
Documented utilizing data base tool.
Progress tracked and Project published.
Goals
Complete shutdown work.
Prepare for structured pre-startup checkout.
Minimize start up (time to Physics).
April 13th, 2011
Paul W. Sampson

10. Unscheduled Maintenance
Major failure or other prolonged downtime.
Goal of maximizing use of time and access while minimizing the recovery.
Coordinated by Maintenance Support Group.
Schedule are created as necessary, depending on length of downtime.
Each day run similar to a scheduled maintenance day.
April 13th, 2011
Paul W. Sampson

11. During the run: “Minor” Failure (>4hours)
Small low impact jobs to be performed.
“Fast” experimenter access
Coordinated by on duty Operations Coordinator, with assistance and advice from Maintenance Support Group as needed.
Documented by standard methods.
April 13th, 2011
Paul W. Sampson

12. During the run: Scheduled maintenance days.
Usually 8-12 hours, biweekly.
Driven by experimental access.
Accelerator access for:
Preventative Maintenance.
Repair
Modification
Installation
Commissioning.
Shutdown planning
Inspection/work observation
Continuation of shutdown/installation work.
April 13th, 2011
Paul W. Sampson

13. How maintenance days directly impact availability…
Scheduled maintenance days demand that the program is off.
Many required tasks require prolonged access.
Some systems have long lead time before work can be performed.
Source cool down.
Cryogenic warm up.
Radiation cool down periods.
Setup and recovery after maintenance period.
Human error:
Systems left in inoperable state following maintenance period.
Failure of improperly maintained equipment or systems.
April 13th, 2011
Paul W. Sampson

14. Reliability and Maintenance days:
Preventative maintenance periods should be focused on reducing failures.
Repairs to systems and infrastructure.
Accelerator tunnel/Radiation area access:
PM and repair
Installation, testing and commissioning of new and modified systems.
Low intensity test cycles following maintenance period:
Used for machine development.
Tests and implementation modified and modes before they are adapted as operational.
Scheduled as part of the maintenance period.
April 13th, 2011
Paul W. Sampson

15. Reliability and maintenance days
Maintenance day goals:
Complete tasks necessary to maintain current mode of running at the accelerator complex.
Prepare for any new running conditions expected.
Minimize the total duration and the time from end of maintenance to a return to smooth running.
Some tasks performed can cause unstable running and increased failure in the days that follow maintenance periods.
Vacuum bleed-up.
Ion source maintenance.
Cryogenic.
RF systems.
Super Conducting power supply systems.
April 13th, 2011
Paul W. Sampson

16. Reliability after maintenance:
New running modes can cause increased failure for various reasons:
More aggressive magnetic cycles.
Increased high voltage.
Additional interlocks.
Re-integration of repaired systems:
Feedback elements.
Interlocking devices- loss monitors, chipmunks/radiation monitors, beam permit or inhibit systems.
Failure of new or un-commissioned systems:
Apertures in the beam pipe.
Improperly grounded devices.
Miss-configured interlocks or inhibits
Miss-configured devices (current loops).
April 13th, 2011
Paul W. Sampson

17. Maintenance methods for increased reliability
Scheduled Maintenance Days
Job Requests
Benefit and impact analysis
Prioritization
Individual work planning
Overall work planning and scheduling
System recovery
Accelerator recovery
Return to physics
April 13th, 2011
Paul W. Sampson

18. Before maintenance days
Determine which tasks are actually necessary.
Avoid low return, high risk jobs.
Carefully document accelerator status before turn off.
Define parameters for recovery.
Schedule accesses for efficiency- 4 hours to secure accelerators.
Communicate with all groups
Scheduling meeting, web, CCTV, face to face …
April 13th, 2011
Paul W. Sampson

19. Accelerator Physics/Administration Experimental Support and Facilities
Maintenance Support
Accelerator Physics/Administration
Experimental Support and Facilities
Accelerator systems
Integrated Facility Management

20. Job Request: Create
April 13th, 2011
Paul W. Sampson

21. Pic of list
April 13th, 2011
Paul W. Sampson

22. Work planning:
Individual jobs are planned by cognizant techs or engineers.
Job request is submitted to the MSG for review.
Primary approval given and schedule created.
Final review with upper management, experimenters and run coordinators.
Staff briefing.
Schedule posts.
April 13th, 2011
Paul W. Sampson

23. Schedule: Time Activity Personnel By 0600
Turn down OPPIS source for maintenance (After store to last to 0800hrs has been established)
MCR
0600
Apply CA LOTO to AGS
AGS Maintenance Begins (items outside the ring)
CAS
CAD
0800
RHIC areas to RA, RHIC Maintenance Begins
LINAC Maintenance begins
MSG
LINAC
0900
CA Access into the AGS
MCR/CAS
1100
AGS Access complete
1130
RHIC sweeps begin in select areas
1400
PHENIX tunnel sweep
1430
Hysteresis ramp, PHENIX IR sweep
1500
Physics setup in RHIC
MCR/AP…
April 13th, 2011
Paul W. Sampson

24. Submission: Individual Jobs
Initial review: Task Analysis, job approval (denial)
Initial Schedule Proposal
Final Schedule publication, review with staff
Execution of scheduled and Emergent tasks
System Recovery, Test ramp
Return to Physics
April 13th, 2011
Paul W. Sampson

25. Execution Complex formally handed over to Maintenance Support Group
MCR staffing:
Maintenance Coordinator (MC) to run activities, normally from MCR.
Utilize Operations Coordinator, Operators and support staff.
Leads keep MC informed of progress, emergent issues and job completion.
MSG leader appraised of major change, facilitates review, approval and schedules changes as necessary.
Maintenance coordinator executes changes.
April 13th, 2011
Paul W. Sampson

26. Communication:
During the maintenance, it is communication is essential.
CCTV
Web, elogs, Job request system.
Radio, pagers and telephone…
April 13th, 2011
Paul W. Sampson

27. End of maintenance:
Job closeout statements reviewed by MC before final closure.
Tiered access allows for recovery of systems to begin as soon as they are ready.
Injectors RF
Power supply
Vacuum
Instrumentation
Specialist confirms with MSG that systems have been returned to operational condition.
Specialist remains on alert until program is recovered
April 13th, 2011
Paul W. Sampson

29. Return to Physics
Accelerators are returned to operating condition as available.
Schedule work accordingly.
After injectors have recovered, RHIC test ramp performed.
Structured hand off to MCR for routine operation.
Maintenance Coordinator’s Journal closeout.
Operations Coordinator’s journal entry.
Maintenance day summary/closeout.
April 13th, 2011
Paul W. Sampson

30. April 13th, 2011
Paul W. Sampson

31. April 13th, 2011
Paul W. Sampson

32. April 13th, 2011
Paul W. Sampson

33. Results Since structured systems introduction in 2005:
The methods described have produced a marked improvement in performance on and after maintenance periods.
Return to productive running has steadily improved.
In many instances, machine performance is actually better immediately following maintenance periods.
Could maintenance have actually helped??!!
April 13th, 2011
Paul W. Sampson

34. April 13th, 2011
Paul W. Sampson

35. Maintenance Effectiveness Trend
April 13th, 2011
Paul W. Sampson

36. Other comments: Maintenance and repairs done behind physics stores.
Software releases.
Systems maintenance: BPMs.
Systems commissioning, improvement.
Studies following Maintenance days.
Dual mode days- Maintenance on one machine and science on another.
April 13th, 2011
Paul W. Sampson

37. Acknowledgements:
Availability, “Top Ten” and integrated luminosity plots adapted from presentation “Operating Statistics” by P. F. Ingrassia, Brookhaven National Laboratory.
“RHIC from space” courtesy of BNL Photo archives.
April 13th, 2011
Paul W. Sampson

38. Thank you
April 13th, 2011
Paul W. Sampson

39. April 13th, 2011
Paul W. Sampson

40. April 13th, 2011
Paul W. Sampson

41. April 13th, 2011
Paul W. Sampson