Accelerator Operations and Efficiency Presentation by Roger Erickson SLAC Operations Review SLAC, June 15-16, 2004

06/16/04 Accelerator Operations Review Outline  Accelerator Operations Section and Matrix Organization  Communications with Stakeholders  MCC Control Room Staffing  Operator Profile and Training  Conduct of Operations: Formality of Safety Procedures  Maintenance Management  ROD and Down Time Planning  Hardware Availability

06/16/04 Accelerator Operations Review Accelerator Operations Section

06/16/04 Accelerator Operations Review Matrix Management Accelerator programs are managed through a matrix organization structure with links to:  Program Coordinators for PEP-II and Linac Pulsed Programs.  Liaison persons representing each experiment.  System Physicists for each major section of the accelerator facility.  Area Managers – one for each major section, who report to the System Physicists.  Accelerator physicists and engineers responsible for subsystems or commissioning tasks.  Technical support departments with engineering and technician support for on-call maintenance.  Accelerator Department Safety Office.  ES&H Departments, including Radiation Physics and OHP.

06/16/04 Accelerator Operations Review Communications with Stakeholders  8:00 meeting every day, seven days a week, in MCC conference room. Operations report by EOIC, including run time statistics, progress in carrying out scheduled program, descriptions of problems, etc. Reports by representatives of major experimental user groups. Attended by Tech Div, Accel Dept, and Operations management reps, Program Deputy, System Physicists, Area Managers, representatives of major technical support departments, and interested users; typical attendance: 45 persons.  8:15 Maintenance meeting, five days each week. Run by Operations Section maintenance managers. Attended by supervisors of all maintenance groups, Area Managers, and interested physicists and engineers. Attendance: typically 45 persons.  Accelerator Plans Meeting, weekly. Review of operational progress, problems, and plans. Attended by Tech Div AD and Dept Heads, Program Coordinators, Accelerator Section Heads and ADSO.

06/16/04 Accelerator Operations Review Control Room Staffing  Program Deputy Duty rotates among Accelerator Department physicists. Oversees program for 3 or 4 day period. Not always present in control room.  Operations Engineer 1 EOIC per shift (from group of 7), on rotating 7-week cycle. Responsible for safely carrying out the scheduled program. Exempt position; directs the Accelerator Systems Operators. Must be present if accelerator is capable of producing beam. All have experience as accelerator systems operators.  Accelerator Systems Operators Typically 3 operators per shift on rotating 5-week cycle: Linac and sources PEP-II rings Secondary program, PPS control, and backup.

06/16/04 Accelerator Operations Review Typical Operator Profile Typical applicant is a recent graduate with a bachelor’s degree in physics. 15 have bachelor’s degrees 4 have master’s degrees 4 have a technical background and extensive SLAC experience. Accelerator Systems Operator years of experience Accelerator Systems Operator years of experience Accelerator Systems Operator 3 5+ years of experience Operations Engineer 2+ years of experience

06/16/04 Accelerator Operations Review Operator Training Program  Workbook and evaluation for each classification: Accelerator Systems Operator 1 Accelerator Systems Operator 2 Accelerator Systems Operator 3 Operations Engineer  ES&H Training (including RWT-II and LRCA)  Accelerator Operations Safety Procedures Training (including PPS operation, searching, interlock testing for each zone.)  US Particle Accelerator School  Downtime seminars and special projects

06/16/04 Accelerator Operations Review Main Control Center

06/16/04 Accelerator Operations Review Documentation Hierarchy

06/16/04 Accelerator Operations Review Safety Procedures Operators are trained to carry out certain well-defined tasks, following formally controlled procedures. Examples: Interlock Checklists Search Procedures Entry and Exit Procedures Safety Inspection Checklists Safety procedures: undergo a formal review and approval process. are updated as conditions change. are available on the AD Documentation web site, although the signed originals are preserved in paper form. Completed checklists and log books are archived.

06/16/04 Accelerator Operations Review Accelerator Department Safety Documents Accelerator Operations BAS Key and Lock ProceduresHot Sheet Accelerator Operations BAS InstructionsIncident Response Procedures Alarm Response ProceduresLaser Safety Procedures Arcs PPS LogLinac etc Building Emergency Plan Beam Containment System ProceduresLinac PPS Log BSOIC Certification Checklists Logging Requirement for Work… BSY PPS LogMCC Building Manager Checklists CID PPS LogMCC Emergency Plan DRIP/DR PPS LogMCC Key Safe Checkout Electrical Hazard Test ProceduresPEP PPS Log Emergency Notification FormPEP PPS Overview Entry and Exit ProceduresPLS Keys Procedures and Log Equipment Checkout LogPositron PPS Log ESA PPS LogPPS Interlock Checklists FFTB PPS LogPPS Zone Maps Final Focus PPS LogSafety Configuration Control Fire Alarm System Reference ManualSafety Inspection Checklists Guidelines for Work in SLAC Accelerator HousingsSearch Procedures Gun Test Lab Safety Procedure VVS Remote Turn-on Procedure High Voltage Test Facility Safety ProceduresWarning Response Procedures

06/16/04 Accelerator Operations Review Maintenance Management  Problems that interrupt the scheduled program trigger immediate repairs to restore operations.  24-hour on-site staff to repair: Mechanical utilities Power supplies Modulators Cryogenics  On-call staff for: Electrical power Magnets and other mechanical systems RF systems Vacuum systems OHP  Extensive call-in lists identify first responders, alternates, and experts for all systems.

06/16/04 Accelerator Operations Review Opportunistic Maintenance  Opportunistic Approach Whenever possible, identified maintenance, repair, and installation jobs are pre- planned, but deferred until an opportunity arises.  Coordination Daily 8:15am meeting. Coordinated by Operations Maintenance Office. Area Managers are advocates for their areas.  Standby Maintenance List (SML) Jobs that are planned and can be launched on short notice. Provides options for EOIC to take advantage of unscheduled interruptions.  Repair Opportunity Day (ROD) Launched only when required by a driving event. Carefully planned, and typically involve 100 or more jobs.  ARTEMIS Electronic Database

06/16/04 Accelerator Operations Review ARTEMIS  ARTEMIS (Accelerator Remedy Trouble Entry and Maintenance Information System) is an Oracle database with a Remedy interface, accessible over the internet by everyone involved with SLAC accelerator maintenance.  Operators report problems as they occur by making entries in ARTEMIS.  Repair work is launched immediately if the problem is inhibiting the scheduled accelerator program.  New ARTEMIS entries are reviewed at the daily maintenance meeting. Open questions are addressed and jobs are assigned.  Jobs may be assigned to the Standby Maintenance List (SML), or to the Repair Opportunity Day (ROD) list, or marked for the next long down time.  Jobs are tracked using ARTEMIS until they are finally closed.

06/16/04 Accelerator Operations Review Daily Maintenance Meeting

06/16/04 Accelerator Operations Review Review of New Problems Sample from the June 10 maintenance meeting.

06/16/04 Accelerator Operations Review Standby Maintenance List

06/16/04 Accelerator Operations Review Repair Opportunity Days  The accelerator systems are not turned off for maintenance on a routine, scheduled basis.  A ROD list of pending repair jobs is derived from the ARTEMIS database and maintained by the Operations Section. Installation and PM jobs are added as needed.  All ROD jobs are reviewed in the daily maintenance meetings.  The ROD list can be viewed over the internet.  When a “driving event” occurs, a detailed plan can be assembled and launched quickly, using the ROD list.  A typical ROD day includes about 100 distinct jobs.  Three ROD days so far in 2004.

06/16/04 Accelerator Operations Review Long Down Time Planning  Long Down Times usually occur once each year (although the linac hasn’t seen a long down time for two years) and typically last two to three months.  Area Managers collect job requests from system physicists, project engineers, maintenance groups, and the ARTEMIS database.  A comprehensive schedule is prepared, updated, and reviewed at weekly meetings, starting about three months before the down time.  This schedule lists each job with an assigned shop, and lists the Project or Group Manager, Responsible Task Person, and Area Manager.  Schedule specifies time and duration of each job (with 1 day time resolution), and number of persons required.  Detailed (shift by shift) schedule of pre-run testing, safety certification, and turn-on activities is prepared by the Operations Section and revised as needed on a real-time basis.

06/16/04 Accelerator Operations Review Maintenance Statistics  ARTEMIS entries since Jan 1989: 66,883 (as of June 14, 2004)  Total identified problems not yet resolved: 620 Many of these involve non-critical problems that were observed, but went away without clear explanation; e.g., a flow switch trip or a power supply found to be out of tolerance.  ARTEMIS entries since September 1, 2003: 2306 new problems (8.0 per day). 339 problems resulted in lost PEP-II run time problems closed (7.7 per day).  Active jobs: SML: 83 Next ROD: 89 Aug-Sept down time: 109

06/16/04 Accelerator Operations Review Accelerator Performance Statistics Revealed failures: malfunctions resulting in lost beam time. Also called “events”. Unscheduled down time: hours lost from scheduled program due to malfunctions. Mean Time to Fail: MTTF = Scheduled beam time Events Mean Time to Repair: MTTR = Unscheduled down time Events Availability = 1 - Unscheduled down time Scheduled beam time NOTE: PEP-II aborts are not counted as downtime, unless the event is reported; i.e., unless we stop to fix something and make a database entry.

06/16/04 Accelerator Operations Review Unscheduled Down Statistics Jan 10, 2000 – June 29, 2003

06/16/04 Accelerator Operations Review Down Time Analysis Details Example: PEP-II systems, Run 3

06/16/04 Accelerator Operations Review Availability, MTTF, and MTTR

06/16/04 Accelerator Operations Review PEP-II Time Accounting by Quarter Start of Trickle Charging