1. Operational Experience @ SNS Karen S. White Controls Group Leader 10/11/10

2. 2Managed by UT-Battelle for the U.S. Department of Energy EPICS Collaboration Meeting - October 2010 The Spallation Neutron Source The world’s most powerful pulsed neutron source

3. 3Managed by UT-Battelle for the U.S. Department of Energy EPICS Collaboration Meeting - October 2010 Outline Machine Performance Controls Performance Issues New Stuff

4. 4Managed by UT-Battelle for the U.S. Department of Energy EPICS Collaboration Meeting - October 2010 Accelerator Daily Performance

5. 5Managed by UT-Battelle for the U.S. Department of Energy EPICS Collaboration Meeting - October 2010 Accelerator Lifetime Performance

6. 6Managed by UT-Battelle for the U.S. Department of Energy EPICS Collaboration Meeting - October 2010 Focus on Availability Availability considered more important than increasing the beam power Machine designed for 1.4 MW, but routinely operates at ~1MW to maximize availability and production hours

7. 7Managed by UT-Battelle for the U.S. Department of Energy EPICS Collaboration Meeting - October 2010 Accelerator Availability

8. 8Managed by UT-Battelle for the U.S. Department of Energy EPICS Collaboration Meeting - October 2010 Controls Availability FY07 – FY10

9. 9Managed by UT-Battelle for the U.S. Department of Energy EPICS Collaboration Meeting - October 2010 Controls Downtime Includes MPS and PPS FY09 = 116.8 hours – Dominated by PLC and PLC/IOC communications problems ~60 hr – PPS ~26 hr – MPS ~17 hr – Radiation Monitors ~14 hr FY10 = 40.8 hours – DeviceNet ~13 hr – PPS ~12 hr – MPS ~6 – Radiation Monitors ~3 hr

10. 10Managed by UT-Battelle for the U.S. Department of Energy EPICS Collaboration Meeting - October 2010 PLC and PLC/IOC Communication Late 2008, increase in PLC module failures and communication problems Two failures impacted CHL resulting in trips and the downtime associated with 4K to 2K transition (16.3 and 19.4 hours) Failure analysis revealed eight different contributing problems Worked with Allen-Bradley engineers to understand failures and plan improvements Analyzed control system response to failures in CHL and implemented improvements

11. 11Managed by UT-Battelle for the U.S. Department of Energy EPICS Collaboration Meeting - October 2010 PLC/IOC Failure Causes and Mitigation CauseMitigation HW failures in PLC processorsUpgraded processors (99) HW failure in PLC Ethernet modulesUpgraded Ethernet modules (50) HW failure in PLC I/O modulesReplaced failed modules Damaged fiber optic cableReplaced cable Memory corruption bug in IOC/PLC communication driver Bug fix installed on all IOCs Failing Automatic Transfer SwitchReplaced ATS PLC Firmware bugsFirmware upgraded; monitoring AB bulletins Single memory bit error in PLC processors Upgraded processors correct single bit errors

12. 12Managed by UT-Battelle for the U.S. Department of Energy EPICS Collaboration Meeting - October 2010 Machine Protection System Easy to “shoot the messenger” Most downtime due to noisy input signals Testing verified beam coupling related noise in the Ring Service Building Several improvements were made to reduce the noise and help the MPS better tolerate the noise Timing measurements showed many MPS nodes did not trip the beam within the required 20  s Primary causes for MPS delay – Large internal capacitors installed for noise filtering – Open collector sublink drive circuit slow response (capacitors make this problem a lot worse)

13. 13Managed by UT-Battelle for the U.S. Department of Energy EPICS Collaboration Meeting - October 2010 Performance Issues – LLRF Controls LLRF IOCs run automatic feed forward algorithm Increasing beam power -> longer beam pulses -> more data MVME2100 CPU could not processing all data for each 60Hz pulse Upgrading CPUs to MVME5500

14. 14Managed by UT-Battelle for the U.S. Department of Energy EPICS Collaboration Meeting - October 2010 Increasing Controls Scope Increasing support for Instruments – Applying accelerator controls technology where it makes sense for instrument construction and support Instrument Personnel Protection Systems Instrument CF*, Vacuum, selected device controls Sharing engineering expertise for problem areas – Instrument work not in original Controls scope Infrastructure not in place Standards not in place – propagating accelerator controls standards *LonWorks talk Wednesday 9:30 am

15. 15Managed by UT-Battelle for the U.S. Department of Energy EPICS Collaboration Meeting - October 2010 Lessons Take data – listen to what it tells you Sometimes what appears to be “a problem” has more than one cause Send clear messages for faults or errors Monitor and evaluate bug reports; apply patches as needed Plan for how the control system should respond to various types of failures Important to strike a balance between new technology and need for high availability

16. 16Managed by UT-Battelle for the U.S. Department of Energy EPICS Collaboration Meeting - October 2010 CSS Tools CSS applications for archiving (BEAUTY) and alarm handling (the BEAST) deployed in ops – Configured using RDB – Data and logging also stored in RDB New DM, BOY under development to interoperate with B&B BOY – Thursday 11am and 3 pm The BEAST Thursday 5 pm

17. 17Managed by UT-Battelle for the U.S. Department of Energy EPICS Collaboration Meeting - October 2010 New Stuff We now have the SNS Café in our building Spring 2011, a nearby Guest House will open for visitors Working on CD-2 for Power Upgrade Project Working on CD-1 for Second Target Station