1. The Evolution of the APS Beamline Personnel Safety System (PSS) Work supported by U.S. Department of Energy, Office of Science, Office of Basic Energy Sciences, under Contract No. DE-AC02-06CH11357 Phil McNamara-Presenter Kenneth Belcher Greg Markovich Advanced Photon Source Safety Interlocks Group

2. 2 PSS History Overview

3. 3 GEN1 PSS  Processors: –Dissimilar: Chain A - Allen Bradley (AB) PLC5 – ESD, C&C intermixed Chain B - General Electric (GE) 90-70 – ESD only – Faults were not latched  Programming Languages –Dissimilar : Chain A – Ladder Logic Chain B – State Logic, Ladder Logic and MegaBasic  HMI: –LEDs and mechanical PBs  I/O Interface: –Hardwired to terminal blocks –Inconsistent addressing  DIW Monitoring –LOVE process controllers, binary output, 1 channel per station  Validations –Invasive, 2-3 days Hardwired Front-End Simulator Laptops running DOS (Windows 98)  Communication –Relay isolation for signals between chains (status, permits, and heart beat) –One-way communication to EPICS Chain A – DH+ through a Data Communications Module (PLC writes only, no read functions) Chain B – RS232 through a Processor Control Module (PLC writes only, no read functions)

4. 4 GEN1 PSS Chain A Emergency Shutdown and Command and Control PLC Chain B Emergency Shutdown PLC ACIS Air Supply Global Online Solenoid Safety Shutter 1 Safety Shutter 2 Photon Shutter 2 Integral Shutters Beam Line Station A WDT SR Trip/Fault Photon Shutter 1 Open status Closed status Open status Closed status SR Trip/Fault WDT Station A Searched Station A Beam Ready Closed status Door closed Estop Open status Closed status Note: No closed switch status for PS1 to ACIS

5. 5 GEN1 PSS Beamline vacuum sensors Beamline components DIW flow sensors Beamline EPS PSS Beamline Control System (Station Enclosure) User Panels (Pushbuttons, key switches and LED Indicators) PSS PLC Front-End Control System (Mezzanine Rack) PSS EPICS IOC FEEPS EPICS IOC Storage Ring MPS Front-End vacuum valves position (SV,FV,FEV,BIV) Front-End components DIW flow sensors Front-End vacuum sensors FEEPS (Mezzanine Rack) Remote IO link ACIS Sector Interface Enclosure ACIS IOC Front-End Shutters BL Shutter status Beamline component DIW flow sensors Relay contact from Love Controllers Solenoid control Solenoid control Closed switches FE shutter permit GOL key status Open switches Closed switches (except PS1) ACIS PLC (Main Control Room) FES Closed status PS1open request FE shutter permit FES Open status PSS SR trips Shutter air supply Pressure sensors BL Shutter permits Beamline Shutters (1,2or3) Hutch doors Search stations E-stop buttons Remote I/O link ACIS Controlled Equipment (RF&Dipole) FE shutter & vacuum permit FE shutter & valve status Beamline PSS interface diagram Up to 6 stations Beamline valves position Remote Shutter Interface Air supply shutoff (except PS1)

6. 6 GEN1 PSS 1. The Front-End Shutter (FES) Simulator is connected. The Front-End Shutters are critical devices that can’t be operated during the validation. Therefore their operations and statuses must be simulated. The connectors on the Front-End Relay Distribution Panel (FERDP) for the Front-End Shutters (FES), Access Control Interlock System (ACIS), Front-End Equipment Protection System (FEEPS), De-Ionized Water (DIW) and Beamline Equipment Protection System (BLEPS) systems are unplugged and the simulator connectors are plugged in their place. The simulator allows the operator to manipulate and monitor the signals normally supplied by these devices and/or systems. 2. An Input/Output Validation is performed. Verify that each device is properly functioning and wired to the assigned input or output of the Emergency Shut Down (ESD) system’s PLC. Verify chain independence. 3. A Functional Test (Validation) is performed. The system is monitored for proper response to inputs from the field devices and/or the simulator for each test case. In order to facilitate specific test cases for critical devices, the Station User Panel (15U) connectors are unplugged, and a test box is plugged in series with the existing devices. This test box is used to interrupt the signals from the field devices to simulate individual tests cases. After these particular tests are done, the test box is unplugged and the connectors are plugged back into their original receptacles. This process is repeated at each 15U several times during the validation. 4. At the end of the Functional Validation, the FERDP simulator is unplugged and the connectors are plugged back in, and a functional test of the front end components is performed. Testing Methodology

7. 7 GEN1 PSS Front-End Relay Distribution Panel (FERDP) GEN1 PSS

8. 8 Validation FES Simulator Cables Validation FES Simulator Rear Panel

9. 9 Validation FES Simulator GEN1 PSS

10. 10 GEN1 PSS DOS-based GE validation laptop

11. 11 1.Disconnecting the real I/O for validation is invasive. 2.Problems in reconnecting the real I/O. Not reconnected in the proper location Not seated fully Connectors or wires broken during the process 3.Maintaining laptops that are compatible with Windows 98 & DOS. GEN1 PSS Testing Methodology Problems

12. 12 GEN2 PSS  Processors: –Dissimilar Chain A - AB PLC5, ESD Chain B - GE 90-70, ESD Chain C - Industrial computer utilizing Siemens I/O, Command & Control  Programming Languages –Dissimilar : Chain A – Ladder Logic Chain B – State Logic, Ladder Logic and MegaBasic Chain C – Cimplicity soft PLC (Wonderware)  HMI: –Soft Panel displays & controls  I/O Interface: –Hardwired to terminal blocks  DIW Monitoring –LOVE process controllers, binary output, 1 channel per station  Validations –Invasive, 2-3 days Front-End Simulator Laptops running DOS (Windows 98)  Communication –Relay isolation for signals between chains (status, permits, and heart beat) –One-way communication from ESD systems to Chain C using Profibus (ESD systems write only, no read functions) –Profibus between Chain C and EPICS

13. 13 GEN3 PSS  Processors: –Similar - AB ControlLogix L61 Chain A – ESD Chain B – ESD Chain C – Command & Control  Programming Languages –Ladder Logic  HMI: –Soft Panel displays and controls  I/O Interface: –Hardwired to Circuit Boards –Consistent locations  DIW Monitoring –Initially LOVE process controllers, binary output, 1 channel per station –Migrated quickly to Analog modules in the ESD PLCs  Validations –2-3 days All I/O simulated (Diode injection) Touch Panel HMI  Communication –Electronic isolation for signals between chains (status, permits, and heart beat) –One-way communication from ESD systems to Chain C using Profibus (ESD systems write only, no read functions) –Profibus between Chain C and EPICS

14. 14 Chain-C Command & Control (C&C) PLC EPICSEPICS Chain A Emergency Shutdown (ESD) PLC ACIS –PSS Interface Air Supply Global On/Off Line Station A Door Control HMI & Non Critical I/O Photon Shutter 2, Safety Shutter 1, Safety Shutter 2 Integral Shutters Beam Line Station B Safety I/O HMI Station A Safety I/O HMI WDT, Cross Trip, Search Status, etc GEN3 PSS FES limit switches are not shown but are interfaced the same as GEN1 PSS with ACIS connected directly to FES closed limit switches. ACIS does not monitor PS1 closed limit switches. Chain B Emergency Shutdown (ESD) PLC Station B Door Control data

15. 15 Beamline vacuum sensors Beamline components DIW flow sensors Beamline EPS PSS Beamline Control System (Station Enclosure) User Panels (HMI’s and key Switches) PSS PLC Front-End Control System (Mezzanine Rack) PSS EPICS IOC - RSI & Clock Sync FEEPS EPICS IOC Storage Ring MPS Front-End vacuum valves position (SV,FV,FEV,BIV) Front-End components DIW flow sensors Front-End vacuum sensors FEEPS (Mezzanine Rack) Remote IO link ACIS Sector Interface Enclosure ACIS IOC Front-End Shutters BL Shutter status Beamline component DIW flow sensors Analog PLC modules direct to transducers (8) Solenoid control Closed switches FE shutter permit GOL key status Open switches Closed switches (except PS1) ACIS PLC (Main Control Room) FES closed status PS1 Open request FE shutter permit FES Open status PSS SR trips Shutter air supply Pressure sensors Chain A only BL Shutter permits Beamline Shutters (1,2 or 3) Hutch doors Search stations E-stop buttons Remote I/O link ACIS Controlled Equipment (RF&Dipole) FE shutter & vacuum permit FE shutter & valve status Up to 8 stations Beamline valves position Air supply shutoff (except PS1) GEN3 PSS Beamline PSS interface diagram PSS Trip Test Solenoid Control MEZZIE

16. 16 1. A testing system is connected in parallel. The Front-End Shutters are critical devices that can‘t be operated during the validation. A HMI & PLC are connected to all 3 PLCs via Ethernet. When the Test Cover is lifted the Front-End Shutters solenoid outputs and ACIS SR Permit are disconnected through relays. The HMI system will allow the operator to monitor signals coming from the PSS. 2. An I/O Validation is performed. This will still be done to verify that each device is properly functioning and wired to the assigned input or output of the Emergency Shut Down (ESD) system’s PLC. Verify chain independence. 3. A Functional Test (Validation) is performed. The Validation injection hardware is connected with no field device disconnections. The power supplies are disconnected from the field devices via electro-mechanical relays via the Validation system. The Validation system will allow the operator to manipulate all inputs going to the PSS. The ESD system can be monitored for proper response to inputs. All test case scenarios can be simulated. 4. At the end of the Functional Test, the Validation system is disconnected. 5. The field devices, inputs and outputs are re-enabled with none of the PSS wiring being disturbed. 6. The method of disabling and re-enabling the PSS inputs and outputs is failsafe. GEN3 PSS Testing Methodology

17. 17 1.Once the Validation system is connected, there is no unplugging and plugging in of connectors during the functional test. 2.The I/O Validation uses a touch panel HMI instead of laptop computers. 3.All inputs are disabled via relays instead of being unplugged. 4.All functional testing is done via injection of test signals. This allows for a full functional test of the system to be performed. 5.Upon completion of the functional test, the critical devices are re-enabled instead of reconnected. GEN3 PSS Testing Methodology – Key Changes

18. 18 Door Switch Shutter Closed Limit Switch Shutter Open Limit Switch +24V PSS 1Power is supplied to the PSS field device inputs via CR1 2Signals from the field devices pass through the series diodes D1 D2 D3 CR1 1 Connections for the Validation System 2 GEN3 PSS Testing Methodology Normal Operation-Validation System not connected

19. 19 Door Switch Shutter Closed Limit Switch Shutter Open Limit Switch +24V PSS Disable power to field devices Injected Test Signals Test Mode Enabled Bus Monitor 1 Power is still supplied to the PSS field device inputs via CR1 2 Chain C verifies Validation System is connected to turn OFF its outputs CR1 1 Removable Test Connector 2 GEN3 PSS Testing Methodology - Validation System connected

20. 20 Door Switch Shutter Closed Limit Switch Shutter Open Limit Switch +24V PSS Disable power to field devices Inject Test Signals Test Mode Enabled Bus Monitor CR1 1 Removable Test Connector GEN3 PSS Testing Methodology Validation System connected 1 Power is removed from the PSS field device inputs via CR1 2 The Validation System verifies field device power is disabled 3 Test signals can now be injected for functional testing 3 2

21. 21 1Front-End Shutter (FES) operations and Storage Ring Permit enabled via 4-pole relays 2All systems can verify FES is disabled 3 Outputs are monitored for proper operation GEN3 PSS Testing Methodology - Disabling of FES Validation System connected 1 3 2

22. 22 GEN3 PSS

23. 23 Validation System Main menu allows for configuration of beamline GEN3 PSS

24. 24 Front-End Shutter Functional Validation screen GEN3 PSS

25. 25 Station Functional Validation screen GEN3 PSS

26. 26 1. The connectors used in the Validation system are Multi Pin (~120) and are very fragile. 2. Real equipment is only exercised during the I/O validation. GEN3 PSS Testing Methodology Problems

27. 27 The GEN1 Mini-Upgrade GEN1 PSS with the following important but quick and cheap fixes  Hardware Changes –Changed the DIW monitoring to analog modules in the ESD PLCs (default set points are loaded at power up and allow adjust via HMI thereafter). –Added Chain B Fault Lights to the Panel.  Software Changes –Modified Chain B code to require a reset before re-granting Storage Ring Permit. –Modified Chain B code to indicate a flow fault condition by blinking the new Chain B Minor Fault LED. –Modified Chain B code so that any remote block to PLC communication loss will latch a fault, remove all shutter permits, and indicate a fault via blinking Chain B Fault LED. –Modified Chain A code so that the Storage Ring Permit was not a latched signal. –Modified Chain A code to lose station "Searched" status when it sees a Crash button pressed. –Modified Chain A code to generate a “Chain B Shutter Permit Lost" minor fault when a shutter is open and the corresponding Chain B shutter permit goes low. –Modified Chain A code to add a “quick search” when a signal, only available when the FES simulator is connected to the FERDP during validations, is asserted. –Modified Chain A code to lose search status for all stations any time a remote I/O communications fault exists. –Modified both Chains A & B to add a DIW bypass signal which is only available when the FES simulator is connected to the FERDP during validations, is asserted. –Modified both Chains A & B to remove all Shutter permits when the Storage Ring Permit is removed.

28. 28 The GEN1 Upgrade  Processors: –Redundant and Dissimilar: Chain A - AB PLC5 ESD and C&C in Separate routines Chain B - GE RX7i, ESD  Programming Languages –Similar : Chain A – Ladder Logic Chain B – Ladder Logic and C++  HMI: –LEDs and mechanical PBs  I/O Interface: –Hardwired to terminal blocks –Software mapping of all I/O for standardization  DIW Monitoring –Analog modules in the ESD PLCs  Acceptance testing –Done in Lab (when changes are made or every 5 years) using a Wonderware-based software simulator  Validations –Annual, Noninvasive, 1/2 day Touch Panel HMI Simulating the Front-End Shutters  Communication –Relay isolation for signals between chains (status, permits, and heart beat) –One-way communication to EPICS Chain A – DH+ through a Data Communications Module (PLC writes only, no read functions) Chain B – RS232 through a Processor Control Module (PLC writes only, no read functions)

29. 29 The GEN1 Upgrade  A thorough and concise specification document  Each PLC chain separates logic routines - functionally –Input and Output map(s) –ESD – immediate radiation hazard - logic Storage Ring Permit removing faults and trips –ESD – potential radiation hazard - logic Front-End Shutter Permit removing faults and trips Beamline Shutter Permit removing faults and trips –Command and Control logic (Chain A only) –Status logic –Warning logic  Major, Serious, and Minor Faults were changed to Faults, Trips, and Warnings with much clearer diagnostics for troubleshooting  ESD Permits driven by real equipment status rather than resulting fault codes  Standardized station addressing (100s = Station A, 200s = Station B)  Chain B reports the faults and trips  Chain B looks at Search status from Chain A at the leading edge of the searched signal  Doors locked when the search is complete, eliminating many Lock and Unlock buttons.  Each station now has Beam Ready, Fault/Trip, and No Access Allowed indicators. Software Changes

30. GEN 1 Upgrade PSS 30 Chain A Emergency Shutdown and Command and Control PLC Chain B Emergency Shutdown PLC ACIS Air Supply Global Online Solenoid Safety Shutter 1 Safety Shutter 2 Photon Shutter 2 Integral Shutters Beam Line Station A WDT SR Trip/Fault Photon Shutter 1 Open status Closed status Open status Closed status SR Trip/Fault WDT Station A Searched Crosstrip Station A Beam Ready Closed status Door closed Estop Closed status Open status Closed status ACIS now monitors PS1 as a backup for PS2 Chain B PS1 Permit added Search Status from A to B is now leading edge trigger Cross-trips were added Hardware Changes

31. 31 Beamline vacuum sensors Beamline components DIW flow sensors Beamline EPS PSS Beamline Control System (Station Enclosure) User Panels (Pushbuttons, key switches and LED Indicators) PSS PLC Front-End Control System (Mezzanine Rack) PSS EPICS IOC FEEPS EPICS IOC Storage Ring MPS Front-End vacuum valves position (SV,FV,FEV,BIV) Front-End components DIW flow sensors Front-End vacuum sensors FEEPS (Mezzanine Rack) Remote IO link ACIS Sector Interface Enclosure ACIS IOC Front-End Shutters BL Shutter status Beamline component DIW flow sensors Analog PLC modules direct to transducers (8) Solenoid control Solenoid control Closed switches FE shutter permit GOL key status Open switches ALL Closed switches ACIS PLC (Main Control Room) FES closed status PS1 Open request FE shutter permit EPICS clock sync FES Open status PSS SR trips Shutter air supply Pressure sensors Chain A only BL Shutter permits Beamline Shutters (1,2 or 3) Hutch doors Search stations E-stop buttons Remote I/O link ACIS Controlled Equipment (RF&Dipole) FE shutter & vacuum permit FE shutter & valve status Up to 6 stations Beamline valves position Remote Shutter Interface Air supply shutoff (except PS1) GEN1 Upgrade PSS Beamline PSS interface diagram

32. 32 Internal memory tables were created to map all beamline I/O to a common addressing scheme (I/O wiring is not consistent across all beamlines).  This allows the Acceptance and Validation Systems to have a single program for all beamlines.  This allows the Acceptance and Validation Systems to control where the inputs to the ESD logic originate (i.e., from either real world devices or software signals).  This allows the Acceptance and Validation Systems to simulate the Front-End Shutters and other critical signals. Testing Methodology GEN1 Upgrade PSS

33. 33 Before PLC code is downloaded for testing at a beamline, testing is performed in a lab simulator called the Acceptance Test System (ATS) The ATS procedures tests ALL of the PLC logic ESD – immediate radiation hazard - logic (each fault and trip tested n+1 times) Storage Ring Permit removing faults and trips ESD – potential radiation hazard - logic (each fault and trip tested n+1 times) Front-End Shutter Permit removing faults and trips Beamline Shutter Permit removing faults and trips Command and Control logic Warning logic Testing Methodology – The ATS GEN1 Upgrade PSS

34. 34 Acceptance Test System GEN1 Upgrade PSS

35. 35 I/O validation is performed first to verify that all beamline input and output devices operate correctly and are mapped properly to internal memory tables The VTS procedures perform testing for ONLY the following PLC logic ESD – immediate and potential radiation hazard - logic Storage Ring Permit removing faults and trips Shutter Permit removing faults and trips After a Validation is complete, the Validation System is disconnected from the PSS and the PSS returns to normal with no residual data (created by the Validation System) remaining in the data tables of the PLCs. It is of utmost importance to ensure the real inputs are mapped to the internal files/bits and no faults or trips are blocked after the Validation System is removed. This is accomplished by the following method: –A pulse, originating from the Validation System, must be present to activate any of these Validation System functions in the PLCs. –Perform End-to-End Test (PSS input through ACIS output to dump beam). Testing Methodology – The VTS GEN1 Upgrade PSS

36. 36 Validation Test System GEN1 Upgrade PSS

37. 37 Validation System Main menu allows for configuration of beamline GEN1 Upgrade PSS

38. 38 Functional Validation screen manipulates FES switches to create trips and faults GEN1 Upgrade PSS

39. 39 Main Functional Validation screen contains mezzanine based signals and allows Validator to monitor SR Permit status GEN1 Upgrade PSS

40. 40  Processors: –Similar - AB ControlLogix L61 Chain A – ESD Chain B – ESD Chain C – Command & Control  Programming Languages: –Ladder Logic  HMI: –Soft Panel displays and controls  I/O Interface: –Hardwired to Circuit Boards –Software mapping of all I/O –Diagnostic Modules on Front-End Shutter I/O  DIW Monitoring –Analog modules in the ESD PLCs  Acceptance Testing –Done in Lab (when changes are made or every 5 years) using a Wonderware-based software simulator  Validations –Annual, Noninvasive, 1/2 day Touch Panel HMI simulating the Front-End Shutters  Communication –Electronic isolation for signals between chains (status, permits, and heart beat) –One-way communication from ESD systems to Chain C using Produced and Consumed Tags over Ethernet –Ethernet between EPICS and Chain C using Produced and Consumed Tags GEN3 Upgrade PSS

41. 41 APS PSS Comparison Gen 1Gen 2Gen 3Gen 1UGen 3U Hardware AB - PLC5-30 3 - AB Contrologix L61‘AB - PLC5-303 - AB Contrologix L61 GE - 90-70 GE - RX7i Industrial Computer with Semiens I/O Discrete panel interfacesGE - Soft panel displaysAB - Soft Panel displaysDiscrete panel interfacesAB - Soft Panel displays Love Controllers Love Controllers /Analog ModulesAnalog Modules I/O Interface Hardwired - No Standard Circuit board - StandardizedHardwired - No StandardCircuit board - Standardized Mapped in Logic to standardizeMapped in Logic Communication to EPICS AB - DH+Profibus through Chain C AB - DH+Ethernet through Chain C GE - Serial Produce & Consumed Tags Software Chain A - ESD and C&C combinedChain A - ESD only Chain A - ESD and C&C in separate routinesChain A - ESD only Ladder logic Chain B - ESD only State, Ladder and MegaBasic Ladder logic Ladder logic & C++Ladder logic Chain C - C&C Wonderware Ladder logic Acceptance Testing None Done in Lab utilizing Wonderware Testing All Logic (safety, C&C and diagnostic) System Validation Invasive Non invasive - DOS Based PCs Touch Panel HMI - Hardwired FES Simulator - Diode injection- simulating FES - all other I/O is real - Contrologix processor with Seimens I/O- all other I/O is real - simulating all I/O All Logic is tested (safety, C&C and diagnostic) Only ESD – prompt radiation hazard - logic is tested