1. EPICS Collaboration Meeting Fall 2012 @ PAL October 22 ~ 26, 2012 LCLS-I/LCLS-II Timing System Low Level Kukhee Kim for LCLS Timing Team ICD Software, SLAC National Accelerator Laboratory October 22, 2012

2. EPICS Collaboration Meeting Fall 2012 @ PAL October 22 ~ 26, 2012 2 Event System is a most complicated system!

3. EPICS Collaboration Meeting Fall 2012 @ PAL October 22 ~ 26, 2012 3 Accelerator complex for current and future (LCLS-I, LCLS-II and …) TS4, and TS1 TS5 and TS2 TS6, and TS3

4. EPICS Collaboration Meeting Fall 2012 @ PAL October 22 ~ 26, 2012 4 360Hz – constraint: What is the timeslots? LCLS-I LCLS-II FACET

5. EPICS Collaboration Meeting Fall 2012 @ PAL October 22 ~ 26, 2012 5 Legacy Timing System SLC timing system -The Linac is a Pulsed Machine (get a packet of beam per pulse) runs at a max of 360Hz (120Hz) -Three Main Timing Signals: Hardware -476MHz Master Accelerator Clock (runs down 2mile Heliax Main Drive Line cable) -360Hz Fiducial Trigger / encoded onto the 476MHz master clock Software – MPG (Master Pattern Generator) -128-Bit PNET (Pattern Network) Digital Broadcast (contains time slot, trigger setup, beam type & rate information, beam destination) -Provide the timing pattern 3 fiducial prior Advantage of the Legacy system -Physicists & Operator are used to the old system -Provide event patterns 3 fiducial prior give a time margin to slow device, or pre-trigger up to 2 fiducial ahead

6. EPICS Collaboration Meeting Fall 2012 @ PAL October 22 ~ 26, 2012 6 Old architecture for the LCLS-I Timing System DEVDEV ~ Linac main drive line Sync/Div SLC MPG PNETPNET 119 MHz 360 Hz SLC events LCLS events PNETPNET PP PDUPDU EVREVR OCOC TTL-NIM convert. Digitizer LLRF BPMs Toroids Cameras Wire Scanner SLC klystrons TTL SLC Trigs FANFAN OCOC Low Level RF EPICS Network Precision<10 ps fiber distribution LCLS Timing System components are in RED *MicroResearch * * EVGEVG LCLS Master Oscillator 476 MHz Linac Master Osc System is based around the EVent Generator and EVent Receiver FIDOPDU Raw 360 Hz LCLS Timeslot Trigger

7. EPICS Collaboration Meeting Fall 2012 @ PAL October 22 ~ 26, 2012 7 LCLS-I Timing System LCLS Timing System Use MRF EVG and EVRs Generate event code at 360 Hz based on: -PNET pattern input (beam code + 128 bits pattern that define beam path and other conditions, scheduled information) -Receive MPS *) information (real-time information), and queue it up into the pattern pipeline (pipeline bypass for quasi real-time) -Add LCLS information (MPS, BPM calibration, diagnostics and ….) Extend the pattern to 192 bits -Hardwired BCS fault Send out timing pattern, including EPICS timestamp with encoded pulse ID on the timing fiber Manage user-defined Beam Synchronous Acquisition (BSA) MPS *) Please, see the talk for LCLS Machine Protection System (M. Boyes) in hardware solution session (16:30 – 16:50 Oct 22, 2012)

8. EPICS Collaboration Meeting Fall 2012 @ PAL October 22 ~ 26, 2012 8 How to work the EVG PNET message broadcast from the old timing system (following a fiducial, 360 Hz rate) PNET receiver in EVG IOC generates IRQ EVG software takes the PNET data and uses it to assign the proper event codes (the PNET data pipelined 3 fiducial ahead) Event code setup in EVG one cycle head Next fiducial is sent: 360 Hz fiducial signal received by EVG The EVG begins to send out the event codes in its Sequence RAM The event codes get sent across the timing fiber links to the EVRs Inside the EVR, its mapping RAM is set to map a specific HW trigger to an event code when the Event Code matches the same value in the mapping RAM, a hit is generated and after a programmed delay, a HW trigger is output form the EVR to the device

9. EPICS Collaboration Meeting Fall 2012 @ PAL October 22 ~ 26, 2012 9 How to work the EVR Set trigger delays, pulse width, and enable/disable via user requests Set event code per trigger Receive timing pattern ~8.3 msec (3 fiducials prior) before corresponding pulse. Provides EPICS timestamp to record processing Perform BSA based on tags which are set by EVG in the timing pattern Process pre-defined records when specific event codes are received

10. EPICS Collaboration Meeting Fall 2012 @ PAL October 22 ~ 26, 2012 10 Timeline for the EVR F0 0 Fiducial Time (usec) B0 Acq Trigger 1023 Beam Kly Standby Record processing (event, interrupt) Fiducial Event Received Event Timestamp, pattern records, and BSA ready Receive pattern for 3 pulses ahead Hardware Triggers ~500 0.3 ~40 100 Triggering Event Codes Kly Accel F1 Fiducial … 2778 110 StartEnd

11. EPICS Collaboration Meeting Fall 2012 @ PAL October 22 ~ 26, 2012 11 Upgrade Timing system for both LCLS-I and LCLS-II Main Goals Remove the dependency with the SLC timing system Independent EVG for each beam program -LCLS-I, LCLS-II, FACET, XTA and future beam programs -Promise for independent operation -Relieves the pain from limited event number (255 limitation) Keep compatibility with old system, keep SLAC specific features (advantage of the legacy system) -Beam code and timing pattern pipeline: For pattern aware operation -BSA -Event Code Sequence for photon group Covers various platforms for EVR: RTEMS, linux, and linuxRT -LCLS-I, and LCLS-II both are going to use various hardware platforms VME, PMC, Standard PC, uTCA, and COM-X Need to support linux/linuxRT for new hardware platform

12. EPICS Collaboration Meeting Fall 2012 @ PAL October 22 ~ 26, 2012 12 Timing System Upgrade Cont’d Upgrade Develop a new hardware called VMTG: VME based Master Trigger Generator which provides 360 Hz IRQ to drive EVG software instead of the PNET receiver (VMTG in EVG side) Develop new software module PABIG: Pattern Bits Generator which mimics the PNET Develop a new software tool to design the event/timing pattern visually: evGUI 1) Work on the EVR software to support linux/linuxRT 2) for the new hardware platform: uTCA, and COM-X evGUI 1) Please, see the talk for LCLS timing system for pattern design, evGUI and high level (M. Zelazny) in the timing system session (15:00 – 15:20 Oct 22, 2012) Support for linux/linuxRT 2) Please, see the talk for Real-Time Performance Issues on linux/linuxRT (K. Kim) in the low level application session (14:10 – 14:30 Oct 23, 2012)

13. EPICS Collaboration Meeting Fall 2012 @ PAL October 22 ~ 26, 2012 13 New architecture for the timing system MPS Link Node MPS Network LCLS-II CD-2 DOE Review / Aug 21 - 23, 2012

14. EPICS Collaboration Meeting Fall 2012 @ PAL October 22 ~ 26, 2012 14 VMTG  Sector 0 (LI00) VMTG – Master Trigger Generator Provides the synchronized 360Hz trigger to the fiducial generator Input: 360Hz AC trigger train + 476 MHz MDL RF Ouput: 360Hz synchronized trigger Then, the fiducial generator makes synchronization with the Master Oscillator  Sector 20 (IN20) VMTG with the EVG – Master Trigger Generator Provide 360 Hz interrupts with the Timeslot counter Provide Timeslot Synchronization Input: 60 Hz AC trigger (AC phase C) + Synchronized 360 Hz trigger Output: 360 Hz interrupts + timeslot counters

15. EPICS Collaboration Meeting Fall 2012 @ PAL October 22 ~ 26, 2012 15 Disabling PNET allows VMTG control PNET task needs to be delayed ~ 150 usec after the PNET IRQ to wait the MPS message. PNET IRQ is using HW_TIMER to make the delay The PNET ISR implement 150 usec delay on the HW_TIMER The callback for the HW_TIMER releases signal to execute the PNET Task. VMTG has configurable IRQ delay Configure the VMTG IRQ delay as ~570 usec The VMTG IRQ occurs 150 usec after the PNET IRQ Fiducial PNET VMTG Unknown delay PNET IRQ ISR implements the timer Timer callback releases the signal to the PNET task PNET task VMTG IRQ The ISR releases the signal to the PNET task ~150 usec ~ 570 usec MPS message arrives somewhere before the signal.

16. EPICS Collaboration Meeting Fall 2012 @ PAL October 22 ~ 26, 2012 16 Software Stacks for the new EVG application and the PABIG PNETVMTG Acromag DIOEVG BCS SBI Permit BCS Beam Permit 119MHz Clock 360Hz Fiducial TriggerPNET broadcasting Asyn moduleVMTG module EVG Drv/Dev PABIG GenVar Master Beam Control Bits 0 to 8 via CA HW input bits in PABIG EVG Application EPICS DB EDEF reservation for the BSA PNBN PVs from SoftIOC SNL program 476MHz Clock 360Hz Trigger 360Hz Fiducial Trigger 2 nd NIC udpComm mpsComm PNET Task MPS message (beam destination & Permit)

17. EPICS Collaboration Meeting Fall 2012 @ PAL October 22 ~ 26, 2012 17 EVR Software Stacks/ Form factor & OS dependency VME EVR HardwarePMC EVR Hardware mrfCommon/mrfVme64 drvErMrf devErMrf erRecord BSA EVR Processing Logic Event Module for RTEMS/vxWorks erapi drvLinuxEvr devErMrf erRecord BSA EVR Processing Logic Event Module for linux/linuxRT PMC EVR Hardware Works with old register mapWorks with modular register map (new)

18. EPICS Collaboration Meeting Fall 2012 @ PAL October 22 ~ 26, 2012 18 Summary Successfully upgraded timing system Standalone from the SLC system Same code base for LCLS-I, LCLS-II and future projects Backward compatibility, keep the advantages from the legacy system Our Team has done the following work: Develop new VMTG hardware: John Dusatko Develop PABIG software: Till Straumann Develop evGUI: Mike Zelazny and Partha Natampalli EVG software integration, EVR software support for linux/linuxRT and real-time performance issues: Kukhee Kim, Till Straumann, Stephanie Allison

19. EPICS Collaboration Meeting Fall 2012 @ PAL October 22 ~ 26, 2012 19 Related Talks LCLS Machine Protection System (M. Boyes) in hardware solution session (16:30 – 16:50 Oct 22, 2012) LCLS timing system for pattern design, evGUI and high level (M. Zelazny) in the timing system session (15:00 – 15:20 Oct 22, 2012) Real-Time Performance Issues on linux/linuxRT (K. Kim) in the low level application session (14:10 – 14:30 Oct 23, 2012)

20. EPICS Collaboration Meeting Fall 2012 @ PAL October 22 ~ 26, 2012 Thank You!