1. Timing System of the Swiss Light Source Timo Korhonen Paul Scherrer Institute, Switzerland 1. Introduction 2. Components and technology 3. SLS Timing Application 4. Future plans and possibilities 5. Conclusions

2. Fundamental task : Injection control -reference generation, RF & AC line synchronization -distribution of fiducial signals But: a timing system can be more: -a means to synchronize operations in several IOCs -operation sequencing tool -global timebase (timestamp support) ideal: to have "single cable" delivering all the timing to (practically) all the IOCs -a means to bind a distributed system into a single coherent entity. We did not want (nor had the time) to start from scratch. Starting point and requirements

3. Roots Looking around for possible solutions, we found that the APS event system (ANL) had almost all that we wanted: -synchronous timing signal distribution -hardware timestamping mechanism -local integrated delay generators -interrupt support and -well integrated into EPICS But the original hardware did not have enough bandwidth and an additional (fast) timing system would have been necessary. (also component availability was becoming a problem) Decided to upgrade to a recent technology (but to let an external company to do the work)

4. Event System Concept (APS) Event generator Event receiver Event code The stimulus to send an event can be: -pulse on a hardware input -software event (write to a register) -an entry in an event playback RAM. Upon receival of an event code the receiver can: -output a pulse, of specified delay and width -process an EPICS record Each event receiver can be programmed to respond in a different way to the same event code. Optical fiber link Event receiver Event receiver(s) -star/tree topology, unidirectional transmission -raw data frames are transmitted at the hardware frame rate to event receivers (no handshaking)) -external stimuli are encoded to 8-bit event codes; when no stimulus is present, null frames are sent out

5. SLS Event system Technical Features Technology: Gigabit Ethernet (physical layer) short wavelength (860 nm) fiberoptic transceivers ->industry standard, several sources Multimode (50/125 um) fiber (long wavelength transceivers optional, drop-in) 50 MEvents/second (50 MHz event rate), 20 ns resolution. Directly clocked from the main RF oscillator XILINX Virtex FPGA for the logic, loaded from flash ROM (in-system reprogrammable) FPGA code about 4000 lines of VHDL for EVR & EVG Device support (EPICS) Register model(VHDL) Hardware (Gigabit) As a side benefit, the functionality was extracted into a portable description (VHDL) Any of the functional layers can be modified according to needs and availability of technology. Hardware (10 Gigabit)

6. (SLS) Event Generator 3 possible event sources: -hardware (TTL) inputs. 8 per card -event generation by software (register write) -event RAMs(2) for event storage and playback (sequencer) Time Injection A pulse triggers the sequence in event RAM The sequence is clocked out with steps synchronized to the main clock, for example at Booster revolution frequency The sequence RAMs are the central workhorse of SLS -the step clock can be set (to multiples of revolution period) -alternate mode: one RAM runs while the second is loaded. -any sequence can be programmed through EPICS records. A single reference signal input triggers the sequence. All other fiducials are sent from the event RAM.

7. SLS Timing Master Consists of one Event Generator card, using presently two signal inputs: RF bucket zero-crossing marker (synchronized with AC line) and a utility frequency Even these could be generated internally in EG from RF and AC frequency inputs (when using newer EVG version)

8. Event Receiver Receives event input stream, actions for events of interest can be programmed individually. -programmable pulse and set/reset "flip-flop" outputs. -delay & width generators down to 20 ns resolution -trigger outputs phase locked to RF with a jitter of about 25 ps RMS (PLL of Gigabit transceiver) -VME interrupt facility for software synchronization (also a hardware-delayed interrupt) -timestamp counter can count received frames (or "tick" events). This enables a very fine timestamp synchronization without using bandwidth for clock transmission. The receiver latches the timestamp counter and the event number into a FIFO when event is received. Synchronous actions across IOCs: EPICS records can be triggered to process when an event arrives and stamped with the time the event occurred (vs. time of processing); Timestamps across IOCs match exactly.

9. Master Oscillator(s) @ 500 Mhz Bunch Clock Timing System Structure -Booster revolution clock (1.111 Mhz) -Storage ring rev. clock (1.042 Mhz) -BO & SR coincidence clock (69.4 kHz) Reference generation EVREVR EVGEVG Linac gun timing: Presettable delay cards (KEK TD4V) -precision to RF (2 ns) -jitter 4 ps RMS -pulsed magnets -BPMs -magnet & RF ramp trigger -diagnostics over 100 EVRs Line Sync EVREVR -AC line sync & clock divider -synchroniser to external (RF) pulse fan out Master Timing IOC Event generation and distribution Operation sequences Filling pattern control RF frequency control Overall timing Linac timing

10. Beamline Timing at SLS EVREVR EVGEVG Beamline can have its own timing events multiplexed with machine timing, but not interfering with other branches Needs coordination of event allocation EVREVR fan out Master Timing IOC Event generation and distribution Operation sequences Filling pattern control RF frequency control Beamline timing Beamline master EVGEVG Event receivers and generators can be cascaded -> add a sub-branch for a beamline

11. Future Plans: Diamond Event system Further development by Diamond Light Source (UK), Micro-Research and PSI (to ensure compatibility with our existing system) Higher frame rates, 2.5 Gbit data rate (or higher) Integrated RF clock recovery (in receivers) High-resolution delay generators (and online jitter measurement) integrated Full VME64x implementation (geographical addressing, hot swap support, etc.) Improved VME interface (higher bandwidth, simplified access) XILINX Virtex-II Pro FPGA (with an integrated PowerPC CPU core) Better possibilities for customization (mezzanine card slots for piggyback cards)

12. Hardware Capabilities Raw data transfer 50 MHz 16-bit frame rate (SLS HW), corresponds to 100 Mbytes/second data rate SLS implementation: 8 bits used for events, 8 bits “free”. At SLS, the free bits are used for distributing discrete clock signals. With a simple extension these could be used to broadcast arbitrary data at a maximal rate of 50 MBytes/s (in practice, would be limited by the IOC/VME bus bandwidth.) Protocol can be modified. For example: Transmit beam “codes” as data and the strict timing as events. In this way, pulse-to-pulse control information can be transmitted together with the events. Delays from events can be generated locally with the receivers. Local delay generators At the moment, purely counters. In future, fine tunable delay generators will be available (mezzanine card) RF signal regeneration, using tunable local oscillators for better jitter performance. The functionality can be customized to do what the hardware allows.

13. Integration with EPICS  Specific records to control the hardware (inherited from APS, extended for new features) delays, widths, prescalers, etc. controllable via EPICS records software events can be generated via EPICS records error handling mechanisms (alarm when hardware failure detected)  Binding with EPICS “soft” events: dispatch soft events when a hardware event is received generate sequences of actions when an event happens  Timestamp support hardware-supported timestamps match exactly. Simplifies correlation of measurements and actions done at different IOCs.

14. Timestamp mechanism Timestamp facility: –Each EVR has a timestamp counter Can count incoming events or frame clock cycles (after prescaler) –Received events that have been programmed to cause an interrupt cause the event code and timestamp counter value to be put in FIFO -interrupt handler reads the FIFO and updates event times in a table -records can retrieve these timestamps. They are synchronous in all crates that have an even receiver -timestamp counters have to be synchronized and reset periodically (done through network broadcasts)

15. Summary  Gigabit Ethernet as a base technology –Widely used, several component sources, reliable –Room for extensions  Functionality implementation in VHDL –Had a good starting point (APS) –Portable to new platforms –Easy to add and modify functionality  EPICS integration –Could use existing drivers and concepts, very big help for a new project

16. Master timing IOC LINAC timing IOC Pulsed magnets IOC Injection timing control Gun trigger (sync &delay) Gun trigger (sync &delay) Event generator Event receiver The SLS Storage Ring has 480 RF buckets, the Booster has 450. At the coincidence of SR and Booster period pulses (superperiod) the rings have their #0 bucket in alignment. From that instant the buckets proceed as BO 000 001 002 003... 449 000 001... 448 449 000 001... SR 000 001 002 003... 449 450 451... 418 419 420 421... The #0 buckets will be in alignment every 16th Booster period. By delaying extraction from booster one turn, the injection to SR is shifted by 30 buckets. Finer adjustment is done in the linac (e-gun trigger)

17. Beamline experiments Beamline experiments Injector Top-up Injection Gun trigger (sync &delay) Gun trigger (sync &delay) Event generator Event receiver The basic logic for top-up: -inject every N:th cycle when the current is below a defined threshold operations: e-gun trigger, SR injection kickers trigger enabled when a top-up cycle occurs Possibly switch off booster cycle (not tried yet) Warn beamlines, send a gate event to disable DAQ Need to synchronize several IOCs -synchronously count cycles to determine when to trigger -cycle data preset through channel access, synchronized with an event Event receiver