1. Diamond Light Source FPGA Development at Diamond
Austen Rose
Intelligent Controls for Particle Accelerators Meeting
January 2018

2. FPGA Development at Diamond
The very first thing to do is to say that I am NOT an FPGA developer, nor have I ever been one. I am presenting the work of others.
Introduction
Accelerator FPGA Developments
Beamline FPGA Developments
Detector FPGA Developments
Licensing hints
Meetings
Summary

3. FPGA Development at Diamond
Introduction – What is Diamond:
UK’s national synchrotron science facility on the Harwell campus
Electron storage ring with an energy of 3 GeV, 562m circumference
Three accelerators:
100 MeV Linac (including electron gun)
100 MeV to 3 GeV Booster Synchrotron
3 GeV Storage Ring
Generates synchrotron light from Hard X-rays to far infrared
> 30 operational beamlines

4. FPGA Development at Diamond
Introduction – Why FPGAs:
FPGA development has been important to Diamond
Many systems are dependent on the technology
The way FPGAs work is fundamentally different to the way that standard processors work
FPGA logic can be configured to offer pipelining and/or parallelism
Can deliver low latency that is predictable – not dependent on schedulers or interrupts
Extremely flexible in processing digital/digitised signals
High bandwidth
System-On-Chip (SOC) offered on many FPGA Designs, and there are two types:
Soft core – uses FPGA logic blocks to implement a processor
Hard core – has an embedded processor on the silicon with an interface to the FPGA logic
This will be a whistle-stop tour of developments over the life of Diamond
Not particularly detailed in any area
Want to know more:

5. FPGA Development at Diamond
Accelerator FPGA Developments:
Fast Orbit Feedback Overview
The Fast Feedback Communication Controller 
Libera EPBM
FA sniffer
FofbPMC
IsaPhoton
Multibunch Feedback Overview
TMBF / LMBF
Digital Low-Level RF
Power Supply Systems

6. FPGA Development at Diamond
Fast Orbit Feedback Overview:
A global Fast Orbit Feedback (FOFB) system to enhance electron orbit stability towards an ideal orbit.
Allow beam steering
The SR consists of 24 cells in the Storage Ring Lattice*
The FOFB system consists of:
~ 168 electronic Beam Position Monitors (BPMs)
24 VME-based feedback processors
~ 168 corrector magnet and power supplies
Communication network to distribute the beam position values to the feedback processors.
* No longer identical due to mini-beta cells and the DDBA cell.

7. FPGA Development at Diamond
The Fast Feedback Communication Controller (CC):
This project dates back to more or less the start of Diamond.
The CC subsystem is central to the data distribution over FOFB communication network
Low-latency, high-speed, network independent data distribution
VHDL IP*
This is part of all our Libera EBPMs and is integrated into the IsaPhoton machines, the FA Sniffer and a special card the FofbPMC
Placed on the PMC locations of the feedback processors.
The module has a Xilinx XCV2VP30-6 Virtex II Pro FPGA with four RocketIO links
See:
*An IP (intellectual property) core is a block of logic / data that is used in an FPGA Design.

8. FPGA Development at Diamond
Libera EPBM:
Mostly firmware provided by Instrumentation Technologies (i-Tech) with some local patches by us, together with integration of the communication controller
Has a Xilinx XCV2VP30-5 Virtex-II Pro FPGA
Libera will acquire electron beam position data and provide down-sampled and filtered beam position values to the fast feedback Communication Controller (CC) subsystem at fast feedback rate around 10kHz

9. FPGA Development at Diamond
FA sniffer:
Used for the fast archiver, and available for a number of different target hardware platforms
The FA Archiver captures full beam position orbit data at 10 kHz to a short term rolling archive, and republishes the live data stream interested client applications
The Fast Acquisition communication network communicates with the FA archiver through a communication controller FPGA firmware image installed on a Virtex-5 PCI Express development board, the LXT ML555* development kit
See:
*Found a couple on e-bay!

10. FPGA Development at Diamond
FofbPMC
The FofbPMC firmware is responsible for distributing the beam position values to the FOFB feedback processors. It targets PMC-SFP modules connected on the MV5500 feedback processors. The module has a Xilinx XCV2VP30-6 Virtex II Pro FPGA with four RocketIO links used for connecting the module to the FOFB communication network.
Can also operate in EVR mode and copies EVR events onto the fast feedback network.

11. FPGA Development at Diamond
IsaPhoton:
A digital back-end beam position processor monitor for analog XBPMs
Each IsaPhoton can interface to 2x Locum devices
Hardware is composed of 3 main hardware components:
8-Channel ADC board:
16-Bit, 200 kSPS ADC device.
Interface to XBPM blade currents: A,B,C,D
Xilinx XUPV2P FPGA development board:
Virtex-II Pro FPGA
Serial UART
Ethernet
2-channel SFP module interfacing to FPGA’s gigabit transceivers via SATA and SMA connectors available on the XUPV2P board.

12. FPGA Development at Diamond
Multi-bunch Feedback Overview:
The interaction of the electromagnetic fields (associated with the very high intensity electron beam) with the vacuum chamber and/or RF cavity creates wake fields which act back on the electron beam and produces growth of oscillations at the betatron frequency. If the growth rate of these oscillations is stronger than the natural damping, the oscillations become unstable creating so called “multibunch instabilities”. Although each electron bunch oscillates at the betatron frequency, they oscillate with different phase relationships between them, resulting in different multibunch modes.
These multibunch instabilities degrade the quality of the beam (e.g. increased beam emittance, energy spread) and can lead to beam loss. Such instabilities can be cured by using an active bunch-by-bunch feedback system where bunches are individually sampled and corrected.
Two variants:
Transverse MultiBunch Feedback (TMBF)
Longitudinal MultiBunch Feeback (LMBF)

13. FPGA Development at Diamond
Transverse MultiBunch Feedback (TMBF):
Longitudinal MultiBunch Feeback (LMBF)
Diamond is in the process of introducing two normal conducting cavities in their storage ring, along with the original super-conducting cavities.
The concern is that the cavity Higher-Order-Modes (HOMs) will cause increased longitudinal instabilities.
We are in the process of commissioning a new LMBF system
Uses a kicker cavity installed in the SR
We have developed code on two platforms:
Libera EBPM
uTCA based system
(More interesting to this talk!)

14. FPGA Development at Diamond
Libera TMBF:
This is a project with a long history, based originally on a Simulink implementation then converted to System Verilog, and then converted to VHDL.  This project is based on the Libera platform provided by i-Tech and includes firmware provided by them.  This system has been stable for a few years now and is unlikely to see further development, and the FPGA is now full!

15. FPGA Development at Diamond
MicroTCA MBF (multi-bunch feedback):
Currently referred to as LMBF.  Very active, currently deployed on machine, will replace existing Libera TMBF by end of Q2 this year
Although developing an LMBF system this can be extended to do TMBF and thus it becomes simply MBF
uTCA is a relatively new technology to us
We are using an AMC525 uTCA card with a Virtex FPGA from VadaTech
Interest shown from four other facilities
See:

16. FPGA Development at Diamond
Digital LLRF:
A development with ALBA based on their DLLRF system
Digital LLRF system for Booster and Storage Ring SC and NC cavities on microTCA platform
Xilinx Virtex-6 based Perseus platform form Nutaq
The DLLRF hardware is composed of two main subsystems:
uTCA Chassis + Linux PC + FPGA Motherboard
RF Front Ends

17. FPGA Development at Diamond
Power Supply development:
SLS Power supply design:
Quite an old design, goes back to early 2000s
Purchased rights to manufacture and develop the SLS digital power supply controller. Has a DSP processor and FPGA:
DSP - State control, closed loop control and interlocking
FPGA - PWM generation.
Altera FLEX 10K FPGA
Has separate ADC card with another small FPGA to manage data acquisition and communication
ADC Cards have been found to suffer from bit faults on conversion
For the EPICS supervisory and FOFB control there a VME based IP carrier card that is FPGA based

18. FPGA Development at Diamond
Power Supply development:
DLS Version I FPGA Based design:
Xilink Spartan 3E FPGA
Colibri PXA270 Processor
FPGA -PWM and control loop
Colibri processor – Direct EPICS interface
Delivers much reduced latency
ADC No longer available!
Currently controlling all the steerer magnets in the booster and has successfully run a FOFB system

19. FPGA Development at Diamond
Power Supply development:
DLS Version II:
Primarily an ADC card to replace cards in the SLS type ADC card
But designed with large enough FPGA and a Ethernet port to allow evolution into a closed loop control and PWM generation on one board
Xilinx Artix-7 FPGA

20. FPGA Development at Diamond
Beamline Developments:
Zebra
PandABox

21. FPGA Development at Diamond
Beamline Developments, Zebra:
Configurable Signal level converter and position capture unit
Front panel takes signals from multiple single channel inputs in TTL, LVDS, PECL, NIM or Open Collector format, converts the levels to LVTTL and passes the signals to an FPGA
Rear panel passes encoder signals to the FPGA input via Sub-D connectors
The FPGA implements logic gates and position capture circuitry, then signals are translated to be output in the same format as their respective input
Based on Xilinx Spartan-6
The system was designed by the Diamond Controls Group and the unit has seen great success at Diamond across many beamlines from tomography to MX and EXAFS, the unit simplifies and enables more flexibility at once
One-off SSI Encoder version
Sales of Zebra are ~67 with ~28 to DLS

22. FPGA Development at Diamond
Beamline Developments, PandABox:
The next generation configurable Position and Acquisition processing instrument developed in collaboration with Soleil. 
PandaBox is an upgrade of Zebra with significant functional and system integration improvements. It is based on Zynq 7030 and Spartan-6 FPGAs:
Gigabit Ethernet connectivity for control systems integration and high-speed data acquisition.
Capable Zynq-7030 SoC Zedboard device for significant improvement of existing processing, programmable functionality and tighter system integration.
16-Channel TTL and 4-Channel LVDS I/Os for synchronous triggering and clocking with higher timing resolution.
4-Channels of encoder interface I/Os, supporting a wider range of protocols.
3-Channels of SFP Gigabit Transceiver interface for possible applications including IpBus, Timing System, or custom high-speed serial connectivity.
A fully compliant Low-Pin Count FMC slot for interfacing to analog and digital off-the-shelf boards or custom I/O modules.
Sales of PandABoxes currently 14 (12 to DLS).

23. FPGA Development at Diamond
Beamline Developments, PandABox:

24. FPGA Development at Diamond
Detector developments:
LabView FPGA based
Merlin
STFC Work
Excalibur
Xspress3
FEM-II
Tristan (Large Area Time Resolved Detector)
Xspress4

25. Merlin Using LabView FPGA and NI PXIe FlexRIO
FPGA Development at Diamond
‘controller card’ and In rack ‘host’ PC
Merlin Using LabView FPGA and NI PXIe FlexRIO
FPGA Card, Adapter Card and power card
LabView Pros
Good platform integration.
DMA data transfer FPGA to host, or FPGA to FPGA.
Can mix in Lab Windows CVI to host code for performance boost.
Can add VHDL blocks.
LabView Cons
Cannot target non NI hardware.
Immediate interface must be in LabView.
Some built in primitives have low performance.
Medipix3
LabView FPGA targets NI hardware. In this case we use a Compact PCI Express chassis*, controller and FlexRIO FPGA card. This system uses a power card in a PCIe slot, to avoid drawing too much power through the FPFA card.
* Actually a PXI Express chassis. PXI: PCI eXtenstion for Instrumentation.

26. FPGA Development at Diamond
Excalibur System Overview:
Enhanced X-ray CAmera for Live Imaging and BURst mode operation:
Excalibur is a large area photon counting detector based on the Medipix3 read out chip.
3 sensor modules
3M pixels (11 x 10 cm)
2 FPGA cards / module
STFC FPGA hardware with STFC RTL code. Development funded by Diamond, with STFC and Diamond hardware design.
FEM
Designed by STFC for XFEL LPD
Reused in Excalibur and Xspress3
Virtex5
Front-End Module (FEM) FPGA Card
DLS/STFC Project

27. FPGA Development at Diamond
Xspress3 (STFC DL & QD):
The Xspress 3 was developed to maximise the throughput and resolution of such detectors and remove the bottleneck at the readout stage.
STFC FPGA hardware with STFC RTL code (FEM)
Xspress3 is a digital pulse processor for spectroscopic X-Ray measurement
It is an evolution of earlier STFC work
Virtex5

28. FPGA Development at Diamond
Front End Readout (FEM-II):
Large FPGA
Xilinx Virtex7 XC7VX690T
Zynq with ARM A9
8x 10GbE Data
1x 1GbE Control
DDR3 RAM
Lots of IO:
192 LVDS (32 CC)
96 SE
6 MGT
Developed by STFC
Used in LATRD and Xspress4 Diamond developments.

29. FPGA Development at Diamond
Large Area Time Resolved Detector (LATRD):
Based around CERN hosted Medipix3 Collaboration
Timepix3 ASICs, with STFC designed FEM-II FPGA card.
Electronic Hardware design by Diamond.
FPGA RTL by Diamond using Vivado.

30. FPGA Development at Diamond
Xspress4 DLS & STFC DL:
STFC FPGA hardware with STFC RTL code
Development funded by Diamond, with Diamond hardware design
Xspress4 is a digital pulse processor for spectroscopic X-Ray measurement
It is an evolution of earlier STFC work, using the larger FPGAs to increase the feature set
Xpress4 also uses FEM-II with Virtex7 XC7VX690T and Zynq
The main enhancement is cross talk correction

31. FPGA Development at Diamond
Licensing hints:
As a research institute we benefit from access to the EDA* tools via Europractice
Need to be wary of the terms of the licence agreements with the tool vendors. Although the terms of each EULA** differ, all will preclude commercial use with an academic licence, but some are overly restrictive. For example, they may not allow redistribution of the tool products - this has been interpreted by one vendor to include source code developed with the aid of the licenced software
Meetings:
We have formed a working group at Diamond bringing together all the FPGA code developers in different areas of the organisation (Primarily the Detector Group and the Controls Group). One aim is to discuss and perhaps rationalise our FPGA coding styles / build environments / tools. Its also really useful to be able to call on some local help when developing FPGA code
We have held one meeting with CCFE*** discussing our approaches to FPGA development. We hope to hold more
*Electronic Design Automation
**End User Licence Agreement
*** Culham Centre for Fusion Energy (

32. FPGA Development at Diamond
Summary:
Project
FPGA
Date
Comments
FoFBPMC
Xilinx XCV2VP30-6 Virtex II Pro FPGA
~2000
Legacy
Libera EBPM
Xilinx XCV2VP30-5 Virtex-II Pro FPGA
FA Sniffer
Xilinx Virtex-5 PCI Express dev board
~2006
Legacy?
IsaPhoton
Xilinx XUPV2P VirtexII 2 Pro FPGA dev board
Libera TMBF
Same unit as Libera EBPM
MicroTCA MBF
Xilinx Virtex FPGA - AMC525 uTCA card
Recent
Digital LLRF:
Xilinx uTCA Virtex-6
~2010
SLS Power Supply
Altera FLEX 10K FPGA
DLS I Power Supply
Xilink Spartan 3E FPGA
~2013
DLS II Power Supply
Xilinx Artix-7 FPGA
Zebra
Xilinx Spartan-6
~2011
PandABox
Xilinx Zynq 7030 System on Chip (SoC)
Xilinx Spartan-6 FPGA
Merlin
Xilinx Virtex-5
FEM
Xilinx Virtex5
FEM-II
Xilinx Virtex7 and Zynq

33. FPGA Development at Diamond
Summary:
If you have interest in any of these projects then contact me and I will pass on your details
Need to deal with Legacy systems – Electronics industry is fast-paced and components do go obsolete
A lot can be achieved with development board based solutions – do not need to invest in electronic design
uTCA seems to be gaining considerable momentum
FPGA development is a skilled activity

34. FPGA Development at Diamond
Thank you for listening!
Acknowledgements:
Isa Uzun – formally of Diamond Light Source, now at STFC Doing Detector Development
Michael Abbott – Diamond Light Source, Senior Software Engineer for Diagnostics
Glenn Christian –Diamond Light Source, Senior FPGA Engineer
Tom Cobb - Diamond Light Source, Senior Software Engineer for Beamline Developments
David Omar - Diamond Light Source, Senior Software Engineer for Detector Systems
Andy Cousins - Diamond Light Source, Electronics Engineer
Guenther Rehm – Diamond Light Source, Head of Diagnostics
Colin Abraham – Diamond Light Source, Senior Power Supply Engineer
Paul Hamadyk – Diamond Light Source, Senior Software Systems Engineer