1. XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser Wojciech Jalmuzna, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 Software Architecture for LLRF based on ATCA

2. Wojciech Jalmuzna, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 2 Agenda Overview of the distributed system Requirements Possible algorithm locations Definitions of different software groups Control System Communication Links Diagnostics, Automation and Simulation Other software aspects Software Layout

3. Wojciech Jalmuzna, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 3 Overview of the distributed system cryomodule klystron

4. Wojciech Jalmuzna, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 4 Overview of the distributed system cryomodule klystron 24 channels 8 channel board 3x 8 channel board 3x 8 channel board 3x 8 channel board 3x 8 channel board 3x 8 channel board 3x 8 channel board 3x

5. Wojciech Jalmuzna, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 5 Overview of the distributed system cryomodule klystron 24 channels 8 channel board 3x 8 channel board 3x 8 channel board 3x 8 channel board 3x 8 channel board 3x 8 channel board 3x 8 channel board 3x Computation board DAC board multiple FPGAs multiple DSPs embedded systems surrounding devices

6. Wojciech Jalmuzna, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 6 Requirements The software architecture must fit to the distributed and redundant hardware platform. The scheme must integrate and interact with all possible applications to allow them to fulfill their requirements.

7. Wojciech Jalmuzna, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 7 Possible algorithm locations FPGADSP Embedded system CPUs on site Calculation Clusters Remote CPUs Low Level Applications High Level Applications Controller 5 ns200 ns50 ms

8. Wojciech Jalmuzna, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 8 Definitions of different software groups Controller The controller is the element of the system which has direct influence on the RF station. The main task of this application is to provide RF power to the cavities in a controlled way. During normal operation it controls vector sum of the cavities with a given precision. To achieve that, its reaction time must be as short as possible. Moreover, it provides interfaces for other software components giving them possibility to change the RF field parameters. Low Level Applications This is a set of applications designed mostly to execute algorithms and measurements between pulses. Parameters calculated during that time can be uploaded to the controller and used in the next pulse. These applications may also function as support to the controller during on-line operation. In this case, only DSP processor can be considered – other elements of the system cannot meet timing constraints. High Level Applications This is a set of applications designed mostly to run in the background of working system in long execution intervals. Parameters calculated by these algorithms are uploaded to the controller every few pulses. The main execution platforms for these applications are control servers and computation cluster.

9. Wojciech Jalmuzna, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 9 Controller Computation Pipe of the controller Field detection module Feedback module Output module To Vector Modulator Cavity signals - detection of the components of the field - I-Q - Amplitude, Phase - parametrized control function - P-I-D - MIMO - output corrections - klystron chain linearization - beam load compensation - etc.

10. Wojciech Jalmuzna, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 10 Piezo Control The controller is used for interaction with RF part of the RF station. The piezo sensors and actuators are handled by piezo controller. It executes algorithms and measurements related to the detuning and microphonics compensation. The piezo controller implemented on the separate hardware platform with high analogue channel count interacts with the controller, low and high level applications, using dedicated interfaces.

11. Wojciech Jalmuzna, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 11 Low Level Applications This includes: - Adaptive Feed-Forward - System Identification - Loop gain and loop phase calculation - detuning and loaded Q calculation - Vector sum calibration - Beam Diagnostic - Exception Detection and Handling

12. Wojciech Jalmuzna, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 12 High Level Applications This includes: - Adaptive Feed-Forward - Vector sum calibration - Beam Diagnostic - Automated frequency tuning - Exception Detection and Handling - RF-Gun control

13. Wojciech Jalmuzna, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 13 Applications summary The functionality of the low and high level applications overlaps. So how can we know where to put given algorithm ? It depends on our timing constraints: if we want to execute given algorithm as fast as possible (during a pulse or just between) it should be implemented as low level if we want to execute given algorithm with extended precision and functionality we implement it as high level

14. Wojciech Jalmuzna, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 14 Control System The applications presented above display thousands of operation parameters and provide tenths of signals for monitoring and diagnostics. It is not possible even for a large team of operators to monitor them separately using only the interfaces provided by each application. The role of gathering all necessary controls into one place belongs to the control system. It is the main tool for non-expert user for interaction with the software environment. The control system is executed simultaneously with the use of different facilities such as embedded systems, crate CPUs, computation clusters and client machines.

15. Wojciech Jalmuzna, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 15 Communication Links All interconnections amongst applications are impemented using one of the following communication links: ● Low Latency Link – for on-line communication amongst FPGA chips which perform computations during the pulse; they provide relatively high bandwidth and guarantee low latency of the transmission ● PCIe – for non-critical communication amongst different boards, used mostly for data acquisition system and operation parameters distribution (used also by LLA) ● Gigabit ethernet – used for communication of the system with on-site CPUs and control system.

16. Wojciech Jalmuzna, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 16 Diagnostics, Automation, Simulation To maximize the availability of the system, it is necessary to provide good fault detection and localization. This can be done by global diagnostic system. This system communicates with the critical elements of each layer, using dedicated diagnostic interfaces. The information acquired together with the results of on-line simulation allows to localize existing faults and notify operators by the control system interface. In addition, it interfaces the automation subsystems. This allows to implement and execute automatic fault recovery. The automation system is also responsible for machine state transitions and procedure execution. It acts on top of the control system.

17. Wojciech Jalmuzna, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 17 Other software aspects Operating systems Libraries IPMI controller Software used for development: Xilinx ISE (provided by FPGA vendor – license at reasonable price) Open source C compilers running on Linux VisualDSP for TigerSharc

18. Wojciech Jalmuzna, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 18 Development environment probe signals Down- converte r IF=54M Hz SIMCO N DSP Vector Modulat or control signal Down- converte r IF=54M Hz SIMCO N DSP Down- converte r IF=54M Hz SIMCO N DSP forward signals reflected signals ACB probe signals to klystron Before ATCA system arrives:

19. Wojciech Jalmuzna, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 19 Software layout RF station ControllerPiezo control Low Level Applications High Level Applications Control system Diagnostics piezoRF Automatio n

20. Wojciech Jalmuzna, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 20 Thank you for your attention