1. Instituto de Plasmas e Fusão Nuclear Instituto Superior Técnico Lisbon, Portugal http://www.ipfn.ist.utl.pt M.Correia| Lisboa, May 24th 2010 | RT2010 ATCA/xTCA-based hardware for Control and Data Acquisition on Nuclear Fusion fast control plant systems Miguel Correia Control and Data Acquisition Group IPFN-IST

2. Author’s name | Place, Month xx, 2007 | Event M.Correia| Lisboa, May 24th 2010 | RT2010 2 CDAQ in Fusion Applications of CDAQ in modern Fusion plant systems Equipment integrity and interlock System state monitoring Recording system control actions Tokamak system operations control Preparing plasma discharge Ensuring plasma discharge quality Record system performance Human safety/security to enable control of the plant system

3. Author’s name | Place, Month xx, 2007 | Event M.Correia| Lisboa, May 24th 2010 | RT2010 3 Requirements for fast CDAQ Example of requirements for tokamak vertical stabilization of plasma Measurement resolution ENOB = 16 bit (ideally 18 bit). Sampling rate: 20 kSample/s, 200k Sample/s for micro-instabilities detection 200 kSample/s as normal operation mode Dynamic 20/200kSample/s with data decimation, Event distribution network required Loop cycle latency (general rule for all PCS diagnostics) Maximum of ~10% of the characteristic time of the phenomenon to be controlled. To implement a prototype of a Fast Controller for Vertical stabilization the system shall have 150 acquisition channels. (with redundancy)

4. Author’s name | Place, Month xx, 2007 | Event M.Correia| Lisboa, May 24th 2010 | RT2010 4 Need for MIMO control Some actuators are driven by controllers using data from several different diagnostics. Other (secondary control) diagnostics serve as backup in case of failure of the primary. Many variables are shared between several controllers. MIMO required – full-mesh support between boards and low- latency deterministic links between systems for global variable sharing.

5. Author’s name | Place, Month xx, 2007 | Event M.Correia| Lisboa, May 24th 2010 | RT2010 5 CDAQ in Fusion Challenges presented by modern Fusion experiments to the CDAQ subsystem increasing number of interdependent parameters to be controlled Increasingly faster loop-cycle response Implications to the CDAQ Massive processing power (parallel, multi-processing support) High bandwidth for data-transfer Advanced, intelligent, flexible timing & syncronization Real-time multi-input-multi-output (MIMO) control Steady-state operation for subsystems (including CDAQ) demands High-availability Failure prevention Redundancy

6. Author’s name | Place, Month xx, 2007 | Event M.Correia| Lisboa, May 24th 2010 | RT2010 6 CDAQ in Fusion Engineering point-of-view Modularity Expandability Programmability Compatibility with high-performing industry standards Multi-purpose Integrate further developed hardware Adopting commercial-off-the-shelf (COTS) products

7. Author’s name | Place, Month xx, 2007 | Event M.Correia| Lisboa, May 24th 2010 | RT2010 7 ATCA ATCA already successfully delivered valid solutions for MIMO CDAQ Configurable network topologies Supports various high-performance serial gigabit protocols Multiple processor support ATCA also provides important non-functional features, catering for the performance and security demands of such complex subsystems hardware management hot-swapping fail-safe large area form-factor and front-panel large power dissipation capabilities n+1 redundancy Obstacles (hardware development) Complexity – lenghty development ATCA (as yet) still undefined for Physics/instrumentation applications

8. Author’s name | Place, Month xx, 2007 | Event M.Correia| Lisboa, May 24th 2010 | RT2010 8 xTCA for Physics xTCA – specification based on ATCA, specialized for Physics Standardizes and facilitates hardware development for device operation in a Fusion control plant environment Extensive, hot-swappable use of the Rear Transition Module (RTM) panel, for input-output (IO) HA Concise data port assignment Improved signals for timing and synchronization (with compatibility between ATCA and AMC timing specs) Zone 3 (RTM) connectors specification

9. Author’s name | Place, Month xx, 2007 | Event M.Correia| Lisboa, May 24th 2010 | RT2010 9 Hardware design overview builds upon previous ATCA implementations (e.g. For JET and COMPASS) AMC carrier cards support Targeting compliance with xTCA specification Purpose: develop base prototypes for multiple aplications. Focus on MIMO control – high-bandwidth, low-latency communications networks (e.g. PCIe, SRIO, hardware assisted GbE)

10. Author’s name | Place, Month xx, 2007 | Event M.Correia| Lisboa, May 24th 2010 | RT2010 10 Target boards and networks 3 types of boards3 communications networks PCIe-hubsPCIe network (dual-star) Timing & Synchronization-hubsT&S network (dual-star) Node blades Node blade network (full-mesh)

11. Author’s name | Place, Month xx, 2007 | Event M.Correia| Lisboa, May 24th 2010 | RT2010 11 PCIe network PCIe-hub (redundant set) Dual-star (slots 1 and 2) Handles data from every PCIe endpoint, in every blade of the shelf 4 lanes (×4 PCIe Gen 1 or Gen 2)

12. Author’s name | Place, Month xx, 2007 | Event M.Correia| Lisboa, May 24th 2010 | RT2010 12 PCIe data hub PEX 8696 (PCIe switch) 96 PCIe Gen 2 (5 GT/s) lanes up to 24 flexible ports up to 8 upstream ports on-chip NT port for dual-host and fail-over applications hot-plug support 13 ×4 PCIe →ATCA fabric ch. (PCIe network) 4 ×4 PCIe → AMC 1 ×16, ×8 PCIe → RTM 1 ×4 PCIe → FPGA Virtex-6 LXT FPGA 128000 logic cells up to 600 user IO ports up to 20 6.6 Gb/s GTX transceivers ATCA/AMC/ARTM clk switch 5 ×1 SRIO → AMC/RTM 1 ×4 PCIe → PEX8696

13. Author’s name | Place, Month xx, 2007 | Event M.Correia| Lisboa, May 24th 2010 | RT2010 13 T&S network Timing & Synchronization network T&S-hub (redundant set) Dual-star (slots 3 and 4) distributes a customized group of timing and synchronization signals to all Node-blades. LVDS clock/gate/trigger signals Deterministic Gb timing link 4 lanes (×1 SRIO / ×3 LVDS)

14. Author’s name | Place, Month xx, 2007 | Event M.Correia| Lisboa, May 24th 2010 | RT2010 14 Timing & Synchronization hub PEX 8696 2 ×4 PCIe →PCIe network 4 ×4 PCIe → AMC 1 ×16, ×8 PCIe → RTM 1 ×4 PCIe → FPGA Virtex-6 LXT FPGA 128000 logic cells up to 600 user IO ports up to 20 6.6 Gb/s GTX transceivers ATCA/AMC/ARTM clock switch 5 ×1 SRIO → AMC/RTM 1 ×4 PCIe → PEX8696 11 ×3 LVDS → T&S network 11 ×1 SRIO → T&S network

15. Author’s name | Place, Month xx, 2007 | Event M.Correia| Lisboa, May 24th 2010 | RT2010 15 Node network (full-mesh) Node blade network P2P between every Node card Full-mesh (slots 5 to 14) Each port has ×1 SRIO / ×3 LVDS allows, in conjunction with the PCIe and T&S networks, to the CDAQ subsystem to achieve real-time MIMO connectivity to every endpoint.

16. Author’s name | Place, Month xx, 2007 | Event M.Correia| Lisboa, May 24th 2010 | RT2010 16 Node-blade PEX 8696 2 ×4 PCIe →PCIe network 4 ×4 PCIe → AMC 1 ×16, ×8 PCIe → RTM 1 ×4 PCIe → FPGA Virtex-6 LXT FPGA 128000 logic cells up to 600 user IO ports up to 20 6.6 Gb/s GTX transceivers ATCA/AMC/ARTM clock switch 5 ×1 SRIO → AMC/RTM 1 ×4 PCIe → PEX8696 (9+2) ×3 LVDS → Node/T&S (9+2) ×1 SRIO → Node/T&S

17. Author’s name | Place, Month xx, 2007 | Event M.Correia| Lisboa, May 24th 2010 | RT2010 17 T&S-hub / Node-blade 2 boards-types one PCB one BOM same assembly process one DIFFERENCE: FIRMWARE Virtex-6 LXT FPGA ATCA/AMC/ARTM clock switch 5 ×1 SRIO → AMC/RTM 1 ×4 PCIe → PEX8696 11 ×3 LVDS → T&S network 11 ×1 SRIO → T&S network (T&S-hub) or (Node blade) (9+2) ×3 LVDS → Node/T&S (9+2) ×1 SRIO → Node/T&S

18. Author’s name | Place, Month xx, 2007 | Event M.Correia| Lisboa, May 24th 2010 | RT2010 18 Overall ATCA fabric network setup Resulting ATCA fabric interface configuration, denoting the implemented network topologies and respective fabric channel connections.

19. Author’s name | Place, Month xx, 2007 | Event M.Correia| Lisboa, May 24th 2010 | RT2010 19 Examples of use IO Events FPGA Events FPGA Node TRG IO Clk GEN IO PCIe CPU storage REFCLK T&S-hub IO storage PCIe CPU Clk GEN IO PCIe CPU PCIe-hub PCIe Over cable Clock sync IO DSP Clock sync T&S-hub

20. Author’s name | Place, Month xx, 2007 | Event M.Correia| Lisboa, May 24th 2010 | RT2010 20 Summary A hardware architecture solution for fast control plant systems was presented. It is based upon a development of the ATCA specification, the most promising architecture to substantially enhance the performance and capability of existing standard systems, which has already delivered very encouraging results. The hardware designed consists of 3 different prototypes. Quad-AMC carrier format provides flexible, modular, expandable CDAQ. xTCA compliancy provides diverse IO solutions (e.g. IO just on RTM for improved hot- swapping capabilities), clock and trigger signaling and parallel processing, making it suitable for real-time MIMO control. Being designed according to the currently available xTCA specifications, this hardware aims to offer a set of base prototypes which can perform highly on many future CDAQ applications, in a Fusion control plant environment.