XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser Communication in ATCA-LLRF System LLRF Review, DESY, December 3rd, 2007 Communication in ATCA-LLRF System Presenter: Waldemar Koprek

2 Agenda Requirements Concept Communication interfaces in ATCA standard –Base Interface –Fabric Interface –Links on a Carrier Board –Links to RTM –Links to Other Systems Schedule 2 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser Agenda Requirements Concept Communication interfaces in ATCA standard Channel definition on a fabric interface Proposed backplane topology of an ATCA shelf Schedule Communication in ATCA-LLRF System LLRF Review, DESY, December 3rd, 2007

Agenda Requirements Concept Communication interfaces in ATCA standard –Base Interface –Fabric Interface –Links on a Carrier Board –Links to RTM –Links to Other Systems Schedule 3 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser Requirements 1.Fast, low latency links for fast feedback loop, piezo control 2.High throughput between pulses for DAQ systems – sending a lot of data recorded during pulse 3.Boards management mechanism 4.Boards control in ATCA Shelf 5.Control of RTMs 6.Interfaces to other systems Communication in ATCA-LLRF System LLRF Review, DESY, December 3rd, 2007

XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser Communication interfaces offered by ATCA standard PICMG3.0 – General Specification, ATCA Backplane Topology, IPMI E-Keying Subsidiary Specifications: –PICMG3.1 Ethernet / Fibre Channel –PICMG3.2 Infiniband –PICMG3.3 PCI Express / Advanced Switching –PICMG3.4 Starfabric –PICMG3.5 RapidIO –PICMG3.6 PRS fabric 4 Communication in ATCA-LLRF System LLRF Review, DESY, December 3rd, 2007

XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser General concept RTM1 AMC1 CPU HUB RTM2 ATCA Shelf Shelf Manager I 2 C - IPMI over ATCA Backplane - IPMI on a carrier card - Control link to RTM Low Latency Link - Data transport from ADC to a carrier card - Partial vector sum between carrier cards - Interface to piezo system Other systems 5 Other systems Ethernet - Modules configuration and software management - Control system to all modules - NFS over backplane - IPMI over ethernet - Interfaces to other systems Other systems Carrier Card PCIe - DAQ system between AMC modules, carrier cards for LLA and HLA

XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser Latency in selected communication standards Rocket IO –direct peer-to-peer connections –latency around 300 ns (Virtex 2 Pro) and 100 ns (Virtex 5)‏ PCI Express –connections via switch –minimum latency ns depending on payload size –up to 10 us with higher switch load Gigabit Ethernet –2.5 us 10 Gigabit Ethernet – ns 6 Communication in ATCA-LLRF System LLRF Review, DESY, December 3rd, 2007

XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser Agenda 7 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser ATCA-LLRF Channel Definition for Shelf Backplane Rx+ Rx- Tx+ Tx- Rx+ Rx- Tx+ Tx- Fabric Channel PICMG 3.0 Port 3 Port 1 Port 2 Port 0 Lane 2 (Optional) Lane 1 Custom Link 2 Custom Link 1 Port 3 Port 1 Port 2 Port 0 PCIe Link Custom Link ATCA-LLRF Channel 1000BASE-BX (Optional) 1000BASE-BX Custom Link 2 Custom Link 1 Port 3 Port 1 Port 2 Port 0 Ethernet Custom Link Option 1 Option 2 Communication in ATCA-LLRF System LLRF Review, DESY, December 3rd, 2007

XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser How many slots do we need in one ATCA shelf? 5 x Carrier Boards (2 slots each) – 10 slots –4 cards for 4 cryo-modules –1 card as a main controller 1 HUB (or 2 for redundancy) – 1 or 2 slots CPU – 0,1 or 2 slots –1 CPU without redundancy –2 CPUs with redundancy –0 CPU in case of distributed control system (AMC- CPU cards, processors on carrier cards) TOTAL: slots ATCA shelf 8 Communication in ATCA-LLRF System LLRF Review, DESY, December 3rd, 2007

XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser P24 P23 P22 P21 P Base Interface Fabric Interface Update Clock Base Interface in 14 Slots Shelf To Shelf Manager HUB

CPU C5C4 C3 C2 C1 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser P24 P23 P22 P21 P20 Base Interface Fabric Interface Update Clock Backplane Links in the 14 Slots Full Mesh Shelf Utilization 8 of 14 channels 57% 100% with redundant HUB Utilization 10 of 91 channels 11% Maximum 16 of 91 channels 17% To Shelf Manager 1000BASE-T 1000BASE-BXRocketIO Optional PCIe or 1000BASE-BX

HUB CPU C5C4 C3 C2 C1 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser P24 P23 P22 P21 P20 Base Interface Fabric Interface Update Clock Utilization 8 of 14 channels 57% Utilization 10 of 91 channels 24% Backplane Links in the 14 Slots Dual-Dual Star Shelf 1000BASE-BX RocketIO To Shelf Manager 1000BASE-T

XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser Derived Requirements for the Carrier Card and E-Keying One 1000BASE-T ethernet on base interace Configurable RocketIO connection to any of 14 channels on fabric interface for low latency link – port 2 and 3 in the channel Configurable 1000BASE-BX ethernet on any of 14 channels on fabric interface for DAQ I 2 C connection from carrier board to RTM Other requirements Universal AMC card with optical link to other systems 12 Communication in ATCA-LLRF System LLRF Review, DESY, December 3rd, 2007

XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser Schedule time January FebruaryMarchAprilMayJuneJuly August September OctoberNovemberDecember Carrier ready 1.E-Keying definition for IPMI 2.Purchasing HUB card 3.Ethernet communication 1Gb Ethernet on base interface tests using CPU 1Gb Ethernet on fabric interface using CPU HUB configuration 4. PCIe tests Communication between carrier cards on shelf backplane Communication between AMC and carrier cards 5. Low latency link tests on a carrier cards and on a shelf backplane 6. Control link between carrier card and RTM 7. System integration 5 Carrier ready 6 RTM ready Communication in ATCA-LLRF System LLRF Review, DESY, December 3rd, 2007

XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser Thank You 14 Communication in ATCA-LLRF System LLRF Review, DESY, December 3rd, 2007