1. 5th April, 2005JEM FDR1 JEM FDR: Design and Implementation JEP system requirements Architecture Modularity Data Formats Data Flow Challenges : Latency Connectivity, high-speed data paths JEM revisions JEM 1.1 - implementation details Daughter modules Energy sum algorithms FPGA resource use Performance Production tests

2. 5th April, 2005JEM FDR2 JEP system requirements Process –4.9 < η < 4.9 region ~32×32×2 = 2k trigger towers of Δη×Δφ=.2×.2 9 bit input data (0-511 GeV) 32x32 10-bit “jet elements” after em/had pre-sum 2 multiplications per jet element: E T  (E X,E Y ) 3 Adder trees spanning the JEP (JEMs, CMMs) Sliding window jet algorithm, variable window size within 3×3 environment Output data to CTP Thresholded E T, E T Jet hit count Output data to RODs Intermediate results, mainly captured from module boundaries RoI data for RoIB

3. 5th April, 2005JEM FDR3 JEP system design considerations Moderate data processing power Tough latency requirements Large amount of signals to be processed  partition into parallel operating modules Algorithm requiring environment to each jet element  high bandwidth inter-module lanes Data concentrator functionality, many  few Severely pin bound design, dominated by input connectivity Modules Processors (FPGAs) Benefit from similarities to cluster processor Common infrastructure (Backplane) Common serial link technology

4. 5th April, 2005JEM FDR4 System modularity Two crates, each processing two quadrants in φ  32 × 8 bins (jet elements) per quad η range split over 8 JEMs  4 × 8 jet elements per JEM Four input processors per JEM Single jet processor per JEM Single sum processor per JEM

5. 5th April, 2005JEM FDR5 Replication of environment elements - system and crate level - JEM has 32 core algorithm cells 4 × 8 jet elements Directly mapped : 4 PPMs (e,h)  1 JEM JEM operates on a total of 77 jet elements including ‘environment’ : 7 × 11 Replication in φ via multiple copies of PPM output data Replication in η via back- plane fan-out

6. 5th April, 2005JEM FDR6 JEM data formats – real-time data JEM Inputs from PPM: Physical layer : LVDS, 10 bits, 12-bit encoded w. start/stop bit D0 odd parity bit D(9:1) 9 bit data, D1 = LSB= 1 GeV Jet elements to jet processor: No parity bit D(9:0) 10 bit data, D0 = LSB= 1 GeV 10 data bits muxed to 5 lines, least significant first Energy sums to sum processor: No parity bit E T (11:0) 12 bit data, D0 = LSB= 1 GeV E X (13:0) 14 bit data, D0 = LSB=.25 GeV E Y (13:0) 14 bit data, D0 = LSB=.25 GeV JEM output to CMM: J(23:0) 8 x 3 bit saturating jet hits sent on bottom port J24 odd parity bit S(23:0) 3 x 8 bit quad-linear encoded energy sums on top port 6 bit energy 2 bit range Resolution 1GEV, 4 GeV, 16 GeV, 64 GeV S24 odd parity bit

7. 5th April, 2005JEM FDR7 JEM data formats - readout Physical layer : 16bits, 20-bit encoded (CIMT, alternating flag bit, fill-frames 1A/1B, HDMP 1022 format) Event separator : Minimum of 1 fill-frame sent after each event worth of data All data streams odd parity protected (serial parity) DAQ readout : 67-long stream per L1A / slice being read out Input data on D(14:0) : 11 bit per channel, nine bit data, 1 bit parity error, 1 bit link error 12 bit Bcnum & 25 bit sum & 25 bit jet hits on D15 RoI readout : 45-long stream per L1A D(1:0) : total of 8ROIs 2 bits location & saturation flag & 8 bits threshold passed D2 : 12 bits Bcnum D(4:3) : used on FCAL JEMs only (forward jets) D(15:5) : always zero

8. 5th April, 2005JEM FDR8 JEM data flow LVDS deserialiser Input processor Jet processor + readout controller To CMM 400 Mbit/s serial data (480 Mbit/s with protocol) 40 MHz parallel 80 Mb/s 40 Mb/s parallel Sum processor + readout controller Link PHYTo CMMLink PHY 640 Mbit/s serial data (800 Mbit/s with protocol) Not synchronous to bunch clock Multiple protocols and data speeds and signaling levels used throughout board Multiplexing up and down takes considerable fraction of latency budget Re-synchronisation of data generally required on each chip and board boundary FiFo buffers Phase adjustment w. firmware-based detection Delay scans 40Mb/s

9. 5th April, 2005JEM FDR9 Challenges : latency & connectivity Latency budget for energy sum processor:18.5 ticks (TDR) Input cables : ~2 ticks CMM : ~ 5 ticks Transmission to CTP <2 ticks  ~ 9.5 ticks available on JEM from cable connector to backplane outputs to CMM Module dimensions imposed by use of common backplane Large module : 9U*40cm Full height of backplane used for data transmission due to high signal count  long high-speed tracks unavoidable  need to use terminated lines throughout  need to properly adjust timing High input count : 88 differential cables

10. 5th April, 2005JEM FDR10 Connectivity : high-density input cabling 24 4-pair cable assemblies arranged in 6 blocks of 4 (2 φ bins × em, had) Same coordinate system now on cables and crate: φ upwards, η left to right (as seen from front) V cable rotated Different cabling for FCAL JEMs  re-map FCAL channels in jet FPGA firmware

11. 5th April, 2005JEM FDR11 Connectivity : details of differential data paths Differential 100Ω termination at sink 400 (480) Mbit/s input data Use de-serialisers compatible to DS92LV1021 (LVDS signal level, not DC-balanced) 88 signals per JEM arriving on shielded parallel pairs Run via long cables (<15m) and short tracks (few cm) Require pre-compensation on transmitting end 640 (800) Mbit/s readout data PECL level  electro-optical translator HDMP1022 protocol, 16-bit mode Use compatible low-power PHY

12. 5th April, 2005JEM FDR12 Connectivity : details of single ended data paths CMOS signals point-to-point 60Ω DCI source termination throughout on all FPGAs 40Mb/s (25ns) at 1.5V, no phase control Energy sum path into sum processor : 40 lines per input processor General control paths At 2.5V : CMM merger signals via backplane (phase adjustment on receiving end) 80Mb/s (12.5ns) at 1.5V : jet elements 7x11x5bit =385 lines into jet processor 2x3x11x5bit=330 lines on backplane from/to adjacent modules Global phase adjustment via TTCrx All signals latched into jet processor on same clock edge

13. 5th April, 2005JEM FDR13 JEM history JEM0.0 built from Dec. 2000 LVDS de-serialiser DS92LV1224 11 input processors covering one phi bin each, Spartan2 Main processor performing jet and energy algorithms, Virtex-E Control FPGA, ROC, HDMP1022 PHY, coaxial output Complete failure due to assembly company JEM 0.x built from Dec. 2003 Minor design correction wrt to JEM0.0 New manufacturer (PCB / assembly ) Fully functional prototype except CAN slow control and FPGA flash configuration TTC interface not to specs due to lack of final TTCrx chip Successfully tested all available functionality

14. 5th April, 2005JEM FDR14 JEM 0 11 input processors Main 88 x DS92LV1224 ROC VME-Interface 2 x HDMP1022 Backplane Conn. TTCrx CAN

15. 5th April, 2005JEM FDR15 JEM history (2) JEM1.0 built in 2003 All processors Virtex-2 Input processors on daughter modules (R,S,T,U) LVDS de-serialiser SCAN921260 (6-channel) 4 input processors covering three phi bins each 1 Jet processor on main board 1 Sum processor on main board 1 Board control CPLD (CC) Readout links (PHY & opto) on daughter module (RM) Flash configurator : system ACE Slow control / CAN : Fujitsu microcontroller Successfully tested algorithms and all interfaces Some tuning required on SystemACE clock CAN not to new specs (L1Calo common design)

16. 5th April, 2005JEM FDR16 History: JEM 1.0 power Jet Sum R S T U VME CC RM ACE CAN Flash TTC JEM1.0 successfully tested Algorithms All interfaces LVDS in FIO inter-module links Merger out Optical readout VME CAN slow control Mainz, RAL slice test, CERN test beam

17. 5th April, 2005JEM FDR17 JEM 1.1 JEM1.1 in production now Identical to JEM 1.0 Additional daughter module: Control Module (CM) CAN VME control Fan-out of configuration lines Expected back from assembly soooon

18. 5th April, 2005JEM FDR18 JEM details –main board 9U*40cm*2mm, bracing bars, ESD strips, shielded b’plane connector 4 signal layers incl. top, bottom, 2*Vcc, 4*GND  total 10 layers Micro vias on top, bottom, buried vias All tracks controlled impedance : controlled / measured by manufacturer Single ended 60Ω Differential 100Ω Point-to-point links only All hand-routed 60Ω DCI source termination on processors (CMOS levels) Power distribution All circuitry supplied by local step-down regulators, fused 10A (estimated maximum consumption < 5A on any supply, 50W tot.) 10A capacity, separate 1.5V regulator for daughter modules Defined ramp-up time (Virtex2 requirement) staged bypass capacitors, low ESR VME buffers scannable 3.3V (DTACK: open drain 3*24mA), short stubs on signal lines, 20-75 mm Vccaux for FPGAs : dedicated quiet 3.3V Merger signals (directly driven by processors) on 2.5V banks FPGA core and inter-processor and inter-module links 1.5V

19. 5th April, 2005JEM FDR19 JEM details –main board (2) Timing TTC signals terminated and buffered (LVPECL, DC) near backplane TTCdec module with PLL and crystal clock automatic backup DESKEW1 bunch clock used as a general purpose clock Low skew buffers (within TTCdec PLL loop) with series terminators DESKEW2 clock used for phase-controlled sampling 80Mb/s jet element data (local & FIO) on jet processor only VME Synchronised to bunch clock Sum processor acts as VME controller Basic pre-configure VME access through CM Readout located on RM (ROCs on sum and jet processor) DCS/CAN located on CM (except PHY - near backplane) Configuration via SystemACE / CF P2P links to keep ringing at bay Multiple configurations, slot dependent choice

20. 5th April, 2005JEM FDR20 JEM details –main board (3) JTAG available on most active components. Separate chains FPGAs (through SystemACE) Non-programmable devices on input daughters TTCdec and Readout Module Buffers Control Module JTAG used for Connectivity tests at manufacturer & MZ CPLD configuration FPGA configuration (ACE)

21. 5th April, 2005JEM FDR21 Input modules 24 LVDS data channels per module 12 layer PCB with micro vias Impedance controlled tracks 60 Ω single ended 100 Ω differential LVDS signals entering via 100Ω differential connector on short tracks (<1cm) Differential termination close to de-serialiser 4 × SCAN921260 6-channel de-serialiser PLL and analogue supply voltage only (3.3V) supplied from backplane Digital supply from step-down regulator on main board Reference clock supplied via FPGA XC2V1500 input processor 1.5V CMOS 60Ω DCI signals to sum and jet processor SMBus device for Vcc and temperature monitoring (new)

22. 5th April, 2005JEM FDR22 Readout Module RM 2 channels, 640 Mb/s 16bit  20 bit CIMT coded, fill-frame FF1, alternating flag bit, as defined in HDMP1022 specs 2xPHY, 2xSFP opto transceiver, so far 2-layer boards High-speed tracks <1cm PHYs tested: HDMP1022 serialiser 2.4W/chip (reference, tested in 16-bit and 20-bit mode) HDMP1032A serialiser 660mW/chip, €27.86 @ 80pc (16-bit) TLK1201A serdes 250mW/chip, < €5.00 @ 80pc, uncoded, requires data formatter firmware in ROC (16-bit, 20-bit) Successfully run off bunch clock Converted to Xtal clock due to unknown jitter situation on ATLAS TTC clock Problems with Xtal clock distribution to ROI PHY (RAL, MZ) RM seems to work with clock linked from DAQ PHY to ROI PHY  Want a local crystal oscillator on RM  Need new iteration of RM (HDMP1032A, TLK1201A)

23. 5th April, 2005JEM FDR23 Control Module CM Combines CAN/DCS, VME pre-configure access and JTAG fanout CAN Controller to L1Calo specs now (common design for all processors, see CMM/CPM Link to main board via SMBus only (Vcc, temperatures) VME CPLD (pinout error corrected) generating DTACK for all accesses within module sub- address range to avoid bus timeout Providing basic access for FPGA configuration via VME configuration reset ACE configuration selection / slot dependent ACE configuration selection via VME Buffers for SystemACE-generated JTAG signals to FPGAs TTCdec parallel initialisation (ID from geographical address)

24. 5th April, 2005JEM FDR24 JEM Cost @ 40pcs Main board, 10 layer PCB 21,393 € Assembly 7,657 € Components 49,725 € Sub-total 78,775 € Input module, 12 layer PCB 11,625 € Assembly 6,500 € Components 43,000 € Sub-total 61,152 € Total 139,927€ +control + readout + SFP

25. 5th April, 2005JEM FDR25 Energy Sum Algorithm In all stages saturate outputs if input is saturated or arithmetic overflow occurs Operate on 40Mb/s data from LVDS de-serialisers : 88 channels per JEM, 9-bit E T data, parity, link error Latch incoming data on bunch clock, 2 samples per tick Select the stable sample under VME control Automatic phase detection in f/w (remove that feature ?) Delay scan (VME) Correct for upstream latency differences, up to 3 ticks (shift register, VME controlled) Send data to readout and spy circuitry Zero data on parity error Apply channel mask Sum up electromagnetic and corresponding hadronic channel to 10-bit jet element Multiplex jet elements to 80Mb/s and send to jet processor and backplane

26. 5th April, 2005JEM FDR26 Energy Sum Algorithm (2) Threshold jet elements and sum to E T (12 bits, 1GeV resolution) Threshold jet elements and multiply (cosφ,sinφ),.25GeV resolution Sum to 2*14 bit (E X,E Y ) missing energy vector Transmit (E X,E Y,E T ) to sum processor Calculate board-level total vector sum Quad-linear encoding to 8 bit each 6-bit value and 2-bit range indicator Resolution 1,4,16,64 GeV, full scale 4 TeV Send 25 bits of data incl. odd parity bit D(24) to backplane

27. 5th April, 2005JEM FDR27 FPGA resources used Fully synchronous designs, I/O Flip-flops used on all data lines Input FPGAs XC2V1500-4FF896C Slice Flip Flops: 27% LUTs: 59% total IOBs 90% Block RAMs: 68% Multipliers 50% GCLKs: 12% DCMs: 12%40.6MHz SUM FPGA XC2V2000-4BF957C Slice Flip Flops: 7% LUTs: 11% total IOBs 83% Block RAMs: 12% GCLKs: 25% DCMs: 12%42.8MHz

28. 5th April, 2005JEM FDR28 Performance All interfaces and the algorithms have been tested on JEM1.0 in Mainz, at the RAL slice test and in the CERN test beam. Problems revealed: SystemACE configuration fails if incoming clock or TCK signal are of insufficient quality : signal distortions confirmed  re-layout of crystal clock and TCK distribution on JEM1.1 At CERN 2 out of 4 PPR channels could not be received error- free : signal distortions confirmed  modifications required on the PPR LCD module Errors observed on ROI readout only recently : problems with on-JEM crystal clock distribution confirmed  re-layout of readout module, use local clock Apart from the above problems all interfaces and the algorithms have shown to work error free in all tests

29. 5th April, 2005JEM FDR29 Test setup Up to 3 JEMs in a 9U crate allowing for FIO tests either direction, along with VMM, TCM, CMM (and CPMs!) Control: Concurrent CPU on VMM or via flat cable External data sources for TTC : TTCvx, TTCvi, TTCex (CERN/RAL) via TCM LVDS : 1 DSS 16-channel (MZ) Several DSS (RAL) LSM (RAL) PPR (CERN) (4 channels) External data sinks for Merger signals : 2 CMMs (RAL) Readout path: Complete ROS (RAL) G-link tester with f/w pattern comparison (MZ)

30. 5th April, 2005JEM FDR30 Test strategies Test the full system including all interfaces and algorithms at moderate statistics. Generally use physics-like test vectors  Requires operation of a ROS and data comparison on a computer. Therefore even in relatively long test runs very low bit error rates would go undetected Test interfaces with firmware-based test adapters and on-JEM diagnostic firmware allowing for real-time detection of pattern errors  These tests will reveal even low-level errors quickly Choice of test patterns - have a look at possible failure mechanisms: FIO data and merger data on backplane source-terminated lines at moderate speed: no signal dispersion expected nor observed 800 Mb/s readout data : due to optical transmission no dispersion expected nor observed LVDS links : the pre-compensation circuitry is required to compensate at a single time constant only, well below a single bit period. At the receiving end a slight overshoot should be observed  no inter-symbol interference expected on neither of the transmission lines. Main source of errors: system noise. Any non-constant data pattern should do. Use binary counter pattern. Useful on serial links: Has long stretches of many ones / many zeroes Has transitions all-one to all-zero Easy to detect errors

31. 5th April, 2005JEM FDR31 System test at RAL (slice test) Setup with 2-stage merging in a single crate: DSS  JEM  crate CMM  system CMM  ROD  ROS  ROD  ROS Comparing readout data against simulation. ROD type : 6U modules Data format : old format (6U module specific) Results (June 2004): Data taken up to 5 slices of JEM DAQ data. Trigger rate up to 60kHz, 4*10 6 events analysed, no errors observed on JEM readout.

32. 5th April, 2005JEM FDR32 Interface tests At RAL: Playback from JEM (ramps) into CMM (parity detection). Merger signals crossing 2/3 of backplane length: no error in 10 13 bits In Mainz: FIO tests 3 JEMs (ramps, pattern comparison on central JEM) : no error in 10 13 bits LVDS input tests Source : DSS, 16 inputs exercised at a time, pattern comparison (ramp) in input module : no error in 10 15 bits Readout link tests : G-link tester with pattern comparison(ramp) no error in 10 14 bits (problems with crystal clock from jet processor)

33. 5th April, 2005JEM FDR33 FIO tests : delay scan All data latched into jet processor on a common clock edge Sweep TTCrx delay setting, 104ps steps Measure data errors on each channel : 10 bits, 5 signal lines Single channel 8ns error free All channels 6.5ns error free

34. 5th April, 2005JEM FDR34 latency Latency Energy path 183ns Jet path 234ns < 9.5BC

35. 5th April, 2005JEM FDR35 CERN test beam Within a wider test setup the following modules were available to generate / analyse JEM ‘test vectors’ based on true calorimeter signals PPR  JEM  CMM  CTP  ROD  ROS ROD type : 6U modules Data received from PPR error free on 2 channels Readout from PPR not possible  could not verify input signal integrity except parity error check Energy sum signal processing verified internally

36. 5th April, 2005JEM FDR36 Test beam results Sum algorithm error-free (see effects of quad- linear encoding) Input data Eem+Ehad Energy sum to CMM

37. 5th April, 2005JEM FDR37 Production tests Boundary scan at manufacturer: high coverage due to large fraction of scannable components  verify connectivity (static test) Standalone tester for input module LVDS inputs, pattern comparison in firmware (high statistics) Standalone tester for readout module, pattern comparison in firmware (high statistics) DCI operation verified w. oscilloscope (drive unterminated 50 Ω cable into scope, record pulse shape)  dynamic test System-level tests in Mainz : 1 crate, 1 JEM supplied with LVDS data at a time, playback and spy facilities used to generate / capture data on board boundaries. FIO delay scan High statistics FIO BER tests, pattern detection in firmware, test full crate at a time with maximum activity on LVDS, VME, readout System-level tests at CERN