1. 1 iTOP Electronics Effort LYNN WOOD PACIFIC NORTHWEST NATIONAL LABORATORY JULY 17, 2013

2. Topics Belle II DAQ System COPPER/FINESSE iTOP DAQ System FINESSE firmware COPPER software Next Steps BASF2 COPPER-III FINESSE redesign 2

3. Belle II DAQ System Goal: unified architecture Smooth transition from Belle I Common readout hardware: “COPPER” Common data link protocol: “Belle2Link” Common readout software framework: “roobasf/basf2” Goal: scalability Luminosity at start of experiment will be several times lower than design luminosity, but trigger rate may be just as high Nominal L1 trigger rate: 20 kHz, design average rate set as 30 kHz 3

4. Brief Belle II DAQ History Belle I had dead-time issues with original LeCroy FastBus-based DAQ system Started in-house design in 2002 COPPER-I demo successful, took latency from 29us down to 3us Replaced all LeCroy systems in Belle I with COPPER-II (2005-08) For Belle-II, expect to use a combination of COPPER-II and redesigned COPPER –III 4

5. COPPER-II 5 COPPER = COmmon Pipelined Platform for Electronics Readout 9U VME board Four detector front-end slots Remote boot (no local filesystem) COPPER-III redesign for Belle II will replace obsolete components, add new CPU and Gigabit Ethernet, fix bugs

6. COPPER-II CPU CPU is Linux-based embedded PC PCI Mezzanine Card = PMC COPPER-II uses 800MHz Pentium-III Issues: 512MB, lack of support COPPER-III will use 1.6 GHz Intel Atom “New” features: VGA, USB, GbE, etc. Issues: power requirements 6

7. COPPER-II FINESSE Interface Interface to specific detectors handled by custom daughtercards FINESSE = “ Front-end INstrumentation Entity for Sub- detector Specific Electronics” Standard interface to COPPER FIFOs for data and local bus for control/status (NOTE: most detectors settled on common HSLB (High Speed Logic Board) FINESSE design for Belle II) 7

8. COPPER-II Other features Global clock and trigger come in on “TTRX” board Signals from “FTSW” (Frontend Timing Switch) board Distributes deskewed clock and trigger to multiple destinations FIFO control/status on dedicated COPPER FPGA VME, secondary Ethernet available, but not commonly used Second PMC slot for expansion (not pictured) 8

9. COPPER-II Data Flow Data read into FINESSE Processed then stored in 1MB FIFOs on local bus DMA on COPPER CPU monitors FIFO status, reads out data when full/etc. COPPER CPU sends data to host PC 9

10. iTOP DAQ for Cosmic/Beam Tests Previous testing (CERN, Fermilab, bench) used custom PCIe board from U. Hawaii for readout DAQ goal for 2012-13 cosmic ray/beam tests: demonstrate COPPER-based readout Requirements: Base hardware: COPPER-II, custom 9U VME crate iTOP-specific FINESSE (hardware, firmware) COPPER-based readout software for iTOP FINESSE 10

11. Trigger sequence: TOF trigger via NIM logic Trigger passed to SCRODs via FTSW Data readout from SCROD by FINESSE/COPPER Trigger data readout by USB daughtercard on COPPER Trigger clear from FINESSE back to NIM bin iTOP DAQ for Cosmic/Beam Tests 11 USB iTOP bar COPPER CPU (PC1) CAMAC crate TOF NIM FTSW Fiber TTL COPPER server (PC2) USB CMD Tx/Rx FINESSE B FINESSE A Ethernet FIFO SCROD Remote boot/ Data Readout TOF Data Readout Trigger Clear Local bus

12. FINESSE Support for Cosmic/Beam Tests U. Hawaii has “DSP_FINESSE” design Up to 4 fiber links to SCRODs Spartan-6 FPGA Two dual-core BlackFin DSPs Only basic example firmware and code – no fiber readout, data handing, etc. Minimal DMA example No existing COPPER drivers, only examples from other FINESSE boards COPPER runs old (2.4), customized version of Linux Effort during 2011-12: wrote COPPER drivers to communicate with FINESSE Added COPPER DMA support for FIFO readout Wrote FINESSE firmware to read data from SCRODs, send to COPPER FIFOs Wrote demonstration code to loop data through DSP before writing to COPPER 12

13. COPPER-II Software for Cosmic/Beam Tests To provide data-taking capabilities for cosmic and beam tests, overall framework was developed in 2012-13 Python-based scripts Speed-critical portions (readout) in C libraries “Experiment” directory structure records all configuration settings, log files, and data for each run Scripts generated for pedestal, pulser, laser, and beam runs Used to debug electronics at KEK in 2013 Used for data-taking at KEK and LEPS Scripts updated during electronics bring-up and during LEPS data-taking to eliminate bugs, better match expected behavior and utilization Multiple staff present at KEK/LEPS almost continuously in Jan-Jun 2013 13

14. Next Steps Conversion to BASF2 readout Belle II readout settling on BASF2 framework on COPPER CPU Plan to use current DAQ software for next beam test(s), but will start porting on BASF2 readout FINESSE redesign Processing required for real-time analysis is not yet clear Unclear that current configuration will work Spartan-6 on SCROD “packed to the gills” Data transfer rate between FPGA and DSP on FINESSE limited to 75MHz Transition to COPPER-III Currently developing with COPPER-II, but iTOP will use COPPER-III COPPER-III boards in short supply (all at KEK) Hardware should be largely “transparent”, but CPU will be running new version of Linux (2.6 kernel) – possible non-trivial driver changes 14

15. Future Processing Requirements Possible changes: Replace FPGA on SCROD (likely to happen anyway) If we replace Spartan-6 with Virtex-7 – enough to handle all processing? Could then use “standard” FINESSE design (HSLB) Replace DSP_FINESSE with new design Some redesign already required (obsolete components) We have started investigating FINESSE redesign Researched different processing units: Vertex-7, Vertex-5 w. PowerPC, Zynq SoC (FPGA + dual-core ARM), quad-core ARM processor Wrote draft design document based on Zynq Kintex-7 FPGA (125-350K logic cells) 0.7-1GHz dual-core ARM Cortex-A9 256KB on-chip memory, 8 DMA controllers High-speed interconnect between CPU and FPGA Starting on benchmarking effort with dev kit 15

16. Data Handling Combine FPGA-based and CPU-based processing FPGA receives data and stores to memory via DMA, notifies CPU CPU manages data, passes commands to FPGA for dedicated processing FPGA writes data to COPPER FIFO CPU can make decisions based on data rates, quality metrics, etc. 16

17. Looking into using existing Zynq “module” instead of full custom design Simplifies layout, memory, etc. Possibility of easy upgrade/downgrade later Challenges: Only a few vendors with larger Zynq on modules Adds peripherals unnecessary for our application Clocking requirements may not match Belle II requirements Cost trade-offs: up-front design cost vs. higher per-unit cost In unofficial discussion with Enclustra SODIMM form factor (68 x 30mm) Xilinx Zynq XC6Z030 512MB SDRAM, 512MB flash Potential Design Simplification 17