Eric Hazen1 Ethernet Readout With: E. Kearns, J. Raaf, S.X. Wu, others... Eric Hazen Boston University

Eric Hazen2 Ethernet Readout ● Motivation – Simple, inexpensive solution ● Easily scaled as needed ● Hardware Needed: – Ethernet Daughterboard for ATM – Commercial network switches – Standard computers ● Some thoughts on system architecture

Eric Hazen3 Requirements ● Record all PMT hits ● Distribute data to CPU farm for triggering ● Handle 1.5 MB/s* average rate per ATM ● Full control of ATM (replace TKO access) ● Ethernet Daughter Board Proposed: – 100BaseT Ethernet output (max 5MB/s) – On-board buffering for >1s data – FPGA with SDRAM, Ethernet PHY chip * MB/s = Mbytes/second

Eric Hazen4 Daughter Board on ATM signal input (0~11ch) signal input (12~23ch) QTC chip (X 8) TDC chip (X 4) FPGA TKO board specification Ethernet Daughterboard RJ-45 Ethernet on front panel Power, signals from ATM

Eric Hazen5 Daughterboard Layout TKO Bus Signals Ethernet / LED Signals (to ATM front panel) Power from ATM Board-to-Board Connectors

Eric Hazen6 Requirements from ATM ● The ATM should provide to daughterboard: – Power ● Single voltage – DC/DC converters on DB ● Prefer a “high” voltage, 5V or (8V, 15V) – Clock – 60MHz (jitter < 100ps p-p preferred) – PMT data and Control Path ● Modified TKO protocol – can control ATM – JTAG (for firmware programming) ● Controlled via Ethernet or direct cable

Eric Hazen7 Ethernet Daughterboard Xilinx FPGA Spartan 3 or Virtex 4 SDRAM PMT Data, Control (TKO proto) Clock ATM Connector Power Ethernet PHY MII * * MII = Media Independent Interface (industry standard) RJ BaseT Ethernet JTAG Flash Memor y JTA G Buffer to handle > 1.5 Mbyte (1 sec data) Peak rates: Writing:16-100MB/s (depends on protocol) Reading:10MB/s (Ethernet)

Eric Hazen8 Ethernet FPGA s MicroBlaze CPU core Block RAM (instructions, data) 10/100 Ethernet MAC SDRAM controller FIFO PMT Data from ATM Control SDRAM (ICs or DIMM) Ethernet PHY Flash Control Flash Memory Program Store OPB DMA Controller TKO Master Xilinx core On-chip bus (IBM std) Block RAM Custom logic IOs (pins)

Eric Hazen9 Online Architecture (One Option) HUT 1 HUT 2 HUT 3 HUT 4 Ethernet Switch CPU Ethernet Switch CPU Trigger / Event Builder Processor Farm Storage Manager Permanent Storage

Eric Hazen10 One Hut 10 ATM per Crate (240 ch) 1.5MB/s 100BaseT per ATM (10kHz dark rate) Network Switch (48) 100BaseT ports (1) GbE port 14 crates (ID + OD) (5) GbE to counting house GbE = Gigabit Ethernet Total ~ 20Mb/s (10kHz dark rate)

Eric Hazen11 24 port GbE/ 10GbE Switch Central Switch (One Possible Implementation) (20) GbE Links from Huts 48 port GbE/ 10GbE Switch 10GbE Up to 48 CPUs (can easily expand)

Eric Hazen12 Hit Assignment to CPUs T Q to IT-1 to IT-2 to IT-3 ATM Switch CPU Data stream partitioned into time periods for processing (as proposed by Hayato-San) Each CPU receives packets for it's time period ATM assigns packet destination address based on time stamp Each hit arrives at the correct CPU using standard TCP/IP routing... no intermediate processing needed

Eric Hazen13 Efficiency Considerations ● Packets should be maximum size (1.5k) to use Ethernet efficiently. – Each ATM should only send data when it has accumulated ~250 PMT hits (1ms at 10kHz/PMT) ● UDP vs TCP protocol? Need to study... ● Overlap between time slices > maximum event duration is required (100  s) – Some PMT hits are sent twice

Eric Hazen14 Online System Summary ● “One-Way” Model (described here) – Send data direct from ATM to trigger one CPU in processor “farm” – No intervening computers needed – Ethernet board on ATM assigns a destination I/P address to each packet based on pulse time ● “Event Builder” Model – Send data to event builder and to IT CPUs – Event builder buffers data until trigger

Eric Hazen15 Possible Development Plan 1.Establish DB Specs (preliminary version done) 2.Build prototype daughterboard 3.Write firmware/embedded software for above 4.Build a simple motherboard to hold ~10 daughterboards (simulates on crate of ATMs) 5.Connect via network switch to a cluster of CPUs 6.Write test DAQ software 7.Test DAQ using simulated PMT data This is a significant amount of work. It would be nice to have some collaborators to help, particularly with tasks 4-6.

Eric Hazen16 Summary / Plans ● Establish DB Specifications ASAP – Preliminary spec for Iwatsu this week (Hazen) ● Provide prototype to test with new ATM – April-May 2006 (simple firmware only) ● Full-function prototype in six months ● Could be used for new online system instead of new SCH – Full control of ATM via ethernet

Eric Hazen17 Daughter Board Spec