Status Report of the PC-Based PXD-DAQ Option Takeo Higuchi (KEK) 1Sep.25,2010PXD-DAQ Workshop

Quick Review of the “Option #3” 2 RocketIO  PCIe card for PXD data RX is one of the key R&D issues in this option.

Rough Sketch of the Card Xilinx FPGA XC5VFX70T-2? Xilinx FPGA XC5VFX70T-2? Clocking Crystal (312MHz) Buffer Optical link x8 PCIe (Gen1) x4 PCIe (Gen2) >6.25Gbps Buffer full indicating signal LVDS/RJ45 AURORA on RocketIO

Status of the Option #3 As the first step for the backup preparation, we are to put an order of a “pre-study” of the RocketIO  PCIe card performance to a company. – We pay mainly for the R&D of the firmware. Purpose of the “pre-study” – Verify that the data transfer speed from the SFP+ through the PCIe server via an FPGA exceeds 6.25Gbps (before the 8b10b encoding) using a prototype RocketIO  PCIe card. – Measure the CPU load of the PCIe server at when it receives the data at 6.25Gbps. 4

Hardware Platform of the Pre-Study EK-V6-ML605-G-J – Virtex-6 FPGA evaluation kit provided by Xilinx. – FPGA = XC6VLX240T. – Capability of PCIe Gen2 (x4). – x2 FMC connectors interfacing to optional daughter card. – Unit price = €2,420. 5

Hardware Platform of the Pre-Study TD-BD-FMC-OPT4BOARD – FMC daughter card with x4 SFP+. – AVAGO’s optical transceivers. – Released on Aug.11,2010 from Japanese company. – Unit price ~€4,400. PCI-express host server 6

Schematic Drawing of the Pre-Study LX240T PCIe FMC ML605 TD-BD-FMC-OPT4BOARD Loopback optical link PCIe server 7

Firmware Platform of the Pre-Study IP core for PCIe TX/RX and DMA controller – “SYPCIE” … product of Japanese IP vendor (SYSTEC) – Capable of PCIe Gen2 (x4) in combination with Virtex-6. – Free of charge (for evaluation). Free-of-charge version runs for up to 120mins. To reuse the IP, cold start of the FPGA is needed. 8

Software Platform of the Pre-Study OS = Linux – The linux distribution will depend on a PCIe device driver provided by SYSTEC. PCIe device driver – SYSTEC’s device driver provided as a compiled object. – Free of charge for performance evaluation. User software – SYSTEC’s evaluation software (free of charge). – Company’s software. 9

Details of the Pre-Study [1] Initialization – Once at the cold start, the PCIe server downloads a data pattern stored in a file to the Virtex-6’s internal memory through the PCIe. Data path in the running mode Virtex-6’s internal memory  FMC  daughter card’s SFP+ (TX-only)  optical cable  loopback  daughter card’s SFP+ (RX-only)  FMC  Virtex-6’s internal memory (for PCIe block buffer)  PCIe on ML605 (out)  PCIe server (in)  TX/RX data comparison  discarded 10 Repeated forever (120min)

Details of the Pre-Study [2] Firmware functionality [1] – Data pattern download 11 # of words in patternPattern FIFO Pattern #n Pattern #(n–1) PCIe address space … Pattern download the PCIe server to the Virtex-6’s internal memory needs not be very fast.

Details of the Pre-Study [3] Firmware functionality [2] – Firmware reset from the PCIe server Initialization of the TX data pointer to the FMC. Initialization of the TX data pointer to the PCIe. Zero clear of the stored data pattern. Zero clear of the buffer for an accidental PCIe block. – Trivially, the firmware interfaces to the RocketIO and PCIe. Even in a case of TX/RX faults on the RocketIO link, no retransmission is performed. No checksum. 12

Details of the Pre-Study [4] Software functionality on the PCIe server – Throughput monitor of the incoming data to the server. – CPU load monitor of the server. – Word-by-word comparison of the downloaded data pattern to Virtex-6 and the incoming data to the server; error rate monitor. 13

Schedule Ordering schedule – Right after we return to Japan (very beginning of Oct), we put an order to a company. R&D schedule – The performance study by the company and by ourselves will be finished by the end of this year (Dec.,2010).  (Hopefully) we can present some report of the option #3’s performance in the option decision meeting expected to be held around the end of this year. 14

Summary We are to put order in early October of the development of RocketIO  PCIe firmware, peripheral affairs, and performance pre-studies. Results of the pre-study by the company and by ourselves are expected to be made by the end of this year, i.e.: before we make option decision. 15