1. Sep. 17, 2002BESIII Review Meeting BESIII DAQ System BESIII Review Meeting IHEP · Beijing · China Sep. 16 - 17, 2002

2. Sep. 17, 2002BESIII Review Meeting BESIII DAQ Scheme

3. Sep. 17, 2002BESIII Review Meeting Readout Crate System Controller – MVME2431-1  PowerPC 750™ 32-bit microprocessor  32MB of on-board ECC SDRAM  Up to 1MB capacity for on-board firmware  8MB on-board Flash memory  One asynchronous serial debug port  10/100Mbps Ethernet interface  4-level requester, 7-level interrupter, and 7-level interrupt handler for VMEbus

4. Sep. 17, 2002BESIII Review Meeting 3 Levels of Event Builder

5. Sep. 17, 2002BESIII Review Meeting BESIII DAQ Tasks  Event Data Readout from FEE  Event Building  Event Filtering  Event Recording to Persistent Media  Run Control System  Monitoring (event, histogram display...)  Message Reporting

6. Sep. 17, 2002BESIII Review Meeting Data Rate Estimation  Level-1 Trigger Rate: 4KHz (2KHz Good Events)  Event Size: 12KBytes  Data Rate to be Processed by DAQ: 48MByte/sec  To be Recorded on Disk Array/Tape: 36MByte/sec

7. Sep. 17, 2002BESIII Review Meeting Technologies  Multi-level Buffering  Switch Network  Parallel Computing  Easily Upgrade and Port  Easily Scale  Reliable / Stable / Error Recovery

8. Sep. 17, 2002BESIII Review Meeting Online Data Flow

9. Sep. 17, 2002BESIII Review Meeting PowerPC/VxWorks Based Test Environment

10. Sep. 17, 2002BESIII Review Meeting Test Results – Programmed I/O System Controller: MVME2431-1 (350 MHz PowerPC 750) Operating System: VxWorks 5.4 Writing from CPU DRAM into VMEbus 32-bit transfers: 400 ns/32-bit write, 10MB/s 16-bit transfers: 400 ns/16-bit write, 5MB/s Reading from VMEbus into CPU DRAM 32-bit transfers: 1,100 ns/32-bit read, 3.6MB/s 16-bit transfers: 1,100 ns/16-bit read, 1.8MB/s

11. Sep. 17, 2002BESIII Review Meeting Note:  The above VME access speeds were Programmed I/O speeds.  The program was written in C language and run in the context of the VxWorks shell.  The tests were conducted on an idle system with no other competing processes nor any concurrent activity on the VMEbus.

12. Sep. 17, 2002BESIII Review Meeting Question: Can we use Programmed Read to read out data from ADCs/TDCs modules? Can we use Programmed Read to read out data from ADCs/TDCs modules?

13. Sep. 17, 2002BESIII Review Meeting Estimation of One VME Readout Crate (1) Basic Assumptions:  1 crate = 16 modules  1 module = 32 channels  Trigger rate: 4000 Hz  When one L1 trigger comes, 10% channels have signals.  Each channel has its own Buffer containing 16-bit data to be read.  Each module has a Hit Map (32-bit)  Data readout is in A24/D16 mode and 1.1 microsec/16bit read

14. Sep. 17, 2002BESIII Review Meeting Estimation of One VME Readout Crate (1) (con’t) For each module: Data Ready Flag:1 word Trigger Number:1 word Hit Map:2 word Data:32 × 10% = 3.2 words For each crate: 16 × ( 1 + 1 + 2 + 3.2 ) = 115.2 words 16 × ( 1 + 1 + 2 + 3.2 ) = 115.2 words When trigger rate is 4000 Hz, readout time will be: 4000 × 115.2 bytes × 1.1 microsec = 0.51 sec

15. Sep. 17, 2002BESIII Review Meeting Estimation of One VME Readout Crate (2) Basic Assumptions:  1 crate = 16 modules  1 module = 32 channels  Trigger rate: 4000 Hz  When one L1 trigger comes, 10% channels have signals.  Each module has a Global Buffer containing 16- bit channel number and 16-bit data to be read.  Data readout is in A24/D32 mode and 1.1 microsec/32bit read

16. Sep. 17, 2002BESIII Review Meeting Global Buffer Data Format 31 0 … Event Data Channel Number …… …… Tail … Event Data Channel Number Trigger Number Tail Event Data Channel Number Event Data Channel Number Trigger Number

17. Sep. 17, 2002BESIII Review Meeting Estimation of One VME Readout Crate (2) (con’t) For each module: Data Ready Flag:1 DWord Header:1 DWord Data:32 × 10% = 3.2 DWords Tail:1 DWord For each crate: 16 × ( 1 + 1 + 3.2 + 1 ) = 99.2 DWords 16 × ( 1 + 1 + 3.2 + 1 ) = 99.2 DWords When trigger rate is 4000 Hz, readout time will be: 4000 × 99.2 bytes × 1.1 microsec = 0.44 sec

18. Sep. 17, 2002BESIII Review Meeting So, the answer is:  The system controller does not have enough time to read out data from ADCs/TDCs by using Programmed Read.  The way to get faster reads is to use the DMA engine resident on the Universe VME PCI bridge chip. It is designed to copy larger chunks of data from one bus to the other.

19. Sep. 17, 2002BESIII Review Meeting MVME2431 DMA Controller  The Universe II has a DMA controller for high performance data transfer between the PCI bus and VMEbus.  Transfer size Any Number of bytes from 1 byte to 16 Mbytes. Any Number of bytes from 1 byte to 16 Mbytes.  Transfer Data Width D08 (EO) D08 (EO) D16 D16 D16BLT D16BLT D32 D32 D64 D64 D32BLT D32BLT D64BLT (MBLT) D64BLT (MBLT)

20. Sep. 17, 2002BESIII Review Meeting Readout Mode: Global Buffer + DMA  Block size must be counted somewhere on board.  To avoid too many DMA operations (DMA Interrupts), it’s better for each ADC/TDC board to buffer data of multiple triggers (e.g. 64, 128 triggers), and then interrupt the system controller to start a DMA operation.

21. Sep. 17, 2002BESIII Review Meeting DMA Transfer Some typical values for DMA transfers: D32 Read:20 -- 23 MB/s D32 Write:22 -- 25 MB/s D64 Read:40 -- 45 MB/s D64 Write:45 -- 50 MB/s Note: This is the raw speed of the hardware without any software overhead.

22. Sep. 17, 2002BESIII Review Meeting Main Tasks of Readout  DMA transfer  Calculation  Data unpacking & packing  Network transmission.

23. Sep. 17, 2002BESIII Review Meeting R & D Prototype

24. Sep. 17, 2002BESIII Review Meeting File Server  DELL PowerEdge 2500 Dual Intel Pentium III 1.4 GHz Processors with 512K Cache  Integrated 10/100 Ethernet Adapter  1 GBytes SDRAM  SCSI RAID Controller  Six 18GB SCSI Hard Drive, 10K RPM  Intel Pro 1000 Fiber Network Adapter  Red Hat Linux 7.3 Kernel 2.4.18-3 smp

25. Sep. 17, 2002BESIII Review Meeting Online Farm & Readout PC Nodes  DELL GX260 Intel Pentium 4 1.8 GHz Processor with 256K Cache  Integrated 10/100/1000 Ethernet Adapter  512 MBytes SDRAM  40GB EIDE 7200 RPM Hard Drive  Red Hat Linux 7.3 Kernel 2.4.18-3

26. Sep. 17, 2002BESIII Review Meeting Ethernet Switch  Catalyst 2948G-L3  48 Ports of 10/100-Mbps Fast Ethernet  2 Ports of 1-Gbps Gigabit Ethernet

27. Sep. 17, 2002BESIII Review Meeting Network Performance (Preliminary) Block size Throughput Throughput Throughput(Bytes)(Mbps)(Bytes)(Mbps)(Bytes)(Mbps) 80.47102426.1513107287.69 160.93204836.5126214488.77 321.82409652.5852428889.26 643.54819265.60104857689.52 1286.541638475.76209715289.66 25611.813276881.91419430489.72 51218.616553685.96838860889.76

28. Sep. 17, 2002BESIII Review Meeting More Studies on Prototype  Performance Studies Network performance in parallel mode Network performance in parallel mode Raid disk performance Raid disk performance write/read speed vs. config (raid0/raid5)  Data flow control software  Online filter software  Distributed histogram processing  Single event display

29. Sep. 17, 2002BESIII Review Meeting Thanks!