1. CHEP 2010, 18-22 October 2010, Taipei, Taiwan 1 18 th International Conference on Computing in High Energy and Nuclear Physics This research project has been supported by a Marie Curie Early Initial Training Network Fellowship of the European Community’s Seventh Framework Programme under contract number (PITN-GA-2008-211801-ACEOLE) Studies of future readout links for the CMS experiment Sebastian Bukowiec on behalf of the CMS DAQ group

2. CHEP 2010, 18-22 October 2010, Taipei, Taiwan 2 18 th International Conference on Computing in High Energy and Nuclear Physics CMS trigger and DAQ design Current readout link: SLINK64 Motivation for upgrade Difficulties for upgrade Possible future architectures Hardware prototype under development Ongoing studies Reliable Data Protocol - RDP Summary Outline

3. CHEP 2010, 18-22 October 2010, Taipei, Taiwan 3 18 th International Conference on Computing in High Energy and Nuclear Physics 700 sub-detector specific Front Ends CMS event builder system High Level Trigger 700 sub-detector specific Front Ends CMS event builder system High Level Trigger Filter farm CMS trigger and DAQ design Subject of the studies

4. CHEP 2010, 18-22 October 2010, Taipei, Taiwan 4 18 th International Conference on Computing in High Energy and Nuclear Physics Current readout link: SLINK64 FED SLINK64 Front-End DriverFront-End Readout Link Readout Unit RU FED – custom sub-detector “Front-End Driver” SLINK64 : protocol for the interface FED – Readout link Readout Link Implementation: a custom developed Common Mezzanine Card (CMC), directly plugged into the sub-detector FED Copper LVDS 400 Mbytes/sec 3.2 Gbit/s effectively FRL (Front End Readout Link): Custom Compact PCI card First stage of the CMS event builder Data integrity check Generation of fixed size data packet (avoid memory fragmentation at receiving PC) Possibility of merging data from 2 links Frontend Readout Links CMS Event Builder

5. CHEP 2010, 18-22 October 2010, Taipei, Taiwan 5 18 th International Conference on Computing in High Energy and Nuclear Physics Better performance Currently SLINK64 operates on 400 MB/s The first stage of the event builder operates on 200 MB/s Non standard protocols Dependence from one vendor (Myrinet Network Technology) Aging hardware Motivation for upgrade

6. CHEP 2010, 18-22 October 2010, Taipei, Taiwan 6 18 th International Conference on Computing in High Energy and Nuclear Physics The foreseen upgrade for some sub-detectors will imply hardware modifications. The new readout links have to be compatible with existing and foreseen front ends. Difficulties for upgrade

7. CHEP 2010, 18-22 October 2010, Taipei, Taiwan 7 18 th International Conference on Computing in High Energy and Nuclear Physics Possible future architecture #1 FED OPT. LINK Front-End Driver Readout Unit RU FED – custom sub-detector “Front-End Driver” Readout Link Implementation: a custom developed Common Mezzanine Card (CMC), directly plugged into the sub-detector FED (like in case of SLINK64) Optical fibre 1280 Mbytes/sec 10 Gbit/s effectively CMS Event Builder

8. CHEP 2010, 18-22 October 2010, Taipei, Taiwan 8 18 th International Conference on Computing in High Energy and Nuclear Physics Possible future architecture #2 μTCA crate with FEDs and DAQ readout card Readout Unit RU FED – custom sub-detector “Front-End Driver” Readout Link Implementation: a custom developed μ TCA board Optical fibre 1280 Mbytes/sec 10 Gbit/s effectively CMS Event Builder

9. CHEP 2010, 18-22 October 2010, Taipei, Taiwan 9 18 th International Conference on Computing in High Energy and Nuclear Physics Ongoing studies - Sender and Receiver Commercial Switch 2 1 RDP 1 2 FED 1. The link designed in a way that it can be interfaced to legacy FEDs of sub-detectors which do not need to be upgraded. 2. The link interfaced to newly designed FEDs which will be able to transfer substantially higher data volumes to the DAQ. 1. Commercial hardware (PCs) for point-to-point connections. 2. Commercial hardware (network switches) for multi-destination data transfer capability.

10. CHEP 2010, 18-22 October 2010, Taipei, Taiwan 10 18 th International Conference on Computing in High Energy and Nuclear Physics FPGA DATA Memory Control Memory DATA Opt. 6 Gb/s Data 8 pairs Ongoing studies: evaluation board diagram of essential components (PCIe-link) to study protocol RDP 8 pairs

11. CHEP 2010, 18-22 October 2010, Taipei, Taiwan 11 18 th International Conference on Computing in High Energy and Nuclear Physics Hardware Prototype under development FPGA: ALTERA Stratix IVGT (up to 24 links @11 Gb/s) Stratix V (links @ 28 Gb/s) CONTROL MEMORY (maintains MMU/RDP tables) SRAM QDRII 200 MHz ~2Mbytes (def. by data memory) PROCESSOR with 1 Gb/sethernet PowerQuiccII – FreeScale (Motorola) 400 MHz 32 bit MEMORY Processor SDRAM DDR2 133Mhz DATA Memory SRAM QDRII 200MHz 32 Mbytes SDRAM DDRII 200 MHz 256 Mbyte CONNECTOR SLINK64 μTCA

12. CHEP 2010, 18-22 October 2010, Taipei, Taiwan 12 18 th International Conference on Computing in High Energy and Nuclear Physics Protocol – Reliable Data Protocol - RDP The Reliable Data Protocol (RDP) is designed to provide a reliable data transport service for packet-based applications. Datagram protocol Layered on the Internet Protocol (IP) Simple to implement Reliable message services Sequence control Acknowledgment Frame data Frame header IP data RDP data Data Frame header IP header RDP header RDP is defined by IETF (http://www.ietf.org ) in RFC 908 and RFC 1151.

13. CHEP 2010, 18-22 October 2010, Taipei, Taiwan 13 18 th International Conference on Computing in High Energy and Nuclear Physics Why RDP? Reliable protocol Datagram protocol No packets splitting as it could happen using TCP The sequence number is a packet number not a byte number (TCP) Implementing datagram protocol is easier on hardware side It is specified as one of the encapsulated protocols in IP

14. CHEP 2010, 18-22 October 2010, Taipei, Taiwan 14 18 th International Conference on Computing in High Energy and Nuclear Physics RDP - Implementation The prototype of the receiver software was developed on the Linux platform using RAW socket interface. Socket(PF_PACKET, SOCK_RAW, ETH_P_IP) domain type protocol PF_PACKET - packet interface on device level SOCK_RAW - RAW socket type ETH_P_IP - listens only for incoming IP packets RAW socket gives the possibility to receive every packet with all the headers (i.e. Ethernet, IP, RDP etc) from the network and process them.

15. CHEP 2010, 18-22 October 2010, Taipei, Taiwan 15 18 th International Conference on Computing in High Energy and Nuclear Physics Protocol – RDP Header SYNSYN SYNSYN ACKACK ACKACK EAKEAK EAKEAK RSTRST RSTRST NULNUL NULNUL 0 0 VER NO. VER NO. HEADER LENGTH SOURCE PORT DESTINATION PORT DATA LENGTH SEQUENCE NUMBER ACKNOWLEDGEMENT NUMBER VARIABLE HEADER AREA 15 7 0 1234512345 67896789 CHECKSUM 10 Bit#Bit Name Description 0 SYN Establish connection and synchronize sequence numbers. 1 ACK Acknowledge field significant. 2 EACK Non-cumulative (Extended) acknowledgement. 3 RST Reset the connection. 4 NUL This is a null (zero data length) segment. 5 Unused. RDP uses the 16-bit TCP checksum, which is specified on page 16 of RFC-793. This area is used to transmit parameters for the SYN and EACK segments.

16. CHEP 2010, 18-22 October 2010, Taipei, Taiwan 16 18 th International Conference on Computing in High Energy and Nuclear Physics Summary The first version of the firmware of the evaluation board was created ( 1Gb/s ). It provides main functionalities like: sending and acknowledging the packets. The beta version of the software which implements the RDP protocol was developed in order to receive and acknowledge packets coming from the evaluation board.

17. CHEP 2010, 18-22 October 2010, Taipei, Taiwan 17 18 th International Conference on Computing in High Energy and Nuclear Physics Thank you for your attention!