1. LNL CMS G. MaronCPT Week CERN, 23 April 2001 1 Legnaro Event Builder Prototypes Luciano Berti, Gaetano Maron Luciano Berti, Gaetano Maron INFN – Laboratori Nazionali di Legnaro

2. LNL CMS G. MaronCPT Week CERN, 23 April 2001 2 GE Event Builder Test conditions: No command or event aggregation (each packet transports a command or data frame relative to a single event) No command or event aggregation (each packet transports a command or data frame relative to a single event) full data transfer from/to PC memory full data transfer from/to PC memory recovery from packet loss recovery from packet loss fixed fragment sizes are varied 400-4000 bytes fixed fragment sizes are varied 400-4000 bytes Components:Hardware: switch: FoundryNet FastIron switch: FoundryNet FastIron NIC: SysKonnect SK9821 NIC: SysKonnect SK9821 PC: Supermicro PIII (i840) PC: Supermicro PIII (i840) Software: vxWorks based 15 x 15

3. LNL CMS G. MaronCPT Week CERN, 23 April 2001 3 Event builder layout 1 1514 1312 1110 9 8 7 6 543 2 1 1514 1312 1110 9 8 7 6 543 2 EVM RUs BUs Slot 1 Slot 2 Slot 3 Slot 4 RUs and BUs distributed in all switch slots: – Part of the traffic localized within the slot – Reduces switch backplane utilization RU performance problem found with problem found with this configuration this configuration

4. LNL CMS G. MaronCPT Week CERN, 23 April 2001 4 Modified Event Builder layout 1 1514 1312 1110 9 8 7 6 543 2 1 1514 1312 1110 9 8 7 6 543 2 EVM RUs BUs Slot 1 Slot 2 Slot 3 Slot 4 Request data commands Fast Ethernet Slot Request data commands - RU fast control message over FE (PCI 32/33) - RU data transfer on GE (PCI 64/66)

5. LNL CMS G. MaronCPT Week CERN, 23 April 2001 5 The GE Event Builder

6. LNL CMS G. MaronCPT Week CERN, 23 April 2001 6 EB protocol BU allocate EVM confirm 1 23n RUs send cache

7. LNL CMS G. MaronCPT Week CERN, 23 April 2001 7 Concurrent building threads in the same BU BU EVM 1 23n RUs BU thread 1 BU thread 2 BU thread 3

8. LNL CMS G. MaronCPT Week CERN, 23 April 2001 8 BU allocate EVM confirm Sequential reading 1 23n RUs BU allocate EVM confirm Random reading 1 23n RUs 45 send cache send cache Sequential vs Random reading

9. LNL CMS G. MaronCPT Week CERN, 23 April 2001 9 BU allocate EVM confirm 1 23n RUs 45 send cache “Sliding Window” multiple send to Rusmultiple send to Rus reduce the total rebuilding timereduce the total rebuilding time less events in the Busless events in the Bus not yet testednot yet tested

10. LNL CMS G. MaronCPT Week CERN, 23 April 2001 10 No difference on performanceNo difference on performance But more allocated event needed on BUs,But more allocated event needed on BUs, All the measurements with random readingAll the measurements with random reading Sequential reading Random reading Sequential - random reading comparison

11. LNL CMS G. MaronCPT Week CERN, 23 April 2001 11 Recovery from Packets loss BU Req. Data RU timer timer start EvtData Req. Data ( retry ) start cancel timeout EvtData BU Req. EvtId EVM timer timer start EvtId Req. EvtId (retry) start cancel timeout EvtId BU – EVM communication BU – RU communication Timeouts 80 - 160 ms

12. LNL CMS G. MaronCPT Week CERN, 23 April 2001 12 EVB 15x15 performance - Throughput Throughput up to 116 MB/s, ie 93% link speed Throughput up to 116 MB/s, ie 93% link speed no packet loss observed (as expected) no packet loss observed (as expected) 15 x 15

13. LNL CMS G. MaronCPT Week CERN, 23 April 2001 13 EVB Scaling

14. LNL CMS G. MaronCPT Week CERN, 23 April 2001 14 EVB Performance – Event Rate Nominal fragment size 2kbytes: event rate = 52 kHz 15 x 15

15. LNL CMS G. MaronCPT Week CERN, 23 April 2001 15 Conic Event Builder Event Manager Manager Builder Network RU BU FU FU FU FU Event Manager Manager RU FUFUFUFUFUFUFUFU Builder Network FUFUFUFU symmetric EVB conic EVB faster ports at Rus faster ports at Rus slower ports at BUs slower ports at BUs

16. LNL CMS G. MaronCPT Week CERN, 23 April 2001 16 Conic Event Builder: Layout 143 2 1 2927 2523 2119 17 15 13 11 975 3 EVM RUs FUs GE Slot 1 Request Data Command 3937 3533 31 2 3028 2624 2220 18 16 14 12 1086 4 4038 3634 32 FE Slot 1 FE Slot 2

17. LNL CMS G. MaronCPT Week CERN, 23 April 2001 17 EVB throughput – Conic vs Symmetric conic EVB: no performance degradation vs symmetric

18. LNL CMS G. MaronCPT Week CERN, 23 April 2001 18 EVB Conic – Scaling

19. LNL CMS G. MaronCPT Week CERN, 23 April 2001 19 Conic: RU/FU Throughput ratio

20. LNL CMS G. MaronCPT Week CERN, 23 April 2001 20 To be done and test variable size eventsvariable size events EB performances with the new implemented “ window” mechanismEB performances with the new implemented “ window” mechanism latency times measurementslatency times measurements Fault generation with the new implemented Random Error Generator to check the error recovery procedureFault generation with the new implemented Random Error Generator to check the error recovery procedure

21. LNL CMS G. MaronCPT Week CERN, 23 April 2001 21 Multistage Event Builder All our results have been obtained with a single switch event builder configurationAll our results have been obtained with a single switch event builder configuration We propose to extend our tests to a multistage ethernet switches topology and to study the behavior of this configuration.We propose to extend our tests to a multistage ethernet switches topology and to study the behavior of this configuration.

22. LNL CMS G. MaronCPT Week CERN, 23 April 2001 22 Plain Topology In the Event Builder application data flows in only one directionIn the Event Builder application data flows in only one direction The inter-switch Gigabit Ethernet links are full-duplexThe inter-switch Gigabit Ethernet links are full-duplex Result : half of the inter-switch bandwidth available is wastedResult : half of the inter-switch bandwidth available is wasted RUs BUs

23. LNL CMS G. MaronCPT Week CERN, 23 April 2001 23 Full Mesh Topology Full Mesh Topology RU and BU distributed in all the switchesRU and BU distributed in all the switches Inter-switch links are used in both directionInter-switch links are used in both direction Same number of ports of the plain topologySame number of ports of the plain topology Twice of the bandwidth of the plain topology in the inter-switch linksTwice of the bandwidth of the plain topology in the inter-switch links RUs BUs

24. LNL CMS G. MaronCPT Week CERN, 23 April 2001 24 Plain and mesh topology limits Plain and mesh topology limits Each couple of switches is connected by a single linkEach couple of switches is connected by a single link This is a bottleneck if the traffic is not uniformly randomThis is a bottleneck if the traffic is not uniformly random The network is blocking for certain traffic patternsThe network is blocking for certain traffic patterns

25. LNL CMS G. MaronCPT Week CERN, 23 April 2001 25 Traffic with patterns If traffic has patterns (for example this could happens in the case the event builder is performed in steps) it could make sense to introduce an artificial mechanism that randomise the traffic.If traffic has patterns (for example this could happens in the case the event builder is performed in steps) it could make sense to introduce an artificial mechanism that randomise the traffic. This mechanism exist and it is called Universal RoutingThis mechanism exist and it is called Universal Routing

26. LNL CMS G. MaronCPT Week CERN, 23 April 2001 26 Universal Routing Reference Discovered by L.G. Valiant in 1980Discovered by L.G. Valiant in 1980 See: M.D May, P.W. Thompson, P.H. Welch NETWORKS,ROUTERS & TRASPUTER available on : http://www.pact.srf.ac.uk/macrame/papers/bluebook.htmlSee: M.D May, P.W. Thompson, P.H. Welch NETWORKS,ROUTERS & TRASPUTER available on : http://www.pact.srf.ac.uk/macrame/papers/bluebook.html Those papers describe the Universal Routing applied to Transputers Networks, a wormhole routing based networkThose papers describe the Universal Routing applied to Transputers Networks, a wormhole routing based network We adapted the same concept to a packet switched network like Gigabit EthernetWe adapted the same concept to a packet switched network like Gigabit Ethernet

27. LNL CMS G. MaronCPT Week CERN, 23 April 2001 27 Universal Routing with GigaEthernet Based on Clos topologyBased on Clos topology Multiple path available between each couple of switchesMultiple path available between each couple of switches Every packets is sent to a randomly chosen intermediate switchEvery packets is sent to a randomly chosen intermediate switch The intermediate switch send the packet to the final destinationThe intermediate switch send the packet to the final destination Full bandwidth between each couple of switches and uniform buffer utilizationFull bandwidth between each couple of switches and uniform buffer utilization

28. LNL CMS G. MaronCPT Week CERN, 23 April 2001 28 Universal Routing Transformation of the CLOS topology to a folded CLOSTransformation of the CLOS topology to a folded CLOS The resulting number of ports is the same of the plain topologyThe resulting number of ports is the same of the plain topology Half duplex links Full duplex links RUs BUs RUs BUs

29. LNL CMS G. MaronCPT Week CERN, 23 April 2001 29 Large (500x500) multistage GE network (1) 1 1 2 25 3 2 20 BUs 20 4 20 RUs 20 Ports 25 Ports 40 Ports 25 switches with 60 x 1Gb ports25 switches with 60 x 1Gb ports 20 switches with 25 x 1 Gb ports20 switches with 25 x 1 Gb ports

30. LNL CMS G. MaronCPT Week CERN, 23 April 2001 30 Large (500x500) multistage GE network (2) 25 switches with 40 x 1Gb ports + 2 x 10 Gb uplinks25 switches with 40 x 1Gb ports + 2 x 10 Gb uplinks 2 switches with 25 x 10 Gb ports2 switches with 25 x 10 Gb ports 1 1 2 25 2 20 BUs 20 RUs 2 Ports 10G 25 Ports 10G 40 Ports

31. LNL CMS G. MaronCPT Week CERN, 23 April 2001 31 Proposal for a multistage event builder demonstrator Multistage event builders can be emulated using the much cheaper fast ethernet connections and switches. The GE speed is not needed in these topological investigationsMultistage event builders can be emulated using the much cheaper fast ethernet connections and switches. The GE speed is not needed in these topological investigations The proposal is to have prototypes for:The proposal is to have prototypes for: –Full Mesh Topology –Folded CLOS topology with (and without) Universal Routing mechanism

32. LNL CMS G. MaronCPT Week CERN, 23 April 2001 32 Full Mesh 64x64 Event Builder Prototype 1 5 7 3 2 6 8 4 8 RUs 8 BUs - 1 host node 4 Rus / 4 BUs or a mix of them a mix of them - 32 hosts - 128 FE NICS (56 + 72) - 8 24 FE ports switch Missing components

33. LNL CMS G. MaronCPT Week CERN, 23 April 2001 33 Folded CLOS 64x64 Event Builder Prototype 1 1 2 4 3 2 16 BUs 16 4 16 RUs 16 Ports 4 Ports 32 Ports 3 FastIron with 3 24 FE ports mods - 1 host node 4 Rus / 4 BUs or a mix of them a mix of them - 32 hosts - 128 FE NICS (56 + 72) - 4 48 FE ports switch - 2 24 FE ports FastIron module + 1 24 FE ports FastIron module Missing components - 1 host node 3 Rus / 3 BUs or a mix of them a mix of them - 32 hosts - 96 FE NICS (56 + 40) - 4 36 FE ports switch - 2 24 FE ports FastIron module 64x64 48x48

34. LNL CMS G. MaronCPT Week CERN, 23 April 2001 34 Folded CLOS 80x80 Event Builder Prototype 1 1 2 4 20 BUs 20 RUs 2 GE Ports 4 Ports 40 FE Ports 3 FastIron with 8 GE ports module (1000 BaseT or 1000 BaseSX) - 1 host node 4 Rus / 4 BUs or a mix of them a mix of them - 40 hosts (32+8) - 160 FE NICS (56 +104) - 4 48 FE ports + 2 GE links switch - 1 8 GE (Base SX) ports FastIron module Missing components 80x80 2

35. LNL CMS G. MaronCPT Week CERN, 23 April 2001 35 8x80 Conic Event Builder Prototype 1 2 4 20 FUs 2 GE Ports 20 FE Ports 3 RU1 RU2 RU3 RU4 RU5 RU6 RU7 RU8 FastIron with 2 8 GE ports modules (1000 BaseT or 1000 BaseSX) - 1 host node 4 FUs - 20 hosts - 80 FE NICS (56 + 24) - 4 24 (48) FE ports + 2 GE up links switch - 2 8 GE (Base SX) ports FastIron module Missing components 8x80

36. LNL CMS G. MaronCPT Week CERN, 23 April 2001 36 Material for the event builder multistage prototypes Mesh 64x64 - 72 FE NICs - 8 24 FE ports Folded CLOS 80x80 - 8 PCs - 104 FE NICS - 4 48 FE ports with 2 GE uplinks 1000 baseT if the 1000 baseT uplinks are not available: 1) Folded CLOS 64x64: 72 FE NICs 4 48 FE ports switch 1 24 FE ports FastIron module 2) Folded CLOS 48x48: 40 FE NICs 4 >36 FE ports switch