1. QoS Testing between CNAF and Pisa Test results Nov 29 2005

2. Outline Objectives QoS scenarios Best-effort TCP performance results Test on differentiation of outgoing traffic from INFN-Pisa to CNAF Best-effort UDP performance results Future work

3. Objectives Test traffic differentiation techniques on the customer edge routers of the INFN local area networks, for allocation of a minimum guaranteed bandwidth to a number of traffic classes –Traffic classification and marking –Scheduling: Weighted Round Robin (WRR) –No policing (for the moment)

4. Testbed configuration CNAF: –Juniper M10 (dedicated to testing) –GigaEthernet switch Extreme Summit 400 –Two end-nodes (64 bit PCI slot network interface, 1 GEthernet), connected to the Service Challenge GigaEthernet switch –Capacity to/from GARR: 2 Gbit (boundling of two GEthernet interfaces) PISA –Juniper M7 (production router) –Two end-node (64 bit PCI-X slot network interface, 1 GEthernet; 1 Fast-Ethernet interface ) –Capacity to/from GARR: 1 Gbit

5. Traffic Classes 1.User traffic: Bandwidth range: [300, 1000] Mb/s Minimum 30% of link capacity guaranteed in case of congestion TOS Preference codepoint: 000 (best-effort) 2.Service Challenge traffic: Bandwidth range: [700, 1000] Mb/s Minimum 70% of link capacity guaranteed in case of congestion TOS Preference codepoit: 001 (assured-rate) Purposes of traffic differentiation: –Allocation of minimum guaranteed bandwidth to input/output legacy and Service Challenge traffic classes in case of congestion –Fair distribution of link capacity in case of congestion –Possibility to get more bandwidth than the minimum guaranteed in case of spare link capacity

6. Scenario 1: differentiation of outgoing traffic (from Pisa) GARR GARR CNAF INFN Pisa 70% 30% Juniper M10Juniper M7 1 Gb/s 2.0 Gb/s Service Challenge bottleneck Users

7. Scenario 2: differentiation of incoming traffic (to Pisa) GARR GARR CNAF INFN Pisa 70% 30% Juniper M10Juniper M7 1 Gb/s 2.0 Gb/s Service Challenge bottleneck Users

8. Scenario 3: differentiation of outgoing traffic (from CNAF) GARR CNAF Pisa Juniper M10 Torino Legnaro Milano Bari 2.0 Gb/s 20% bottleneck

9. Scenario 4: differentiation of incoming traffic (to CNAF) GARR GARR CNAF Pisa Juniper M10 Torino Legnaro Milano Bari 2.0 Gb/s 1Gb/s 100Mb/s 1Gb/s bottleneck

10. Best-effort TCP performance Initially asymmetric throughput to/from Pisa (probably due to misconfiguration in the PISA MAN), now solved Performance (with network configuration fixed):

11. Best-effort TCP performance: test CNAF INFN Pisa

12. Differentiation of outgoing traffic (from Pisa) GARR GARR CNAF INFN Pisa SC 1 70% Juniper M10Juniper M7 1 Gb/s 2.0 Gb/s Service Challenge bottleneck Users 30% SC 2 BE1 BE2 BE1 AR

13. Test on differentiation of outgoing traffic (1/2) N° test Sender (command - line) [Mb/s] Riceiver [Mb/s] Protocol Dropped Packets Percentuage 1 BE1 = 90 BE2 = 250 AR = 750 BE1 = 72 BE2 = 198 AR = 670 UDP BE1 = 8 BE2 = 21 AR = 71 2 BE1 = 90 BE2 = 280 AR = 700 BE1 = 67 BE2 = 208 AR = 663 UDP BE1 = 7 BE2 = 22 AR = 71 3 BE1 AR TCP × BE1 = 10,5 AR = 89,5 4 BE1 = BE2 = 280 AR = BE1 = 68,5 BE2 = 280 AR = 565 TCP UDP TCP × BE1 = 7,5 BE2 = 30,7 AR =61,8

14. N° test Sender (command - line) [Mb/s] Riceiver [Mb/s] Protocol Dropped Packets Percentuage 5 BE1 = 90 BE2 = AR = 750 BE1 = 90,5 BE2 = 47 AR = 783 UDP TCP UDP × BE1 = 9,9 BE2 = 5,1 AR = 85 6 BE1 = 90 BE2 = 250 AR = BE1 = 90,4 BE2 = 250 AR = 583 UDP TCP × BE1 = 10,6 BE2 = 27 AR = 62,4 7 BE1 = BE2 = AR = 750 BE1 = 76,3 BE2 = 68 AR = 784 TCP UDP × BE1 = 9,7 BE2 = 7,2 AR = 83,1 8 BE1 = 80 BE2 = 250 AR = BE1 = 80 BE2 = 250 AR = 600 UDP TCP (*20) × BE1 = 8,4 BE2 = 26,9 AR = 64,5 9 BE1 = 80 BE2 = 250 AR = BE1 = 80 BE2 = 250 AR = 600 UDP TCP (*100) × BE1 = 8,4 BE2 = 26,9 AR = 64,5 Test on differentiation of outgoing traffic (2/2)

15. Test Marker TrafficTos binaryTos hexadecimal Best-effort000(0x00) Assured-rate0010x20 [datatag1test] /home/bencivenni > tcpdump -c 20 host 212.189.152.177 -i eth0 tcpdump: listening on eth0 11:18:33.254874 192.135.23.200.20000 > qos1.pi.infn.it.33233:. ack 2468187583 win 18432 (DF) 11:18:33.254878 qos1.pi.infn.it.33233 > 192.135.23.200.20000:. 1:1461(1460) ack 0 win 22 (DF) [tos 0x20]

16. Best-effort UDP performance (1/3) Performance issues in the LAN CNAF, probably due to the NIC hardware used

17. Best-effort UDP performance 1/3: traffic profile Purpose of test: verify the inpact of production traffic on the UDP constant bit rate traffic profile generated at the source (constant inter-packet gap) Production traffic mixing with test traffic on most of the end-to-end path under test

18. Best-effort UDP performance 3/4

19. Differences between CNAF and INFN Pisa PISA Operating System: Linux 2.4.21- 32.0.1.ELsmp iperf version 1.7.0 MTU = 1500 Byte BUS PCI CNAF Operating System: Linux 2.4.21- 20.EL.cernsmp iperf version 2.0.1 MTU = 5000 Byte BUS PCI-X

20. Xeon GARR Xeon optero n datatag1test test7200a nettetst1 nettest2 192.135.23.200 Eth0 1Gb/s 64 bit 192.135.23.201 Eth0 1Gb/s 64 bit 192.135.23.202 Eth1 1Gb/s 64bit 192.168.1.11 Eth1 1Gb/s 32 bit 192.168.1.14 Eth1 1Gb/s 32 bit Eth2 10Gb/s 64 bit/133 Mhz 192.168.1.101 192.168.1.102 Eth2 10Gb/s 64 bit/133 Mhz Switch Extreme Summit 400-48pt Jumbo frames 9216 Switch Extreme Summit 400-48pt Jumbo frames 9216 Router Juniper M10 131.154.99.32 131.154.99.201 131.154.100.1 131.154.100.2 Test SC 2 Gb/s Balanced Boundling SC Eth2 Eth0 Athlon Router Juniper M7 INFN Pisa 212.189.152.177 Eth0 1gb/s 64 bit/133 Mhz 1 Gb/s

21. Iperf: Command line option -l (iperf_len): The length of buffers to read or write. Iperf works by writing an array of len bytes a number of times. Default is 8 KB for TCP, 1470 bytes for UDP. Note for UDP, this is the datagram size and needs to be lowered when using IPv6 addressing to 1450 or less to avoid fragmentation.

22. UDP Test : Pisa CNAF Sender (command line) (Mb/s) -l (Byte) Sender (real) (Mb/s) Riceiver (Mb/s) -l (Byte) Packet loss 800×840 470- 819 ×20% 900×905 422- 800 ×16% 8001500800 15000% 900169090190016900%* 900169090190020000%

23. UDP Test : CNAF Pisa Sender (command line) (Mb/s) -l (Byte) Sender (real) (Mb/s) Riceiver (Mb/s) -l (Byte) Packet loss 800× 463- 805 449- 788 ×2,2% 900× 500- 785 482- 780 ×2,2% 8001500800×1500× 9001690901×1690× 9001690901×2000×

24. Sender (command line) (Mb/s) -l (Byte) Sender (real) (Mb/s) Riceiver (Mb/s) -l (Byte) Packet loss 800× 466- 696 460- 648 ×0,2% 900× 509- 739 503- 718 ×16% 8001500800670150014% 9001690901702169022% 9001690901702200022% UDP Test : LAN CNAF

25. Router Configuration: Class of Service (1/2) class-of-service { classifiers { # Definisco traffico best-effort tutto ciò inet-precedence inet-precedence-service-challenge{ che entra con un TOS = 000 # forwarding-class best-effort { loss-priority high code-points 000; } forwarding-classes { # Definisco le 2 code # queue 0 best-effort; queue 2 ip-premium; } interfaces { ge-y/y/y { scheduler-map service-challenge-ar-be; unit 0 { rewrite-rules { # Allinterfaccia duscita marco i pacchetti del inet-precedence ar-mark; traffico assured-rate con il TOS opportuno } che verrà poi specificato nelle rewrite-rules # }............ # Associo alle interfacce fastethrnet il traffico best-effort #

26. Router Configuration: Class of Service (2/2) rewrite-rules { inet-precedence ar-mark { forwarding-class assured-rate { loss-priority low code-point 001; # TOS = 001 Assured forwarding # } scheduler-maps { service-challenge-ar-be { forwarding-class assured-rate scheduler sch-assured-rate; forwarding-class best-effort scheduler sch-best-effort; } schedulers { # Servo per il 70% del tempo la coda del sch-assured-rate traffico assurde-srate e per il 30% quella transmit-rate percent 70; del traffico best-effort # buffer-size percent 70; priority high; } sch-best-effort { transmit-rate percent 30; buffer-size percent 30; priority low;

27. Router Configuration: Firewall firewall { filter assured-rate-traffic { #Etichetto come assured-rate- term term-192.135.23.200{ traffic il traffico generato da from { 212.189.152.177 e destinato source-address { a 192.135.23.200 # 212.189.152.177/32; } destination-address { 192.135.23.200/32; } protocol [ udp tcp ]; } then { accept; forwarding-class assured-rate; } term term-192.135.23.201{ #Etichetto come best-effort- from { traffic il traffico generato da source-address { 212.189.152.177 e destinato 212.189.152.177/32; a 192.135.23.201 # } destination-address { 192.135.23.201/32; } protocol [ udp tcp ]; then { accept; forwarding-class best-effort; } term default-action { # Tutto ciò che non è stato specificato then { precedentemente viene accettato e accept; classificato come traffico best forwarding-class best-effort; effort # }