2. Sequin Technical Summary Mauro Campanella INFN-GARR Mauro.Campanella@garr.it

3. M. Campanella - Sequin Workshop - February 1st, 2002 2 Agenda - Approach to the problem - Users’ requirements - Quantitative definition of QoS - Which QoS service - Premium IP service - Service Level Agreements - Monitoring - Premium IP status - Other QoS services

4. M. Campanella - Sequin Workshop - February 1st, 2002 3 Synergy with - A joint and task force on advanced networking research http://www.dante.net/tf-ngn

5. M. Campanella - Sequin Workshop - February 1st, 2002 4 Approach to the task (end to end QoS across multiple domains) QoS Definition Top - down Qualitative through user’s questionnaire Bottom - up Quantitative, using a minimum and sufficient set of QoS parameters QoS service(s) definition, architecture, testing and implementation in NREN networks

6. M. Campanella - Sequin Workshop - February 1st, 2002 5 Basic components of QoS the applicationthe operating system and the transport protocols the User the network

7. M. Campanella - Sequin Workshop - February 1st, 2002 6 Intuitive definition of QoS The network offers a QoS service when it’s capable of handling selected packets in such a way to fulfill application’s requirements. “some packets are more equal than others…”

8. M. Campanella - Sequin Workshop - February 1st, 2002 7 Agenda - Approach to the problem - Users’ requirements - Quantitative definition of QoS - Which QoS service - Premium IP service - Service Level Agreements - Monitoring - Premium IP status - Other QoS services

9. M. Campanella - Sequin Workshop - February 1st, 2002 8 Users’ Interview A questionnaire has been developed to asses user’s needs of QoS. The questionnaire was articulated in 26 questions in 4 sections: - geography; - qualitative perception of QoS; - quantitative perception of QoS; - network options and expectation. It was sent to 20 groups of pan-European and large users, out of which 11 responded.

10. M. Campanella - Sequin Workshop - February 1st, 2002 9 Interview results outside Europe to which connectivity is needed used

11. M. Campanella - Sequin Workshop - February 1st, 2002 10 Interview results (continued) QoS (QoS need)

12. M. Campanella - Sequin Workshop - February 1st, 2002 11 Interview results (continued) Overall the users showed medium to low knowledge of their QoS needs and QoS techniques, but unanimously requested it, as a way to have a better service from network for their work. Present difficulties are mainly due to congestion. Willingness to pay is proportional to the real benefits, granularity of the service, provisioning time and flexibility and behaviour of Best Effort. Need for simple, fast access to the QoS service.

13. M. Campanella - Sequin Workshop - February 1st, 2002 12 Questionnaire summary

14. M. Campanella - Sequin Workshop - February 1st, 2002 13 Agenda - Approach to the problem - Users’ requirements - Quantitative definition of QoS - Which QoS service - Premium IP service - Service Level Agreements - Monitoring - Premium IP status - Other QoS services

15. M. Campanella - Sequin Workshop - February 1st, 2002 14 QoS parameters From users’ requirements and technical considerations :  - one-way delay;  - IP packet delay variation;  - capacity (rate);  - one-way packet loss. The set is common to IETF and ITU-T. Naming and definitions are chosen to be comply to RFC 2330 (Framework for IP Performance metrics) and follow the ongoing IPPM IETF working group work.

16. M. Campanella - Sequin Workshop - February 1st, 2002 15 QoS parameters sample value ranges

17. M. Campanella - Sequin Workshop - February 1st, 2002 16 QoS parameters (continued) Memento To build a QoS service based on the previous listed parameters, some basic requirements on the network should be fulfilled: - physical and data link stability; - exhibit a Bit Error Rate better than 10 -12 ; - overall network hardware performance. The minimum MTU size should be chosen large enough to avoid fragmentation. Duplicate and out-of-order packets at the physiological level (which is not null, but very small)

18. M. Campanella - Sequin Workshop - February 1st, 2002 17 Agenda - Approach to the problem - Users’ requirements - Quantitative definition of QoS - Which QoS service - Premium IP service - Service Level Agreements - Monitoring - Premium IP status - Other QoS services

19. M. Campanella - Sequin Workshop - February 1st, 2002 18 Which QoS service Start with the “very good” service and call it “Premium IP”: - it satisfies all the users’ requests - it is “the best” achievable service possible - it maps to very high priority scheduling techniques available now - it is similar to a “virtual wire”

20. M. Campanella - Sequin Workshop - February 1st, 2002 19 Agenda - Approach to the problem - Users’ requirements - Quantitative definition of QoS - Which QoS service - Premium IP service - Service Level Agreements - Monitoring - Premium IP status - Other QoS services

21. M. Campanella - Sequin Workshop - February 1st, 2002 20

22. M. Campanella - Sequin Workshop - February 1st, 2002 21 Provision QoS for the European research users in the form of an end to end network service offering the equivalent of a leased line. The service has to be implemented by combining border to border services provided by the NRENs and networks The architecture has to be simple, modular, scalable, adapt to network changes easily, based on IP and independent from the transport technology. The implementation and Service Level Agreements have to match the current status of hardware availability and network topology Premium IP goals

23. M. Campanella - Sequin Workshop - February 1st, 2002 22 Increasing complexity QoS Protocols and Architectures Overview StatelessStateful Stateless Stateful Overprovisioning Diffserv 802.1p RSVP (aggregates) Diffserv - 802.1p RSVP (per flow) Diffserv - 802.1p RSVP - Intserv MPLS -Traffic engineering ATM - ATM signaling

24. M. Campanella - Sequin Workshop - February 1st, 2002 23 Which QoS framework to use ? Differentiated Services - RFC2475 - Integrated Services - RFC 1633 - Overprovisioning  

25. M. Campanella - Sequin Workshop - February 1st, 2002 24 Premium IP Specification   Differentiated Services Architecture and use the expedited forwarding per hop behavior (EF PHB)   interface definition between domains that behaves as an EF PHB   do not starve best effort traffic (limited percentage of link capacity devoted to Premium IP, about 5%)   initial provisioning structure: static, no dynamic signaling   IETF IPPM QoS parameters measurement framework   QoS parameters monitoring system is a key element

26. M. Campanella - Sequin Workshop - February 1st, 2002 25 Premium IP Specification   minimize number of action per node   modular approach that allows different implementation schemes at every hop or domain and allows domain to join the service when readyand do not try to solve the most general problem, but rather develop a model that can be implemented in parallel with the start of GÉANT, using available tools

27. M. Campanella - Sequin Workshop - February 1st, 2002 26 IP v4 IP v6

28. M. Campanella - Sequin Workshop - February 1st, 2002 27 Simplifying the actions for each node - monitoring and accounting- QoS rules propagation - congestion control - admission control and classification - scheduling In principle, each node might perform an awful lot of tasks: - marking- policing - shaping

29. M. Campanella - Sequin Workshop - February 1st, 2002 28 Use the information in the IP - IP source and destination (prefixes) as near to the source as possible - the DSCP (or IP precedence equivalent value) along the path - perform an optional, suggested, admission control based on AS source and destination at inter-domain links (safety measure) - rules might be based on additional parameters, as time-of-day Admission control

30. M. Campanella - Sequin Workshop - February 1st, 2002 29 The consequences are: - allowing the computation of total requested Premium IP capacity at each network node in the default case (and for main backup cases too) - short access list near users’ premise (few users) - simple control at backbones (IP addresses are not propagated) - choosing destination-aware service (next slide) Admission control (continued)

31. M. Campanella - Sequin Workshop - February 1st, 2002 30 Examining the tasks for each node - monitoring and accounting- QoS rules propagation - congestion control - admission control and classification - scheduling In principle, each node might perform an awful lot of tasks: - marking- policing - shaping always

32. M. Campanella - Sequin Workshop - February 1st, 2002 31 - Mark each “EF” legal packet at first classification point - Use the same DSCP value on all domains (Class selector 5 - decimal 40 [RFC 2474] to have interoperability with ToS-only capable hardware) - strongly suggested - - valid DSCP coupled to invalid IP addresses implies discard to allow easy debugging - packets with other DSCP values are left untouched Marking is mandatory at the first classification point, remarking is optional. Marking

33. M. Campanella - Sequin Workshop - February 1st, 2002 32 Examining the tasks for each node - monitoring and accounting- QoS rules propagation - congestion control - admission control and classification - scheduling- marking- policing - shaping always Selected locations

34. M. Campanella - Sequin Workshop - February 1st, 2002 33 Microflow policing should be done as close as possible to the source according to agreed (through SLA) Premium IP capacity. This step is mandatory Policing will be done using a token bucket. The depth of the token bucket will be two MTU close to the source and increase to 5 or more along the path if additional policing is required It is suggested to perform only one additional policing stage at the ingress to GÉANT from an NREN, with a larger aggregated capacity value than the sum of the agreements. “Avoid unwanted packet loss” is the motto. Policing

35. M. Campanella - Sequin Workshop - February 1st, 2002 34 The additional policing stage at the ingress to GÉANT from an NREN serves the purpose of protecting Premium IP traffic from misconfiguration/DoS coming from a single source. It creates virtual “pipes” for the aggregated Premium flows from each NREN to each other (when needed). The failure of one “pipe” does not influence the others. Policing (continued)

36. M. Campanella - Sequin Workshop - February 1st, 2002 35 L1L2 N1 N2 N3 CORE L1, L2 : end user domain (for example LANs) N1, N2, N3 : intermediate transport domains (for example NRENs backbones) CORE : interconnection domain (for example GÉANT) : router/switch Sample multidomain network Classification on IP addresses Strict policing Policing can be avoided at ingress when receiving from a trusted backbone Classify by DSCP Police by (AS source,dest) aggregate capacity on all border nodes Policing not needed

37. M. Campanella - Sequin Workshop - February 1st, 2002 36 Examining the tasks for each node - monitoring and accounting- QoS rules propagation - congestion control - admission control and classification - scheduling- marking- policing - shaping always Selected locations Selected locations

38. M. Campanella - Sequin Workshop - February 1st, 2002 37 Use the highest priority queueing mechanism (PQ or WRR). Limit total Premium capacity when assigning service to users at about 5% of each core link. Assigned Premium capacity can be larger near users’ sites. Scheduling Total link capacity Premium traffic Best Effort traffic Suggested Premium limit This should never happen, but it works..

39. M. Campanella - Sequin Workshop - February 1st, 2002 38 Examining the tasks for each node - monitoring and accounting- QoS rules propagation - congestion control - admission control and classification - scheduling- marking- policing - shaping always Selected locations Selected locations always

40. M. Campanella - Sequin Workshop - February 1st, 2002 39 Shaping The compliance of the Premium user flow to the contracted capacity is the key for the result of the service. Shaping is intended here as limiting the rate of transmission to a specific value. The speed of the core link and the highest priority in scheduling for the packets of the Premium IP service make delay variation small even at aggregation points. At 2.5 Gb/s the transmission time of a 1500 bytes packet is about 5 microseconds. The consideration suggests to start the service without enabling shaping in the core and it shaping may be optional also at the border, provided the sources produce a well shaped flow.

41. M. Campanella - Sequin Workshop - February 1st, 2002 40 Shaping The sending source is hence required to shape the traffic it produces. Shaping inside the sending host itself is the preferred way, shaping by the network will in most case lead to packet losses TCP Application Scheduling IP Network Interface No Packet/Data losses host

42. M. Campanella - Sequin Workshop - February 1st, 2002 41 Examining the tasks for each node - monitoring and accounting- QoS rules propagation - congestion control - admission control and classification - scheduling- marking- policing - shaping always Selected locations Selected locations always NO Done by source not needed Selected locations Selected locations

43. M. Campanella - Sequin Workshop - February 1st, 2002 42 Shape ONLY here Classify (IP pair prefixes) Police - Strict, Capacity Mark Classification (DSCP) and High priority scheduling on all nodes Do not police on egress Do not shape anywhere Policing can be avoided at ingress when receiving from a trusted backbone Police by (AS source,dest) aggregate capacity on all border nodes Premium IP Summary

44. M. Campanella - Sequin Workshop - February 1st, 2002 43 Summary Innovations: - interdomain behaviour specification - end to end service level agreement The architecture allows: - different implementation strategy in each domain - asynchronicity in implementation - sub-domain implementation - explicit rate limitation only near sending source - enabling a user just adding few lines of access control

45. M. Campanella - Sequin Workshop - February 1st, 2002 44 Example (one direction) Domain 1 802.1p VLAN Or dedicated wire Classification (IP) Policing (strict 2 MTU) Marking - scheduling Domain 5 802.1p VLAN Or dedicated wire Domain 2 ATM Dedicated PVC Domain 3 Backbone Classification (DSCP) Policing (AS aggregate) Classification (DSCP) scheduling Domain 4

46. M. Campanella - Sequin Workshop - February 1st, 2002 45 Agenda - Approach to the problem - Users’ requirements - Quantitative definition of QoS - Which QoS service - Premium IP service - Service Level Agreements - Monitoring - Premium IP status - Other QoS services

47. M. Campanella - Sequin Workshop - February 1st, 2002 46 SLA/SLS Basic implementation In the first phase the SLS negotiation will be performed manually (no bandwidth broker). The analytical computation of the QoS metric in a IP based network is extremely complex and the SLA specification will require extensive testing of the available infrastructure. Usually only QoS parameter ranges can be specified and assurances as percentages of total time. There are always two SLA, one for each direction. The contracted values might be different (asymmetric capacity for example)

48. M. Campanella - Sequin Workshop - February 1st, 2002 47 The service must be defined both as an end to end service level agreement and be accepted as a modification in the chain of service level agreements between all involved domains. The SLA/SLS is in reality a chain of SLA/SLS between neighbour domains and a final end-to-end one. Fundamental concepts (continued) User 1 User 2 User 1 NREN1 GÉANT NREN2 User 2 +

49. M. Campanella - Sequin Workshop - February 1st, 2002 48 Fundamental concepts (continued) Users must understand the application requirements in term of the QoS parameter, at least the requi- rement for the maximum sending/receiving rate of the application. There is the need of a central database to keep up to date track of allocate resources and check their availability. Debugging can be assigned to just one specific entity.

50. M. Campanella - Sequin Workshop - February 1st, 2002 49

51. M. Campanella - Sequin Workshop - February 1st, 2002 50 Agenda - Approach to the problem - Users’ requirements - Quantitative definition of QoS - Which QoS service - Premium IP service - Service Level Agreements - Monitoring - Premium IP status - Other QoS services

52. M. Campanella - Sequin Workshop - February 1st, 2002 51 Monitoring Highly distributed measurement of QoS parameters that can measure the end to end and single hops performance. Use a mixture of active (in-band) and passive methodologies In-house developed tool for GÉANT (Taksometro)

53. M. Campanella - Sequin Workshop - February 1st, 2002 52 Active injects measurement traffic at small capacity use low cost dedicated hardware like RIPE TT boxes, surveyor, chariot suitable for loss, delay, jitter can be implemented in key locations Passive used also for regular (BE) traffic known tools based on SNMP, like netflow, that read counters on nodes suitable for packet loss, policing, queue depth… can access every node Methodologies

54. M. Campanella - Sequin Workshop - February 1st, 2002 53 Per-domain measurement taksometro Web interface router NREN A NREN B NREN C Ripe ttm

55. M. Campanella - Sequin Workshop - February 1st, 2002 54 Agenda - Approach to the problem - Users’ requirements - Quantitative definition of QoS - Which QoS service - Premium IP service - Service Level Agreements - Monitoring - Premium IP status - Other QoS services

56. M. Campanella - Sequin Workshop - February 1st, 2002 55 Premium IP road map QoS Definition Deliverable D2.1 Sequin Premium IP Architecture Deliverable D9.1 QoS Testbed Definition Deliverable D3.1Sequin Premium IP SLA/SLS Deliverable D9.1- Addendum 2 Premium IP Implementation Deliverable D9.1- Addendum 1 QoS Monitoring Deliverable D9.4 Testing Activity in GÉANT and Testbeds - Pilot users

57. M. Campanella - Sequin Workshop - February 1st, 2002 56 Premium IP status Currently tests are running between Switzerland, Italy, Germany and Greece. Goal is to validate the model, measure the network performance (end to end) and measure the effect using a videoconferencing application based on H.323 Premium IP is configured on GÉANT routers.

58. M. Campanella - Sequin Workshop - February 1st, 2002 57 Premium IP in progress Fine tuning of buffering and token bucket depth in routers. As a rule of thumb the token bucket depth can be assumed to be 1.2 * (number of Diffserv active interfaces on router) Scalability - the maximum amount of aggregated Premium IP capacity the network can offer - hardware capabilities Fast provisioning of the service Widespread availability and tuning of “last mile” (LANs)

59. M. Campanella - Sequin Workshop - February 1st, 2002 58 Agenda - Approach to the problem - Users’ requirements - Quantitative definition of QoS - Which QoS service - Premium IP service - Service Level Agreements - Monitoring - Premium IP status - Other QoS services

60. M. Campanella - Sequin Workshop - February 1st, 2002 59 Other QoS services ? Less than Best Effort - Less than Best Effort (Scavenger) Already working in Internet2, requires queuing and marking, not access control Assured Forwarding - Assured Forwarding based services. Sequin could not find a implementation scenario for NRENs The limit is the number of hardware queues in the routers and the hardware performance.

61. M. Campanella - Sequin Workshop - February 1st, 2002 60 References All the deliverables, presentation and relevant documentation can be found on the web in: http://www.dante.net/sequin and http://www.dante.net/tf-ngn

62. M. Campanella - Sequin Workshop - February 1st, 2002 61 Thank you and user QoS is even tougher

63. M. Campanella - Sequin Workshop - February 1st, 2002 62 Overprovisioning Two possible definitions: - istantaneous link load never greater than 30% - no packet losses (weaker) It works for 99.9 % of the cases, but capacity is far from being overprovisioned all over Europe (yet). Even many LANs have not enough capacity. It’s not perfect (yet) though...

64. M. Campanella - Sequin Workshop - February 1st, 2002 63 99.99% clean Tier 1 US backbone [ From Casner @ Nanog 22]

65. M. Campanella - Sequin Workshop - February 1st, 2002 64 Tier 1 US backbone (continued) 99.99% clean [ From Casner @ Nanog 22] 99.99%

66. M. Campanella - Sequin Workshop - February 1st, 2002 65 Overprovisioning (continued) Deviation from 99.99 % of delay variation constant value due to : - routing problems; - routing timers set-up; - ARP cache timeouts; -... It’s mostly instability/misbehaviour of the software layer on routers/switches.