We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published bySamuel Quinn
Modified about 1 year ago
Diagnosing Performance Degradation with Route Analytics Cengiz Alaettinoglu
It all started with a Jitter Study (2000) Studied jitter on 2 US, and 1 European backbones for several weeks For 99.99% packets, measured jitter < 1ms Copyright © 2014 Packet Design2
However, 0.01% of Jitter was Severe Copyright © 2014 Packet Design3 10+ Second of packet drops 7 seconds jitter
Severe Packet Reordering Copyright © 2014 Packet Design4
Theory: Packets being spewed out from an unwinding routing loop… Did we really have long routing loops in the network? Did ISIS really take 10+ seconds to convergence? So, we analyzed routing along with jitter Copyright © 2014 Packet Design5 Test host is passive peer, sends no routes IP Backbone R traceroute every 5s IS-IS hellos + tcpdump traffic Test Host packet trace file sk gig- ether Test Host R
Excessive ISIS Churn caused excessive LSP Propagation Delay Copyright © 2014 Packet Design6
Explanation Link state databases were not in sync: Very large LSP databases High churn rate many LSPs to flood LSP rate-control slowed down flooding SPF updates may also have been delayed by rate limits Any topology change could result in a loop under these conditions We realized being able to look at routing was key for powerful network performance analysis Today, we see very high churn in very large TE databases with auto-bandwidth with large number of tunnels Copyright © 2014 Packet Design7
Route Analytics Today Copyright © 2014 Packet Design8 Collection Analysis Presentation BGP IGPs Flows Multicast Path Reports RCA Exit Routers Traffic Reports Tunnels Reporting Alerting Planning Trouble- shooting Trouble- shooting
Route Analytics Applications Troubleshooting and visualization Ability to look at network state at any given time Inspecting and playing events learned both from the routers and routing protocols Comparing routing state and paths when a service/application is performing well and when it is not Service/application monitoring and alerting Monitor paths for changes in hops, metric, delay, and bandwidth Monitor excessive protocol behavior Copyright © 2014 Packet Design9
Route Analytics Applications (cont.) Network health assessment Capacity planning Anomaly detection Path bottleneck analysis (bandwidth, delay, metric) Failure analysis Topology-aware traffic analysis Where is the traffic coming from, going to, its path and why? Feed this back into path computation as a traffic matrix Proactive change modeling Add/drop routers/switches/links/prefixes/peerings Add/drop applications/services Analyze the impact on paths and traffic levels Feed this into provisioning Copyright © 2014 Packet Design10
Use Case: Diagnosing Black Holing A peering router to a major service provider crashed Hot swappable card was not quite so… Traffic to the SP was black-holed network-wide Traffic that exited in all 6 locations was all black-holed About 3 minutes of routing outage 3 minutes was too short to diagnose the issue at human speed Had a 45 minute ad-revenue impact on the services Users who can not use the service do something else Copyright © 2014 Packet Design11
Expected Exit-Points Before Incident Copyright © 2014 Packet Design12 6 Exit-Points (circled) Blue routers take the blue exit router…
The Incident ISIS activity during incident Copyright © 2014 Packet Design13
Exit-Points During Incident Copyright © 2014 Packet Design14 6 more Exit-Points popped up Core routers with no EBGP
A Path Before and After the Incident Copyright © 2014 Packet Design15 Before After Route Recursion BGP Next hop resolution: before /32 in ISIS vs after /19 in BGP
Cause of Black Holing When the peering router crashed IGP routes were withdrawn in seconds BGP routes were not withdrawn 3 KEEPALIVEs of 60 seconds each – router rebooted before this The BGP routes were now resolved by the /19 prefix in BGP The /19 BGP announces internal address space – not meant for this Injected by 6 core routers – cost from any router to a core router is very low Remedy: Insert a really expensive static route for the /19 to ISIS Cost more than longest possible path in IGP Now, when a peering router crashes, the traffic will choose a true exit See Do not: Make IBGP session to converge faster (like running BFD) You will lose the IBGP session each time the IGP path of the session changes Copyright © 2014 Packet Design16
Use Case: BGP Peering Traffic Analysis For most regional networks, incoming traffic from the Internet is higher than outgoing traffic What ASs (sources) is the traffic coming from What neighbor AS did it come from How is it distributed across the edge routers Need answers for: Should I upgrade external links or get new links? To whom? - The same neighbor ASs or new neighbor ASs? Who can I peer with to cut transit cost? Copyright © 2014 Packet Design17
Routing-Aware Traffic Flow Analysis Route-Flow Fusion determines each flow’s ingress/egress and path across the network and onto the Internet Delivers traffic matrices for planning and path computation Helps make peering decisions to save transit cost Copyright © 2014 Packet Design18
Traffic by Source AS Copyright © 2014 Packet Design19 Peering with Google can save the most cost Not all sources may be feasible to peer with
Neighbor ASs Copyright © 2014 Packet Design20 Reveals where the traffic enters your network Is the traffic balanced the way you would like? Capacity planning with this data tells when it is time to upgrade or add new peering
Upstream Transit ASs Copyright © 2014 Packet Design21 Offers the good choices for peering – very hard to compute This is not in your control, rather in the source AS’s control A heuristics-based solution: Alternate routes for a given source are exposed during BGP convergence Use these and graph search to compute these values
Modeling a New BGP Peer Copyright © 2014 Packet Design22
Before-and-After Comparison Reveals Transit Traffic Shifts Copyright © 2014 Packet Design23
Route Analytics Simplifies Planning for BGP Peering Analysis Traffic volumes are obtained by fusing flows with routes Create a bunch of “mocked-up” BGP routes Route-Flow Fusion will now follow these routes Simple and elegant No simulation, emulation or inaccurate modeling used Impact analysis Route analytics knows what your network will do Copyright © 2014 Packet Design24
Challenges in Software Defined Networking What governs whether or not these programmatic changes should be made? What will be their impact? SDN makes networks programmable for Network overlays Bandwidth reservation Demand placement Service deployment Etc. Northbound APIs SDN Controller OpenFlow, i2RS, PCE, NETCONF, ForCES, SNMP, CLI, etc. Southbound APIs Orchestration Management Interface Admissions API Interface Physical & Virtual Network Devices Application Services Bandwidth Reservation Demand Placement Service Deployment Applications App 1 App 2App n … App 3 Copyright © 2014 Packet Design25
Network Virtualization SDN Analytics Tying Overlay and Underlay Networks 26 Copyright © 2014 Packet Design Overlay Network Underlay Network SDN Analytics VMWare or OpenStack
Concluding Remarks Routing impacts network performance Availability and reachability Sub-optimal paths with longer delays, jitter Route analytics proves to be very effective Troubleshooting, monitoring, alerting Reporting and network health assessment Routing-aware traffic analysis BGP peering analysis Traffic matrices Route analytics provides foundation for SDN Analytics Copyright © 2014 Packet Design27
The Role of Analytics in Routing, Network Performance and SDN Cengiz Alaettinoglu.
© 2005 Cisco Systems, Inc. All rights reserved. BGP v3.2—6-1 Scaling Service Provider Networks Scaling IGP and BGP in Service Provider Networks.
© 2005 Cisco Systems, Inc. All rights reserved. BGP v3.2—2-1 BGP Transit Autonomous Systems Forwarding Packets in a Transit AS.
© 2006 Cisco Systems, Inc. All rights reserved. MPLS v2.2—2-1 Label Assignment and Distribution Introducing Typical Label Distribution in Frame-Mode MPLS.
Deployment of MPLS VPN in Large ISP Networks Luyuan Fang IP Network Architecture AT&T.
© Janice Regan, CMPT 128, CMPT 371 Data Communications and Networking BGP, Flooding, Multicast routing.
More on Internet Routing A large portion of this lecture material comes from BGP tutorial given by Philip Smith from Cisco (ftp://ftp- eng.cisco.com/pfs/seminars/APRICOT2004.
Internet Traffic Engineering Motivation: –The Fish problem, congested links. –Two properties of IP routing Destination based Local optimization TE: optimizing.
Routing Basics 1 Routing Concepts IPv4 Routing Forwarding Some definitions Policy options Routing Protocols 2.
Packet Design Real-World Uses of Route Analytics Greg Hooten Senior Routing Engineer Packet Design Inc.
Multihoming and Multi-path Routing CS 4251: Computer Networking II Nick Feamster Spring 2008.
Vytautas Valancius, Nick Feamster, Akihiro Nakao, and Jennifer Rexford.
Courtesy: Nick McKeown, Stanford 1 Routing Algorithms Tahir Azim.
1 EL736 Communications Networks II: Design and Algorithms Class3: Network Design Modeling Yong Liu 09/19/2007.
Information-Centric Networks04a-1 Week 4 / Paper 1 Open issues in Interdomain Routing: a survey –Marcelo Yannuzzi, Xavier Masip-Bruin, Olivier Bonaventure.
© 2005 Cisco Systems, Inc. All rights reserved. BGP v3.2—5-1 Customer-to-Provider Connectivity with BGP Connecting a Multihomed Customer to a Single Service.
IP Routing Principles. Network-Layer Protocol Operations Each router provides network layer (routing) services X Y A B C Application Presentation Session.
MPLS and Traffic Engineering December 08, 2003 MPLS and Traffic Engineering Zartash Afzal Uzmi Department of Computer Science Lahore University of Management.
1 Interdomain Routing Protocols. 2 Autonomous Systems An autonomous system (AS) is a region of the Internet that is administered by a single entity and.
Computer Networks Layering and Routing Dina Katabi
December 20, 2004MPLS: TE and Restoration1 MPLS: Traffic Engineering and Restoration Routing Zartash Afzal Uzmi Computer Science and Engineering Lahore.
1 Computer Communication & Networks Lecture 22 Network Layer: Delivery, Forwarding, Routing (contd.)
1 Version 3.1 Module 6 Routed & Routing Protocols.
Network Monitoring for Internet Traffic Engineering Jennifer Rexford AT&T Labs – Research Florham Park, NJ 07932
ICS 156: Networking Lab Magda El Zarki Professor, ICS UC, Irvine.
Evolving Toward a Self-Managing Network Jennifer Rexford Princeton University
Chapter 8 Routing. Introduction Look at: –Routing Basics (8.1) –Address Resolution (8.2) –Routing Protocols (8.3) –Administrative Classification (8.4)
Automatic network configuration: Position presentation Cristel Pelsser WODNAFO, Feb
A Routing Control Platform for Managing IP Networks Jennifer Rexford Princeton University
Introduction to OSPF Nishal Goburdhan. Routing and Forwarding Routing is not the same as Forwarding Routing is the building of maps Each routing protocol.
CCNA 2 Week 6 Routing Protocols. Copyright © 2005 University of Bolton Topics Static Routing Dynamic Routing Routing Protocols Overview.
© J. Liebeherr, All rights reserved 1 Border Gateway Protocol This lecture is largely based on a BGP tutorial by T. Griffin from AT&T Research.
+ Dynamic Routing Protocols 2 nd semester
COS 461 Fall 1997 Routing COS 461 Fall 1997 Typical Structure.
1 Routing Algorithms (Ch5 of Computer Network by A. Tanenbaum) –Shortest path routing static simple and easy to understand. Metric for a shortest path:
IETF San Francisco Multicast-only Fast ReRoute (MoFRR) Dino Farinacci Apoorva Karan Clarence Filsfils March, 2009.
Border Gateway Protocol Ankit Agarwal Dashang Trivedi Kirti Tiwari.
A Snapshot on MPLS Reliability Features Ping Pan March, 2002.
CS 3700 Networks and Distributed Systems Inter Domain Routing (It’s all about the Money) Revised 8/20/15.
Authors Renata Teixeira, Aman Shaikh and Jennifer Rexford(AT&T), Tim Griffin(Intel) Presenter : Farrukh Shahzad.
1. 2 Anatomy of an IP Packet IP packets consist of the data from upper layers plus an IP header. The IP header consists of the following:
Inter-domain Routing Outline Border Gateway Protocol.
Border Gateway Protocol (BGP) W.lilakiatsakun. BGP Basics (1) BGP is the protocol which is used to make core routing decisions on the Internet It involves.
Traffic Engineering With Traditional IP Routing Protocols B. Fortz, J. Rexford, and M. Thorup.
UNICAST ROUTING PROTOCOLS Major Functions: Define the domain of operation (Internal/External to the ISPs), and interaction with other protocols.
© 2010 Cisco and/or its affiliates. All rights reserved. 1 Segment Routing Clarence Filsfils – Distinguished Engineer Christian Martin –
Why SDN and MPLS? Saurav Das, Ali Reza Sharafat, Guru Parulkar, Nick McKeown Clean Slate CTO Summit 9 th November, 2011.
ROUTING ON THE INTERNET COSC Jun-16. Routing Protocols routers receive and forward packets make decisions based on knowledge of topology.
© 2005 Cisco Systems, Inc. All rights reserved. BGP v3.2—2-1 BGP Transit Autonomous Systems Monitoring and Troubleshooting IBGP in a Transit AS.
© 2017 SlidePlayer.com Inc. All rights reserved.