2Goal of This TutorialTeach engineers the basics of networking and ISP operationsNetworks todayBusiness modelsOperations (NOC, operators)Common problemsMeasurement, Monitoring, and Security
3Today’s Networks Service provider business models Network operations centerNetwork operators and engineers
4Business ModelsIncreasingly commoditized (see Geoff Huston’s talk at NANOG)Status quo: Establish transit costs, bill at 95th percentile of usageFuture: differential pricing, preference for certain groups of users, applications
5Billing for Internet Usage 95th Percentile billingCustomer network pays for “committed information rate” (CIR)Throughput measured every 5 minutes (typically with SNMP; flow statistics also can be used for billing)Customer billed based on 95th percentile
11Internet Routing Overview Autonomous Systems (ASes)AbileneComcastGeorgiaTechAT&TCogentIntradomain (i.e., “intra-AS”) routingInterdomain routing
12Internet Routing Protocol: BGP Autonomous Systems (ASes)Route AdvertisementDestination Next-hop AS Path/16174… 2637SessionTrafficDiagram of routing table is very confusing because it’s not pointing to anythingGreen arrow shorter, and too thick… green is a msgMore intuition about how the system actually works.Don’t say “interdomain”DESTINATION-BASED RoutingTables look like a set of possible routes and a rankings over these routes(pop up a simplified table fragment)
13Question: What’s the difference between IGP and iBGP? Two Flavors of BGPiBGPeBGPExternal BGP (eBGP): exchanging routes between ASesInternal BGP (iBGP): disseminating routes to external destinations among the routers within an ASQuestion: What’s the difference between IGP and iBGP?
14IPv4 Addresses: Networks of Networks Topological Addressing32-bit number in “dotted-quad” notation130207736Network (16 bits)Host (16 bits)Problem: 232 addresses is a lot of table entriesSolution: Routing based on network and host/16 is a 16-bit prefix with 216 IP addresses
15Pre-1994: Classful Addressing 3281624Class ANetwork IDHost ID/8 blocks (e.g., MIT has /8)Class B10/16 blocks (e.g., Georgia Tech has /16)Class C110/24 blocks (e.g., AT&T Labs has /24)Class DMulticast Addresses1110Class E1111Reserved for experimentsSimple Forwarding: Address range specifies network ID length
16Classless Interdomain Routing (CIDR) Use two 32-bit numbers to represent a network.Network number = IP address + MaskExample: BellSouth Prefix: /22IP Address: “Mask”:Address no longer specifies network ID range. New forwarding trick: Longest Prefix Match
17Benefits of CIDREfficiency: Can allocate blocks of prefixes on a finer granularityHierarchy: Prefixes can be aggregated into supernets. (Not always done. Typically not, in fact.)Customer 1/24/8AT&TInternetCustomer 2/24
26Passive vs. Active Measurement Passive Measurement: Collection of packets, flow statistics of traffic that is already flowing on the networkPacket tracesFlow statisticsApplication-level logsActive Measurement: Inject “probing” traffic to measure various characteristicsTraceroutePingApplication-level probes (e.g., Web downloads)
27Billing for Internet Usage 95th Percentile billingCustomer network pays for “committed information rate” (CIR)Throughput measured every 5 minutes (typically with SNMP; flow statistics also can be used for billing)Customer billed based on 95th percentile
28Passive Traffic Data Measurement SNMP byte/packet counts: everywherePacket monitoring: selected locationsFlow monitoring: typically at edges (if possible)Direct computation of the traffic matrixInput to denial-of-service attack detectionDeep Packet Inspection: also at edge, where possible
29Simple Network Management Protocol Management Information Base (MIB)Information storeUnique variables named by OIDsAccessed with SNMPSpecific MIBs for byte/packet counts (per link)SNMPManagerAgentManaged ObjectsDB
30SNMP (Passive) Advantage: ubiquitous Disadvantages Utility Supported on all networking equipmentMultiple products for polling and analyzing dataDisadvantagesCoarse granularityCannot express complex queries on the dataUnreliable delivery of the data using UDPUtilityLink utilization (billing)Traffic matrix inference
31Packet-level Monitoring Passive monitoring to collect full packet contents (or at least headers)Advantages: lots of detailed informationPrecise timing informationInformation in packet headersDisadvantages: overheadHard to keep up with high-speed linksOften requires a separate monitoring device
33What is a flow? Source IP address Destination IP address Source port Destination portLayer 3 protocol typeTOS byte (DSCP)Input logical interface (ifIndex)
34Cisco NetFlow Basic output: “Flow record” Most common version is v5Current version (9) is being standardized in the IETF (template-based)More flexible record formatMuch easier to add new flow record typesCore NetworkCollector (PC)Approximately 1500 bytes20-50 flow recordsSent more frequently if traffic increasesCollection and Aggregation
35Flow Record Contents Source and Destination, IP address and port Basic information about the flow…Source and Destination, IP address and portPacket and byte countsStart and end timesToS, TCP flags…plus, information related to routingNext-hop IP addressSource and destination ASSource and destination prefix
36Aggregating Packets into Flows Criteria 1: Set of packets that “belong together”Source/destination IP addresses and port numbersSame protocol, ToS bits, …Same input/output interfaces at a router (if known)Criteria 2: Packets that are “close” together in timeMaximum inter-packet spacing (e.g., 15 sec, 30 sec)Example: flows 2 and 4 are different flows due to time
37Reducing Measurement Overhead Filtering: on interfacedestination prefix for a customerport number for an application (e.g., 80 for Web)Sampling: before insertion into flow cacheRandom, deterministic, or hash-based sampling1-out-of-n or stratified based on packet/flow sizeTwo types: packet-level and flow-levelAggregation: after cache evictionpackets/flows with same next-hop ASpackets/flows destined to a particular service
38Packet Sampling for Flow Monitoring Packet sampling before flow creation (Sampled Netflow)1-out-of-m sampling of individual packets (e.g., m=100)Create of flow records over the sampled packetsReducing overheadAvoid per-packet overhead on (m-1)/m packetsAvoid creating records for a large number of small flowsIncreasing overhead (in some cases)May split some long transfers into multiple flow records… due to larger time gaps between successive packetstimenot sampledtimeouttwo flows
39Sampling: Flow-Level Sampling Sampling of flow records evicted from flow cacheWhen evicting flows from table or when analyzing flowsStratified sampling to put weight on “heavy” flowsSelect all long flows and sample the short flowsReduces the number of flow recordsStill measures the vast majority of the trafficsample with 0.1% probabilityFlow 1, 40 bytesFlow 2, bytesFlow 3, 8196 bytesFlow 4, bytesFlow 5, 532 bytesFlow 6, 7432 bytessample with 100% probabilitysample with 10% probability
40Two Main Approaches Packet-level Monitoring Flow-level Monitoring Keep packet-level statisticsExamine (and potentially, log) variety of packet-level statistics. Essentially, anything in the packet.TimingFlow-level MonitoringMonitor packet-by-packet (though sometimes sampled)Keep aggregate statistics on a flow
41Packet Capture on High-Speed Links Example: Georgia Tech “OC3Mon”Rack-mounted PCOptical splitterData Acquisition and Generation (DAG) cardSource: endace.com
42Characteristics of Packet Capture Allows inspection on every packet on 10G linksDisadvantagesCostlyRequires splitting optical fibersMust be able to filter/store data
45What is Multihoming?The use of redundant network links for the purposes of external connectivityCan be achieved at many layers of the protocol stack and many places in the networkMultiple network interfaces in a PCAn ISP with multiple upstream interfacesCan refer to having multiple connections toThe same ISPMultiple ISPs
46Why Multihome? Redundancy Availability Performance Cost Interdomain traffic engineering: the process by which a multihomed network configures its network to achieve these goals
47Redundancy Maintain connectivity in the face of: Physical connectivity problems (fiber cut, device failures, etc.)Failures in upstream ISP
48PerformanceUse multiple network links at once to achieve higher throughput than just over a single link.Allows incoming traffic to be load-balanced.30% of traffic70% of traffic
49Multihoming in IP Networks Today Stub AS: no transit service for other ASesNo need to use BGPMulti-homed stub AS: has connectivity to multiple immediate upstream ISPsNeed BGPNo need for a public AS numberNo need for IP prefix allocationMulti-homed transit AS: connectivity to multiple ASes and transit serviceNeed BGP, public AS number, IP prefix allocation
50BGP or no? Advantages of static routing Advantages of BGP Cheaper/smaller routers (less true nowadays)Simpler to configureAdvantages of BGPMore control of your destiny (have providers stop announcing you)Faster/more intelligent selection of where to send outbound packets.Better debugging of net problems (you can see the Internet topology now)
51Same Provider or Multiple? If your provider is reliable and fast, and affordably, and offers good tech-support, you may want to multi-home initially to them via some backup path (slow is better than dead).Eventually you’ll want to multi-home to different providers, to avoid failure modes due to one provider’s architecture decisions.
52Multihomed Stub: One Link Multiple links between same pair of routers.Default routes to “border”“Stub” ISPUpstream ISPDownstream ISP’s routers configure default (“static”) routes pointing to border router.Upstream ISP advertises reachability
53Multihomed Stub: Multiple Links Multiple links to different upstream routersBGP for load balance at edge“Stub” ISPUpstream ISPInternal routing for “hot potato”Use BGP to share loadUse private AS number (why is this OK?)As before, upstream ISP advertises prefix
54Multihomed Stub: Multiple ISPs Upstream ISP 1“Stub” ISPUpstream ISP 2Many possibilitiesLoad sharingPrimary-backupSelective use of different ISPsRequires BGP, public AS number, etc.
55Multihomed Transit Network ISP 1Transit ISPISP 3ISP 2BGP everywhereIncoming and outcoming trafficChallenge: balancing load on intradomain and egress links, given an offered traffic load
56Interdomain Traffic Engineering The process by which a network operator configures the network to achieveTraffic load balanceRedundancy (primary/backup), etc.Two tasksOutbound traffic controlInbound traffic controlKey Problems: Predictability and Scalability
57Outbound Traffic Control Easier to control than inbound trafficDestination-based routing: sender determines where the packets goControl over next-hop AS onlyCannot control selection of the entire pathProvider 1Provider 2Control with local preference
58Outbound Traffic: Load Balancing Control routes to provider per-prefixAssign local preference across destination prefixesChange the local preference assignments over timeUseful inputs to load balancingEnd-to-end path performance dataOutbound traffic statistics per destination prefixChallenge: Getting from traffic volumes to groups of prefixes that should be assigned to each linkPremise of “intelligent route control” preoducts.
59Traffic Engineering Goals PredictabilityEnsure the BGP decision process is deterministicAssume that BGP updates are (relatively) stableLimit overhead introduced by routing changesMinimize frequency of changes to routing policiesLimit number of prefixes affected by changesLimit impact on how traffic enters the networkAvoid new routes that might change neighbor’s mindSelect route with same attributes, or at least path length
60Managing Scale Destination prefixes Routing choices More than 90,000 destination prefixesDon’t want to have per-prefix routing policiesSmall fraction of prefixes contribute most of the trafficFocus on the small number of heavy hittersDefine routing policies for selected prefixesRouting choicesAbout 27,000 unique “routing choices”Help in reducing the scale of the problemSmall fraction of “routing choices” contribute most trafficFocus on the very small number of “routing choices”Define routing policies on common attributes
61Achieving Predictability Route prediction with static analysisHelpful to know effects before deploymentStatic analysis can helpBGP policyconfigurationTopologyBGP routingmodeleBGProutesOfferedtrafficFlow of traffic through the network
62Challenges to Predictability For transit ISPs: effects on incoming trafficLack of coordination strikes again!
63Inter-AS Negotiation Coordination aids predictability Destination 1Coordination aids predictabilityNegotiate where to sendInbound and outboundMutual benefitsHow to implement?What info to exchange?Protecting privacy?How to prioritize choices?How to prevent cheating?Provider Bmultiplepeeringpoints“Hot Potato”routingProvider ADestination 2
64Outbound: Multihoming Goals RedundancyDynamic routing will failover to backup linkPerformanceSelect provider with best performance per prefixRequires active probingCostSelect provider per prefix over time to minimize the total financial cost
65Inbound Traffic Control More difficult: no control over neighbors’ decisions.Three common techniques (previously discussed)AS path prependingCommunities and local preferencePrefix splittingHow does today’s paper (MONET) control inbound traffic?
66How many links are enough? K upstream ISPsNot much benefit beyond 4 ISPsAkella et al., “Performance Benefits of Multihoming”, SIGCOMM 2003
67Problems with Multihoming in IPv4 Routing table growthProvider-based addressingAdvertising prefix out multiple ISPs – can’t aggregatePoor control over inbound trafficExisting mechanisms do not allow hosts to control inbound traffic
68Internet Routing Overview Autonomous Systems (ASes)AbileneComcastGeorgiaTechAT&TCogentIntradomain (i.e., “intra-AS”) routingInterdomain routing
69Configuration Problems: “AS 7007” “…a glitch at a small ISP… triggered a major outage in Internet access across the country. The problem started when MAI Network Services...passed bad router information from one of its customers onto Sprint.” news.com, April 25, 1997UUNetSprintFlorida InternetBarn
70Diagnosis and Troubleshooting “…a glitch at a small ISP… triggered a major outage in Internet access across the country. The problem started when MAI Network Services...passed bad router information from one of its customers onto Sprint.” news.com, April 25, 1997“Microsoft's websites were offline for up to 23 hours...because of a [router] misconfiguration…it took nearly a day to determine what was wrong and undo the changes.” wired.com, January 25, 2001“WorldCom Inc…suffered a widespread outage on its Internet backbone that affected roughly 20 percent of its U.S. customer base. The network problems…affected millions of computer users worldwide. A spokeswoman attributed the outage to "a route table issue." cnn.com, October 3, 2002"A number of Covad customers went out from 5pm today due to, supposedly, a DDOS (distributed denial of service attack) on a key Level3 data center, which later was described as a route leak (misconfiguration).“-- dslreports.com, February 23, 2004
71Operator Mailing List (NANOG) Date: Mon, 18 Oct :15:Subject: Level 3 US east coast "issues"Level 3 experiencing widespread "unspecified routing issues" on the US east coast. Master ticket Anyone have more specific information?Date: Mon, 18 Oct :20: (EDT)Subject: Re: Level 3 US east coast "issues"Level 3 is currently experiencing a backbone outage causing routing instability and packet loss. We are working to restore and will be sending hourly updates…
72Compare: 83 power outages, 1 fire Operator Mailing ListCompare: 83 power outages, 1 fire“Two rats crawled through an underground cable conduit into a cabinet of power switching gear adjacent to the Stanford University cogeneration plant, and caused an explosion that cut off power to the Stanford area.” (October 12, 1996)XXX need to regenerate bar graphs to give equal weighting to years (i.e., do it in three year chunks, starting 96-98, 99-01, 02-04? I think that way the comparisons are easier to make.How fatal were these errors?Note: Only includes problems openly discussed on this list.
73Routing Configuration Filtering: route advertisementRanking: route selectionCustomerPrimaryDissemination: internal route advertisementXXX What problem does factoring solve? Need to tie into challenge this approach solves (“dealing with complexity”)This slide seems like an orphan slide. Can’t see the transition both to and from. The next slide is on vis which flows well from the prev slide!CompetitorBackup
74Internet Business Model (Simplified) ProviderPreferences implemented with local preference manipulationFree to usePay to usePeerGet paid to useCustomerDestinationCustomer/Provider: One AS pays another for reachability to some set of destinations“Settlement-free” Peering: Bartering. Two ASes exchange routes with one another.
75Peering Contracts: Consistent Export Rules of settlement-free peering:Advertise routes at all peering pointsAdvertised routes must have equal “AS path length”Sprint“equally good”routesAT&TEnables “hot potato” routing.
76Two different Export Policies Consistent ExportPossible CausesNeighbor AS ExportExport Clause PrependMalice/deceptioniBGP signaling partitionInconsistent export policyTwo different Export Policiesneighborroute-map PEER permit 10set prependneighborroute-map PEER permit 10set prepend 123
77Inconsistent Export in Practice Feamster et al., “BorderGuard: Detecting Cold Potatoes from Peers”. ACM IMC, October 2004.
78Blackholes Date: Thu, 18 Jul 2002 06:05:10 -0400 (EDT) From: Chad OlearySubject: Re: problems with 701To:We're starting to see the same issues with UUNet, again. Anyone elseseeing this? Trying to reach Qwest...traceroute to ( ), 30 hops max, 38 byte packets1 esc-lp2-gw.e-solutionscorp.com ( ) ms ms msSerial2-10.GW1.TPA2.ALTER.NET ( ) ms ms msat XL4.ATL1.ALTER.NET ( ) ms ms ms4 0.so XL2.ATL5.ALTER.NET ( ) ms ms ms5 POS7-0.BR2.ATL5.ALTER.NET ( ) ms ms ms6 * * *7 * * *…
80Security: “Bogon” Routes Feamster et al., “An Empirical Study of ‘Bogon’ Route Advertisements”. ACM CCR, January 2005.
81Can IP addresses from which spam is received be spoofed? Spam, Phishing, etc.Unsolicited commercialAs of about August 2008, estimates indicate that about 95% of all is spamCommon spam filtering techniquesContent-based filtersDNS Blacklist (DNSBL) lookups: Significant fraction of today’s DNS traffic!Can IP addresses from which spam is received be spoofed?
84What is a Worm? Code that replicates and propagates across the network Often carries a “payload”Usually spread via exploiting flaws in open services“Viruses” require user action to spreadFirst worm: Robert Morris, November 19886-10% of all Internet hosts infected (!)Many more since, but none on that scale until July 2001
85Example Worm: Code Red Initial version: July 13, 2001 Exploited known ISAPI vulnerability in Microsoft IIS Web servers1st through 20th of each month: spread 20th through end of each month: attackPayload: Web site defacementScanning: Random IP addressesBug: failure to seed random number generator
86Code Red: Revisions Released July 19, 2001 Payload: flooding attack onAttack was mounted at the IP address of the Web siteBug: died after 20th of each monthRandom number generator for IP scanning fixed
88Designing Fast-Spreading Worms Hit-list scanningTime to infect first 10k hosts dominates infection timeSolution: Reconnaissance (stealthy scans, etc.)Permutation scanningObservation: Most scanning is redundantIdea: Shared permutation of address space. Start scanning from own IP address. Re-randomize when another infected machine is found.Internet-scale hit listsFlash worm: complete infection within 30 seconds
89Botnets Bots: Autonomous programs performing tasks Plenty of “benign” botse.g., weatherbugBotnets: group of botsTypically carries malicious connotationLarge numbers of infected machinesMachines “enlisted” with infection vectors like worms (last lecture)Available for simultaneous control by a masterSize: up to 350,000 nodes (from today’s paper)
90“Rallying” the Botnet Easy to combine worm, backdoor functionality Problem: how to learn about successfully infected machines?OptionsHard-coded address
91Botnet Controller (IRC server) Dynamic DNSBotnet Controller (IRC server)Infected MachineBotnet master typically runs some IRC server on a well-known port (e.g., 6667)Infected machine contacts botnet with pre-programmed DNS name (e.g., big-bot.de)Dynamic DNS: allows controller to move about freely
93Idea #1: Ingress Filtering Drop all packets with source address other than /24Internet/24RFC 2827: Routers install filters to drop packets from networks that are not downstreamFeasible at edgesDifficult to configure closer to network “core”
94Idea #2: uRPF Checks Unicast Reverse Path Forwarding Accept packet from interface only if forwarding table entry for source IP address matches ingress interfaceStrict ModeuRPF Enabled“A” Routing TableDestination Next Hop/ Int. 1/ Int. 2from wrong interfaceUnicast Reverse Path ForwardingCisco: “ip verify unicast reverse-path”Requires symmetric routing
96S-BGP Address-based PKI: validate signatures Authentication ofownership for IP address blocks,AS number,an AS's identity, anda BGP router's identityUse existing infrastructure (Internet registries etc.)Routing origination is digitally signedBGP updates are digitally signedRoute attestations: A new, optional, BGP transitive path attributecarries digital signatures covering the routing information in updates
97Practical Problems with S-BGP Requires Public-Key InfrastructureLots of digital signatures to calculate and verify.Message overheadCPU overheadCalculation expense is greatest when topology is changingCaching can helpRoute aggregation is problematic (maybe that’s OK)Secure route withdrawals when link or node fails?Address ownership data out of dateDeployment