1. How to Multi-Home Avi Freedman VP Engineering AboveNet Communications

2. What is Multi-Homing? Multi-homing is the process of selecting, provisioning, and installing a redundant connection to the Internet. Could be the same provider, or a different provider.

3. Why Multi-Home? Slow is 1,000,000% better than dead. You may be out of bandwidth. And –Telco circuits die. –Routers die. –Providers networks fail. –Different networks have better performance to different sites.

4. A Multi-Homed Architecture Ideally, take advantage of the opportunity to multi-home to remove all single points of failure in your network. Use - –Multiple providers, unless your current provider will let you have cheap backup –Multiple routers –Multiple telco vendors

5. Multi-Homed Architecture Two routers, each with a different WAN connection from a different telco vendor. Use HSRP or VRRP internally to make both routers look like one virtual router. Eventually, multiple providers. Upcoming Boardwatch article with configs.

6. How the Internet Works Well, it breaks more than it works but when it does work - The Internet is a network of networks. Each network (called Autonomous System) on the Internet announces routes, which are lists of the IP addresses of the boxes on their network. You need to be able to send packets *to*, and get packets *from*, everywhere.

7. Inbound Traffic - Routes Routes are announced via BGP4 (the Border Gateway Protocol) Routers are announced to BGP peers. Each BGP peer can be a network peer or a transit peer. Network peers exchange just lists of customer routes. Each route is tagged by the ASNs it passes through.

8. Inbound Traffic - Routes So when AboveNet and UUNET peer, only AboveNet and UUNET routes are exchanged. No Sprint, PSI, etc... Transit peers - –Announce to their customers all of the routes on the net (AboveNet, UUNET, Sprint, PSI, and the 60,000+ routes on the net). –Announce to their peers all routes heard via transit.

9. Inbound Traffic - Routes So if you advertise 207.106.96.0/19 to AboveNet, - –If youre a network peer, they only re-announce 207.106.96.0/19 to customers (and use it internally); –If youre a transit peer/customer, they announce 207.106.96.0/19 to all of their network peers. Thats how you get global *inbound* reachability.

10. Address Space Issues Noone wants to hear a route for you unless - –You are multi-homed (even then, some people dont want to hear routers), or –You have your own direct IP space allocation from ARIN, RIPE, or APNIC. So, when youre single-homed without your own space, your IPs are reachable because theyre part of your providers aggregate block.

11. Address Space Issues For example, your provider has 207.8.128.0/17. You have 27.8.197.0/24 from them. Youre single-homed. The only route on the net for you is the 207.8.128.0/17 route, originated by your providers ASN (and you dont have to do anything special).

12. Address Space Issues If you have your own CIDR block and are single-homed, your provider will originate it. So, if you have 219.190.64.0/19, itll be visible as an announcement by your provider, originated into the BGP mesh with your providers ASN as the origin.

13. Address Space Issues If you have your own IP space and want to multi-home, addressing issues are simple. Your other provider will start also originating your IP blocks. Or youll start speaking BGP, originate your IP blocks, and your providers will re- advertise them to the world.

14. Address Space Issues If you dont have your own IP space, its a bit more complicated. So, normally your ISP will only be advertising 207.8.128.0/17 if you have 207.8.200.0/23. If youre multi-homed, your other provider will have to advertise 207.8.200.0/23. But *so will your first provider*. Why?

15. Address Space Issues Routes are chosen first by specificity. That is, to how many IP addresses they refer. The route covering the fewest IP is the most specific, and wins. (Otherwise default would always win and nothing would work.)

16. Address Space Issues So, if ISP 1 advertises only 207.8.128.0/17 and ISP 2 advertises only 207.8.200.0/23, all inbound traffic from the net will come in on ISP2. So, ISP 1 needs to blow a hole in their filters to leak the more specific 207.8.200.0/23 route.

17. Address Space: Filtering Some ISPs do or did filter on routes smaller than (more specific than) /19s in > 205.0.0.0 space. But it doesnt matter as long as your two upstreams have good connectivity. Why?

18. Address Space: Filtering If Sprint doesnt see 207.8.200.0/23 from ISP1 or ISP2, theyll still see your providers 207.8.128.0/17 route. So if your connectivity to ISP1 (the owner of 207.8.128.0/17) goes down, all will be well as long as ISP1 still sees 207.8.200.0/23 from ISP2. Sprint -> ISP1 -> ISP2 This is why people dont let you take IPs...

19. Load-Balancing Outbound You can use static default routes to control outbound packets. –ip route 0.0.0.0 0.0.0.0 serial0/0 –ip route 0.0.0.0 0.0.0.0 serial1/0 If theyre equal-cost (no metric at the end), itll load-balance based on *destination*, by default.

20. Load-Balancing Outbound Why load-balance based on destination? For internal networking, sometimes per- packet-load balancing makes sense. But if youre trying to talk to England and one provider has a 60ms path and the other has a 150ms path, packets will arrive out of order and TCP and UDP apps get unhappy and slow.

21. How it works, Single-Homed Outbound (easy): –Use a default route to your provider. Inbound: –Your provider originates a large (aggregate) BGP route, and gives you some space from inside it; and/or –Your provider originates BGP routes for your ARIN/RIPE/APNIC CIDR blocks as well.

22. How it Works, Multi-Homed, Static Outbound (easy): –Load-balance default routes to deal with outbound packets. Inbound: –Your providers both originate BGP routes for just the address space youre using, even if its out of one providers space; and/or –Your providers both originate BGP routes for your ARIN/RIPE/APNIC CIDR blocks as well.

23. How it Works, Multi-Homed, Static Special note: –When providers configure BGP for single- homed customers, they will generally nail up your routes (even your directly-issued) CIDR blocks, so that if your connection goes down and up and down and..., they dont have to flap that route out to the whole Internet. This is a good thing.

24. How it Works, Multi-Homed, Static Special note (ctd): –But you NEED to make sure, when youre multi-homed, that the providers are NOT nailing your routes up. –Why? –Because if they do, when one T1 goes down, that provider will still advertise you to the world, thus blackholing you.

25. How it Works, Multi-Homed, BGP Topic of next talk. You either load-balance outbound with statics, or take full routes from your providers (if you can). You originate advertisements under your ASN for your directly-issued CIDR blocks, AND for the parts of your providers space that youre using (with their permission).

26. The Transition: Static Routing To transition: –Turn up the other T1/T3/Ethernet. –Put IPs on the interface. –Run tests end-end. –Start load-balancing default to the new T1. –Then, in the middle of the night, have the new provider start advertising your IP space. Make sure you have reachability to every other ISP you can think of afterwards.

27. The Transition: Static Routing To transition (ctd): –After testing it live, turn off your other transit pipes and make sure that, after a few minutes, you still have connectivity.

28. The Transition: BGP Routing To transition: –Turn up the other T1/T3/Ethernet. –Put IPs on the interface. –Run tests end-end. –Start load-balancing default to the new T1. –Then, undo that and bring up a BGP session that permits no routes either way. –Then start taking routes, and watch outbound traffic.

29. The Transition: BGP Routing To transition (ctd): –Then, start announcing your routes. –Then, in the middle of the night, have your ISP take out the static route and BGP announcement they were making. –Make sure your route is propagating. –Test reachability. –Turn off your other pipes. –Test reachability.

30. BGP or no? Advantages of doing static - –Cheaper/smaller routers (less true nowadays) –Simpler to configure Advantages of doing BGP - –More control of your destiny (have providers stop announcing you) –Faster/more intelligent selection of where to send outbound packets. –Better debugging of net problems (you can see the Internet topology now)

31. Same Provider or Multiple? If your provider is reliable and fast, and affordably, and offers good tech-support, you may want to multi-home initially to them via Frame, SMDS, or some backup path (slow is 1,000,000% better than dead). Eventually youll want t multi-home to different providers, to avoid failure modes due to one providers architecture decisions.

32. Questions? avi@freedman.net inet-access mailing list

33. Nailing routes