Presentation is loading. Please wait.

Presentation is loading. Please wait.

Inter-Peer NOC Communication Mike Hughes

Similar presentations


Presentation on theme: "Inter-Peer NOC Communication Mike Hughes"— Presentation transcript:

1 Inter-Peer NOC Communication Mike Hughes mike@linx.net

2 Scene Setting: Straw Poll Who here in this room does peering?

3 Scene Setting: Straw Poll Who here in this room does peering? Have you ever had issues resolving problems with your peerings? –Difficulties contacting peers, finding the right contact, communication problems?

4 Scene Setting: Straw Poll Who here in this room does peering? Have you ever had issues resolving problems with your peerings? Do you maintain a local db of contacts? –Why? Issues with freshness of data?

5 Scene Setting: Straw Poll Who here in this room does peering? Have you ever had issues resolving problems with your peerings? Do you maintain a local db of contacts? When a peer needs to talk to you, where does their call/email arrive? –Main NOC contact? Dedicated peering contact? “Customer Care”?

6 Scene Setting: Straw Poll Who here in this room does peering? Have you ever had issues resolving problems with your peerings? Do you maintain a local db of contacts? When a peer needs to talk to you, where does their call/email arrive? Some names have been changed to protect the innocent… and guilty…

7 Why do you go peering? Long term money savings Less Transit Lower latency, better performance Traffic Control Diversity, Reliability Presence …and so on…

8 Where’s the problem? Poor inter-peer communication seems to be common –Friendly IX operator called in to “mediate” Communication hitting the wrong place –Customer NOCs –IX Operator –IP address maintainer (e.g. whois contact)

9 Identifying the right contact Sources of information: –Whois queries to databases –IXP-maintained NOC and Peering contact db –Internal databases –Third-party voluntary databases http://puck.nether.net/netops list peeringdb.com All above are vulnerable to information “rot”

10 How to drive RIPEdb/RA, etc Some really subtle differences in the implementations –RIPE expects “AS” before an AS number! Which contacts are useful Which objects to look up –Like the Peer ASN, not the Peer IP address! Why can’t ASN be logged in adjacency changes on routers? –This seems to drive IP-based lookups

11 Drive the Data Sources Properly! Example: using WHOIS queries “Oh, I have an outage on WAIX, I’ll look up the IP address” $ whois -h whois.arin.net 198.32.212.11|less … OrgName: Exchange Point Blocks … RTechHandle: WM110-ARIN RTechName: Manning, Bill RTechPhone: +1-310-322-8102 RTechEmail: bmanning@karoshi.com

12 Bad Data Enters the System “Okay, I’ll phone Bill Manning” –But all Bill did was give WAIX some v4 space –Bill doesn’t run WAIX, and isn’t an operational contact for WAIX So, Bill either ignores your voicemail, or tells you to call someone else Whatever – it’s added delay, increased frustration – it’s how not to do it

13 Driving Whois Properly Always lookup the PEER ASN –Not the IP address! –It’s a BGP problem, we use ASNs in BGP $ whois -h whois.ra.net AS3856|less aut-num: AS3856 as-name: UNSPECIFIED descr: Packet Clearing House www.pch.net admin-c: Bill Woodcock tech-c: Bill Woodcock remarks: peering@pch.net, +1 866 BGP PEER

14 Driving Whois Properly Always lookup the PEER ASN –Not the IP address! –It’s a BGP problem, we use ASNs in BGP $ whois -h whois.ra.net AS3856|less aut-num: AS3856 as-name: UNSPECIFIED descr: Packet Clearing House www.pch.net admin-c: Bill Woodcock tech-c: Bill Woodcock remarks: peering@pch.net, +1 866 BGP PEER

15 So you’ve found the contact How do they respond to you? –Confusing recursive call trees? –Recalcitrant ticketing systems? –First-line NOC – “Is it switched on?” –“You’re not a customer, go away” Once negotiated, peering is an engineering relationship –So backbone ops, not “customer care”

16 Expectations of Peer Contacts Choose your points of contact carefully Big problems with –What’s peering/BGP/WAIX? –Are you a customer? –What’s your circuit ID? –Go away, you aren’t a customer All serious no-no’s – be nice to your peers!

17 PCH INOC-DBA Phones PCH operate a “dial by ASN” NOC hotline system –They run the SIP registry/proxy –“Bring your own” SIP compliant phone The idea is that it should get through to someone clueful –No call-trees, no music-on-hold http://www.pch.net/inoc-dba/

18 Suggested Role Contacts Peering@ –For setting up new peerings, changing existing ones, no 24x7 expectation –Shouldn’t go to exclusively to sales@ ;-) NOC@ –Reaches your 24x7 NOC, which is either BGP friendly and has enable, or knows when, how and where to escalate Support@ –Is generally your “customer-care”/call center

19 Getting the message across Okay, so you’ve made contact –Now, make your point Provide the peer with useful information –Start with the subject line –Be informative, who, when, what –Messages like “Help” and “Peering down” aren’t helpful

20 How not to do it… Where? How does it affect me? –All detail buried in wordy message body When? No TZ stamp! Help me handle my huge NOC inbox! -----Original Message----- From: Joe Schmoe Sent: Wednesday, January 25, 2006 5:41 PM Subject: Maintenance Notification Dear Peers, …

21 Example: Useful Subject Headers AS7132’s preferred subject line format: - Example subject line: Equinix-Ashburn - RCN/6079 - SBC/7132 - new session turn-up - 29- Mar-06 - 9:45 am EST Thanks to Ren Provo

22 Look clueful What does this say about your peer? –Don’t you think they look silly? Run tools to help you answer these questions yourself –Netflow, MAC accounting, etc. Subject: Traffic Drop Dear Peer, We suddenly noticed a 300Mb drop in traffic on our connection to the PIE-IX. Can you investigate, and help us find where the traffic has gone? Regards, …

23 How to escalate Check your equipment first Ask your peer - “What’s up?” –Often you can resolve a problem bi-laterally Go to the IX only if you need to –Not all IX operators can provide a 24x7 contact When to escalate a customer fault –Don’t stonewall customer reports –Don’t point them to the IX operator –Co-ordinate directly with your peers

24 How the IXP Op can help Provide an up-to-date list of IX participants and their NOC/Peering contact information –Usually password protected Help break comms deadlock –Help fix “dead ends” Otherwise, they can only help with “physical” problems –“link down”, packet loss, broken cables, packet corruption to all destinations connected to the IXP

25 In Summary Keep your own information up to date –Whois db objects, third party dbs Make sure your peering and NOC contacts are appropriate –No-one likes call-trees and holding Find the right contacts at your peers Be nice to your peers!

26 Thanks mike@linx.net


Download ppt "Inter-Peer NOC Communication Mike Hughes"

Similar presentations


Ads by Google