2. Customer Satisfaction 201 Howard C. Berkowitz hberkowi@nortelnetworks.com hcb@clark.net (703)998-5819 ESN 451-5819

3. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 2 What is the Problem to be Solved? What are you? What characteristic applications do you support? To the customer, what are the perceived needs? How do you manage Service Expectations? How do you care for customers?

4. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 3 What are you? ISP Bandwidth provider IPSP BIPSP Do you want to be seen as: lowest cost best service something inbetween?

5. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 4 A Question Why do vendors always offer solutions? Always remember you offer solvents. Solutions exist only when the customer problem dissolves in the solvent you provide.

6. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 5 Characteristic Applications Internet access by internal staff B2C: customer access to public servers Intranet —Usual entry point for voice/video convergence B2C/Extranet Bandwidth provider —Traditional telco space —But VPNs, especially with SLAs, get close

7. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 6 Topology Questions for Applications What part, if any, needs to be in global routing system? —Address space —AS Trust topology Who does operational support? —In-house or outsource —Separate application and network? Single number?

8. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 7 Customer Perceived Needs Uptime Affordable Adequate performance —(but may be perceived as "the best") Scalability Reasonable business relationship

9. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 8 Defining Service Expectations Availability "Classic SLA" —SLA for interactive applications —SLA for mission-critical data (computer-to-computer) —SLA for voice —SLA for imaging

10. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 9 What breaks? Application Server Server farm/host site Connectivity

11. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 10 Availability Expectations: Your Site (1)

12. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 11 Your Site (2)

13. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 12 Your Site (3)

14. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 13 Don't Forget Backup Power...

15. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 14 Oh, and I want to pay $39 per month?

16. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 15 Specifying Availability Period of coverage Restrictions on offered load? Maintenance windows? When does an outage begin? end? —see quality discussion later in this presentation Opportunities for less-than-ideal backup? Pricing incentives?

17. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 16 Traffic Engineering Throughput Need for consistent latency (minimize jitter) Availability Enough bandwidth Bandwidth in the right place Transient congestion avoidance Alternative ways to supply resources

18. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 17 Higher Layer Threats & Responses Single server failure or maintenance downtime Individual overloaded servers at single site Overloaded site or servers, but sufficient overall capacity Server crash Clustered servers at site; cold, warm, hot standby Local load distribution inside cluster Global load distribution among multiple clusters and sites Backups, checkpoints, mirroring

19. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 18 Lower Layer Threats & Responses Routing system failure Failure of direct provider or upstream links Failure of customer router on LAN Single medium failure between customer and ISP Multiple ISPs Multiple connection to single provider. Diversity contracts. VRRP/HSRP. BGP peering to loopbacks. Inverse multiplexing. SONET. Dial/ISDN backup. Local loop diversity

20. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 19 Some Routing Scenarios Registered address space Provider 1 Provider 2 Registered or private address space Private address space

21. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 20 Single point of failure: single-homed routing ISP Enterprise Registered address space directly allocated or ISP suballocation ISP-assigned private address space optional NAT Static Routing or Keepalive Default Route Router

22. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 21 Multilink single-homed routing ISP Enterprise Registered address space directly allocated or ISP suballocation ISP-assigned private address space Static Routing or Keepalive Default Route optional NAT Router

23. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 22 Simple Multihoming to a Single Provider Enterprise Registered address space directly allocated or ISP suballocation ISP-assigned private address space ISP POP 1 optional NAT Router ISP POP 1 optional NAT

24. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 23 Have I been solving the right problem? Enterprise Registered address space directly allocated or ISP suballocation ISP-assigned private address space ISP POP 1 optional NAT Router ISP POP 1 optional NAT Hosted Server 1

25. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 24 Local Distribution Enterprise Registered address space directly allocated or ISP suballocation ISP-assigned private address space ISP POP 1 optional NAT Router ISP POP 1 optional NAT TCP Load Distributor

26. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 25 Global Distribution, Single ISP Router Smart DNS/NAT Smart DNS/NAT

27. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 26 Simple Multihoming to Two Providers Enterprise Registered address space directly allocated or ISP suballocation Router Primary ISP POP More Preferred Default Route Router Backup ISP POP Less Preferred Default Route

28. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 27 RFC 1998 Multihoming POP1POP2 adver- tises Internet routes adver- tises Internet routes ISP with 192.0.0.0/16 assigns 192.0.2.0/22 to customer eBGP customer with private AS numbers west side hosts in 192.0.2.0/23 and east side in 192.0.3.0/23 iBGP Advertises 192.0.2.0/22 192.0.2.0/23 Advertises 192.0.2.0/22 192.0.3.0/23 advertises subset of provider space marked NO-EXPORT Router 1Router 2

29. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 28 RFC 2270 Multihoming customer with private AS numbers west side hosts in 96.0.2.0/23 and east side in 96.0.3.0/23 ISP 1ISP 2 advertises subset of provider space marked NO-EXPORT adver- tises Internet routes adver- tises Internet routes Router 2 iBGP Router 1 eBGP Advertises 96.0.2.0/22 96.0.2.0/23 Advertises 96.0.2.0/22 96.0.3.0/23 Remove private AS Remove private AS

30. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 29 Multihoming & DFZ Table Bloat Enterprise 1.0.3.0/24 ISP 1 1.0.0.0/20 ISP 1 2.0.0.0/20 Rest of Internet 1.0.3.0/24 1.0.0.0/20 1.0.3.0/24 2.0.0.0/20 1.0.3.0/24

31. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 30 Scalability Abilities to: —Add more sites –Add more users at large sites –Support telecommuters and road warriors —Add more total users in enterprise —Add new application types —Improve availability when needed/perceived But it all has to be affordable

32. Address Space Issues Rules are always subject to interpretation Ferengi Rule of Acquisition #284

33. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 32 Address Registries Like Efficient Usage Techniques Dynamic addressing — LAN – DHCP – BOOTP — WAN – Local address pools – PPP IPCP – DHCP proxy services Aggregated routing announcements But...

34. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 33 Question: What is the most important machine in the hospital?

35. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 34 Operational Aspects: Dynamic addressing & violations of the end-to-end assumption How do you ping/traceroute? —DHCP/DNS linkage —IPCP linkage —Layer 2 information What about tunnels? What about NAT?

36. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 35 Midboxes: who troubleshoots? Classic NAT PAT/NAPT Packet Filter Frame Filter Stateful Packet Filter Circuit Proxy Application Proxy Traffic-Aware Proxy Content-Aware Proxy Load Sharing NAT Load Aware DNS Application Caches IPsec Tunnels

37. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 36 Efficient Addressing can be Harder to Manage May complicate management — Registry policy (RFC2050) response: life is hard and then you die. So? — Link DNS and dynamic assignment If something is boring and repetitive Use a computer

38. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 37 Question: How many of your customers fill out addressing templates?

39. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 38 Question: How many of your competitors call you mean and nasty for making your customers do things?

40. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 39 Information Gathering: Think Requirements, not Subnetting Number of sites —Schedule for growth Requirements for flow among sites —Degree of application level meshing Backup and recovery Questions meaningful for the customer What is your name? What is your quest? How fast is your sparrow?

41. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 40 Think Requirements, not Subnetting

42. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 41 Schmoozing customers Price for amount of address space Possibly lower overall charges Possibly tie to bandwidth

43. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 42 Categorize Space in Use Both yours and customer

44. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 43 What Addresses do you Need to Manage? Customer assigned blocks POP/Dialup — xDSL, cable, etc. NOT a solved problem Infrastructure — Inter-router links — Server farms – Virtual domains – DNS, DHCP, SNMP, etc. — Inter-AS links

45. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 44 Laws of Customer Address Administration Avoid entering an address more than once Automate configuration updating — TFTP or telnet/expect — Replace vs. merge — Scheduled reboot Remember "the most important machine in the hospital" (M. Python) Document automatically — For troubleshooting — For justifying address allocation

46. Customer Care What impression do you want to give?

47. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 46 What are you offering? Minimal cost residential/SOHO service Business SOHO Large site service Hosting services Serious question: In your business model, how important is customer perception?

48. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 47 Components of Customer Care Sales Pro-active quality Problem reporting Support

49. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 48 What's the difference between used car salesmen and network service sales?

50. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 49 Proactive Quality Hard to define -- "I know it when I see it" Track —Bandwidth utilization —Downtime —Number of support requests and response time —Per-user growth (when known) Inform sales or customer BEFORE critical limit reached —Consider very low-key notifications —Avoid perception of sales pressure —Consider customer-accessible information

51. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 50 Problem Management for SOHO Provide non-labor-intensive status information —when a failure affecting dedicated user access exceeds more than (4?) hours, email to subscribers and/or post something on an internal web —"Technicians are aware of the problem" is minimally helpful —Give customers an idea if they need to go to serious backup

52. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 51 Quality Suggestions Establish a "quality suggestions" email —Use email, not just web, for minimal screening —And possibly get more informative messages Top technical management, and possibly marketing, should routinely read. JW Marriott does.

53. NANOG 21 Customer Satisfaction Tutorial 2/17/2001 52 Trouble Reporting for SOHO & Business Customers are busy —NEVER use music on hold -- people may have you on hold, in conference, etc. —A periodic, professional "you're still in queue," preferably with a waiting time estimate can be useful. —DON'T tell me "it will be just a moment" if it won't. —NEVER NEVER blather about how important my call is. If it were really that important, you would have answered it already. —NEVER NEVER NEVER tell me about new products on the problem reporting line First line support doesn't have all the answers —Don't get defensive or pretend knowledge that isn't there —Provide meaningful ways to get more detailed information