Managing Services and Networks Using a Peer-to-peer Approach Henning Schulzrinne (with Vishal Singh and other IRT members) Dept. of Computer Science Columbia University July 2007

Overview The transition in IT cost metrics End-to-end application-visible reliability still poor (~ 99.5%) –even though network elements have gotten much more reliable –particular impact on interactive applications (e.g., VoIP) –transient problems Lots of voodoo network management Existing network management doesn’t work for VoIP and other modern applications Need user-centric rather than operator-centric management Proposal: peer-to-peer management –“Do You See What I See?” Using VoIP as running example -- most complex consumer application –but also applies to IPTV and other services Also use for reliability estimation and statistical fault characterization

Network management  transition in cost balance Total cost of ownership –Ethernet port cost  $10 –about 80% of Columbia CS’s system support cost is staff cost about $2500/person/year  2 new PCs/year much of the rest is backup & license for spam filters Does not count hours of employee or son/daughter time PC, Ethernet port and router cost seem to have reached plateau –just that the $10 now buys a 100 Mb/s port instead of 10 Mb/s All of our switches, routers and hosts are SNMP-enabled, but no suggestion that this would help at all

VoIP user experience Only % call attempt success –“Keynote was able to complete VoIP calls 96.9% of the time, compared with 99.9% for calls made over the public network. Voice quality for VoIP calls on average was rated at 3.5 out of 5, compared with 3.9 for public-network calls and 3.6 for cellular phone calls. And the amount of delay the audio signals experienced was 295 milliseconds for VoIP calls, compared with 139 milliseconds for public-network calls.” (InformationWeek, July 11, 2005) Mid-call disruptions common Lots of knobs to turn –Separate problem: manual configuration

Issues in automated VoIP diagnosis Increasingly complex and diverse network elements Complex interactions & relationships between different network elements Different run time bindings for each application usage instance –e.g., different calls may use different DNS, SIP proxy servers, media path Problem in one network element may manifest itself as user perceived failure of another element

Circle of blame OS VSP app vendor ISP must be a Windows registry problem  re-install Windows probably packet loss in your Internet connection  reboot your DSL modem must be your software  upgrade probably a gateway fault  choose us as provider

Diagnostic undecidability symptom: “cannot reach server” more precise: send packet, but no response causes: –NAT problem (return packet dropped)? –firewall problem? –path to server broken? –outdated server information (moved)? –server dead? 5 causes  very different remedies –no good way for non-technical user to tell Whom do you call?

VoIP diagnosis What is automated VoIP diagnosis? –Determining failures in network: when, where –Automatically finding the root cause of the failure: why Why VoIP diagnosis? –networks are complex --> difficult to troubleshoot problems –automatic fault diagnosis reduces human intervention Issues in VoIP diagnosis –Detecting failures/faults –Finding the cause of failure, determining dependency relationships among different components for diagnosis Solution steps and approaches

Traditional network management model SNMP X “management from the center”

Old assumptions, now wrong Single provider (enterprise, carrier) –has access to most path elements –professionally managed Problems are hard failures & elements operate correctly –element failures (“link dead”) –substantial packet loss Mostly L2 and L3 elements –switches, routers –rarely APs Problems are specific to a protocol –“IP is not working” Indirect detection –MIB variable vs. actual protocol performance End systems don’t need management –DMI & SNMP never succeeded –each application does its own updates

Management element inspection configuration fault location network understanding we’ve only succeeded here what causes the most trouble?

Managing the protocol stack RTP UDP/TCP IP SIP no route packet loss TCP neg. failure NAT time-out firewall policy protocol problem playout errors media echo gain problems VAD action protocol problem authorization asymmetric conn (NAT)

Types of failures Hard failures –connection attempt fails –no media connection –NAT time-out Soft failures (degradation) –packet loss (bursts) access network? backbone? remote access? –delay (bursts) OS? access networks? –acoustic problems (microphone gain, echo)

Examples of additional problems ping and traceroute no longer works reliably –WinXP SP 2 turns off ICMP –some networks filter all ICMP messages Early NAT binding time-out –initial packet exchange succeeds, but then TCP binding is removed (“web-only Internet”) policy intent vs. failure –“broken by design” –“we don’t allow port 25” vs. “SMTP server temporarily unreachable”

Fault localization Fault classification – local vs. global –Does it affect only me or does it affect others also? Global failures –Server failure e.g. SIP proxy, DNS failure, database failures –Network failures Local failures –Specific source failure, e.g., node A cannot make call to anyone –Specific destination or participant failure, e.g., no one can make call to node B –Locally observed but global failures, e.g., DNS service failed, but only B observed it

Proposal: “Do You See What I See?” Each node has a set of active and passive measurement tools Use intercept (NDIS, pcap) –to detect problems automatically e.g., no response to HTTP or DNS request –gather performance statistics (packet jitter) –capture RTCP and similar measurement packets Nodes can ask others for their view –possibly also dedicated “weather stations” Iterative process, leading to: –user indication of cause of failure –in some cases, work-around (application-layer routing)  TURN server, use remote DNS servers Nodes collect statistical information on failures and their likely causes

Architecture “not working” (notification) inspect protocol requests (DNS, HTTP, RTCP, …) “DNS failure for 15m” orchestrate tests contact others ping can buddy reach our resolver? notify admin ( , IM, SIP events, …) request diagnostics

Solution approach Store context information of past failures experienced by each node –E.g., specific server that was acting as the proxy server (for my call which failed) Store location of past failures instances –LAN, domain, subnet –First hop at each layer e.g., switch (MAC), default gateway (IP), domain’s proxy (application layer), Failure count for each network element (statistical) Last failure timestamp for each network element Last known-good timestamp for each network element –why do I need to test the proxy for you, my call just went through Temporal correlation of past failures –“proxy seems to be failing after DNS fails” Each node has a runtime dependency list based on past failures and diagnostic tests

Solution architecture DNS Server P2P Service Provider 1 Service Provider 2 P1 P2 P3 Domain A P5 P4 P6 P7 P8 DNS Test PESQ Test SIP Server SIP Test Call Failed at P1 Nodes in different domains cooperating to determine cause of failure

Solution architecture: logical view Dependencies encoded as decision tree, static and dynamic rules Admin input [Dependency relationships and tests (XML) ] Triggers to perform TESTS. (Peer selection and Probe selection. Alerts Dependency graph generation [Bayesian network based, Inference, other models ] Failures in Network Decision Tree updates Test results The above figure shows logical entities and separation of dependency graph generation and distributed diagnostic infrastructure (enclosed in blue).

Solution requirements Request-response protocol between the node which experiences the failure and the peer nodes Nodes needto perform diagnostic tests (probes), probe selection based on cost/result Encoding the dependency relationship into a decision tree –from an expert e.g., as XML Peer node discovery, based on –Location (local network, domain) –Capability to perform tests (based on specific tests) Dependency graph generation and updation, based on –Network failure events –Diagnostic test results correlated with failures

Failure detection tools STUN server –what is your IP address? ping and traceroute Transport-level liveness and QoS –open TCP connection to port –send UDP ping to port –measure packet loss & jitter TBD: Need scriptable tools with dependency graph –initially, we’ll be using ‘make’ TBD: remote diagnostic –fixed set (“do DNS lookup”) or –applets (only remote access) media RTP UDP/TCP IP

Test & probe selection Which diagnostic probe to run? network layer or application layer and for what kind of failures. A probe covering broad range of failures can give faster but less accurate results e.g., ping vs. TCP connect vs. SIP OPTIONS tests Cost of probing –messages –CPU overhead

Dependency classification Functional dependency –At generic service level e.g., SIP proxy depends on DB service, DNS service Structural dependency –Configuration time e.g., Columbia CS SIP proxy is configured to use mysql database on host metro-north Operational dependency –Runtime dependencies or run time bindings e.g., the call which failed was using failover SIP server obtained from DNS which was running on host a.b.c.d in IRT lab

Dependency classifications: layered approach Vertical and lateral dependencies –application depends on other application layer services e.g., SIP service depends on DB, DNS service as well as lower layer services –OSI layers as service dependency layers Application layer service also depends on transport layer service which in turn depends on network layer service –MAC layer: access point, switch –Network layer: router –Application layer: DNS, SIP, database Topology-based dependency –e.g., calls from CS domain depends on specific SIP server –calls from lab phones depends on specific switches and routers

Dependency Graph

Dependency Graph Encoded to Decision Tree A C B D A Failed, Use Decision Tree Yes Invokes Decision Tree for C No Yes Invokes Decision Tree for B Invokes Decision Tree for D Cause Not Known Report, Add new Dependency A B C D A = SIP Call C = SIP Proxy B = DNS Server D = Connectivity

Diagnostic tests - a bit more detail SIP proxy –Proxy server availability SIP PING –Call routing availability INVITE tests –Call path determination SIP TraceRoute Media path –Quality related Speech quality degradation - MOS Echo jitter- MOS, PESQ QoS – RTCP –NAT/Firewall Checking binding expiration. Firewall failure to open a port - One way media. –which firewall in the path? SIP signaling ?

VoIP example Call failure – possible causes –SIP proxy server database authentication –Media path failure Gateway –Specific call legs – ERL, authentication, etc. –DNS server failure –End station failure –Network failure, e.g., router, switch failure Different calls will have different run time dependencies

Diagnostic tests, cont’d DNS tests DHCP Switch/router –ARP/RARP/multicast –BGP failures Conference mixers Gateway –Echo return loss- readings- analysis DB XCAP server tests Presence service availability tests

Current work Building decision tree system Using JBoss Rules (Drools 3.0)

Future work Learning the dependency graph from failure events and diagnostic tests Learning using random or periodic testing to identify failures and determine relationships Self healing Predicting failures Protocols for labeling event failures --> enable automatically incorporating new devices/applications to the dependency system Decision tree (dependency graph) based event correlation

Failure statistics Which parts of the network are most likely to fail (or degrade) –access network –network interconnects –backbone network –infrastructure servers (DHCP, DNS) –application servers (SIP, RTSP, HTTP, …) –protocol failures/incompatibility Currently, mostly guesses End nodes can gather and accumulate statistics

Conclusion Hypothesis: network reliability as single largest open technical issue  prevents (some) new applications Existing management tools of limited use to most enterprises and end users Transition to “self-service” networks –support non-technical users, not just NOCs running HP OpenView or Tivoli Need better view of network reliability