Scalable Management for Networks and Services Rolf Stadler Laboratory for Communication Networks KTH Royal Institute of Technology Stockholm HP Laboratories, Palo Alto, March 31, 2003

node Management station Management station Manager-Agent based management node AAA M results download & execute P Management Program Manager Agent Centralized Control Management protocols: SNMP, CMIP Program runs on Management Station Decentralized Control Program runs on network nodes P The Shift of a Management Paradigm A P A A

Router Execution Environment Management station Management Program navigation Code Server Architecture for Pattern-based Management

Weaver—A Testbed for pattern-based Management WAN A WAN B WAN C WAN D Management Station Router ARouter BRouter CRouter D FastEthernet Switch

Simple Navigation Patterns

Echo Pattern (expansion)

d root =1 Echo Pattern (expansion)

d root =2 Echo Pattern (expansion)

d root =3 Echo Pattern (expansion)

d root =4 Echo Pattern (expansion)

Echo Pattern d root =5

d root =4 Echo Pattern (contraction)

d root =3 Echo Pattern (contraction)

d root =2 Echo Pattern (contraction)

d root =1 Echo Pattern (contraction)

The Echo Pattern Two phases of traversal –expansion phase: explorers flood network with requests for local operations –contraction phase: echoes return and aggregate results Properties –Generates balanced traffic load –Traffic load depends on network topology, not on speed of traversal –Time complexity increases linearly with network diameter.

Examples of Echo-based Management Get information on topology –compute the current number of leaf nodes, the connectivity distribution –discover current topology within 10 hops of node x Get information on network state –identify 10 most congested links –compute distribution of link utilization, queue lengths –identify sub topologies with highly loaded links –find a resource R closest to node x

Pattern-based Management— An Engineering Approach to Decentralized Management A management program consists of –A navigation pattern (distr. graph traversal algorithm) –An operation on nodes –An aggregation function Relevance of this approach –Provides a basis to analyze management operation for performance, scalability, robustness –Supports concept of re-usable patterns, hides complexity

Composing Management Programs Segall Echo Patterns Navigation Patterns Chang Skip WaitScopeMulti Echo Aggregators Res. Disc. Aggregators Leaf Count Load. Hist. Conn. Hist. CLIHTTP XML SNMP Local Operations Node Access Management Program

Properties of Patterns Echo Aggregators Res. Disc. Aggregators Leaf CountLoad. Hist.Conn. Hist. CLIHTTP XML SNMP Node Access Management Program Segall Simple Echo Robust EchoOthers Echo Patterns Navigation Patterns Chang Skip Wait Scope Multi A pattern can be used for many management operations. A pattern can be chosen according to performance objectives. A pattern hides the complexity of a distributed operation. Network failures can be handled within patterns. Code mobility can be controlled.

visited i : boolean init false; G i : set of integers init neighbors(); parent i : integer init -1; Echo(inmsg: bytes, from: integer) { G i := G i - from; if visited i = false { parent i := from; visited i := true; OnInitiate(inmsg, outmsg); if G i != empty dispatch(parent i, outmsg, i); } else OnAggregate(inmsg); if G i = empty { OnComplete(outmsg); if parent i >= 0 dispatch(parent i, outmsg, i); else OnTerminate(inmsg); } The Interface between Pattern and Aggregator OnBegin OnTerminate OnInitiate OnComplete OnAggregate … av_load i := av_load; … av_load := load(); n:=1; … av_load := (av_load*n + av_load j )/(n+1); n:=n+1; …

SIMPSON: A SIMple Pattern Simulator fOr Large Networks

Analyzing Management Operations Network Graph G=(V,E)Execution Graphs G’=(V’,E’) Centralized ManagementDistributed Management Star PatternEcho Pattern

Traffic Complexity of Management Operations Amount of traffic placed on the network during execution.

Time Complexity of Management Operations Time needed from invocation until completion of a operation.

Performing Echo-based Operations on the Entire Internet Purpose is illustrating the scalability of echo-based operations. What we needed: –Complexity analysis of pattern –Estimation of Internet topological properties diameter connectivity distribution number of nodes

Estimated Performance of Echo-based Operation on the Internet Assumptions: Process-level transmission time: 5ms Network delay per hop: 4ms Message size: 1KB Local operation: 500ms per execution Diameter of Internet: 34 hops

Active Node Manager Source Repository Binaries Repository Preprocessor Transport Access Point Execution Environment Management Operation Results Device Manager C++ Compiler Source, State SNMP sets Management commands Weaver Active Node Source code, Active node management Router Node State Local Program States Source, State Events SNMP gets/traps Source Code results Active Node Engine Management Station

Suboperations in Weaver Node ANode B start Execution (T1) Serialization (T2) Dispatch (T3) Receiving (T4) Loading (T5) or Instantiation (T6) De-serialization (T7) Execution (T1) Serialization (T2) Dispatch (T3) Receiving (T4) Resolving (T8) end Time De-serialization (T7) Execution (T1) T C1 T C2

Measuring Execution Times on Weaver Duration in msPerformed by Module Execution (T1)1.57 (σ = 0.48)Execution Environment Serialization (T2)3.46 (σ = 0.71)Execution Environment Dispatch (T3)1.67 (σ = 0.49)Transport Access Point Receiving (T4)0.62 (σ = 0.30)Transport Access Point Loading (T5)23.42 (σ = 0.70)Execution Environment Instantiation (T6)0.77 (σ = 0.015)Execution Environment De-serialization (T7)2.04 (σ = 0.49)Execution Environment Resolving (T8)0.15 (σ = 0.001)Execution Environment Communications Delay (T C )4.04 (σ = 0.10)---

Estimating Execution Times of Echo-based Operations on Weaver

Skip Echo SkipEcho(inmsg: bytes from: integer) { if visited i = false { parent i := from; visited i := true; OnInitiate(inmsg, outmsg, i); G i = up_neighbors() - from; if G i != empty dispatch(parent i, outmsg, i); } else { G i = G i - from; OnAggregate(inmsg); } if complete i != true and G i = empty { OnComplete(outmsg); completed i := true; if parent i >= 0 dispatch(parent i, outmsg, i); else OnTerminate(inmsg); } alarm(type: {failure, recovery}, affected: integer){ if visited i = true { if type = failure { G i := G i - affected if complete i != true and G i = empty { complete i := true; OnComplete(outmsg); if parent i >= 0 dispatch(parent i, outmsg, i); else OnTerminate(inmsg); } Wait Echo SkipEcho(inmsg: bytes from: integer) { if visited i = false { parent i := from; visited i := true; OnInitiate(inmsg, outmsg, i); G i = up_neighbors() - from; if G i != empty dispatch(parent i, outmsg, i); } else { G i = G i - from; OnAggregate(inmsg); } if complete i != true and G i = empty { OnComplete(outmsg); completed i := true; if parent i >= 0 dispatch(parent i, outmsg, i); else OnTerminate(inmsg); } alarm(type: {failure, recovery}, affected: integer){ if visited i = true { if type == failure { G i = G i - affected B i = B i + affected if complete i != true and G i = empty { complete i := true; OnComplete(outmsg); if parent i >= 0 dispatch(parent i, outmsg, i) else OnTerminate(inmsg); } } else { if affected is in B i { B i = B i - affected G i = G i + affected } Designing Robust Patterns Plain Echo Echo(inmsg: bytes, from: integer) { Gi := Gi - from; if visitedi = false { parenti := from; visitedi := true; OnInitiate(inmsg, outmsg); if Gi != empty dispatch(parenti, outmsg, i); } else OnAggregate(inmsg); if Gi = empty { OnComplete(outmsg); if parenti >= 0 dispatch(parenti, outmsg, i); else OnTerminate(inmsg); }

Network Coverage vs. Execution Time for Skip Echo MTTR=1 min MTTR=11 min MTTR  0 MTTR  inf MTTF = 3.6 hrs MTTF = 7.3 hrs MTTF = 11.0 hrs MTTF = 14.7 hrs MTTF = 29.4 hrs MTTF = 73.6 hrs

Current and Planned Work Self-organizing, adaptable Networks and Systems: Patterns for routing and dynamic construction of network control structures. (Constantin Adam) WQL: A table-based Network Query Language on Weaver. (Koon-Seng Lim) Policy-based Management: Patterns for distribution and dynamic re-computation of policies. (Alberto Gonzalez)

Literature on this Work K.S. Lim, R. Stadler: “Weaver—Realizing a scalable management paradigm on commodity routers,” Eighth IFIP/IEEE International Symposium on Integrated Network Management (IM 2003), Colorado Springs, Colorado, USA, March 24-28, 2003.Weaver—Realizing a scalable management paradigm on commodity routers K.S. Lim and R. Stadler: "Developing pattern-based management programs," IFIP/IEEE International Conference on Management of Multimedia Networks and Services (MMNS 2001), Chicago, IL, October 29 - November 1, 2001.Developing pattern-based management programs K.S. Lim and R. Stadler: "A navigation pattern for scalable Internet management,"IFIP/IEEE International Symposium on Integrated Network Management (IM 2001), Seattle,Washington, May, 2001.A navigation pattern for scalable Internet management R. Kawamura and R. Stadler: "A middleware architecture for active distributed management of IP networks, "IEEE/IFIP Network Operations and Management Symposium (NOMS 2000), Honolulu, Hawaii, April 10-14, 2000.A middleware architecture for active distributed management of IP networks