1. Sylvia Ratnasamy Intel Research
Capturing Complexity in Networked Systems Design: The Case for Improved Metrics
Sylvia Ratnasamy
Intel Research

2. Motivation Our community values “simple”, “clean” system designs
“The advantage of Chord is that it is less complicated…” – Chord, Sigcomm’01
“The complexity of the architectural design…” – Sprint, IEEE Network 2000
“A design for multicast that is simple…” – Perlman. Simple Multicast, IETF Draft.
“Complex routing algorithms may have difficulty scaling… simple floods are sufficient” TinyOS, NSDI’04
“This paper proposed a simple and effective approach …” – SOSR, OSDI’04
“Thus, simplicity is our primary design requirement …” – BVR, NSDI’05

3. Motivation Our community values “simple”, “clean” system designs
But lacks metrics that rigorously capture this aesthetic
Best practice: metrics borrowed from the theory community
e.g., total_messages, total_state
Problem: at times incongruent with our notion of “simplicity”
e.g., flooding: inefficient but simple
e.g., dijkstra’s: O(n) state, but simple
e.g., leaderID vs. neighborID

4. This Paper Can we quantify design “simplicity”?
more rigorously compare-and-contrast design options
align design goals of algorithms, systems communities
First stab: complexity metrics for the algorithmic component of a networked system
Note
just one aspect to system complexity
intent: complement, not replace, existing metrics
Two things to point out…first that focus on proc spec just one aspect that contributes to complexity. Two addtnl aspects that seem at least as important are: selecting the service model and something like the CAP theorem starts to formal reason about this. The other is the architecture of the software implementation. These are crucial but not what’s considered here.
Second is that the intent of complexity metrics is to complement, …. Still need metrics that tell you how good your soln is at the job you want done.

5. Outline Strawman Simple evaluation scenarios
Limitations and Future Directions

6. Strawman Two observations:
much of system design centers around state
what state is required?
how is it constructed/maintained?
how is it used?
what distinguishes wide-area state:
derived from remote nodes  dependencies are distributed
relayed via intermediate nodes  more dependencies
Current metrics mostly treat all state as equal

7. Proposal
For a given piece of state s, measure the ensemble of distributed dependencies that yield s
First cut: measure  counting
akin to existing metrics: #msgs, #state
amenable to evaluation by simple examination, simulation

8. Value vs. Transport Dependencies
next(B)
next(B)
next(B)
next(B) A R1 R2 R3 B
x=30°
s=x
s is value dependent on x,
s is transport dependent on next(B) state at A, R1, R2, R3.
vs = #pieces of state on which s is value dependent
tsx = #pieces of state relied on to transport x to s

9. Counting Dependencies
next(B)
next(B)
next(B)
next(B) A R1 R2 R3 B
x=30°
s=x
vx = txx = 0
vs = 1 ; tsx = 4

10. Counting DependenciesR4 A R1 R2 R3 B
y=20°
x=30+y°
s=x
vy = tyy = 0
vx = 1
txy = 2
vs = 2 tsx = 4
Note: value dependencies accumulate
Add something about the manner in which they accumulate.

11. Dependencies  ComplexityR4 A R1 R2 R3 B
y=20°
x=y+30°
s=x
vs = 2 , tsx = 4
vy = tyy = 0
vx = 1 txy = 2
cy = 0
cx = = 2
cs = = 10
Value dependencies accummulate and interact with transport dependencies. So how do they lead to complexity.
single input: state s, derived from x
complexity of s, cs = (vx + 1) tsx + cx

12. Dependencies  Complexity… A B
x=0
s=1
s=2
s=i
s=d
vx = i tii-1 = 1
vs = d , tss-1 = 1
c = 0
cs = 1
cs = 3
cs = O(d2)
single input: state s, derived from x
complexity of s, cs = (vx + 1) tsx + cx

13. Dependencies  Complexity
multiple inputs: state s, derived from x1, x2, …, xm
two basic variants:
s = ALL (xi) cs = Σ ((vxi + 1) tsxi + cxi)
if vxi = cxi = 0, tsxi = 1, then cs = m
s = ANY (xi) cs = 1/m Σ ((vxi + 1/m) tsxi + cxi)
if vxi = cxi = 0, tsxi = 1, then cs = 1/m

14. Dependencies  Complexity
multiple paths: state s derived from x, multiple paths from x to s
simple case: m disjoint paths, each incurring d transport dependencies
tsx = d/m, cs = vs tsx + cx
if vx = cx = 0, then cs = d/m

15. Complexity: toy scenariosvs cs
s=x; s,x are 1 hop apart 1
O(1)
s=x; s,x are k hops apart
O(k)
s=distance(x); s,x k hops apart k
O(k2)
s=ALL(x1, x2, …, xm) s, xi are 1 hop apart m
O(m)
s=ANY(x1, x2, …, xm)
s, xi are 1 hop apart
1/m
O(1/m)
s=x m distinct 1-hop paths from x to s

16. Preliminary Evaluation (summary)
system vs cs
croute
distance-vector
O(d)
O(d2)
O(d3)
link-state
O(1)
compact [SPAA04]??
O(n)
> O(nd)
> O(nd2 )
tree-based
O(n1/2)
O(n3/2)
geo-routing
O()
O(n)
virtual-geo [mobicom03]
O(n5/2)
Few more results – no detail, just summary.
First 3, routing algorithms for general graphs. Algo from spaa’04 as representative of a large body of work on “”.
Basically, we see that compact is sig > than dv/ls. This is contrary to what #state would suggest. And actually is in agreement with what is deployed.
Next 3, rting from sensor networks. Tree-based, geo-based that has the advantage of.. But limited in applicability which has led to a fair bit of work on virtual-geo that essentially construct synthetic coors. This is one from mobicom’03.
Note that vg > tree mostly due to the complexity of the bootstrap phase that constructs the virtual
Coor sys. This despite state/overhead advantage. Again, in keeping with… Also note, geo looks great. So, a generally-applicable geo-routing that maintains low complexity would be the ideal. Mobicom’03 is not it. Future work to look at alternate proposals.

17. Limitations, Future Directions
scope: no validation of assumptions, correctness, quality, performance, etc.
correlated inputs
discriminating transport dependencies
capturing absent dependencies
capturing robustness
count “reverse” dependencies?
role of time
evaluating the metric: future studies

18. Summary design simplicity: valued, but poorly measured
question: is design complexity measurable?
conjecture: capturing dependencies in state is key (w.r.t. algorithmic component of a system)

19. Thanks! Questions?

20. complexity of operationsB A
x=0
s=1
s=2
s=d
s=i …
cs = O((d-1)2) O(d2)
complexity of forwarding operation: cfwd = O(d3)
an operation acts on different pieces of state
each piece of state has a complexity
compute complexity of an operation by (appropriately) combining complexity of state it acts on

21. complexity of operations
cs = k 1
cs = k 2
cs = k 3
fetch(obj) …
cs = k m
complexity of fetch: cfetch = O(k/m)
Check.
an operation acts on different pieces of state
each piece of state has a complexity
compute complexity of an operation by (appropriately) combining complexity of state it acts on

22. Future Studies centralized solutions simpler? e.g., RCP vs. DV
designing for low dependency
e.g., soft state-based approaches
layered vs. customized
e.g., PHTs [sigcomm’05] vs. Mercury [sigcomm’04]
different routing options
mesh (RON, ESM), DHT-inspired (VRR/ROFL), centralized (RCP), compact routing-based, landmark routing, coordinate-based, …