1. CMPE 252A : Computer Networks
Chen Qian
Computer Engineering
UCSC Baskin Engineering
Lecture 13
Some slides from Eugene Ng

2. Survey paper due
11/21
At the beginning of class, hardcopy or softcopy via to both me and TAs.

3. Global Network Positioning: A New Approach to Network Distance Prediction
Tze Sing Eugene Ng

4. New Challenges Large-scale distributed services and applications
Napster, Gnutella, End System Multicast, etc
Large number of configuration choices
K participants  O(K2) e2e paths to consider
Stanford
MIT
CMU
Berkeley
Stanford
MIT
CMU
Berkeley
Stanford
MIT
CMU
Berkeley

5. Role of Network Distance Prediction
On-demand network measurement can be highly accurate, but
Not scalable
Slow
Network distance
Round-trip propagation and transmission delay
Relatively stable
Network distance can be predicted accurately without on-demand measurement
Fast and scalable first-order performance optimization
Refine as needed

6. Applying Network Distance
Napster, Gnutella
Use directly in peer-selection
Quickly weed out 95% of likely bad choices
End System Multicast
Quickly build a good quality initial distribution tree
Refine with run-time measurements
Key: network distance prediction mechanism must be scalable, accurate, and fast

7. Global Network Positioning (GNP)
Model the Internet as a geometric space (e.g. 3-D Euclidean)
Characterize the position of any end host with coordinates
Use computed distances to predict actual distances
Reduce distances to coordinates
(x2,y2,z2) y
(x1,y1,z1) x z
(x4,y4,z4)
(x3,y3,z3)

8. Landmark Operations y x
(x2,y2)
(x1,y1)
(x3,y3) L1 L2 L3 L1 L2 L3
Small number of distributed hosts called Landmarks measure inter-Landmark distances
Internet
Compute Landmark coordinates by minimizing the overall discrepancy between measured distances and computed distances
Cast as a generic multi-dimensional global minimization problem

9. Landmark Operations
Landmark coordinates are disseminated to ordinary end hosts
A frame of reference
e.g. (2-D, (L1,x1,y1), (L2,x2,y2), (L3,x3,y3))
T. S. Eugene Ng Carnegie Mellon University

10. Ordinary Host Operations
(x2,y2) L2
(x1,y1)
(x4,y4) L1 L1 L3 x L2
Internet L3
(x3,y3)
Each ordinary host measures its distances to the Landmarks, Landmarks just reflect pings
Ordinary host computes its own coordinates relative to the Landmarks by minimizing the overall discrepancy between measured distances and computed distances
Cast as a generic multi-dimensional global minimization problem

11. GNP Advantages High scalability and high speed Enable new applications
End host centric architecture, eliminates server bottleneck
Coordinates reduce O(K2) communication overhead to O(K*D)
Coordinates easily exchanged, predictions are locally and quickly computable by end hosts
Enable new applications
Structured nature of coordinates can be exploited
Simple deployment
Landmarks are simple, non-intrusive (compatible with firewalls)

12. Evaluation Methodology
19 Probes we control
12 in North America, 5 in East Asia, 2 in Europe
Select IP addresses called Targets we do not control
Probes measure
Inter-Probe distances
Probe-to-Target distances
Each distance is the minimum RTT of 220 pings

13. Evaluation Methodology (Cont’d)
Choose a subset of well-distributed Probes to be Landmarks, and use the rest for evaluation T T P2
(x1,y1) P1 T T P3
(x2, y2) P4 T T

14. Computing Coordinates
Multi-dimensional global minimization problem
Will discuss the objective function later
Simplex Downhill algorithm [Nelder & Mead ’65]
Simple and robust, few iterations required
f(x) x

15. Data Sets Global Set 19 Probes
869 Targets uniformly chosen from the IP address space
biased towards always-on and globally connected nodes
44 Countries
467 in USA, 127 in Europe, 84 in East Asia, 39 in Canada, …, 1 in Fiji, 65 unknown
Abilene Set
10 Probes are on Abilene
127 Targets that are Abilene connected web servers

16. Performance Metrics Directional relative error
Symmetrically measure over and under predictions
Relative error = abs(Directional relative error)
Rank accuracy
% of correct prediction when choosing some number of shortest paths

17. GNP vs IDMaps (Global)

18. GNP vs IDMaps (Global)

19. Basic Questions How to measure model error? How to select Landmarks?
How does prediction accuracy change with the number of Landmarks?
What is geometric model to use?
How can we further improve GNP?

20. Measuring Model Error is measured distance is computed distance
is an error measuring function

21. Error Function Squared error
May not be good because one unit of error for short distances carry the same weight as one unit of error for long distances

22. More Error Functions
Normalized error
Logarithmic transformation

23. Comparing Error Functions
6 Landmarks
15 Landmarks
Squared Error
1.03
0.74
Normalized Error
0.5
Logarithmic Transformation
0.75
0.51

24. Selecting N Landmarks Intuition: Landmarks should be well separated
Method 1: Clustering
start with 19 clusters, one probe per cluster
iteratively merge the two closest clusters until there are N clusters
choose the center of each cluster as the Landmarks
Method 2: Find “N-Medians”
choose the combination of N Probes that minimizes the total distance from each not chosen Probe to its nearest chosen Probe
Method 3: Maximum separation
choose the combination of N Probes that maximizes the total inter-Probe distances

25. K-Fold Validation
Want more than just one set of N Landmarks to reduce noise
Select N+1 Landmarks based on a criterion
Eliminate one Landmark to get N Landmarks
i.e., N+1 different sets of N Landmarks that are close to the selection criterion

26. Comparing Landmark Selection Criteria (6 Landmarks)
Clustering
N-Medians
Max sep.
GNP
0.74
0.78
1.04
IDMaps
1.39
1.43
5.57

27. Comparing Landmark Selection Criteria (9 Landmarks)
Clustering
N-Medians
Max sep.
GNP
0.68
0.7
0.83
IDMaps
1.16
1.09
1.74
T. S. Eugene Ng Carnegie Mellon University

28. Number of Landmarks/Tracers
T. S. Eugene Ng Carnegie Mellon University

29. What Geometric Model to Use?
Spherical surface, cylindrical surface
No better than 2-D Euclidean space
Euclidean space of varying dimensions

30. Euclidean Dimensionality

31. Why Additional Dimensions Help?
ISP A B C D 1 5
2-dimensional model
A,B
C,D A B C D
3-dimensional model 1 5
A A B C D A B C D

32. Reducing Measurement Overhead
Hypothesis: End hosts do not need to measure distances to all Landmarks to compute accurate coordinates P1 P3 P2 P5 T P6
(x, y) P4

33. Triangular Inequality Violations
T. S. Eugene Ng Carnegie Mellon University

34. Removing Triangular Inequality Violations
Remove Target (t) from data if
t in {a, b, c}
(a,c)/((a,b)+(b,c)) > threshold
Try two thresholds
2.0; of 869 Targets remain
1.5; of 869 Targets remain
Note: at 1.1, only 19 of 869 Targets remain!!!

35. Removing Triangular Inequality Violations
T. S. Eugene Ng Carnegie Mellon University

36. Removing Triangular Inequality Violations
T. S. Eugene Ng Carnegie Mellon University

37. Removing Triangular Inequality Violations
T. S. Eugene Ng Carnegie Mellon University

38. Why Not Use Geographical Distance?
T. S. Eugene Ng Carnegie Mellon University

39. Summary
Network distance prediction is key to performance optimization in large-scale distributed systems
GNP is scalable
End hosts carry out computations
O(K*D) communication overhead due to coordinates
GNP is fast
Distance predictions are fast local computations
GNP is accurate
Discover relative positions of end hosts
T. S. Eugene Ng Carnegie Mellon University

40. Vivaldi: A Decentralized Network Coordinate System
Frank Dabek, Russ Cox, Frans Kaashoek, Robert Morris
Presented by:
Chen Qian

41. A Solution
Synthetic coordinate systems allow Internet hosts to predict the RTTs to any other hosts.
The distance between the coordinates of two hosts should be an accurate predictor of the RTT.
These systems can be constructed by each host only communicating with a small set of other hosts.

42. Vivaldi
Vivaldi is a simple, adaptive, de-centralized algorithm for computing network coordinates.
No low-dimensional coordinate space would predict RTTS exactly.
Internet latencies violate the triangle inequality.
Vivaldi introduces the notion height that improves the prediction accuracy.

43. Prediction Error Where Lij: the actual RTT between nodes i and j
xi: the coordinates assigned to node i
||xi-xj||: the distance between the coordinates of i and j
Minimizing the squared-error function is equivalent to minimizing the energy in a physical mass-spring network.

44. Centralized Algorithm
Tries to minimize the error of predicted RTT values by simulating the movements of nodes under spring forces.
100 N1 N2
A single spring at rest
150 N1 N2
longer spring 50 N1 N2
shorter spring

45. Algorithm By Hook’s Law:
Scalar quantity: the displacement of the spring from rest
Unit vector which gives the direction of the force on i.
Force vector Fij can be viewed as an error vector, which has a direction

46. Local minimum
But the global minimum is not guaranteed. The system may come to rest in a local minimum. N1 N2
local minimum N3 N4 N5

47. Local minimum
But the global minimum is not guaranteed. The system may come to rest in a local minimum. N1 N2
lower error N3 N5 N4

48. Centralized Algorithm
Calculate sum of forces on node i
Move a step in the direction of the sum of forces

49. Simple Distributed Version
Continuously contact sample nodes
For each sample node
Calculate force (error change) of this sample
Move a step in the direction of the error

50. Coordinates update
Identical to the individual forces calculated in the loop of the centralized algorithm

51. Adaptive Timestep
The main difficulty in implementing Vivaldi is ensuring that it converges to coordinates that predict RTT well.
If the timestep is too small, convergence is slow.
If the timestep is too large, convergence may fail.
optimal
optimal

52. Adaptive Timestep
The system should obtain both fast convergence and avoidance of oscillation.
Simple adaptive timestep
Adaptive timestep to deal with large errors
If the remote node has a large error, it should be given less weight than a remote node with small error.

53. Algorithm with adaptive timestep
Compute error confidence
Update local error
Adjust time step

54. Evaluation Methodology
Latency data
Matrix of inter-host Internet RTTs
Compute coordinates from a subset of these RTTs
Check accuracy of algorithm by comparing simulated results to full RTT matrix
4 Data sets (2 Measured, 2 Synthetic)
192 nodes Planet Lab network, all pair-ping gives fully populated matrix
1740 Internet DNS servers
Collect full matrix using the King method
Continuously measure pairs over a week and take the median value
More geographically diverse at that time

55. King’s method
First DNS query is for a name in the domain of A. It returns the latency to A.
Second query is for a name in the domain of B, but is sent initially to A.
The difference between two queries is the latency between A and B

56. King’s method
Take the median value, because King can report a RTT higher or lower than the true value if there is congestion.
About 10% of the original nodes were removed from the data
High load or queuing at name server A adds a delay that is significantly larger than the network latency.
The initial query (to A) and recursive query (via A to B) will require roughly the same amount of time and the estimated latency between them will be near zero.

57. Setup Simulation test setup Input RTT matrix
Send a packet one a second
Simulator delays each transmission by ½ RTT time
Use measured RTT of the packets to update coordinates
Limitation of the simulator: RTTs do not vary over time; cannot model queuing delay or changes in routing

58. Setup Error definitions Error of Link
Absolute difference between predicted RTT and measured RTT.
Error of Node
Median of link errors involving this node
Error of System
Median of all node errors
A small proportion of nodes have large errors?

59. Timestep choice
(a)Constant timestep: too small and too large values all cause large errors.
(b)Adaptive timestep: c=0.25 yields both quick error reduction and low oscillation.

60. Timestep choice 200 new nodes join a stable 200-node network
Constant timestep, new nodes may confuse the old nodes. The system need to be re-converged.
Timestep with weighted errors allows new nodes to find their places quickly.

61. Communication pattern
Sampling only nearby nodes gives good local coordinates but poor global coordinates.
The second case allow nodes to contact distant nodes as well, improving the accuracy of the coordinates.

62. Communication pattern
Put 4 close neighbors and 4 far-away neighbors. Each node chooses one of the far neighbors with probability p.
p = .5 quick convergence
p < .5 convergence slows. But similar accurate coordinates are eventually chosen.

63. Adapting to network changes
Ability to adapt to changes in the network (tested with “Transit-Stub”)
At time 100 one of the transit stub links is made 10 time larger; after 20 s the system has re-converged. At time 300 the link goes back to its normal size and the system quickly re-converged to original error.

64. Accuracy: Vivaldi vs. GNP
How about communication cost?

65. Model Selection
Almost any coordinate space satisfies the triangle inequality (the distance between A and C should be less than or equal to the distance along the path A-B-C).
100 ms N1 N3
Not always true in Internet
48 ms
48 ms N2

66. Triangle inequality
The best indirect path usually has lower RTT than the direct path.
But luckily only 5% pairs have a significant shorter indirect path.

67. Euclidean Spaces
If geographic distance were the only factor in latency, a 2-D model would be sufficient. However, the fit is not perfect. Adding more dimensions, the accuracy of the fit improves slightly
3D is okay!

68. Spherical coordinates
Does a spherical distance function provide a more accurate model, as the distances are drawn from paths along the surface of the Earth?
No!

69. 2D+Height
The Euclidean portion models a high-speed Internet core with latencies proportional to geographic distance.
The height models the time it takes packets to travel the access link from the node to the core.
The cause of the access link latency may be queuing delay, low bandwidth, etc.
A packet sent from one node to another must travel the source node’s height, then travel in the Euclidean space, then travel the destination node’s height.

70. 2D+Height Performs better than 2D and 3D!
Does not look very promising because they take the median!

71. 2D+Height Nodes with large errors
Height plots results smaller max error and median error

72. Conclusion
Presents a simple, adaptive, decentralized algorithm for computing synthetic coordinates, which help Internet hosts to estimate latencies
Requires no fixed infrastructure. All nodes run the same algorithm.
Converges quickly by adaptive timestep.
Maintains accuracy even as a large number of new hosts join the network that are uncertain of their coordinates.