1. Network Lasso: Clustering and Optimization in Large Graphs
David Hallac, Jure Leskovec, Stephen Boyd
Stanford University
Presented by Yu Zhao

2. What is this paper about
Lasso problem
The lasso solution is unique when rank(X) = p, because the criterion is strictly convex.

3. What is this paper about
Network lasso problem
The variables are , where (The total number of scalar variables is mp.) Here is the variable at node i, is the cost function at node i, and is the cost function associated with edge

4. Outline Convex problem definition Proposed solution(ADMM)
Non-convex extension
Experiments

5. Convex problem definition
(1)
(2)

6. Convex problem definition
A distributed and scalable method was developed for solving the network lasso problem, in which each vertex variable xi is controlled by one “agent”, and the agents exchange (small) messages over the graph to solve the problem iteratively.

7. Convex problem definition
General settings for different applications
e.g. Control system:
Nodes: possible states
xi: actions to take when state i
Graph: state transitions
Weights: how much we care about the actions in neighboring states differing

8. Convex problem definition
General settings for different applications
The sum-of-norms regularization that we use is like group lasso, which encourages not just , for edge , but , consensus across the edge.

9. Convex problem definition
Regularization Path
𝜆=0: simply a minimizer of 𝑓𝑖  local computations
𝜆 →∞ (𝜆 ≥𝜆𝑐𝑟𝑖𝑡𝑖𝑐𝑎𝑙):

10. Convex problem definition
Network lasso and clustering
𝑙2-norms penalty defines network lasso.
Cluster size: 𝜆

11. Convex problem definition
Inference on New Nodes
we can interpolate the solution to estimate the value of 𝑥𝑖 on a new node 𝑗.

12. Proposed solution(ADMM)
Alternating Direction Method of Multipliers(ADMM)
S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning, 3:1–122, 2011.

13. Proposed solution(ADMM)
ADMM in network lasso
1). Introduce a copy of 𝑥𝑖, called 𝑧𝑖𝑗 , at each edge 𝑖𝑗.

14. Proposed solution(ADMM)
ADMM in network lasso
2). Augmented Lagrangian
M. R. Hestenes. Multiplier and gradient methods. Journal of Optimization Theory and Applications, 4:302–320, 1969.

15. Proposed solution(ADMM)
ADMM in network lasso
3). ADMM updates

16. Proposed solution(ADMM)
Regularization Path
compute the regularization path as a function of 𝜆 to gain insight into the network structure
Start at 𝜆= 𝜆𝑖𝑛𝑖𝑡𝑖𝑎𝑙
update: 𝜆 ≔ 𝛼𝜆, 𝛼>1
stop:

17. Non-convex extension
replace the group lasso penalty with a monotonically nondecreasing concave function 𝜙(𝑢), where 𝜙 𝑢 =0 , and whose domain is u𝑢 ≥0,
ADMM is not guaranteed to converge, and even if it does, it need not be to a global optimum

18. Non-convex extension Heuristic solution:
to keep track of the iteration which yields the minimum objective, and to return that as the solution instead of the most recent step.

19. Non-convex extension Non-convex z-Update
Compared to the convex case, the only difference in the ADMM solution is the z-update, which is now

20. Experiments Idea: 1. Network-Enhanced Classification
We first analyze a synthetic network in which each node has a support vector machine (SVM) classifier,
but does not have enough training data to accurately estimate it
Idea:
“borrow” training examples from their relevant neighbors to improve their own results
neighbors with different underlying models has non- zero lasso penalties

21. Experiments Network: 1. Network-Enhanced Classification Dataset:
randomly generate a dataset containing 1000 nodes, each with its own classifier, a support vector machine in R50. Each node tries to predict 𝑦∈ −1, 1 , where
Network:
The 1000 nodes are split into 20 equally-sized groups. Each group has a common underlying classifier while different groups have independent models.

22. Experiments
1. Network-Enhanced Classification
Objective function:

23. Experiments 1. Network-Enhanced Classification
Results(regularization path):

24. Experiments 1. Network-Enhanced Classification
Results(prediction accuracy):

25. Experiments 1. Network-Enhanced Classification Results(timing):
Convergence comparison between centralized and ADMM methods for SVM problem

26. Experiments 1. Network-Enhanced Classification Results(timing):
Convergence time for large-scale 3-regular graph solved at a single (constant) value of 𝜆

27. Experiments 2. spatial clustering and regressors
Attempt to estimate the price of homes based on latitude/longitude data and a set of features.

28. Experiments Network: 2. spatial clustering and regressors Dataset:
a list of real estate transactions over a oneweek period in May 2008 in the Greater Sacramento area.
Network:
build the graph by using the latitude/longitude coordinates of each house
connect every remaining house to the five nearest homes with an edge weight inversely proportional to the distance between the houses
785 nodes, 2447 edges, and has a diameter of 61.

29. Experiments 2. spatial clustering and regressors
Optimization Parameter and Objective Function:
At each nodes, solve for
Objective function:

30. Experiments
2. spatial clustering and regressors
Results:

31. Experiments
2. spatial clustering and regressors
Results:

32. Conclusion
The network lasso is a useful way of representing convex optimization problems, and the magnitude of the improvements in the experiments show that this approach is worth exploring further, as there are many potential ideas to build on.
The non-convex method gave comparable performance to the convex approach, and we leave for future work the analysis of different non-convex functions 𝜙(𝑢)
we could attempt to iteratively reweigh the edge weights to attain some desired outcome
Within the ADMM algorithm, there are many ways to improve speed, performance, and robustness

33. Questions?