Network Lasso: Clustering and Optimization in Large Graphs David Hallac, Jure Leskovec, Stephen Boyd Stanford University Presented by Yu Zhao

What is this paper about Lasso problem The lasso solution is unique when rank(X) = p, because the criterion is strictly convex.

What is this paper about Network lasso problem The variables are , where . (The total number of scalar variables is mp.) Here is the variable at node i, is the cost function at node i, and is the cost function associated with edge .

Outline Convex problem definition Proposed solution(ADMM) Non-convex extension Experiments

Convex problem definition (1) (2)

Convex problem definition A distributed and scalable method was developed for solving the network lasso problem, in which each vertex variable xi is controlled by one “agent”, and the agents exchange (small) messages over the graph to solve the problem iteratively.

Convex problem definition General settings for different applications e.g. Control system: Nodes: possible states xi: actions to take when state i Graph: state transitions Weights: how much we care about the actions in neighboring states differing

Convex problem definition General settings for different applications The sum-of-norms regularization that we use is like group lasso, which encourages not just , for edge , but , consensus across the edge.

Convex problem definition Regularization Path 𝜆=0: simply a minimizer of 𝑓𝑖  local computations 𝜆 →∞ (𝜆 ≥𝜆𝑐𝑟𝑖𝑡𝑖𝑐𝑎𝑙):

Convex problem definition Network lasso and clustering 𝑙2-norms penalty defines network lasso. Cluster size: 𝜆

Convex problem definition Inference on New Nodes we can interpolate the solution to estimate the value of 𝑥𝑖 on a new node 𝑗.

Proposed solution(ADMM) Alternating Direction Method of Multipliers(ADMM) S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning, 3:1–122, 2011.

Proposed solution(ADMM) ADMM in network lasso 1). Introduce a copy of 𝑥𝑖, called 𝑧𝑖𝑗 , at each edge 𝑖𝑗.

Proposed solution(ADMM) ADMM in network lasso 2). Augmented Lagrangian M. R. Hestenes. Multiplier and gradient methods. Journal of Optimization Theory and Applications, 4:302–320, 1969.

Proposed solution(ADMM) ADMM in network lasso 3). ADMM updates

Proposed solution(ADMM) Regularization Path compute the regularization path as a function of 𝜆 to gain insight into the network structure Start at 𝜆= 𝜆𝑖𝑛𝑖𝑡𝑖𝑎𝑙 update: 𝜆 ≔ 𝛼𝜆, 𝛼>1 stop:

Non-convex extension replace the group lasso penalty with a monotonically nondecreasing concave function 𝜙(𝑢), where 𝜙 𝑢 =0 , and whose domain is u𝑢 ≥0, ADMM is not guaranteed to converge, and even if it does, it need not be to a global optimum

Non-convex extension Heuristic solution: to keep track of the iteration which yields the minimum objective, and to return that as the solution instead of the most recent step.

Non-convex extension Non-convex z-Update Compared to the convex case, the only difference in the ADMM solution is the z-update, which is now

Experiments Idea: 1. Network-Enhanced Classification We first analyze a synthetic network in which each node has a support vector machine (SVM) classifier, but does not have enough training data to accurately estimate it Idea: “borrow” training examples from their relevant neighbors to improve their own results neighbors with different underlying models has non- zero lasso penalties

Experiments Network: 1. Network-Enhanced Classification Dataset: randomly generate a dataset containing 1000 nodes, each with its own classifier, a support vector machine in R50. Each node tries to predict 𝑦∈ −1, 1 , where Network: The 1000 nodes are split into 20 equally-sized groups. Each group has a common underlying classifier while different groups have independent models.

Experiments 1. Network-Enhanced Classification Objective function:

Experiments 1. Network-Enhanced Classification Results(regularization path):

Experiments 1. Network-Enhanced Classification Results(prediction accuracy):

Experiments 1. Network-Enhanced Classification Results(timing): Convergence comparison between centralized and ADMM methods for SVM problem

Experiments 1. Network-Enhanced Classification Results(timing): Convergence time for large-scale 3-regular graph solved at a single (constant) value of 𝜆

Experiments 2. spatial clustering and regressors Attempt to estimate the price of homes based on latitude/longitude data and a set of features.

Experiments Network: 2. spatial clustering and regressors Dataset: a list of real estate transactions over a oneweek period in May 2008 in the Greater Sacramento area. Network: build the graph by using the latitude/longitude coordinates of each house connect every remaining house to the five nearest homes with an edge weight inversely proportional to the distance between the houses 785 nodes, 2447 edges, and has a diameter of 61.

Experiments 2. spatial clustering and regressors Optimization Parameter and Objective Function: At each nodes, solve for Objective function:

Experiments 2. spatial clustering and regressors Results:

Experiments 2. spatial clustering and regressors Results:

Conclusion The network lasso is a useful way of representing convex optimization problems, and the magnitude of the improvements in the experiments show that this approach is worth exploring further, as there are many potential ideas to build on. The non-convex method gave comparable performance to the convex approach, and we leave for future work the analysis of different non-convex functions 𝜙(𝑢) we could attempt to iteratively reweigh the edge weights to attain some desired outcome Within the ADMM algorithm, there are many ways to improve speed, performance, and robustness

Questions?