1. Parallel Methods for Neuron Network
Haihao Lu and Yuanchu Dang

2. Machine Learning and Artificial Neuron Network

3. Problem of Interest: ANN
W is the weight
b is the bias
ϕ is activation function
x is feature
y is label
l is loss function

4. Drawback of SGD SGD is a first-order method, thus it converges slow.
SGD suffers from vanishing gradient problem
Most importantly, it is hard to parallelize SGD

5. Equivalent Problem
Relaxed Problem

6. Alternating Minimization
W-update
z-update
one or more steps of damped Newton
cheap to compute Hessian
parallel computing

7. RNN

8. RNN
Equivalent Problem
Relaxed Problem

9. With L_2 Regulation W-update z-update
one or more steps of damped Newton
cheap to compute Hessian
parallel computing

10. Code
neuralNetwork.jl k

11. Numerical Test: Auto-Encoder