A critical review of RNN for sequence learning Zachary C A critical review of RNN for sequence learning Zachary C. Lipton zlipton@cs.ucsd.edu

Time series Definition：A time series is a series of data points indexed (or listed or graphed) in time order. It is a sequence of discrete-time data. Feature: data points space sample from continuous real-word process Example： still images that comprise the frames of videos， clinical media data，natural language

Neural Networks Activation function： add the non-linear elements in network

Neural Networks Activation function： add the non-linear elements in network

Neural Networks Training process: backpropagation algorithm Gradient decent + Chain rule Eg： partial derivative of e=(a+b)*(b+1) respective with respect to a and b

Neural Networks Training process: backpropagation algorithm

Neural Networks Training process: backpropagation algorithm

Neural Networks Training process: backpropagation algorithm

Neural Networks Training process: backpropagation algorithm

Neural Networks Training process: backpropagation algorithm

What is RNN? Feedforward neural network with inclusion of edge that span adjacent step times. Input for every time step contains the input of temporary time step and the output of last time step.

What is RNN? Training method: backporpagation , gradient decent. Limitations: Vanishing gradients.

Vanishing gradient loss function: partial derivative of output: partial derivative of (t-1) layer: partial derivative of (t-q) layer: relationship of gradients between (t-q) and t layer:

LSTM (long short-term memory) To solve the problem of vanishing gradient

RNNs for Outlier Detection Classification problem Training RNN weights to minimise the error by normal data. Since RNN attempts to represent the input patterns in the output, representing outliers are less well produced by the trained RNN have a higher reconstruction error.

Conclusion RNN can remember previous input. When the problems involve continuous, prior knowledge related task, it could show advanced capability. RNN is a data inference method, which can get the probability disribution function from x(t) mapping to y(t).--- finding the relationship between 2 time series.