Mixture Density Networks Qiang Lou

Outline Simple Example Introduction of MDN Analysis of MDN Weights Optimization Prediction

Simple Example The inverse problem of x=t+0.3*sin(2лt) Input: x, output: t

Learn form the example Obviously, we find the problem of the conventional neural networks. multi-valued mapping Reason: f(x,w*)= E[Y|X], the average of the correct target values, sometimes is not correct solution.

Solution: mixture density networks MDN: overcome the limits mentioned above ---- using a linear combination of kernel function: Three parameters: coefficients: means: variances:

How to model the parameters? ---- using the outputs of the conventional NN Coefficients: Variances: Means can be directly represented by output of NN:

Basic structure of MDN

Weights Optimization Similar to the conventional NN: maximum likelihood (minimize the negative logarithm of the likelihood). We try to minimize E(w), which is equivalent to maximize the likelihood.

Weights Optimization Using chain rule and back propagation: start off the algorithm:

Prediction General Way take the conditional average of the target data: Accurate Way take the solution of the most probable components μk , where k = arg maxk( )

Results of example

Problems The number of the outputs of the MDN Assume: L models in the mixture model K outputs in conventional NN Outputs of MDN: (K+2) L 2)

Thank you !