1. Integrating Neural Network and Genetic Algorithm to Solve Function Approximation Combined with Optimization Problem Term presentation for CSC7333 Machine Learning Xiaoxi Xu May 3, 2006

2. Outline Problem description & analysis Neural Network Genetic Algorithm Implementations & experiments Results Remarks Conclusion

3. Problem Description We have plentiful data gathered over time; We are not aware the underlying relationship between the data input ( some are human controllable) and its output; We expect to minimize or maximize the output in the future; We hope to know that what kind of input would generate a minimum or maximum output, so that we could adjust the input to achieve our end.

4. Problem Analysis The characteristics of this problem are: a.Unknown exact nature of relationship between input and output, likely non-linear b. Inputs are likely in N-dimension (N>10) In addition…. d. The global optimum is expected to obtain

5. Problem Break Up 1. Function Approximation 2. Optimization Problem

6. Solution for Function Approximation This solution should meet the following requirements: a. Have the parallel structure to handle N dimension variables b. Be able to model the nonlinear relation between variables and their responses c. Had better to be fault-tolerant (noisy data could appear when data set is large)

7. Solution for Function Approximation (Cont ’ d) Neural Network could be one. Why? (Any function can be approximated to arbitrary accuracy by a network with 3 layers with linear transfer function in output layer, and sigmoid function in hidden layer) We want to train a NN as such: Topology: Multiple Layer Network Connection Type: Feed-forward (From a mathematical point of view, a feed-forward neural network is a function. It takes an input and produces an output.) Transfer Function: Logsigmoid, linear Training Algorithm: Back-propagation

8. Problem Break Up 1. Function Approximation 2. Optimization Problem

9. Solution for Optimization Problem To solve it mathematically? In mathematics, LOCAL optima can ONLY be found for functions with good properties such as convex or unimodel. These line search methods include Conjugate gradient descent, quasi-Newton and so on. This solution should meet the following requirements: a. Be able to recognize the expression of the objective function b. Be able to solve the function c. Should have a better chance to find a global optimum

10. Solution for Optimization Problem (Cont ’ d) Genetic Algorithm could be one. Why? ( GA have most been applied to optimization problems)  We can use GA as such: a. Representation ( Any real number can be represented by a string of numerical numbers in base 10) b. Fitness function ( Neural Net) c. Genetic operators ( Crossover,Selection,Mutation )

11. Implementation & Experiment of NN for 2D Function Approximation Initialization with random selected weights Multiple layer with one hidden layer of 20 hidden nodes Transfer function: sigmoid function (hidden layer), purelin function( output layer) Back-propagation algorithm Learning rate: 0.05 Momentum: 0.1 Stop criteria: 1.MSE below 0.01% 2.Exceed epochs ( training times) 100 Test function: tan(sin(x)) - sin(tan(x)) Training data: [-4,4] -4, -3.6, -3.2, -2.8

12. Implementation & Experiment of Genetic Algorithm for 2D Function Optimization  Representation: A string of numerical number in base 10 to represent a real number and its sign;  Random initialization; Range: [-4,4];  Population size: 25; Chromosome length: 6;  One point crossover pr: 0.7; Mutation pr: 0.3;  Roulette wheel selection: preference to best-fit individual ;  Fitness function: Neural Network (represented by inputs,weights and biases as follows: weight_oh*sigmoid(weight_hi* input + bias_hi*1)+ bias_oh*1  Elitism: best-fit individual goes to next generation;  Stop criteria 1. Value of the fitness function changes less than 0.01 after 10 consecutive generations 2. Maximum generation is 30

13. Experiment Result-2D Real function--red solid line Approximation by NN –blue dash line Optimal by GA for approximation function --- magnate star

14. Implementation of NN for 3D Function Approximation One hidden layer of 30 hidden nodes Learning rate: 0.06 Momentum: 0.5 Stop criteria: 1.MSE below 0.01% 2.Exceed epochs ( training times) 250 Test function: 1.85*sin(x)*exp(-3*(y-1.5)^2)+0.7*x*exp(-4*(x-1.2)^2)-1.4*cos(x+ y)*exp(-5*(y+1.3)^2)-1.9*exp(-8*(x+0.5)^2) Training data: [-1,3] (difference between numbers is 0.17)

15. Implementation & Experiment of Genetic Algorithm for 3D Function Optimization  Random initialization  Range: [-1,3]  Population size: 40  Chromosome length: 12  One point crossover pr: 0.8  Mutation pr: 0.6  Stop criteria: 1. Fitness function changes less than 0.01,after 10 consecutive generations 2. Maximum generation is 100

16. Experiment Result- Mesh

17. Experiment Result - Contour Map Optimum-- black circle

18. One More Experiment sin(0.007*x^5)/cos(exp(0.0009*y^7))

19. Remarks How to adjust some parameters for NN? It’s really application-dependent. With regard to my experiments: Learning rate: too small, no apparent decrease of MSE over long while too large, MSE jumped between decrease and increase Momentum: compared to smaller, larger value could model better but keep it in a proper degree Hidden nodes: keep it in a proper degree,otherwise will be overfitting more nodes, better performance but our computer was down before we knew how best the performance would be Epochs: Longer, better performance, but avoid overfittng trade off between the accuracy and the training time

20. Remarks (cont ’ d) How to adjust some parameters for GA?  Population size: bigger is better, keep in a proper degree, otherwise will be overfitting  Crossover probability: typically, [0.6,0.9] it works in my experiment  Mutation probability: typically, [1/pop_size,1/chromosome_length] larger value in my experiment  Generation size: larger, better performance,but avoid overfitting trade off between the accuracy and the time Random initialization influences on the performance success We used random selected weights in training NN and used random selected individuals for the first generation of GA. We found that sometimes, random initialization value determines the success of the performance

21. Room for Improvements Random data would be used to train the neural network with noisy data added More complex examples would be tested for the performance of NN Building up more knowledge on adjusting NN & GA parameters Error surface would be shown Time complexity would be analyzed

22. Conclusion Integrating GA&NN by using NN as the fitness function of GA could approximate a good global optimum pretty close to that found by GA using the real function GA performs well in searching for the global optimum no matter what the fitness function would be In practical, Multiple Layer NNs with one hidden layer have an overall good performance in function approximation, but sometimes they still have the difficulty The random initialization value could sometimes determine the performance success

23. Thanks! Questions & Comments ??