1. ELeaRNT: Evolutionary Learning of Rich Neural Network Topologies Authors: Slobodan Miletic 3078/2010 sloba10@gmail.com Nikola Jovanovic 3077/2010 nikolaj_ub@yahoo.co.uk

2. Introduction Genetic algorithm Genetic algorithm mimics natural evolution mimics natural evolution candidate solution candidate solution mutation mutation Neural network Neural network based on biological neurons based on biological neurons network consists of neurons grouped in layers network consists of neurons grouped in layers 2/16 Slobodan Miletić sloba10@gmail.com Nikola Jovanovićnikolaj_ub@yahoo.co.uk

3. Problem definition Current design methods are manual and inefficient Current design methods are manual and inefficient Hard to define number of neurons and connections Hard to define number of neurons and connections No automated design method No automated design method for specific optimal topology for specific optimal topology 3/16 Slobodan Miletić sloba10@gmail.com Nikola Jovanovićnikolaj_ub@yahoo.co.uk

4. Problem importance Neural networks have a large use area Neural networks have a large use area Creating new neural networks Creating new neural networks takes money and time 4/16 Slobodan Miletić sloba10@gmail.com Nikola Jovanovićnikolaj_ub@yahoo.co.uk

5. Problem trend Computers are getting more powerful Computers are getting more powerful New neural network uses are found New neural network uses are found If not solved, this problem would slow down the evolution of neural networks If not solved, this problem would slow down the evolution of neural networks 5/16 Slobodan Miletić sloba10@gmail.com Nikola Jovanovićnikolaj_ub@yahoo.co.uk

6. Existing solutions Trial & Error Trial & Error Manual algorithm Manual algorithm Few parameters for optimization Few parameters for optimization Long, costly, and not very efficient Long, costly, and not very efficient 6/16 Slobodan Miletić sloba10@gmail.com Nikola Jovanovićnikolaj_ub@yahoo.co.uk

7. Existing solutions Destructive Algorithm Destructive Algorithm Starts with very big networks Starts with very big networks Gets results by pruning the initial network Gets results by pruning the initial network A lot of time is spent on unnecessary training A lot of time is spent on unnecessary training of big networks 7/16 Slobodan Miletić sloba10@gmail.com Nikola Jovanovićnikolaj_ub@yahoo.co.uk

8. Existing solutions Constructive Algorithm Constructive Algorithm Starts with a small neural network Starts with a small neural network Adds nodes and connections Adds nodes and connections Uses input/error rate to form new nodes Uses input/error rate to form new nodes Can miss optimal solution Can miss optimal solution 8/16 Slobodan Miletić sloba10@gmail.com Nikola Jovanovićnikolaj_ub@yahoo.co.uk

9. Proposed solution What’s better? What’s better? General algorithm – fitness function change enables generation General algorithm – fitness function change enables generation of different neural networks types Created neural networks outperform Created neural networks outperform neural network models designed by hand neural network models designed by hand Besides the best network, it creates several suboptimal networks Besides the best network, it creates several suboptimal networks that can also be used as a solutions 9/16 Slobodan Miletić sloba10@gmail.com Nikola Jovanovićnikolaj_ub@yahoo.co.uk

10. Proposed solution What’s new? What’s new? No similar general algorithm on the market No similar general algorithm on the market Original set of genetic algorithm mutations Original set of genetic algorithm mutations 10/16 Slobodan Miletić sloba10@gmail.com Nikola Jovanovićnikolaj_ub@yahoo.co.uk

11. Proposed solution What’s its future? What’s its future? Computer power and parallelism are increasing Computer power and parallelism are increasing which allows more complex neural networks Automated neural network generation algorithm Automated neural network generation algorithm like this will allow creation of complex neural networks 11/16 Slobodan Miletić sloba10@gmail.com Nikola Jovanovićnikolaj_ub@yahoo.co.uk

12. f 2 g 5 p 2 h 7 f 3 r 2 No changeAdd node a 4 b 3 c 7 d 2 q 5 g 5 h 7 f 3 a 4 b 3 d 2 f 2 c 7 p 2 r 2 f 2 p 2 r 2 g 5 h 7 f 3 c 7 a 4 b 3 d 2 f 3 g 5 a 4 b 3 c 7 d 2 f 2 g 2 p 2 h 7 f 3 r 2 h 7 r 2 a 4 b 3 c 7 f 2 p 2 d 2 Drop nodeActivation Function h 7 r 2 a 4 b 3 c 7 f 2 f 5 p 2 d 2 f 2 f 2 f 2 f 2 f 2 f 2 f 2 f 2 g 5 p 2 h 7 f 3 r 2 f 2 a 3 a 3 f 2 f 2 g 2 h 2 f 1 a 3 b 4 c 5 c 3 c 4 c 5 f 2 d 2 a 3 q 3 r 2 h 1 h 1 h 1 h 1 f 3 h 2 f 3 g 3 g 3 h 2 a 4 b 3 c 7 d 2 Solution details Number of Neurons a 4 b 1 c 7 d 2 a 4 b 3 c 7 d 2 3 f 2 g 5 p 2 h 7 f 3 r 2 Drop Connection a 4 b 3 c 7 d 2 Add Connection f 2 c 7 p 2 r 2 12/16 Slobodan Miletić sloba10@gmail.com Nikola Jovanovićnikolaj_ub@yahoo.co.uk 1 point Crossover 2 point Crossover

13. a 4 b 3 c 7 d 2 q 5 Solution details h 7 r 2 a 4 b 3 c 7 f 2 f 5 p 2 d 2 g 5 h 7 f 3 a 4 b 3 d 2 a 4 b 1 c 7 d 2 f 2 g 5 p 2 h 7 f 3 r 2 a 4 b 3 c 7 d 2 f 2 c 7 p 2 r 2 a 4 b 3 g 5 h 7 f 3 d 2 a 4 b 3 c 7 d 2 q 5 a 4 b 3 d 2 q 5 g 5 h 7 f 3 d 2 a 4 b 3 c 7 g 5 h 7 f 3 d 2 a 4 b 3 c 7 a 1 Add nodeNo change a 4 b 3 c 7 d 2 q 5 Drop connection a 4 b 3 c 7 d 2 q 5 g 5 a 4 b 3 h 7 f 3 d 2 a 4 b 3 c 7 d 2 q 5 h 7 f 3 d 2 a 4 g 5 a 4 b 3 b 3 c 7 d 2 q 5 a 4 b 3 c 7 d 2 q 5 g 5 h 7 f 3 a 4 b 3 d 2 g 5 h 7 b 3 f 3 a 4 d 2 a 4 c 7 d 2 b 3 q 5 a 4 c 7 d 2 g 5 h 7 b 3 f 3 a 4 d 2 b 3 q 5 13/16 Slobodan Miletić sloba10@gmail.com Nikola Jovanovićnikolaj_ub@yahoo.co.uk 1 point Crossover 2 point Crossover

14. f 3 a 4 d 2 b 3 q 5 g 5 h 7 f 3 d 2 a 4 b 3 c 7 a 1 f 3 a 4 d 2 b 3 q 5 Solution details a 4 b 3 c 7 d 2 q 5 a 4 b 3 c 7 d 2 q 5 a 4 c 7 d 2 g 5 h 7 b 3 f 3 a 4 d 2 b 3 q 5 h 7 f 3 d 2 a 4 g 5 a 4 b 3 b 3 c 7 d 2 q 5 a 4 b 3 d 2 q 5 g 5 h 7 f 3 d 2 a 4 b 3 c 7 a 1 g b Activation function f1 5 h 7 f 3 d 2 a 4 b 3 c 7 a 1 f 3 a 4 d 2 a 3 q 5 5 Number of Neurons g 10 h 7 f 3 d 2 a 4 b 3 c 7 a 1 g 5 a 4 d 2 f 3 b 3 q 5 a 1 f 3 b 3 q 5 h 7 f 3 d 2 a 4 b 3 c 7 a 4 d 2 g 5 Drop node f 3 b 3 q 5 h 7 f 3 d 2 a 4 b 3 c 7 Add connection a 4 d 2 g 5 h 7 f 3 d 2 a 4 b 3 c 7 a 1 Drop connection g 5 h 7 f 3 d 2 a 4 b 3 c 7 a 1 Add node f 3 a 4 d 2 b 3 q 5 q 5 Add connection f 3 a 4 d 2 b 3 q 5 q 5 14/16 Slobodan Miletić sloba10@gmail.com Nikola Jovanovićnikolaj_ub@yahoo.co.uk 2 point Crossover

15. Conclusion New way to create neural networks New way to create neural networks Results fully comparable Results fully comparable with hand designed networks Space for further improvement Space for further improvement More then one created network can be used More then one created network can be used 15/16 Slobodan Miletić sloba10@gmail.com Nikola Jovanovićnikolaj_ub@yahoo.co.uk

16. QUESTIONS? 16/16 Slobodan Miletić sloba10@gmail.com Nikola Jovanovićnikolaj_ub@yahoo.co.uk