1. Dimensions of Neural Networks Ali Akbar Darabi Ghassem Mirroshandel Hootan Nokhost

2. Outline Motivation Neural Networks Power Kolmogorov Theory Cascade Correlation

3. Motivation Consider you are an engineer and you know ANN You encounter a problem that can not be solved with common analytical approaches You decide to use ANN

4. But… Some questions  Is this problem solvable using ANN?  How many neurons?  How many layers?  …

5. Two Approaches Fundamental Analyses  Kolmogrov Theory Adaptive Networks  Cascade Correlation

6. Outline Motivation Neural Networks Power Kolmogorov Theory Cascade Correlation

7. Single layer Networks Limitations of the perceptron and linear classifiers

8. A Solution

9. Network Construction (x,y)→(x^2, y^2,x*y) 1 2

10. Network Construction (con…)

11. Network Construction (Con…)

12. Learning Mechanism Using Error Function Gradient Descent

13. Outline Motivation Neural Networks Power Kolmogorov Theory Cascade Correlation

14. Kolmogorov theorem (concept) An example:  Any continuous function of n dimensions can be completely characterized by a dimensional continuous functions

15. An Idea Suppose we want to construct f (x, y) A simple idea: find a mapping  (x, y) → r Then define a function g such that:  g(r) = f(x, y) x y r g

16. An Example Suppose we have a discrete function:  We choose a mapping  We define the 1-dimentional function  So 

17. Kolmogrov theorem In the illustrated example we had: 

18. Applying to the neural networks

19. Universal Approximation Neural Networks with a hidden layer can approximate any continuous function with arbitrary precision  Use independent function from main function  approximate the network with traditional networks

20. A kolmogorov Network We have to define:  Mapping  Function g

21. Spline Function Linear combination of several 3-dimensional functions Used to approximate functions with given points

22. Mapping x y

23. Example X1=2.5 X2=4.5 x1 x2 2.1 1.6 2.5 4.5 3.2 2.5 1.4

24. Function g Now for each unique input value of a we should define a output value g corresponding to f We choose the value of f in the center of the square X1=2.5 X2=4.5

25. Function g (Con…)

26. Reduce Error Shifting defined patterns N different patterns will be generated Use avg y

27. Replace the function With sufficiently large number of knots:

28. Outline Motivation Neural Networks Power Kolmogorov Theory Cascade Correlation

29. Dynamic size, depth, topology Single layer learning in spite of multilayer structure Fast learning

30. Architecture

31. Algorithm step 1

32. Adding Hidden Layer

33. Correlation Residual error for output unit for pattern p  Average residual error for output unit  Computed activation for input vector x(p)  Z(p) Average activation, over all patterns, of candidate unit 

34. Correlation Use Variance as a similarity criteria Update weights similar to gradient descent

35. Algorithm step 2

36. Adding Hiding Neuron

37. Algorithm Step 3

38. Final Result

39. An Example 100 Run 1700 epochs on avg Beats standard backprob with factor 10 with the same complexity

40. Results Cascade Correlation is either better  Only forward pass  Many of epochs are run while the network is very small  Cashing mechanism

41. Network Steps

42. Network Steps (con..)

43. Another Example N-input parity problem Standard backprob takes 2000 epoches on N=8 with 16 hidden neurons

44. Discussion There is no need to guess the size, depth and the connectivity pattern Learns fast Can build deep networks (high order feature detector) Herd effect Results can be cashed

45. Conclusion A Network with a hidden layer can define complex boundaries and can approximate any function The number of neurons in the hidden layer determines the amount of approximation Dynamic Networks