1. CS621 : Artificial Intelligence Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 21 Computing power of Perceptrons and Perceptron Training

2. The human brain Seat of consciousness and cognition Perhaps the most complex information processing machine in nature

3. Brain Map Forebrain (Cerebral Cortex): Language, maths, sensation, movement, cognition, emotion Cerebellum: Motor Control Midbrain: Information Routing; involuntary controls Hindbrain: Control of breathing, heartbeat, blood circulation Spinal cord: Reflexes, information highways between body & brain

4. Maslow’s Hierarchy of Needs

5. Brain’s algorithms? Evolutionarily, brain has developed algorithms most suitable for survival Algorithms unknown: the search is on Brain astonishing in the amount of information it processes –Typical computers: 10 9 operations/sec –Housefly brain: 10 11 operations/sec

6. Brain facts & figures Basic building block of nervous system: nerve cell (neuron) ~ 10 12 neurons in brain ~ 10 15 connections between them Connections made at “synapses” The speed: events on millisecond scale in neurons, nanosecond scale in silicon chips

7. Computing Power of Perceptron

8. The Perceptron Model A perceptron is a computing element with input lines having associated weights and the cell having a threshold value. The perceptron model is motivated by the biological neuron. Output = y wnwn W n-1 w1w1 X n-1 x1x1 Threshold = θ

9. θ 1 y Step function / Threshold function y = 1 for Σw i x i >=θ =0 otherwise ΣwixiΣwixi

10. Concept of Hyper-planes ∑ w i x i = θ defines a linear surface in the (W,θ) space, where W= is an n-dimensional vector. A point in this (w,θ) space defines a perceptron. y x1x1... θ w1w1 w2w2 w3w3 wnwn x2x2 x3x3 xnxn

11. Functions computed by the simplest perceptron (single input) 10101 11000 f4f3f2f1x θ≥0 w≤θ θ≥0 w> θ θ<0 w≤ θ θ<0 W< θ 0-function Identity Function Complement Function True-Function

12. Counting the number of functions for the simplest perceptron For the simplest perceptron, the equation is w.x=θ. Substituting x=0 and x=1, we get θ=0 and w=θ. These two lines intersect to form four regions, which correspond to the four functions. θ=0 w=θ R1 R2 R3 R4 θ w

13. Fundamental Observation The number of TFs computable by a perceptron is equal to the number of regions produced by 2 n hyper-planes,obtained by plugging in the values in the equation ∑ i=1 n w i x i = θ

14. The geometrical observation Problem: m linear surfaces called hyper- planes (each hyper-plane is of (d-1)-dim) in d-dim, then what is the max. no. of regions produced by their intersection? i.e. R m,d = ?

15. Case of 2-input perceptrons Output = y w2w2 w1w1 x1x1 Threshold = θ x2x2

16. Basic equation w 1 x 1 +w 2 x 2 =θ There are 4 values of the input: (0,0), (0,1), (1,0), (1,1) The relevant space is the (w 1,w 2,θ) coordinate system w1w1 w2w2 θ

17. 4 planes All go through the origin –With (0,0), θ= 0 -- (1) –With (1,0), w 1 = θ --(2) –With (0,1), w 2 = θ --(3) –With (1,1), w 1 + w 2 = θ --(4) How many regions do they produce? That is equal to the number of functions computable by a 2-input perceptron

18. How to think about this counting problem? Whenever a new plane comes in, the existing planes intersect the new plane is a set of lines all going through the origin These lines produce some regions on the new plane (notice that we are not worrying about the regions in the space, but are thinking about the regions on the new plane) These regions produced on the new plane is the additional number of regions produced in the (w,θ) space

19. Counting the maximum no. of regions produced by 4 planes passing through origin Plane numberAdditional Regions produced 1 st 2 2 nd (1 st plane cuts this plane to form a line through the origin) 2 3 rd (1 st and 2 nd planes cut this plane to form two lines through the origin) 4 4 th (1 st, 2 nd and 3 rd planes cut this plane to form thre lines through the origin) 6

20. It is clear why 2-input perceptron computes 14 functions 2+2+4+6= 14 14 regions are produced by the planes. Hence only 14 functions computed Terminology: Functions computable by a perceptron are called threshold functions

21. General case A perceptron with n weights and the threshold defines an (n+1)-dimensional space. From the basic equation Σ 1 n w i x i =θ, 2 n planes passing through origin are produced How many regions do they produce in the space?

22. Recurrence relation produced by m hyperplanes in d-dimension –C(m,d)= C(m-1,d)+C(m-1,d-1) Boundary conditions: –C(m,1)=2 (degenerate case of m points ‘passing through’ origin) –C(1,d)=2 Existing regions Additional regions on the d th plance

23. Threshold functions miniscule compared to Boolean Function Solution of the recurrence relation leads to the max. no. of regions as 2^n 2 vs. 2^2 n boolean functions