1. Feed-Forward Neural Networks 主講人 : 虞台文

2. Content Introduction Single-Layer Perceptron Networks Learning Rules for Single-Layer Perceptron Networks – Perceptron Learning Rule – Adline Leaning Rule

3. Feed-Forward Neural Networks Introduction

4. Historical Background 1943 McCulloch and Pitts proposed the first computational models of neuron. 1949 Hebb proposed the first learning rule. 1958 Rosenblatt’s work in perceptrons. 1969 Minsky and Papert’s exposed limitation of the theory. 1970s Decade of dormancy for neural networks. 1980-90s Neural network return (self-organization, back-propagation algorithms, etc)

5. Nervous Systems Human brain contains ~ 10 11 neurons. Each neuron is connected ~ 10 4 others. Some scientists compared the brain with a “complex, nonlinear, parallel computer”. The largest modern neural networks achieve the complexity comparable to a nervous system of a fly.

6. Neurons The main purpose of neurons is to receive, analyze and transmit further the information in a form of signals (electric pulses). When a neuron sends the information we say that a neuron “fires”.

7. Neurons This animation demonstrates the firing of a synapse between the pre-synaptic terminal of one neuron to the soma (cell body) of another neuron. Acting through specialized projections known as dendrites and axons, neurons carry information throughout the neural network.

8. bias x1x1 x2x2 x m =  1 wi1wi1 wi2wi2 w im =  i...... A Model of Artificial Neuron yiyi f (.)a (.) 

9. bias x1x1 x2x2 x m =  1 wi1wi1 wi2wi2 w im =  i...... A Model of Artificial Neuron yiyi f (.)a (.) 

10. Feed-Forward Neural Networks Graph representation: – nodes: neurons – arrows: signal flow directions A neural network that does not contain cycles (feedback loops) is called a feed–forward network (or perceptron).... x1x1 x2x2 xmxm y1y1 y2y2 ynyn

11. Hidden Layer(s) Input Layer Output Layer Layered Structure... x1x1 x2x2 xmxm y1y1 y2y2 ynyn

12. Knowledge and Memory... x1x1 x2x2 xmxm y1y1 y2y2 ynyn The output behavior of a network is determined by the weights. Weights  the memory of an NN. Knowledge  distributed across the network. Large number of nodes – increases the storage “capacity”; – ensures that the knowledge is robust; – fault tolerance. Store new information by changing weights.

13. Pattern Classification... x1x1 x2x2 xmxm y1y1 y2y2 ynyn Function: x  y The NN’s output is used to distinguish between and recognize different input patterns. Different output patterns correspond to particular classes of input patterns. Networks with hidden layers can be used for solving more complex problems then just a linear pattern classification. input pattern x output pattern y

14. Training... xi1xi1 xi2xi2 x im yi1yi1 yi2yi2 y in Training Set di1di1 di2di2 d in Goal:

15. Generalization... x1x1 x2x2 xmxm y1y1 y2y2 ynyn By properly training a neural network may produce reasonable answers for input patterns not seen during training (generalization). Generalization is particularly useful for the analysis of a “noisy” data (e.g. time–series).

16. Generalization... x1x1 x2x2 xmxm y1y1 y2y2 ynyn By properly training a neural network may produce reasonable answers for input patterns not seen during training (generalization). Generalization is particularly useful for the analysis of a “noisy” data (e.g. time–series). without noise with noise

17. Applications Pattern classification Object recognition Function approximation Data compression Time series analysis and forecast...

18. Feed-Forward Neural Networks Single-Layer Perceptron Networks

19. The Single-Layered Perceptron... x1x1 x2x2 x m =  1 y1y1 y2y2 ynyn x m-1... w 11 w 12 w1mw1m w 21 w 22 w2mw2m wn1wn1 w nm wn2wn2

20. The Single-Layered Perceptron... x1x1 x2x2 x m =  1 y1y1 y2y2 ynyn x m-1... w 11 w 12 w1mw1m w 21 w 22 w2mw2m wn1wn1 w nm wn2wn2

21. Training a Single-Layered Perceptron... x1x1 x2x2 x m =  1 y1y1 y2y2 ynyn x m-1... w 11 w 12 w1mw1m w 21 w 22 w2mw2m wn1wn1 w nm wn2wn2 d1d1 d2d2 dndn Training Set Goal:

22. Learning Rules... x1x1 x2x2 x m =  1 y1y1 y2y2 ynyn x m-1... w 11 w 12 w1mw1m w 21 w 22 w2mw2m wn1wn1 w nm wn2wn2 d1d1 d2d2 dndn Training Set Goal: Linear Threshold Units (LTUs) : Perceptron Learning Rule Linearly Graded Units (LGUs) : Widrow-Hoff learning Rule

23. Feed-Forward Neural Networks Learning Rules for Single-Layered Perceptron Networks Perceptron Learning Rule Adline Leaning Rule

24. Perceptron Linear Threshold Unit sgn x1x1 x2x2 x m =  1 wi1wi1 wi2wi2 w im =  i......  +1 11

25. Perceptron Linear Threshold Unit sgn Goal: x1x1 x2x2 x m =  1 wi1wi1 wi2wi2 w im =  i......  +1 11

26. Example x1x1 x2x2 x 3 =  1 22 1 22 y Class 1 (+1) Class 2 (  1) Class 1 Class 2 x1x1 x2x2 g(x) =  2x 1 +2x 2 +2=0 Goal:

27. Augmented input vector x1x1 x2x2 x 3 =  1 22 1 22 y Class 1 (+1) Class 2 (  1) Class 1 (+1) Class 2 (  1) Goal:

28. Augmented input vector x1x1 x2x2 x 3 =  1 22 1 22 y Class 1 (  1,  2,  1) (  1.5,  1,  1) (  1,0,  1) Class 2 (1,  2,  1) (2.5,  1,  1) (2,0,  1) x1x1 x2x2 x3x3 (0,0,0) Goal:

29. Augmented input vector x1x1 x2x2 x 3 =  1 22 1 22 y Class 1 (  1,  2,  1) (  1.5,  1,  1) (  1,0,  1) Class 2 (1,  2,  1) (2.5,  1,  1) (2,0,  1) x1x1 x2x2 x3x3 (0,0,0) Goal: A plane passes through the origin in the augmented input space.

30. Linearly Separable vs. Linearly Non-Separable 0 1 1 0 1 1 0 1 1 ANDORXOR Linearly Separable Linearly Non-Separable

31. Goal Given training sets T 1  C 1 and T 2  C 2 with elements in form of x=(x 1, x 2,..., x m-1, x m ) T, where x 1, x 2,..., x m-1  R and x m =  1. Assume T 1 and T 2 are linearly separable. Find w=(w 1, w 2,..., w m ) T such that

32. Goal Given training sets T 1  C 1 and T 2  C 2 with elements in form of x=(x 1, x 2,..., x m-1, x m ) T, where x 1, x 2,..., x m-1  R and x m =  1. Assume T 1 and T 2 are linearly separable. Find w=(w 1, w 2,..., w m ) T such that w T x = 0 is a hyperplain passes through the origin of augmented input space.

33. Observation x1x1 x2x2 +  d = +1 d =  1 + w1w1 w2w2 w3w3 w4w4 w5w5 w6w6 x Which w ’s correctly classify x ? What trick can be used?

34. Observation x1x1 x2x2 +  d = +1 d =  1 + w 1 x 1 +w 2 x 2 = 0 w x Is this w ok?

35. Observation x1x1 x2x2 +  d = +1 d =  1 + w 1 x 1 +w 2 x 2 = 0 w x Is this w ok?

36. Observation x1x1 x2x2 +  d = +1 d =  1 + w 1 x 1 +w 2 x 2 = 0 w x Is this w ok? How to adjust w ?  w = ? ? ?

37. Observation x1x1 x2x2 +  d = +1 d =  1 + w x Is this w ok? How to adjust w ?  w =  x reasonable? <0 >0

38. Observation x1x1 x2x2 +  d = +1 d =  1 + w x Is this w ok? How to adjust w ?  w =  x reasonable? <0 >0

39. Observation x1x1 x2x2 +  d = +1 d =  1  w x Is this w ok?  w = ? +x+x or  x

40. Perceptron Learning Rule + d = +1  d =  1 Upon misclassification on Define error +   + No error

41. Perceptron Learning Rule Define error +   + No error

42. Perceptron Learning Rule Learning Rate Error (d  y) Input

43. Summary  Perceptron Learning Rule Based on the general weight learning rule. incorrect correct

44. Summary  Perceptron Learning Rule xy d +  ............ Converge?

45. Perceptron Convergence Theorem xy d +  ............ Exercise: Reference some papers or textbooks to prove the theorem. If the given training set is linearly separable, the learning process will converge in a finite number of steps.

46. The Learning Scenario x1x1 x2x2 + x (1) + x (2)  x (3)  x (4) Linearly Separable.

47. The Learning Scenario x1x1 x2x2 w0w0 + x (1) + x (2)  x (3)  x (4)

48. The Learning Scenario x1x1 x2x2 w0w0 + x (1) + x (2)  x (3)  x (4) w1w1 w0w0

49. The Learning Scenario x1x1 x2x2 w0w0 + x (1) + x (2)  x (3)  x (4) w1w1 w0w0 w2w2 w1w1

50. The Learning Scenario x1x1 x2x2 w0w0 + x (1) + x (2)  x (3)  x (4) w1w1 w0w0 w1w1 w2w2 w2w2 w3w3

51. The Learning Scenario x1x1 x2x2 w0w0 + x (1) + x (2)  x (3)  x (4) w1w1 w0w0 w1w1 w2w2 w2w2 w3w3 w 4 = w 3

52. The Learning Scenario x1x1 x2x2 + x (1) + x (2)  x (3)  x (4) w

53. The Learning Scenario x1x1 x2x2 + x (1) + x (2)  x (3)  x (4) w The demonstration is in augmented space. Conceptually, in augmented space, we adjust the weight vector to fit the data.

54. Weight Space w1w1 w2w2 + x A weight in the shaded area will give correct classification for the positive example. w

55. Weight Space w1w1 w2w2 + x A weight in the shaded area will give correct classification for the positive example. w  w =  x

56. Weight Space w1w1 w2w2  x A weight not in the shaded area will give correct classification for the negative example. w

57. Weight Space w1w1 w2w2  x A weight not in the shaded area will give correct classification for the negative example. w  w =  x

58. The Learning Scenario in Weight Space w1w1 w2w2 + x (1) + x (2)  x (3)  x (4)

59. The Learning Scenario in Weight Space w1w1 w2w2 + x (1) + x (2)  x (3)  x (4) To correctly classify the training set, the weight must move into the shaded area.

60. The Learning Scenario in Weight Space w1w1 w2w2 + x (1) + x (2)  x (3)  x (4) To correctly classify the training set, the weight must move into the shaded area. w0w0 w1w1

61. The Learning Scenario in Weight Space w1w1 w2w2 + x (1) + x (2)  x (3)  x (4) To correctly classify the training set, the weight must move into the shaded area. w0w0 w1w1 w2w2

62. The Learning Scenario in Weight Space w1w1 w2w2 + x (1) + x (2)  x (3)  x (4) To correctly classify the training set, the weight must move into the shaded area. w0w0 w1w1 w2w2 =w 3

63. The Learning Scenario in Weight Space w1w1 w2w2 + x (1) + x (2)  x (3)  x (4) To correctly classify the training set, the weight must move into the shaded area. w0w0 w1w1 w2w2 =w 3 w4w4

64. The Learning Scenario in Weight Space w1w1 w2w2 + x (1) + x (2)  x (3)  x (4) To correctly classify the training set, the weight must move into the shaded area. w0w0 w1w1 w2w2 =w 3 w4w4 w5w5

65. The Learning Scenario in Weight Space w1w1 w2w2 + x (1) + x (2)  x (3)  x (4) To correctly classify the training set, the weight must move into the shaded area. w0w0 w1w1 w2w2 =w 3 w4w4 w5w5 w6w6

66. The Learning Scenario in Weight Space w1w1 w2w2 + x (1) + x (2)  x (3)  x (4) To correctly classify the training set, the weight must move into the shaded area. w0w0 w1w1 w2w2 =w 3 w4w4 w5w5 w6w6 w7w7

67. The Learning Scenario in Weight Space w1w1 w2w2 + x (1) + x (2)  x (3)  x (4) To correctly classify the training set, the weight must move into the shaded area. w0w0 w1w1 w2w2 =w 3 w4w4 w5w5 w6w6 w7w7 w8w8

68. The Learning Scenario in Weight Space w1w1 w2w2 + x (1) + x (2)  x (3)  x (4) To correctly classify the training set, the weight must move into the shaded area. w0w0 w1w1 w2w2 =w 3 w4w4 w5w5 w6w6 w7w7 w8w8 w9w9

69. The Learning Scenario in Weight Space w1w1 w2w2 + x (1) + x (2)  x (3)  x (4) To correctly classify the training set, the weight must move into the shaded area. w0w0 w1w1 w2w2 =w 3 w4w4 w5w5 w6w6 w7w7 w8w8 w9w9 w 10

70. The Learning Scenario in Weight Space w1w1 w2w2 + x (1) + x (2)  x (3)  x (4) To correctly classify the training set, the weight must move into the shaded area. w0w0 w1w1 w2w2 =w 3 w4w4 w5w5 w6w6 w7w7 w8w8 w9w9 w 10 w 11

71. The Learning Scenario in Weight Space w1w1 w2w2 + x (1) + x (2)  x (3)  x (4) To correctly classify the training set, the weight must move into the shaded area. w0w0 w 11 Conceptually, in weight space, we move the weight into the feasible region.

72. Feed-Forward Neural Networks Learning Rules for Single-Layered Perceptron Networks Perceptron Learning Rule Adline Leaning Rule

73. Adline (Adaptive Linear Element) Widrow [1962] x1x1 x2x2 xmxm wi1wi1 wi2wi2 w im...... 

74. Adline (Adaptive Linear Element) Widrow [1962] x1x1 x2x2 xmxm wi1wi1 wi2wi2 w im......  Goal: In what condition, the goal is reachable?

75. LMS (Least Mean Square) Minimize the cost function (error function):

76. Gradient Decent Algorithm E(w)E(w) w1w1 w2w2 Our goal is to go downhill. Contour Map (w 1, w 2 ) w1w1 w2w2 ww

77. Gradient Decent Algorithm E(w)E(w) w1w1 w2w2 Our goal is to go downhill. Contour Map (w 1, w 2 ) w1w1 w2w2 ww How to find the steepest decent direction?

78. Gradient Operator Let f(w) = f (w 1, w 2, …, w m ) be a function over R m. Define

79. Gradient Operator ff ww ff ww ff ww df : positive df : zero df : negative Go uphillPlainGo downhill

80. ff ww ff ww ff ww The Steepest Decent Direction df : positive df : zero df : negative Go uphillPlainGo downhill To minimize f, we choose  w =   f

81. LMS (Least Mean Square) Minimize the cost function (error function):

82. Adline Learning Rule Minimize the cost function (error function): Weight Modification Rule

83. Learning Modes Batch Learning Mode: Incremental Learning Mode:

84. Comparisons Fundamental Habbian Assumption Gradient Decent ConvergenceIn finite steps Converge Asymptotically Constraint Linearly Separable Linear Independence Perceptron Learning Rule Adline Learning Rule (Widrow-Hoff)