Presentation is loading. Please wait.

Presentation is loading. Please wait.

Introduction to Artificial Neural Networks 主講人 : 虞台文.

Similar presentations


Presentation on theme: "Introduction to Artificial Neural Networks 主講人 : 虞台文."— Presentation transcript:

1 Introduction to Artificial Neural Networks 主講人 : 虞台文

2 Content Fundamental Concepts of ANNs. Basic Models and Learning Rules – Neuron Models – ANN structures – Learning Distributed Representations Conclusions

3 Introduction to Artificial Neural Networks Fundamental Concepts of ANNs

4 What is ANN? Why ANN? ANN  Artificial Neural Networks – To simulate human brain behavior – A new generation of information processing system.

5 Applications Pattern Matching Pattern Recognition Associate Memory (Content Addressable Memory) Function Approximation Learning Optimization Vector Quantization Data Clustering...

6 Applications Pattern Matching Pattern Recognition Associate Memory (Content Addressable Memory) Function Approximation Learning Optimization Vector Quantization Data Clustering... Traditional Computers are inefficient at these tasks although their computation speed is faster.

7 The Configuration of ANNs An ANN consists of a large number of interconnected processing elements called neurons. – A human brain consists of ~10 11 neurons of many different types. How ANN works? – Collective behavior.

8 The Biologic Neuron

9 樹狀突 軸突 二神經原之神 經絲接合部分

10 The Biologic Neuron Excitatory or Inhibitory

11 The Artificial Neuron x1x1 x2x2 xmxm wi1wi1 wi2wi2 w im yiyi f (.)a (.) ii 

12 The Artificial Neuron x1x1 x2x2 xmxm wi1wi1 wi2wi2 w im yiyi f (.)a (.) ii 

13 The Artificial Neuron x1x1 x2x2 xmxm wi1wi1 wi2wi2 w im yiyi f (.)a (.) ii  w ij positive  excitatory negative  inhibitory zero  no connection w ij positive  excitatory negative  inhibitory zero  no connection

14 The Artificial Neuron x1x1 x2x2 xmxm wi1wi1 wi2wi2 w im yiyi f (.)a (.) ii  Proposed by McCulloch and Pitts [1943] M-P neurons Proposed by McCulloch and Pitts [1943] M-P neurons

15 What can be done by M-P neurons? A hard limiter. A binary threshold unit. Hyperspace separation. w 1 x 1 + w 2 x 2 =  x1x1 x2x2 x1x1 x2x2   y w1w1 w2w2 1 0

16 What ANNs will be? ANN  A neurally inspired mathematical model. Consists a large number of highly interconnected PEs. Its connections (weights) holds knowledge. The response of PE depends only on local information. Its collective behavior demonstrates the computation power. With learning, recalling and, generalization capability.

17 Three Basic Entities of ANN Models Models of Neurons or PEs. Models of synaptic interconnections and structures. Training or learning rules.

18 Introduction to Artificial Neural Networks Basic Models and Learning Rules Neuron Models ANN structures Learning

19 Processing Elements f (.)a (.) ii  What integration functions we may have? What activation functions we may have? Extensions of M-P neurons

20 Integration Functions f (.)a (.) ii  Quadratic Function Spherical Function Polynomial Function M-P neuron

21 Activation Functions f (.)a (.) ii  M-P neuron: (Step function) 1 a f

22 Activation Functions f (.)a (.) ii  Hard Limiter (Threshold function) 1 a 11 f

23 Activation Functions f (.)a (.) ii  Ramp function: 1 a 1 f

24 Activation Functions f (.)a (.) ii  Unipolar sigmoid function:

25 Activation Functions f (.)a (.) ii  Bipolar sigmoid function:

26 x y Example: Activation Surfaces L1L1 L2L2 L3L3 x y L1L1 L2L2 L3L3

27 x y L1L1 L2L2 L3L3 x  1=0 y  1=0  x  y+4=0 x y L1L1 L2L2 L3L3 1 0 1=11=1 0 1 2=12=1 11 11  3 =  4

28 Example: Activation Surfaces x y L1L1 L2L2 L3L3 x y L1L1 L2L2 L3L Region Code

29 x y L1L1 L2L2 L3L3 Example: Activation Surfaces z=1 z=0 L4L4 z x y L1L1 L2L2 L3L3

30 x y L1L1 L2L2 L3L3 Example: Activation Surfaces z=1 z=0 L4L4 z x y L1L1 L2L2 L3L3 1  4 =

31 Example: Activation Surfaces L4L4 z x y L1L1 L2L2 L3L3 M-P neuron: (Step function)

32 Example: Activation Surfaces L4L4 z x y L1L1 L2L2 L3L3 =2 =3 =5 =10 Unipolar sigmoid function:

33 Introduction to Artificial Neural Networks Basic Models and Learning Rules Neuron Models ANN structures Learning

34 ANN Structure (Connections)

35 Single-Layer Feedforward Networks y1y1 y2y2 ynyn x1x1 x2x2 xmxm w 11 w 12 w1mw1m w 21 w 22 w2mw2m wn1wn1 w nm wn2wn2...

36 Multilayer Feedforward Networks... x1x1 x2x2 xmxm y1y1 y2y2 ynyn Hidden Layer Input Layer Output Layer

37 Multilayer Feedforward Networks Pattern Recognition Input Analysis Classification Output Learning Where the knowledge from?

38 Single Node with Feedback to Itself Feedback Loop

39 Single-Layer Recurrent Networks... x1x1 x2x2 xmxm y1y1 y2y2 ynyn

40 Multilayer Recurrent Networks x1x1 x2x2 x3x3 y1y1 y2y2 y3y3...

41 Introduction to Artificial Neural Networks Basic Models and Learning Rules Neuron Models ANN structures Learning

42 Consider an ANN with n neurons and each with m adaptive weights. Weight matrix:

43 Learning Consider an ANN with n neurons and each with m adaptive weights. Weight matrix: To “Learn” the weight matrix. How?

44 Learning Rules Supervised learning Reinforcement learning Unsupervised learning

45 Supervised Learning Learning with a teacher Learning by examples  Training set

46 Supervised Learning x Error signal Generator Error signal Generator d y ANN W

47 Reinforcement Learning Learning with a critic Learning by comments

48 Reinforcement Learning x Critic signal Generator Critic signal Generator y ANN W Reinforcement Signal

49 Unsupervised Learning Self-organizing Clustering – Form proper clusters by discovering the similarities and dissimilarities among objects.

50 Unsupervised Learning x y ANN W

51 The General Weight Learning Rule Input: Output: ii wi1wi1 wi2wi2 w ij w i,m-1 x1x1 x2x2 xjxj x m-1 yiyi ii

52 The General Weight Learning Rule Input: Output: ii wi1wi1 wi2wi2 w ij w i,m-1 x1x1 x2x2 xjxj x m-1 yiyi ii We want to learn the weights & bias.

53 The General Weight Learning Rule Input: ii wi1wi1 wi2wi2 w ij w i,m-1 x1x1 x2x2 xjxj x m-1 ii Let x m =  1 and w im =  i.

54 The General Weight Learning Rule Input: ii wi1wi1 wi2wi2 w ij w i,m-1 x1x1 x2x2 xjxj x m-1 Let x m =  1 and w im =  i. x m =  1 w im =  i

55 The General Weight Learning Rule Input: ii wi1wi1 wi2wi2 w ij w i,m-1 x1x1 x2x2 xjxj x m-1 x m =  1 w im =  i yiyi w i =(w i1, w i2,…,w im ) T  w i (t) = ? We want to learn

56 The General Weight Learning Rule wiwi wiwi x yiyi r didi Learning Signal Generator Learning Signal Generator

57 The General Weight Learning Rule wiwi wiwi x yiyi r didi Learning Signal Generator Learning Signal Generator

58 The General Weight Learning Rule wiwi wiwi x yiyi r didi Learning Signal Generator Learning Signal Generator

59 The General Weight Learning Rule wiwi wiwi x yiyi r didi Learning Signal Generator Learning Signal Generator  Learning Rate

60 The General Weight Learning Rule w i =(w i1, w i2,…,w im ) T We want to learn Discrete-Time Weight Modification Rule: Continuous-Time Weight Modification Rule:

61 Hebb’s Learning Law Hebb [1994] hypothesis that when an axonal input from A to B causes neuron B to immediately emit a pulse (fire) and this situation happens repeatedly or persistently. Then, the efficacy of that axonal input, in terms of ability to help neuron B to fire in future, is somehow increased. Hebb’s learning rule is a unsupervised learning rule.

62 Hebb’s Learning Law +  + 

63 Introduction to Artificial Neural Networks Distributed Representations

64 Distributed Representation: –An entity is represented by a pattern of activity distributed over many PEs. –Each Processing element is involved in representing many different entities. Local Representation: –Each entity is represented by one PE.

65 Example P0P0 P1P1 P2P2 P3P3 P4P4 P5P5 P6P6 P7P7 P8P8 P9P9 P 10 P 11 P 12 P 13 P 14 P 15 + _ ++ ____ ___ + _ ++ ____ + _ + _ ++ _ +++ _ + _ ++ _ + __ ++++ _ Dog Cat Bread

66 P0P0 P1P1 P2P2 P3P3 P4P4 P5P5 P6P6 P7P7 P8P8 P9P9 P 10 P 11 P 12 P 13 P 14 P 15 + _ ++ ____ ___ + _ ++ ____ + _ + _ ++ _ +++ _ + _ ++ _ + __ ++++ _ Dog Cat Bread Advantages Act as a content addressable memory. P0P0 P1P1 P2P2 P3P3 P4P4 P5P5 P6P6 P7P7 P8P8 P9P9 P 10 P 11 P 12 P 13 P 14 P What is this?

67 Advantages P0P0 P1P1 P2P2 P3P3 P4P4 P5P5 P6P6 P7P7 P8P8 P9P9 P 10 P 11 P 12 P 13 P 14 P 15 + _ ++ ____ ___ + _ ++ ____ + _ + _ ++ _ +++ _ + _ ++ _ + __ ++++ _ Dog Cat Bread Act as a content addressable memory. Make induction easy. P0P0 P1P1 P2P2 P3P3 P4P4 P5P5 P6P6 P7P7 P8P8 P9P9 P 10 P 11 P 12 P 13 P 14 P 15 + __ + ____ __ Fido Dog has 4 legs? How many for Fido?

68 Advantages P0P0 P1P1 P2P2 P3P3 P4P4 P5P5 P6P6 P7P7 P8P8 P9P9 P 10 P 11 P 12 P 13 P 14 P 15 + _ ++ ____ ___ + _ ++ ____ + _ + _ ++ _ +++ _ + _ ++ _ + __ ++++ _ Dog Cat Bread Act as a content addressable memory. Make induction easy. Make the creation of new entities or concept easy (without allocation of new hardware). ++ ___ ++ _ + ___ +++ _ Doughnut Add doughnut by changing weights.

69 Advantages P0P0 P1P1 P2P2 P3P3 P4P4 P5P5 P6P6 P7P7 P8P8 P9P9 P 10 P 11 P 12 P 13 P 14 P 15 + _ ++ ____ ___ + _ ++ ____ + _ + _ ++ _ +++ _ + _ ++ _ + __ ++++ _ Dog Cat Bread Act as a content addressable memory. Make induction easy. Make the creation of new entities or concept easy (without allocation of new hardware). Fault Tolerance. Some PEs break down don’t cause problem.

70 Disadvantages How to understand? How to modify? Learning procedures are required.


Download ppt "Introduction to Artificial Neural Networks 主講人 : 虞台文."

Similar presentations


Ads by Google