Presentation is loading. Please wait.

Presentation is loading. Please wait.

Wissenschaftliches Arbeiten - Auswertung Folie 1 Artificial Neural Networks Uwe Lämmel Wismar Business School

Similar presentations


Presentation on theme: "Wissenschaftliches Arbeiten - Auswertung Folie 1 Artificial Neural Networks Uwe Lämmel Wismar Business School"— Presentation transcript:

1 Wissenschaftliches Arbeiten - Auswertung Folie 1 Artificial Neural Networks Uwe Lämmel Wismar Business School

2 Wissenschaftliches Arbeiten - Auswertung Folie 2

3 Wissenschaftliches Arbeiten - Auswertung Folie 3 Literature & Software – Robert Callan: The Essence of Neural Networks, Pearson Education, – JavaNNS based on SNNS: Stuttgarter Neuronale Netze Simulator

4 Wissenschaftliches Arbeiten - Auswertung Folie 4 Prerequisites NO algorithmic solution available or algorithmic solution too time consuming NO knowledge-based solution LOTS of experience (data) Try a NN

5 Wissenschaftliches Arbeiten - Auswertung Folie 5 Content Idea An artificial Neuron – Neural Network Supervised Learning – feed-forward networks Competitive Learning – Self-Organising Map Applications

6 Wissenschaftliches Arbeiten - Auswertung Folie 6 Two different types of knowledge processing Logic Conclusion sequential Aware of Symbol processing, Rule processing precise Engineering traditional" AI Perception, Recognition parallel Not aware of Neural Networks fuzzy Cognitive oriented Connectionism

7 Wissenschaftliches Arbeiten - Auswertung Folie 7 Idea A human being learns by example learning by doing –seeing(Perception), Walking, Speaking,… Can a machine do the same? A human being uses his brain. A brain consists of millions of single cells. A cell is connected with ten thousands of other cell. Is it possible to simulate a similar structure on a computer?

8 Wissenschaftliches Arbeiten - Auswertung Folie 8 Idea Artificial Neural Network –Information processing similar to processes in a mammal brain –heavy parallel systems, –able to learn –great number of simple cells ? Is it useful to copy nature ? –wheel, aeroplane,...

9 Wissenschaftliches Arbeiten - Auswertung Folie 9 Idea we need: software neurons software connection between neurons software learning algorithms An artificial neural network functions in a similar way a natural neural network does.

10 Wissenschaftliches Arbeiten - Auswertung Folie 10 A biological Neuron cell and cell nucleus Axon (Neurit) Dendrits Synapse Dendrits:(Input) Getting other activations Axon:(Output ) forward the activation (from 1mm up to 1m long) Synapse:transfer of activation: –To other cells, e.g.. Dendrits of other neurons –a cell has about to connections to other cells Cell Nucleus:(processing) evaluation of activation

11 Wissenschaftliches Arbeiten - Auswertung Folie 11 Abstraction Dendrits:weighted connections weight: real number Axon:output: real number Synapse:--- (identity: output is directly forwarded) Cell nucleus:unit contains simple functions input = (many) real numbers processing = evaluation of activation output = real number (~activation)

12 Wissenschaftliches Arbeiten - Auswertung Folie 12 An artificial Neuron net: input from the network w: weight of a connection act: activation f act : activation function : bias/threshold f out : output function (mostly ID) o: output w 1i w 2i w ji... oioi

13 Wissenschaftliches Arbeiten - Auswertung Folie 13 A simple switch Set parameters according to function: –Input neurons 1,2 : a 1,a 2 input pattern, here : o i =a i –weights of edges: w 1, w 2 –bias Give values for w 1, w 2, we can evaluate output o a 1 =__a 2 =__ o=__ net= o 1 w 1 +o 2 w 2 a= 1, if net> = 0, otherwise o = a w 1 =__ w 2 =__

14 Wissenschaftliches Arbeiten - Auswertung Folie 14 Questions find values for the parameters so that a logic function is simulated: –Logical AND –Logical OR –Logical exclusive OR (XOR) –Identity We want to process more than 2 inputs. Find appropriate parameter values. –Logical AND, 3 (4) inputs –OR, XOR iff 2 out of 4 are 1

15 Wissenschaftliches Arbeiten - Auswertung Folie 15 Mathematics in a Cell Propagation function net i (t) = o j w j = w 1i o 1 + w 2i o Activation a i (t) – Activation at time t Activation function f act : a i (t+1) = f act (a i (t), net i (t), i ) i – bias Output function f out : o i = f out (a i )

16 Wissenschaftliches Arbeiten - Auswertung Folie 16 Bias function -1,0 -0,5 0,0 0,5 1,0 -4,0-2,00,02,04,0 Identity -4,0 -2,0 0,0 2,0 4,0 -4,0-2,5-1,00,52,03,5 Activation Functions activation functions are sigmoid functions

17 Wissenschaftliches Arbeiten - Auswertung Folie 17 y = tanh(c·x) -1,0 -0,5 0,5 1,0 -0,60,61,0 c=1 c=2 c=3 -1,0 Activation Functions y = 1/(1+exp(-c·x)) 0,5 1,0 -1,0 0,0 1,0 c=1 c=3 c=10 Logistic function: activation functions are sigmoid functions

18 Wissenschaftliches Arbeiten - Auswertung Folie 18 Structure of a network layers input layer – contains input neurons output layer – contains output neurons hidden layer – contains hidden neurons An n-layer network has: n layer of connections which can be trained n+1 neuron layers n –1 hidden layers

19 Wissenschaftliches Arbeiten - Auswertung Folie 19 Neural Network - Definition A Neural Network is characterized by connections of many (a lot of) simple units (neurons) and units exchanging signals via these connections A neural Network is a coherent, directed graph which has weighted edges and each node (neurons, units ) contains a value (activation).

20 Wissenschaftliches Arbeiten - Auswertung Folie 20 Elements of a NN Connections/Links –directed, weighted graph –weight: w ij (from cell i to cell j) –weight matrix Propagation function –network input of a neuron will be calculated: net i = o j w ji Learning algorithm

21 Wissenschaftliches Arbeiten - Auswertung Folie 21 Example XOR-Network , ,5 TRUE

22 Wissenschaftliches Arbeiten - Auswertung Folie 22 Supervised Learning – feed-forward networks Idea An artificial Neuron – Neural Network Supervised Learning – feed-forward networks –Architecture –Backpropagation Learning Competitive Learning – Self-Organising Map Applications

23 Wissenschaftliches Arbeiten - Auswertung Folie 23 Multi-layer feed-forward network

24 Wissenschaftliches Arbeiten - Auswertung Folie 24 Feed-Forward Network

25 Wissenschaftliches Arbeiten - Auswertung Folie 25 Evaluation of the net output NiNi NjNj NkNk net j net k O j =act j O k =act k Training pattern p O i =p i Input-Layerhidden Layer(s)Output Layer

26 Wissenschaftliches Arbeiten - Auswertung Folie 26 Backpropagation Learning Algorithm supervised Learning error is a function of the weights w i : E(W) = E(w 1,w 2,..., w n ) We are looking for a minimal error minimal error = hollow in the error surface Backpropagation uses the gradient for weight approximation.

27 Wissenschaftliches Arbeiten - Auswertung Folie 27 error curve

28 Wissenschaftliches Arbeiten - Auswertung Folie 28 Problem error in output layer: difference output – teaching output error in a hidden layer? output teaching output input layer hidden layer

29 Wissenschaftliches Arbeiten - Auswertung Folie 29 Mathematics –modifying weights according to the gradient of the error function W = - E(W) – E(W) is the gradient – is a factor, called learning parameter -0,6-0,20,20,61

30 Wissenschaftliches Arbeiten - Auswertung Folie 30 Mathematics Here: modification of weights: W = – E(W) – E(W): Gradient – Proportion factor for the weight vector W, : learning factor E(W j ) = E(w 1j,w 2j,..., w nj )

31 Wissenschaftliches Arbeiten - Auswertung Folie 31 Error Function – Error function quadratic distance between real and teaching output of all patterns p: –t j - teaching output –o j - real output –Now: error for one pattern only (omitting pattern index p): – Modification of a weight: (1) (2)

32 Wissenschaftliches Arbeiten - Auswertung Folie 32 Backpropagation rule Multi layer networks Semi linear Activation function (monotone, differentiable, e.g. logistic function) Problem: no teaching outputs for hidden neurons

33 Wissenschaftliches Arbeiten - Auswertung Folie 33 Backpropagation Learning Rule (6.3) Start: 6.1 in more detail: dependencies: f out = Id (6.1) (6.2)

34 Wissenschaftliches Arbeiten - Auswertung Folie 34 The 3 rd and 2 nd Factor 3 rd Factor: dependency net input – weights 2 nd Factor: derivation of the activation function: (6.4) (6.5) (6.7)

35 Wissenschaftliches Arbeiten - Auswertung Folie 35 The 1 st Factor 1 st Factor: dependency error – output Error signal of hidden neuron j: (6.8) (6.10 ) – Error signal of output neuron j: (6.9) j : error signal

36 Wissenschaftliches Arbeiten - Auswertung Folie 36 Error Signal j = f act (net j )·(t j – o j ) Output neuron j: Hidden neuron j: j = f act (net j ) · k w jk (6.12 ) (6.11)

37 Wissenschaftliches Arbeiten - Auswertung Folie 37 Standard Backpropagation Rule For the logistic activation function: f ´ act (net j ) = f act (net j ) (1 – f act (net j )) = o j (1 –o j ) Therefore: and:

38 Wissenschaftliches Arbeiten - Auswertung Folie 38 error signal for f act = tanh For the activation function tanh holds: f´ act (net j ) = (1 – f ² act (net j )) = (1 – tanh² o j ) therefore:

39 Wissenschaftliches Arbeiten - Auswertung Folie 39 Backpropagation - Problems B C A

40 Wissenschaftliches Arbeiten - Auswertung Folie 40 Backpropagation-Problems – A: flat plateau –backpropagation goes very slowly –finding a minimum takes a lot of time – B: Oscillation in a narrow gorge –it jumps from one side to the other and back – C: leaving a minimum –if the modification in one training step is to high, the minimum can be lost

41 Wissenschaftliches Arbeiten - Auswertung Folie 41 Solutions: looking at the values change the parameter of the logistic function in order to get other values Modification of weights depends on the output: if o i =0 no modification will take place If we use binary input we probably have a lot of zero-values: Change [0,1] into [-½, ½] or [-1,1] use another activation function, eg. tanh and use [-1..1] values

42 Wissenschaftliches Arbeiten - Auswertung Folie 42 Solution: Quickprop assumption: error curve is a square function calculate the vertex of the curve slope of the error curve:

43 Wissenschaftliches Arbeiten - Auswertung Folie 43 Resilient Propagation (RPROP) – sign and size of the weight modification are calculated separately: b ij (t) – size of modification b ij (t-1) + if S(t-1) S(t) > 0 b ij (t) = b ij (t-1) - if S(t-1) S(t) < 0 b ij (t-1)otherwise + >1 : both ascents are equal big step 0< - <1 : ascents are different smaller step -b ij (t)if S(t-1)>0 S(t) > 0 w ij (t) = b ij (t)íf S(t-1)<0 S(t) < 0 - w ij (t-1)if S(t-1) S(t) < 0(*) -sgn(S(t)) b ij (t)otherwise (*) S(t) is set to 0, S(t):=0 ; at time (t+1) the 4 th case will be applied.

44 Wissenschaftliches Arbeiten - Auswertung Folie 44 Limits of the Learning Algorithm – it is not a model for biological learning – we have no teaching output in a natural learning process –In a natural neural network there are no feedbacks (at least nobody has discovered yet) –training of a artificial neural network is rather time consuming

45 Wissenschaftliches Arbeiten - Auswertung Folie 45 Development of an NN-application calculate network output compare to teaching output use Test set data evaluate output compare to teaching output change parameters modify weights input of training pattern build a network architecture quality is good enough error is too high quality is good enough

46 Wissenschaftliches Arbeiten - Auswertung Folie 46 Possible Changes – Architecture of NN –size of a network –shortcut connection –partial connected layers –remove/add links –receptive areas – Find the right parameter values –learning parameter –size of layers –using genetic algorithms

47 Wissenschaftliches Arbeiten - Auswertung Folie 47 Memory Capacity - Experiment – output-layer is a copy of the input-layer – training set consisting of n random pattern – error: –error = 0 network can store more than n patterns –error >> 0 network can not store n patterns –memory capacity: error > 0 and error = 0 for n-1 patterns and error >>0 for n+1 patterns

48 Wissenschaftliches Arbeiten - Auswertung Folie 48 Summary – Backpropagation is a Backpropagation of Error Algorithm –works like gradient descent –Activation Functions: Logistics, tanh –Meaning of Learning parameter – Modifications –RPROP –Backprop Momentum –QuickProp – Finding an appropriate Architecture: –Memory Size of a Network –Modifications in layer connection – Applications

49 Wissenschaftliches Arbeiten - Auswertung Folie 49 Binary Coding of nominal values I – no order relation, n-values – n neurons, – each neuron represents one and only one value: –example: red, blue, yellow, white, black 1,0,0,0,0 0,1,0,0,0 0,0,1,0,0... –disadvantage: n neurons necessary, but only one of them is activated lots of zeros in the input

50 Wissenschaftliches Arbeiten - Auswertung Folie 50 Binary Coding of nominal values II – no order-relation, n values – m neurons, of it k neurons switched on for one single value –requirement: (m choose k) n –example: red, blue, yellow, white, black 1,1,0,0 1,0,1,0 1,0,0,1 0,1,1,0 0,1,0,1 4 neuron, 2 of it switched on, (4 choose 2) > 5 –advantage: – fewer neurons – balanced ratio of 0 and 1

51 Wissenschaftliches Arbeiten - Auswertung Folie 51 A1: Credit history A2: debt A3: collateral A4: income Example Credit Scoring network architecture depends on the coding of input and output How can we code values like good, bad, 1, 2, 3,...?

52 Wissenschaftliches Arbeiten - Auswertung Folie 52 Example Credit Scoring A1: A2: A3: A4: class

53 Wissenschaftliches Arbeiten - Auswertung Folie 53 Supervised Learning – feed-forward networks Idea An artificial Neuron – Neural Network Supervised Learning – feed-forward networks Competitive Learning – Self-Organising Map –Architecture –Learning –Visualisation Applications

54 Wissenschaftliches Arbeiten - Auswertung Folie 54 Self Organizing Maps (SOM) A natural brain can organize itself Now we look at the position of a neuron and its neighbourhood Kohonen Feature Map two layer pattern associator -Input layer is fully connected with map-layer -Neurons of the map layer are fully connected to each other (virtually)

55 Wissenschaftliches Arbeiten - Auswertung Folie 55 Clustering -objective: All inputs of a class are mapped onto one and the same neuron f Input set A output B aiai -Problem: classification in the input space is unknown -Network performs a clustering

56 Wissenschaftliches Arbeiten - Auswertung Folie 56 Winner Neuron Kohonen- Layer Input-Layer Winner Neuron

57 Wissenschaftliches Arbeiten - Auswertung Folie 57 Learning in an SOM Choose an input k randomly Detect the neuron z which has the maximal activity Adapt the weights in the neighbourhood of z: neuron i within a radius r of z. Stop if a certain number of learning steps is finished otherwise decrease learning rate and radius, go on with step 1.

58 Wissenschaftliches Arbeiten - Auswertung Folie 58 A Map Neuron –look at a single neuron (without feedback): –Activation: –Output:f out = Id

59 Wissenschaftliches Arbeiten - Auswertung Folie 59 Centre of Activation -Idea: highly activated neurons push down the activation of neurons in the neighbourhood -Problem: Finding the centre of activation: -Neuron j with a maximal net-input -Neuron j, having a weight vector w j which is similar to the input vector (Euklidian Distance): z: x - w z = min j x - w j

60 Wissenschaftliches Arbeiten - Auswertung Folie 60 Changing Weights - weights to neurons within a radius z will be increased: w j (t+1) = w j (t) + h jz (x(t)-w j (t)), j z x-input w j (t+1) = w j (t), otherwise -Amount of influence depends on the distance to the centre of activation: (amount of change w j ?) -Kohonen uses the function : z determines the shape of the curve: - z small high + sharp - z high wide + flat

61 Wissenschaftliches Arbeiten - Auswertung Folie 61 Changing weights -Simulation by a Gauß-curve -Changing Weights by a learning rate (t), going down to zero -Weight change:: w j+1 (t+1) = w j (t) + h jz (x(t)-w j (t)), j z w j+1 (t+1) = w j (t), otherwise - Requirements: -Pattern input by random ! - z (t) and z (t) are monotone decreasing functions in t. Mexican-Hat-Approach 0 0,

62 Wissenschaftliches Arbeiten - Auswertung Folie 62 SOM Training find the winner neuron z for an input pattern p (minimal Euclidian distance) adapt weights of connections winner neuron -input neurons neighbours – input neurons Kohonen layer input pattern m p WjWj

63 Wissenschaftliches Arbeiten - Auswertung Folie 63 A1: Credit History A2: Debts A3: Collateral A4: Income Example Credit Scoring We do not look at the Classification SOM performs a Clustering

64 Wissenschaftliches Arbeiten - Auswertung Folie 64 Credit Scoring – good = {5,6,9,10,12} – average = {3, 8, 13} – bad= {1,2,4,7,11,14}

65 Wissenschaftliches Arbeiten - Auswertung Folie 65 Credit Scoring – Pascal tool box (1991) – 10x10 neurons – 32,000 training steps

66 Wissenschaftliches Arbeiten - Auswertung Folie 66 Visualisation of a SOM Colour reflects Euclidian distance to input NetDemo TSPDemo Weights used as coordinates of a neuron Colour reflects cluster ColorDemo

67 Wissenschaftliches Arbeiten - Auswertung Folie 67 Example TSP – Travelling Salesman Problem –A salesman has to visit certain cities and will return to his home. Find an optimal route! –problem has exponential complexity: (n-1)! routes Experiment: Pascal Program, /32 states in Mexico?

68 Wissenschaftliches Arbeiten - Auswertung Folie 68 Nearest Neighbour: Example – Some cities in Northern Germany: – Initial city is Hamburg Kiel Rostock Berlin Hamburg Hannover Frankfurt Essen Schwerin Exercise: Put in the coordinates of the capitals of all the 31 Mexican States + Mexico/City. Find a solution for the TSP using a SOM!

69 Wissenschaftliches Arbeiten - Auswertung Folie 69 SOM solves TSP input Kohonen layer w 1i = si x w 2i = si y Draw a neuron at position: (x,y)=(w 1i,w 2i ) X Y

70 Wissenschaftliches Arbeiten - Auswertung Folie 70 SOM solves TSP – Initialisation of weights: –weights to input (x,y) are calculated so that all neurons form a circle –The initial circle will be expanded to a round trip – Solutions for problems of several hundreds of towns are possible – Solution may be not optimal!

71 Wissenschaftliches Arbeiten - Auswertung Folie 71 Applications – Data Mining - Clustering –Customer Data –Weblog –... – You have a lot of data, but no teaching data available – unsupervised learning –you have at least an idea about the result – Can be applied as a first approach to get some training data for supervised learning

72 Wissenschaftliches Arbeiten - Auswertung Folie 72 Applications Pattern recognition (text, numbers, faces): number plates, access at cash automata, Similarities between molecules Checking the quality of a surface Control of autonomous vehicles Monitoring of credit card accounts Data Mining

73 Wissenschaftliches Arbeiten - Auswertung Folie 73 Applications Speech recognition Control of artificial limbs classification of galaxies Product orders (Supermarket) Forecast of energy consumption Stock value forecast

74 Wissenschaftliches Arbeiten - Auswertung Folie 74 Application - Summary Classification Clustering Forecast Pattern recognition Learning by examples, generalization Recognition of not known structures in large data

75 Wissenschaftliches Arbeiten - Auswertung Folie 75 Application – Data Mining: –Customer Data –Weblog – Control of... – Pattern Recognition –Quality of surfaces – possible if you have training data...

76 Wissenschaftliches Arbeiten - Auswertung Folie 76 The End


Download ppt "Wissenschaftliches Arbeiten - Auswertung Folie 1 Artificial Neural Networks Uwe Lämmel Wismar Business School"

Similar presentations


Ads by Google