PSY105 Neural Networks 2/5 2. “A universe of numbers”

Lecture 1 recap We can describe patterns at one level of description that emerge due to rules followed at a lower level of description. Neural network modellers hope that we can understand behaviour by creating models of networks of artificial neurons.

Warren McCullock First artificial neuron model Warren McCulloch (neurophysiologist) Walter Pitts (mathematician)

A simple artificial neuron Threshold Add weight input activation Multiply inputs by weights and add. If the sum is larger than a threshold output 1, otherwise output 0 Threshold logic unit (TLU)

0 1 output activation TLU: the output relation threshold The relation is non-linear – small changes in activation give different changes in the output depending on the initial activation

Model neuron function, reminders… Inputs vary, they can be 0 or 1 – Weights change, effectively ‘interpreting’ inputs There is a weight for each input – This can be a +ve number (excitation) or a –ve number (inhibition) – Weights do not change when inputs change Activation = weighted sum of inputs – Activation = input1 x weight1 + input2xweight2 etc If activation>threshold, output = 1, otherwise output=0 – Threshold = 1

States, weights & functions States: all the possible combinations of inputs Weights: how each input is multiplied before contributing to the activation of the unit Functions: a way inputs are combined to produce outputs

Computing with neurons: identify (1) Input 0 1 Weight 0.7 Activation Output 0 X input weight output ? Threshold = 1 Act. State 1 State 2

Computing with neurons: identity (2) Input 0 1 Weight 1 Activation 0 1 Output 0 1 input weight output ? Threshold = 1 Act. State 1 State 2

Question: How could you use these simple neurons (TLUs) to compute the AND function? Input Input Output 0 1

Computing with neurons: AND inputs weights output ? Input Input Activation Output 0 1 Threshold = 1, Weight 1 = 0.5, Weight 2 = 0.5 Act. State 1 State 2 State 3 State 4

Networks of such neurons are Turing complete

Semilinear node Squashing function weight input Add activation

0 1 output activation Semilinear node: the output relation (squashing function) threshold