Neural Network Architecture Session 2 Course : T0293 – NEURO COMPUTING Year : 2013 Neural Network Architecture Session 2

Learning Outcomes At the end of this session, student will be able to: Explain the knowledge representation of a neural network system and differentiate neural network architectures (LO1, LO2) T0293 - Neuro Computing

Lecture Outline Single-Layer Feedforward Networks Multilayer Feedforward Networks Recurrent Networks Knowledge Representation T0293 - Neuro Computing

Network Architectures Neurons (processing units) of a neural network are structured based on the learning algorithm used to train the system. Learning algorithms (rules) in the design of neural networks is structured. In general, there are three types of network architectures (structures): Single-Layer Feedforward Networks Multilayer Feedforward Networks Recurrent Networks T0293 - Neuro Computing

Single-Layer Feedforward Networks Figure 2.1 Feedforwad network with a single layer of neurons. Information Flow Single-layer feedforward network is the simplest form of a layered network. There are two layers: Input Layer Output Layer (Computation Nodes) It is feedforward, means the information flow from input to output and not vice versa. Input layer of source nodes are not counted because no computation is performed. Sources: Haykin, S. (2009). Neural Networks and Learning Machines, 3rd ed. Pearson. ISBN: 978-0-13-129376-2 T0293 - Neuro Computing

Multilayer Feedforward Networks Figure 2.2 Fully connected feedforward network with one hidden layer and one output layer. Information Flow Multilayer feedforward networks has one or more hidden layers. By adding hidden layers, the network is enabled to extract higher-order statistics from its input. In this structure, the computation nodes are called hidden neurons or hidden units. The example architecture in Figure 2.2 is referred to as a 10-4-2 network: 10 source nodes 4 hidden neurons 2 output neurons Fully Connected VS Partially Connected Sources: Haykin, S. (2009). Neural Networks and Learning Machines, 3rd ed. Pearson. ISBN: 978-0-13-129376-2 T0293 - Neuro Computing

Recurrent Networks Recurrent neural network is different from feedforward neural network because it has at least one feedback loop. The presence of feedback loop has a profound impact on the learning capability of the network and its performance. The feedback loops involve the use of particular branches composed of unit-time delay elements (denoted by z-1) Structure at Figure 2.3: No self-feedback loops in the network No hidden neurons Figure 2.3 Recurrent network with no self-feedback loops and no hidden neurons. Sources: Haykin, S. (2009). Neural Networks and Learning Machines, 3rd ed. Pearson. ISBN: 978-0-13-129376-2 T0293 - Neuro Computing

Figure 2.4 Recurrent network with hidden neurons. Structure at Figure 2.4: Contains self-feedback loops in the network Contains hidden neurons The feedback connections originate from the hidden neurons as well as from the output neurons. Figure 2.4 Recurrent network with hidden neurons. Sources: Haykin, S. (2009). Neural Networks and Learning Machines, 3rd ed. Pearson. ISBN: 978-0-13-129376-2 T0293 - Neuro Computing

Knowledge Representation “Knowledge refers to stored information or models used by a person or machine to interpret, predict, and appropriately respond to the outside world.” – Fischler and firschein The primary characteristics of knowledge representation are twofold: What information is actually made explicit How the information is physically encoded for subsequent use In short, knowledge representation is goal directed. “Intelligence” machine, i.e., neural network systems, could be a good solution depends on a good representation of knowledge. PROBLEMS: Knowledge representation form are highly diverse. T0293 - Neuro Computing

Knowledge of the world consists of two kinds of information: In order to achieve their goals, neural network systems have to represent real-world data (environment) and have to model sufficiently consistently with the real-world. Knowledge of the world consists of two kinds of information: The known world state, represent by facts about what is and what has been known (prior information). Observations (measurements) of the world, obtained by means of sensors designed to probe the environment, in which the neural network is supposed to operate. The examples (real-world data) could categories as Labeled and Unlabeled data. A set of input-output pairs, with each pair consisting of an input signal and the corresponding desired response, is referred to as a set of training data or training sample. The training sample is used to train a neural network system. T0293 - Neuro Computing

Case Study (Problem) The handwritten-digit-recognition problem: Input Image with black or white pixels. Each image representing one of 10 digits that are well separated from the background. Desired Output The “identity” of the particular digit whose image is presented to the network as the input signal. The Training Samples Large variety of handwritten digits that are representative of a real-world situation. T0293 - Neuro Computing

Training Samples (or Data) Case Study (Solution) Design of a neural network: Learning Phase An appropriate architecture is selected for the neural network, where: Input Layer Consisting of source nodes equal in number to the pixels of an input image. Output Layer Consisting of 10 neurons (one for each digit). Then, a subset of examples is used to train the network by means of a suitable algorithm. Sources: Haykin, S. (2009). Neural Networks and Learning Machines, 3rd ed. Pearson. ISBN: 978-0-13-129376-2 Training Samples (or Data) Neural Network System is fed to T0293 - Neuro Computing

Testing Phase The recognition performance of the trained network is tested with data not seen before. An input image is presented to the network, but the network is not told the identity of the digit which that particular image represents. The performance of the neural network is asses by comparing the digit recognition reported by the network with the actual identity of the digit in question. Successful performance on the test patterns is called generalization. Sources: Haykin, S. (2009). Neural Networks and Learning Machines, 3rd ed. Pearson. ISBN: 978-0-13-129376-2 Unknown Data Neural Network System is tested by recognized results Results T0293 - Neuro Computing

Summary Single-layer feedforward network is the simplest form of a layered network. Multilayer feedforward networks has one or more hidden layers. Recurrent neural network is different from feedforward neural network because it has at least one feedback loop. Knowledge representation is goal directed. In order to achieve their goals, neural network systems have to represent real-world data (environment) and have to model sufficiently consistently with the real-world. T0293 - Neuro Computing

PerceptronClassification Demo A perceptron is computer code that models the behavior of a single biological neuron. Perceptrons were one of the very earliest types of machine-learning techniques and are the predecessors to neural networks. T0293 - Neuro Computing

Classification T0293 - Neuro Computing http://visualstudiomagazine.com/Articles/2013/04/01/Classification-Using-Perceptrons.aspx?Page=1 T0293 - Neuro Computing

Perceptron Training Method loop until done foreach training data item compute output using weights and bias if the output is incorrect then adjust weights and bias compute error if error < smallest error so far smallest error so far = error save new weights and bias end if increment loop counter end foreach end loop return best weights and bias values found T0293 - Neuro Computing

training Although the training method doesn't have very many lines of code, it's quite clever. The heart of the method is the line that adjusts the weights when a training item is incorrectly classified: weights[j] = weights[j] + (alpha * delta * trainData[i][j]); The adjustment value has three terms: alpha, delta and the input value associated with the weight being adjusted. Delta is computed as desired - output, that is, the desired value stored in the Y array minus the output produced by the current weights and bias. T0293 - Neuro Computing

The learning rate, alpha, is typically some small value such as 0 The learning rate, alpha, is typically some small value such as 0.001 or 0.00001, and throttles the magnitude of change when weights and bias values are adjusted. An advanced alternative is to allow alpha to change, starting out with a relatively large value and gradually decreasing. The idea is to have large adjustment jumps early on but then make the jumps finer-grained later. Another advanced alternative is to set alpha to a small random value each time it's used. The version of the perceptron-training algorithm presented here iterates a fixed maxEpochs times (500 in the demo). T0293 - Neuro Computing

TM1 (group) Create a Neural Network program using Backpropagation using C#, and analyze it: http://visualstudiomagazine.com/articles/2013/08/01/neural-network-back-propagation-using-c.aspx T0293 - Neuro Computing

References Textbook Haykin, S. (2009). Neural Networks and Learning Machines, 3rd ed. Pearson. ISBN: 978-0-13-129376-2 Web http://www.dspguide.com/ch26/2.htm T0293 - Neuro Computing

END T0293 - Neuro Computing