1. 3. Feedforward Nets (1) Architecture
Ref. Paul Werbos, “Backpropagation through Time” Proc. IEEE 90/10
(1) Architecture
Ordered Output Nodes 1 X m X 1 1 m
m+1 i
N+1
N+n
Ordered Hidden Nodes 1 Y n Y

2. (2) Examples (Pseudo-layer form) :
The following diagrams show only extra connections not present in a layered arch.
like direct input-to-output connections.

3. ` (3) Learning Rule For output nodes x j i k net F _ = d net F _ = d
= desired j i k i
net F _ = d k
net F _ = d
For output nodes

4. (4) Example 1 X 2 w 12 w 12 1 X δ2 δ3 δ4 Y 1 1 x 2 x 3 x 4 x 1 4 3 Y 1

5. Section 3.2,3.3 Summary:
Using matrices and vectors, more compact description of the backpropagation equation is possible. It is easier to understand BackProp.
MLP can be generalized to the Feedforward (FF) net that allows new connections for input to output, hidden to hidden, and output to output. The result is a more complex architecture with more connections that is more flexible. This added flexibility can enhance accuracy at the cost of complexity.
Since the general FF net includes MLP as a special case, it will always perform better in learning accuracy. Further, it will stabilize learning, and robust to local minima – AI Expert Magazine, 91/7.
The learning rule for the general FF net is identical to that of the MLP BP in that the error signals are back-propagated through the weights and nonlinear nodes. Although we did not derive it, it is obvious from the MLP BP.
Keywords: Feed-forward net, Ordered nodes, Learning Rule, Chain Rule. Feedback.

6. Student Questions:
1. General Feedforward net or an MLP with more complexity is more flexible. So, it can approximate a more complex function. But, it may poorly generalize by overfitting to the training data. Should Be a way to compromise between model complexity and test data accuracy.
2. FF net has no feedback in its architecture. But, isn’t weight learning another kind of a feedback process ?
3. FF is more flexible than an MLP. How much so ?
4. How can the weight scale affect the output result ?
5. Can the input variable have different dimensions ?

7. 6. The FF net has connections between the hidden nodes, too
6. The FF net has connections between the hidden nodes, too. Since the weight change for the left nodes is the result of accumulating the backpropagation of the error signals, can we say that the FF net will learn with fewer iterations than MLP ?
7. What are the advantages and disadvantages of FF and MLP ?
8. Which is more popular in use ? Or does it depend on specific applications ?