1. Artificial Neural Networks KONG DA, XUEYU LEI & PAUL MCKAY

2. Digit Recognition  Convolutional Neural Network  Inspired by the visual cortex  Our example: Handwritten digit recognition Reference: LeCun et al. Back propagation Applied to Handwritten Zip Code Recognition. 1989Back propagation Applied to Handwritten Zip Code Recognition

3. Method Back Propagation: 1.Propagation: 2.Weight update: http://tex.stackexchange.com/questions/16232 6/drawing-back-propagation-neural-network http://en.wikipedia.org/wiki/Backpropagation

4. Back propagation Le Cun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D. (1990). Back-Propagation Applied to Handwritten Zipcode Recognition. Neural Computation, 1(4).

5. Overfitting  Bias due to experience https://www.youtube.com/watch?v=ZgqsaDnsEq8

6. Overfitting http://pingax.com/regularization-implementation-r/

7. Possible solutions Le Cun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D. (1990). Back-Propagation Applied to Handwritten Zipcode Recognition. Neural Computation, 1(4).

8. Demonstration A neural network based on back-propagation achieves an high accuracy on a modified NIST database of hand-written digits

9. Demonstration

10. Learning Rate(LR) Mean Square Error (MSE) Accuracy

11. Learning Rate Learning rate=0.001 Learning rate=0.01 Learning rate=0.0001

12. MSE http://www.codeproject.com/Articles/16650/Neural-Network-for-Recognition-of-Handwritten-Digi#Backpropagation Back-propagation progress in one epoch The left-hand axis is for MSE the right-hand axis others nitial learning rate (eta) = 0.001 Minimum learning rate (eta) = 0.00005 Rate of decay for learning rate (eta) = 0.794183335 Decay rate is applied after this number of backprops = 120000

13. Accuracy http://www.codeproject.com/Articles/16650/Neural-Network-for-Recognition-of-Handwritten-Digi#Backpropagation Total number of testing set:10000 Total number of errors: 74 (non-distorted) For each pattern: No Expected value => misrecognized value Accuracy=99.26%

14. Multi-digit number recognition  Large-scale deep neural network  11 convolutional layers  Street View House Numbers  Per-digit recognition: 97.84%  Tens of millions of street number annotations  Multi-digit recognition: >90% Reference: Goodfellow et al. Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks. 2013Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks

15. CAPTCHA puzzles  Large-scale deep neural network  9 convolutional layers  Hardest category: 99.8% Reference: Goodfellow et al. Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks (2013)Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks