Handwritten Digits Recognition MNIST Based Handwritten Digits Recognition Li Linjun ECE 539 Introduction to Artificial Neural Network and Fuzzy Systems

Study Result TABLE OF CONTENTS Introduction Approaches Implementation Problems Conclusion Next to do Reference

INTRODUCTION Widely Studied LeCun’s Benchmark Application Problem Description 60,000 images Fit in 28x28 0~9 Recognition

APPROACHES Principal: Studying the Data Trade off: Complexity and Accuracy Devils in the Details: Software Engineering KNN SLP&MLP CNN LeNet-5 VGG-5 VGG-16 Committee/Voting

IMPLEMENTATION KNN L1 L2 L3 Norm,5Neighbors Accuracy: 97.0% CNN LeNet-5 99.3% VGG-5 99.63% VGG-16 99.71% Committee 99.80% SLP & MLP Softmax & Cross Entropy Accuracy 91.75% & 94.12%

IMPLEMENTATION Change the size of kernel is not useful More convolution layers are helpful

PROBLEMS Software Rounding problem Different Frameworks Cross Validation How to -> sklearn Value k For Same Model: Accuracy: Caffe2 and MxNet Easy to use: Tensorflow and Pytorch

CONCLUSION Voting/Committee : engineers’ dirty way 99.80% is even better than committee of 35 conv. net with 0.23% error rate on LeCun’s website. The more complex the better might not be true. Devils are in the details, talk is cheap Understanding your data is the most important thing.

NEXT TO DO Dive into the deep models Extracted feature for KNN or SVM? Something fun: what about RCNN on MNIST?

REFERENCES [1] http://yann.lecun.com/exdb/mnist/ [2] Y. LeCun, L. Bottou, Y. Bengio and P. Haffner: Gradient-Based Learning Applied to Document Recognition, Proceedings of the IEEE, 86(11):2278-2324, November 1998 [3] https://www.kaggle.com/c/digit-recognizer [4] Dan Cireşan, Ueli Meier, Juergen Schmidhuber: Multi-column Deep Neural Networks for Image Classification,CVPR,2012 [5] Liu,Qing,Tang Xianlun, Zhang,Na: Structure Optimized Convolutional Neural Network Based on Unsupervised Pre-training, Gongcheng Kexue Yu Jishu/Advanced Engineering Science , 2017, Vol.49, p.210-215 [6] Quoc V. Le, Navdeep Jaitly, Geoffrey E. Hinton:A Simple Way to Initialize Recurrent Networks of Rectified Linear Units,arxiv:1504.00941v2 [cs.NE] 7 Apr 2015 [7] Li Deng, Dong Yu, Deep Convex Network: A Scalable Architecture for Speech Pattern Classification