1. Convolutional Neural Network

2. Load Data
Assignment 1
Assignment 1

3. Load Data 200,000 10,000 20,000 Validation set Test set Training set
(10 class x 20,000) A
10,000
(10 class x 1,000)
20,000 A A
Validation set
Test set
Training set

4. Reformat Data

5. Reformat Data train_dataset (200000, 28, 28) (200000, 28, 28, 1)

6. Reformat Data labels.shape = (200000, ) labels[:, None].shape
= (200000, 1)
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
array([ [4],
[9], ⋮
[5] ])
(10, )
(200000, 1)

7. Reformat Data
(200000, 10)

8. Reformat Data

9. Batch Data offset = (0 x 16) % (200000 – 16) = 0 % 199984 = 0 step = 1
train_dataset
[0:15, :, :, :]
train_dataset
[16:31, :, :, :]
200,000
200,000
step = 1
step = 2

10. Batch Data step = 1 step = 2 batch_data step = 1 batch_data step = 2
offset=0
train_dataset
[0:15, :, :, :] 28
step = 1
200,000
batch_data 28 16
step = 1
offset=16
train_dataset
[16:31, :, :, :] 28
step = 2
200,000
batch_data 28 16
step = 2

11. Input Data tf_train_dataset = (16, 28, 28, 1)
tf_train_labels = (16, 10)
tf_valid_dataset = (10000, 28, 28, 1)
tf_test_dataset = (10000, 28, 28, 1)

12. Layer-1 Weights
Outputs random values from a truncated normal distribution.
The generated values follow a normal distribution with
specified mean and standard deviation

13. Layer-1 Weights layer1_weights( 5 x 5 x 1 x 16 )
ft.truncated_normal( [5, 5, 1, 16], stddev=0.1 )
layer1_weights( 5 x 5 x 1 x 16 )

14. tf.nn.conv2d(input, filter, strides, padding, …)
Args
Input : A Tensor. Must be one of the following types: float32, float64.(28x28x1,grayscale)
Filter : A Tensor. Must have the same type as input.(5x5x16,patch)
Strides :  A list of ints. 1-D of length 4. The stride of the sliding window for each dimension of input. Must have strides[0] = strides[3] = 1. For the most common case of the same horizontal and vertices strides, strides = [1, stride, stride, 1].( [1,2,2,1] )
Padding: A string from: "SAME", "VALID". The type of padding algorithm to use.(SAME Padding)
return
A Tensor. Has the same type as input.

15. Layer-1 computation conv[0] patch batch_data size = 5 x 5
channel = 1(gray scale)
depth = 16 28
conv[0] 28
stride = 2
padding = same
5 x 5
batch_data 14 28 28 16 14 1 16

16. Layer-1 computation conv = 16 x 14 x 14 x 16 ⋮ batch_data 1 28 2 28 16

17. Layer-1 Bias
depth = 16
layer1_biases = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
layer1_biases.shape = (16, )

18. Layer-1 computation conv + layer1_biases ⋮ layer1_biases =
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
conv + layer1_biases +0 1 2 ⋮ 16

19. tf.nn.relu(features,…) Args return
features: A Tensor. Must be one of the following types: float32, float64, int32, int64, uint8, int16,int8, uint16.
name: A name for the operation (optional).
return
A Tensor. Has the same type as features.

20. Layer-1 computation
conv + layer1_biases
hidden
tf.nn.relu

21. Layer-2 Weights
ft.truncated_normal( [5, 5, 16, 16], stddev=0.1 )

22. Layer-2 computation hidden[0] of layer1 conv[0] of layer2 patch
size = 5 x 5
channel = 1(gray scale)
depth = 16 28
hidden[0] of layer1 14
5 x 5
conv[0] of layer2
5 x 5 7
stride = 2
padding = same 14 7 28 16 16 1

23. Layer-2 computation
conv = 16 x 7 x 7 x 16 ⋮

24. Layer-2 Bias
depth = 16
layer2_biases = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
layer2_biases.shape = (16, )

25. Layer-2 computation conv + layer2_biases tf.nn.relu ⋮ layer1_biases =
[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]
conv + layer2_biases +1
tf.nn.relu ⋮

26. Reshape
reshape
= (hidden, [16, 784])
shape = [16, 7, 7, 16] ⋮

27. Reshape reshape = (hidden, [16, 784]) hidden[0] of layer1
patch
size = 5 x 5
channel = 1(gray scale)
depth = 16 28
hidden[0] of layer1 14
5 x 5
hidden[0] of layer2
5 x 5
reshape[0] 7
stride = 2
padding = same 1 14 7
784 28 16 16 1

28. Reshape
reshape
= (hidden, [16, 784])
reshape
= (16, 784) ⋮

29. Layer-3 Weights
ft.truncated_normal( [ (28 / 4) x (28 / 4) x 16, 64], stddev=0.1 )
= ft.truncated_normal( [ 784, 64], stddev=0.1 )

30. Layer-3 Bias num_hidden = 64
layer3_biases = [1, 1, 1, 1, 1, 1, 1, 1, ..., 1, 1, 1, 1, 1, 1]
layer3_biases.shape = (64, )

31. Layer-3 computation reshape = (16, 784) layer3_weights = (784, 64)
reshape x layer3_weights
= (16, 784) x (784, 64)
= (16, 64)
reshape x layer3_weights = (16, 64)
layer3_biases = (64, )
reshape x layer3_weights + layer3_biases
=(16, 64)
tf.nn.relu

32. Layer-3 computation hidden[0] of layer1 hidden[0] of layer228
hidden[0] of layer1 14
5 x 5
hidden[0] of layer2
5 x 5
reshape[0]
hidden[0] of layer3 7
stride = 2
padding = same 1 1 14 64 7
784 28 16 16 1

33. Layer-4 Weights num_hidden = 64 num_labels = 10
ft.truncated_normal( [ 64, 10], stddev=0.1 )

34. Layer-4 Bias num_labels = 10
layer4_biases = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
layer4_biases.shape = (10, )

35. Layer-4 computation hidden = (16, 64) layer4_weights = (64, 10)
hidden x layer4_weights
= (16, 64) x (64, 10)
= (16, 10)
hidden x layer4_weights = (16, 10)
layer4_biases = (10, )
hidden x layer4_weights + layer4_biases
=(16, 10)

36. Layer-4 computation hidden[0] of layer1 hidden[0] of layer228
hidden[0] of layer1 14
5 x 5
hidden[0] of layer2
5 x 5
reshape[0]
hidden[0] of layer3
output[0] of layer4 7
stride = 2
padding = same 1 1 1 14
784 64 7 10 28 16 16 1

37. Training Computation
16 x 10