1. Upsampling through Transposed Convolution and Max Unpooling

2. Convolution Overview

3. Convolution examples No Padding 𝑖 = 4, 𝑘 = 3, 𝑠 = 1 𝑎𝑛𝑑 𝑝 = 0
𝑜𝑢𝑡𝑝𝑢𝑡 = (𝑖 − 𝑘) + 1
In this example:
Output size = 2
Add information about the figure + output formula

4. Convolution examples Half Padding No Padding
𝑖 =5, 𝑘 = 3, 𝑠 = 1 𝑎𝑛𝑑 𝑝 =1
𝑜𝑢𝑡𝑝𝑢𝑡 = 𝑖 − 𝑘 +2𝑝 + 1
In this example:
Output size = 𝑖=5
It is also called Same Padding
Add information about the figure + output formula

5. Convolution examples Full Padding No Padding Half Padding
𝑖 =5, 𝑘 = 3, 𝑠 = 1 𝑎𝑛𝑑 𝑝 =2
𝑜𝑢𝑡𝑝𝑢𝑡 = 𝑖 − 𝑘 +2𝑝 + 1
In this example:
Output size = 7
Add information about the figure + output formula

6. Convolution as a matrix operation
Convolution can be expressed as a matrix operation
The convolution matrix “C” is inferred from the kernel
The output is obtained by multiplying the “C” with the flattened input column vector and then reshaped

7. Flattening Input

8. Obtaining “C” kernel Slide it over the input 𝑤 0,0 𝑤 0,1 𝑤 0,2 𝑤 1,0
𝑤 1,1
𝑤 1,2
𝑤 2,0
𝑤 2,1
𝑤 2,2
Slide it over the input

9. Convolution as matrix multiplication
𝑜𝑢𝑡𝑝𝑢𝑡=𝐶. 𝑖 𝑇
Where 𝑖 is the flattened input feature map.
From the previous two slides:
Assume the convolution on 𝑖=4 𝑤𝑖𝑡ℎ 𝑘=3, 𝑠=1 :
We get the output as 4x1 vector
Reshape the output

10. Reshaping output 1 2 3 4 1 2 3 4

11. Pooling

12. Pooling Pooling is a down-sampling operation
Summarizes subregions (looses information that is not invertible).
Most common functions of pooling are max-pooling and average-pooling

13. Pooling Examples Max-Pooling
Add information about the figure + output formula

14. Pooling Examples Average-Pooling Max-Pooling
Add information about the figure + output formula

15. Transposed Convolution

16. Transposed Convolution
It is the process of going in the opposite direction of convolution
It is also called deconvolution or fractionally strided convolutions
Expressed mathematically as the transpose of the convolution matrix “ 𝑪 𝑻 ”
Note that transposed convolution does NOT guarantee to recover the original image. But returns a feature map that has the same dimensions as the original

17. Reshape it to the shape of input
Example
Consider the Convolution matrix “C” on slide 7
Consider the flattened output of convolution on slide 9
Output = 𝐶 𝑇 𝑖′
𝐶 𝑇 is 16x4 matrix
𝑖′ is 4x1 column vector
Output = 16x1 vector
Reshape it to the shape of input

18. It is always possible to emulate transposed convolution with direct convolution using the same kernel, given the padding of the input, kernel size, and number of strides of the convolution.
For this, we need to manipulate the output of the convolution (it is also the input for transposed convolution)
This emulation is for the understanding of the transposed deconvolution as it is inefficient to implement due to the sparsity of the convolution matrix. It is rather implemented in practice using the backpropagation of the original convolution.

19. Transpose convolution as direct convolution

20. No padding, 𝑠=1, transposed
A convolution described by
𝑠 = 1, 𝑝 = 0 and 𝑘
has an associated transposed convolution described by
𝑘 ′ = 𝑘, 𝑠 ′ =𝑠 𝑎𝑛𝑑 𝑝 ′ = 𝑘 − 1
and its output size is
𝑜′ = 𝑖′ + (𝑘 − 1)

21. Transposed Convolution can be expressed as
Example
Transposed Convolution can be expressed as

22. Zero padding, 𝑠=1, transposed
A convolution described by
𝑠 = 1, 𝑝 and 𝑘
has an associated transposed convolution described by
𝑘 ′ = 𝑘, 𝑠 ′ =𝑠 𝑎𝑛𝑑 𝑝 ′ = 𝑘 −𝑝− 1
and its output size is
𝑜 ′ = 𝑖 ′ + 𝑘 − 1 −2𝑝

23. Example: Half padding, transposed
Transposed Convolution can be expressed as

24. Example: Full padding, transposed
Transposed Convolution can be expressed as

25. Padded input, non-unit strides, transposed
A convolution described by
𝑠 , 𝑝 and 𝑘
And whose input size “𝒊” is
𝑖+2𝑝−𝑘 is a multiple of 𝑠
has an associated transposed convolution described by
𝑖 ′ , 𝑘 ′ = 𝑘, 𝑠 ′ =𝑠 𝑎𝑛𝑑 𝑝 ′ = 𝑘 −𝑝− 1
Where 𝑖′ is the stretched input obtained by adding 𝑠−1 zeros between each input unit
and its output size is
𝑜 ′ = 𝑠(𝑖 ′ −1)+𝑘−2𝑝

26. Example: Padded input with non-unit strides
Transposed Convolution can be expressed as

27. Max-unpooling

28. Max-Unpooling Max-unpooling is an upsampling procedure
Max-Pooling is an non-invertible operation, instead we obtain an approximation.
I order to retrieve an approximation, the locations of the maximum values obtained during the max-pooling operation are stored.
The approximation is retrieved by placing each input value of the max- unpooling back to its location, and the neighboring values are set to 0.

29. Max-unpooling example

30. Additional Materials

31. Padded input, non-unit strides(odd), transposed
A convolution described by
𝑠 , 𝑝 and 𝑘
And whose input size “𝒊” is
𝑖+2𝑝−𝑘 𝑚𝑜𝑑 𝑠
has an associated transposed convolution described by
𝑎, 𝑖 ′ , 𝑘 ′ = 𝑘, 𝑠 ′ =1 𝑎𝑛𝑑 𝑝 ′ = 𝑘 −𝑝− 1
Where 𝑖′ is the stretched input obtained by adding 𝑠−1 zeros between each input unit, and
𝑎=𝑖+2𝑝−𝑘 where “a” is the no. of zeros added to the bottom and right sides
and its output size is
𝑜 ′ = 𝑠(𝑖 ′ −1)+𝑎+𝑘−2𝑝

32. Example: Padded input with non-unit strides(odd)
Transposed Convolution can be expressed as

33. Dilated Convolution Inflating the kernel with spaces “𝑑”
“𝑑” is a hyper-parameter, which is the number of spaces inserted between kernel elements
𝑑=1 corresponds to the normal convolution
Used to cheaply increase the receptive field without increasing the kernel size

34. Example of Dialated convolution