1. Deeply-Recursive Convolutional Network for Image Super-Resolution
Jiwon Kim, Jung Kwon Lee and Kyoung Mu Lee
Computer Vision Lab.
Dept. of ECE, ASRI
Seoul National University

2. Super-Resolution Problem
Introduction
Super-Resolution Problem

3. Want images Big & Sharp ! Big & Sharp ! Super-Resolution
Low resolution image

4. Observation from VDSR VDSR [CVPR2016]
a successful very deep CNN for SR
We observed that
Convolution layers exactly have the SAME structures reminding us of RECURSIONs
conv3x3-64 / relu
ILR HR
20-layer CNN with same size layers

5. Motivation Receptive field of CNN is important for SR
Determines the amount of contextual information
clue for missing high freq. info.

6. Motivation Two approaches to enlarge receptive field
Increasing depth of conv. layer
Or simply using pooling layer

7. Motivation
Drawbacks
More parameters  overfitting & data management
Discard pixel-wise information
We need a better efficient CNN model to secure large receptive field for SR  Deep Recursive Neural Net

8. Issues on Recursive Neural Network
Problems of conventional Recursive Neural Net models [Eigen et al. ICLR WS2014, Liang et al. CVPR2015]
Shallow (up to 3 layers),
Dimension reduction
Overfitting
We need an efficient Deeply Recursive Convolutional Network (DRCN) for SR
Deep enough
Keep dimension
Simple to avoid overfitting

9. Our Approach

10. Our Basic DRCN Model
All filters are 3x3
Very deep recursive layer of the same convolution (up to 16 recursions)
Very large receptive field (41x41 vs 13x13 of SRCNN)
Can improve performance without introducing new parameters for additional convolutions

11. Problems of Basic DRCN Learning DRCN is very hard
due to exploding/vanishing gradients make the training difficult. Basic model does not converge
Determine optimal no. of recursions is also difficult
To ease the difficulty, we propose two extensions
Recursive-supervision: all recursions are supervised
Final output as an ensemble of intermediate predictions
Skip-connection: input directly goes into reconstruction net

12. Advanced DRCN Model Early recursions are supervised simultaneouslyx
Early recursions are supervised simultaneously
Shared reconstructed net
As outputs reconstructed from all depths are ensembled, cherry-picking the optimal depth is not required

13. Advanced DRCN Model Exact copy of input is not lost during recursionsx
Exact copy of input is not lost during recursions
Input is directly connected to recon net
Network capacity is saved
Exact copy of input can be used to make target

14. Advanced DRCN Model x

15. Loss

16. Training Data 91 clear images from Yang et al.
41 by 41 patches with stride 21
To make low-res image pair, we use bicubic interpolation from MATLAB

17. Test Data
4 Test Datasets
Set5, Set14, B100, Urban100

18. Experimental Results

19. Study of DRCN Model 1. Recursion effect
More recursions yielding larger receptive fields lead to better performances.
Recursion vs. performance for the scale factor ×3 on the dataset Set5.

20. Study of DRCN Model 2. Ensemble effect
Ensemble of intermediate predictions significantly improves performance
Prediction made from intermediate recursions are evaluated.
There is no single recursion depth that works the best across all scale factors.

21. Experimental Results Ground truth (PSNR/SSIM) SelfEx[4] 29.19/0.9000
Ground truth
(PSNR/SSIM)
SelfEx[4]
29.19/0.9000
RFL[3]
29.16/0.8989
DRCN (ours)
29.98/0.9115
SRCNN[2]
29.45/0.9022
A+ [1]
29.18/0.9007
[1] R. Timofte, V. De Smet, and L. Van Gool. A+: Adjusted anchored neighborhood regression for fast super-resolution. In ACCV, 2014
[2] C. Dong, C. C. Loy, K. He, and X. Tang. Image super-resolution using deep convolutional networks. TPAMI, 2014
[3] S. Schulter, C. Leistner, and H. Bischof. Fast and accurate image upscaling with super-resolution forests. In CVPR, 2015
[4] J.-B. Huang, A. Singh, and N. Ahuja. Single image super-resolution using transformed self-exemplars. In CVPR, 2015.

22. Experimental Results Ground truth (PSNR/SSIM) SelfEx[4] 26.90/0.8953
Ground truth
(PSNR/SSIM)
SelfEx[4]
26.90/0.8953
RFL[3]
26.22/0.8779
DRCN (ours)
27.05/0.9033
SRCNN[2]
26.40/0.8844
A+ [1]
26.24/0.8805

23. Experimental Results Ground truth (PSNR/SSIM) SelfEx[4] 29.16/0.9284
Ground truth
(PSNR/SSIM)
SelfEx[4]
29.16/0.9284
RFL[3]
28.44/0.9200
DRCN (ours)
32.35/0.9578
SRCNN[2]
29.40/0.9270
A+ [1]
28.90/0.9278

24. Experimental Results Ground truth (PSNR/SSIM) SelfEx[4] 28.48/0.8998
Ground truth
(PSNR/SSIM)
SelfEx[4]
28.48/0.8998
RFL[3]
28.42/0.8980
DRCN (ours)
29.65/0.9151
SRCNN[2]
28.84/0.9041
A+ [1]
28.44/0.8990

25. Experimental Results Ground truth (PSNR/SSIM) SelfEx[4] 29.93/0.7883
Ground truth
(PSNR/SSIM)
SelfEx[4]
29.93/0.7883
RFL[3]
29.86/0.7830
DRCN (ours)
30.40/0.8014
SRCNN[2]
29.98/0.7867
A+ [1]
30.00/0.7878

28. Quantitative Results
*cpu version
Dataset
A+ [ACCV 2014]
PSNR/SSIM/time
RFL [CVPR 2015]
SelfEx [CVPR 2015]
SRCNN*
[ECCV 2014]
PSNR/SSIM/time
PSNR/SSIM/time
Set5 x4
30.28/0.8603/0.24
30.14/0.8548/0.38
30.31/0.8619/29.18
30.48/0.8628/2.19
37.53/0.9587/0.13
33.66/0.9213/0.13
31.35/0.8838/0.12
37.63/0.9588/1.54
33.82/0.9226/1.55
31.53/0.8854/1.54
Set14
27.32/0.7491/0.38
27.24/0.7451/0.65
27.40/0.7518/65.08
27.49/0.7503/4.39
33.03/0.9124/0.25
29.77/0.8314/0.26
28.01/0.7674/0.25
33.04/0.9118/3.44
29.79/0.8311/3.65
28.02/0.7670/3.63
B100
26.82/0.7087/0.26
26.75/0.7054/0.48
26.84/0.7106/35.87
26.90/0.7101/2.51
31.90/0.8960/0.16
28.82/0.7976/0.21
27.29/0.7251/0.21
31.85/0.8942/2.30
28.80/0.7963/2.31
27.23/0.7233/2.30
Urban 100
24.32/0.7183/1.21
24.19/0.7096/1.88
24.79/0.7374/694.4
24.52/0.7221/18.5
30.76/0.9140/0.98
27.14/0.8279/1.08
25.18/0.7524/1.06
30.75/0.9133/12.72
27.15/0.8276/12.70
25.14/0.7510/12.71
Ours outperforms SRCNN by 0.67 dB / SSIM score

29. Concolusion

30. Conclusion Novel SR method using deeply-recursive convolution network
additional recursion introduces no additional weight parameters (fixed capacity)
Recursive-supervision and skip-connection are used for better training
Achieves the state-of-the-art performance
Can be applied to other image restoration problems easily