1. 1 Video Frames Interpolation Using Adaptive Warping Ying Chen Lou Major Advisor: M.J.T. Smith Co-advisor: Edward Delp Nov. 15, 2010

2. 2 Outline Background Generic motion model Video spatial interpolation Video compression Video frame rate up-conversion Summary and future work

3. 3 Motivation Spatial interpolation –Conversion from SDTV to HDTV –Zooming of region of interest (ROI) Surveillance/forensics Medical imaging Satellite imaging Temporal interpolation (Frame rate up-conversion) –3:2 pull down 24Hz -> 30Hz –Avoid flicker and blurring on LCD –Loss of frames in transmission Frame Rate Up-Converter

4. 4 Challenges and Goal Core: motion estimation To derive a generic motion model which can be used for different applications –Motion needs to be accurate Ill-posed problem (aperture problem) –Suitable for different types of motion Translational (panning) Zoom in/out Rotational –Low to moderate computational intensive Local window used for ME

5. 5 Illustrative Example Block Matching vs. Optical Flow Motion Block Matching Optical Flow

6. 6 Motion Estimation Method (Warping) I[n 1, n 2, k] = I[n 1 + d 1 [n 1, n 2, k], n 2 + d 2 [n 1, n 2, k], k + δk] OFE where

7. 7 Assumes that the pixel displacement functions within some region R of an image I[n 1,n 2,k] can be written as: and are vectors composed of the displacement parameters to be estimated The bilinear displacement parameter are computed by minimizing the mean squared error function (MSEF) Warping Method (cont’d)

8. 8 Quad-tree Divisions Makes the algorithm adaptive and more efficient Quad-treeUniform

9. 9 Application I Spatial Interpolation Motivation Characteristics of lecture videos –Large static backgrounds –Little to medium motion of foreground Possible to store several high resolution frames and retrieve them later Simplest scenario –periodically transmit one high resolution frame and the remaining frames are in the form of low resolution Extend the proposed method to other types of video

10. 10 Interpolation –Bilinear –Adaptive Synthesis Filter Banks Warping –Block matching-based and optical flow-based –Quadtree splitting Proposed method I: Full-band Warping (FWWA)

11. 11 High Motion Breakdown

12. 12 Robustness Issue Address the robustness issue –Obtain reliable motion vectors –Maintain the sharpness Challenges –No corresponding pixels in the reference frame –Ambiguity of improvement in sharpness and distortion objectively

13. 13 Proposed method II The Composite Algorithm Incorporate advanced spatial interpolation algorithms Bidirectional warping –Forward and backward warping –To solve “no corresponding pixels in the reference frame” Hierarchical motion structure –To solve “ambiguity of improvement in sharpness and geometric distortion ”

14. 14 Hierarchical Motion Structure Z F 3 – forward warped frame Z F 3’ – downsampled forward warped frame Z B 3 – backward warped frame Z B 3’ – downsampled backward warped frame X3 – original MxN low resolution frame

15. 15 Experiment Setup Assessment of the composite algorithm –Compare with bilinear, bicubic, NEDI, VA, and the full-band warping algorithm (FWWA) Investigate the impact of different spatial interpolation methods on the composite warping algorithm –Incorporate different spatial interpolation methods in the first step –The remaining stages are the same 3 sets of video sequences, CIF, 50 frames –Low motion (‘talking head’ lecture video ) –High detail –High motion

16. 16 Assessment of the Composite Algorithm Every 5 th frame as a high resolution reference frame Decimate frames to get low resolution frames –Use a 21-tap lowpass filter –Downsample factor 2 Use original frames as ground truth Competing methods –Bilinear, Bicubic simple image interpolation methods –New-edge Directed Interpolation (NEDI) an advanced image interpolation method –A super resolution method proposed by Patrick Vandewalle (VA) 1 –Full band warping (FWWA) [1] P. Vandewalle, S. Susstrunk, and M. Vetterli, “A frequency domain approach to registration of aliased images with application to superresolution”, EURASIP Journal on Applied Signal Processing, 2006

17. 17 Objective Results AkiyoCarphoneNewsSilenceMother Bilinear34.2332.328.8632.9936.46 Bicubic35.0832.9229.8533.6437.45 NEDI34.9832.6428.732.7636.58 VA35.0832.9129.8633.6537.47 FWWA41.8432.7933.3434.5140.74 Comp43.1535.0837.0438.3440.89 SalesBusFlowerTempeteMobile 30.0725.8423.0726.6622.39 30.6326.5523.4727.2322.94 30.0525.3522.7126.3422.11 30.6326.5623.4727.2322.94 32.1318.8526.1127.2824.42 35.3125.8527.9231.1525.79 StefanTableFootball 26.5629.228.47 27.4529.8729.54 25.9129.2528.14 27.4529.8829.54 20.4027.4119.98 26.6930.7527.07

18. 18 Talking Head Video Original frameBilinear interpolation FWWA Comp

19. 19 High Detail Video Original frameBilinear interpolation FWWA Comp

20. 20 High Motion Video Original frameBilinear interpolation FWWA Comp

21. 21 (a) (b) (c) (c) (d) (a) Bilinear (b) NEDI (c) VA (d) Bicubic + Comp

22. 22 Demo

23. 23 Different Spatial Interpolation AkiyoCarphoneNewsSilenceMother Bilinear34.2332.328.8632.9936.46 Bicubic35.0832.9229.8533.6437.45 NEDI34.9832.6428.732.7636.58 VA35.0832.9129.8633.6537.47 Bi-Comp 42.0934.8436.1738.240.54 Cu-Comp 43.1535.0837.0438.34 40.89 NEDI-Comp 41.2434.1435.2336.98 39.53 VA-Comp 43.1335.0837.0438.24 40.68

24. 24 Conclusion The composite methods achieved good spatial interpolation results Accommodate complex motion Outperform competing methods subjectively and objectively –Improvement comes mostly from the warping process –A combination of bicubic interpolation and warping results in best overall performance Complexity not too high Subjective and objective results are satisfactory Particularly perform well for lecture videos and high detail videos

25. 25 Application II Video Compression H.264/AVC Retain edges but remove texture at low bitrates

26. 26 Goal and Proposed Method Goal –Propose a coding method which keeps the high frequency components –Achieve as high visual quality as possible –Maintain integrating of H.264/AVC coder which is well engineered –Be robust Three assumptions –Smaller resolution requires fewer bits –Sequences with low motion don’t need full resolution coding –Key frames more bits Proposed method –Adaptive warping –Spatio-temporal

27. 27 Overall System encoderdecoder

28. 28 Experiment Setup New algorithm compared against H.264 in the following setup H.264 codec –Every N th frame as an I frame, the rest coded as P or B frames Proposed method –Every N th frame used as the reference frame –Other frames are decimated (LL subband) and coded –Total bit rate is sum of full resolution reference frame and the quarter resolution LL subbands Bit rates are the same in both cases

29. 29 Result (1)

30. 30 Subjective Result (a) H.264 (b) proposed method 3 rd frame for Salesman sequence @ 80kbits/s

31. 31 Conclusion The proposed method achieve better visual quality at low bitrates The gap decreases as the bitrates increase At high bitrates, H.264 has more bits to spend on high frequency components and thus achieves better quality The nature of the proposed method works better for sequences with more details Room remained to be improved Explore tradeoffs in spatio-temporal decimation rates More frequently for video with large motion and less often for video with small motion For long lecture video, we can choose full coverage of reference frame and no anymore later

32. 32 Application III Video Frame Rate Up-Conversion Overview of FRUC –No motion vectors Frame repetition, frame average –Use motion vectors Use motion vectors from the decoder directly Advantage: Low complexity Disadvantage: Not true motion Perform motion estimation again Advantage: true motion Disadvantage: High computational complexity Frame Rate Up-Converter

33. 33 Goal and Proposed Method Goal (1) True motion vectors (2) Relative low complexity Challenges (1) Highly accurate MVs (2) Low percentage of MV re-estimation (3) Occlusion (4) Blocking artifacts Approach –Decoded video sequences from the decoder –Additional information from the decoder System diagram Previous Reconstructed frame X1 Current residual frame, reconstructed frame X2, and its MVs MV Reliability Check MV Re-estimation Motion Compensated Interpolation Interpolated Frame Small Block Merging

34. 34 MV reliability check Categoried into 3 groups

35. 35 Small Block Merging Avoid broken edges and want to maintain object structure

36. 36 MV Re-estimation Key in the system Accurate MVs are required for FRUC Low complexity A combination of Optical Flow-based and Block Matching- based motion estimation (Warping method)

37. 37 Motion Compensated Interpolation N=2, k=1

38. 38 Occlusion Uni-directional interpolation N=2, k=1

39. 39 Overlapped Boundary Motion Compensation (OBMC) Goal: To reduce the blocking artifacts. Selectively perform OBMC to reduce the computational complexity (BAD > T)

40. 40 Experiment Setup JM 11.0 GOP: IPP…P, 15 th I frame, fixed QP Code odd frame and skip every other frame 15fps Transform 8x8=2 mode on Search range = 16 Standard CIF Video sequences used: –Akiyo, News, Salesman, Foreman, Carphone –Flower Garden, Tempete –Football, Table Tennis Test against DMCFI, correlation-based motion selection 1 [1] Ai-Mei Huang and T. Nguyen, “Correlation-based motion vector processing for motion compensated interpolation”, ICIP 2008

41. 41 Visual Result (1) 384kb/s (a) Orig (b) DMCFI 20.54dB (c ) Correlat ion- based 20.48dB (d) Proposed 24.19dB

42. 42 Visual Result (2) 512kb/s (a) Orig (b) DMCFI 20.01dB (c ) Correlat ion- based 20.17dB (d) Proposed 20.13dB

43. 43 Conclusion Proposed a FRUC method that combines optical flow and block matching-based motion estimation Reduced computational complexity Reduced blocking artifacts Achieve better visual quality for low motion video sequences and perform on par with other methods for high motion video sequences

44. 44 Summary Provide a generic framework to achieve –Spatial enlargement of video frames –Video compression –Frame rate up-conversion Achiever higher objective and subjective results Improve the robustness by using FW, BW and hierarchical motion structure

45. 45 Future Work Continue to refine the model Apply to higher resolution video Incorporate Subjective Video Quality Analysis Reference frame recycling –Adaptively select the position of high quality reference frame

46. 46 Q & A

47. 47 Application I Spatial Interpolation Related Work Frame restoration Frame interpolation –Bilinear, bicubic, spline, … –Adaptive Synthesis Filter Banks –New edge-directed interpolation Superresolution (SR)

48. 48 Different Spatial Interpolation AkiyoCarphoneNewsSilenceMother Bilinear34.2332.328.8632.9936.46 Bicubi35.0832.9229.8533.6437.45 NEDI34.9832.6428.732.7636.58 VA35.0832.9129.8633.6537.47 Bi-Comp 42.0934.8436.1738.240.54 Cu-Comp 43.1535.0837.0438.34 40.89 NEDI-Comp 41.2434.1435.2336.98 39.53 VA-Comp 43.1335.0837.0438.24 40.68 SalesBusFlowerTempeteMobile 30.0725.8423.0726.6622.39 30.6326.5523.4727.2322.94 30.0525.3522.7126.3422.11 30.6326.5623.4727.2322.94 34.9625.727.3830.2825.66 35.3125.8527.9231.1525.79 34.2125.2527.1330.1525.79 35.325.8527.931.1526.93 StefanTableFootball 26.5629.228.47 27.4529.8729.54 25.9129.2528.14 27.4529.8829.54 26.5930.5527.26 26.6930.7527.07 26.1230.1526.48 26.6930.7327.08