1. Robust global motion estimation and novel updating strategy for sprite generation IET Image Processing, Mar. 2007. H.K. Cheung and W.C. Siu The Hong Kong Polytechnic Univ. ( 香港理工大學 )

2. Outlines  Overview / Introduction  Proposed system New global motion estimation  Combing short- and long-term estimation  Dynamic reference frame 2-pass sprite blending  Preserving frame resolution loss Sprite updating  Overcoming illumination variations & object changing  Experimental results  Conclusions

3. Overview

4.  Sprite High resolution image Composed of information belonging to an object visible throughout a video sequence  Background of a scene

5. Overview  Sprite background of frame 1 (Dimension: 352x288) background of frame 20 Sprite (Dimension: 2670x1072)

6. Overview  Core of sprite generation Global motion estimation (GME)  Finding a set of parameters representing camera motion between frames Image registration Iterative minimization Blending  Temporal (weighted) averaging, median, updating

7. Introduction

8.  Global motion estimation Image registration Short-term motion estimation  Estimation between consecutive frames  Easy and accurate Long-term motion estimation  Estimation between frames with temporal distance  Harder  Required to perform sprite coding Single sprite for all frames in sequence

9. Introduction  Global motion estimation (cont.) Short- to long-term estimation  Converting short-term motion parameters to long- term parameters  Error propagation Directly long-term estimation  Estimation every frames directly to a specified base frame (reference frame) No error propagation  Search range may be huge  Hard to find overlapping area

10. Introduction  Global motion estimation (cont.) Hierarchical estimation  Rough estimation to find coarse parameters  Refining parameters Using coarse parameters as initials Iterative minimization  Some existing methods Dufaux and Konrad Szeliski Smolic et. al. Lu et. al.

11. Introduction  Restrictions Background must be really static  Background objects must be still  No illumination variations Dynamic sprite

12. Introduction  Classification Static sprite  Build offline before coding individual frames  Quality degradation as frame increases Motion estimation errors Illumination variations Background object changes Dynamic sprite  Built dynamically online in both encoder and decoder while coding individual frames Sprite is updated using reconstructed frame  Short-term estimation is employed Error accumulated

13. Introduction  Proposed system New global motion estimation  Directly estimating the relative motion between current image and a chosen reference frame Give accurate, stable and robust estimation Alleviate error accumulation  Hierarchical 3-levels approach Coarse-to-fine approach Sprite updating  Updating sprite only if necessary Sprite update frames are generated and sent

14. Proposed system

15.  Short-term GME to long-term GME Frame 1 A 11 Frame m A m1 Frame m+1 GME reference frame …… A (m+1)1 A m1 + A (m+1)m Registration Error = A (m+1)m  A m(m-1)  …  A 21 A (m+1)k = A (m+1)m  A m1 Registration Error Registration errors are ACCUMULATED More Error

16. Proposed system  Directly measure to reference frame GME Frame 1 A 11 Frame m A m1 Frame m+1 reference frame …… A (m+1)1 A m1 initial guess Registration Error Registration errors are COMPENSATED

17. Proposed system  Weakness Reference frame is temporally far from current frame  Frame contents may change largely Background objects activities Lighting conditions changes Overlapping area could be smaller  Unfavorable to GME

18. Proposed system  Combining the advantages Dividing video into groups of consecutive frames 1st frame of each group is selected as reference  Frames in a group Each frame is directly measured to the 1st frame  Smaller registration error  Merging groups GMEs of reference frames of all groups are merged  Registration error is slightly increased R1R2R3 …… ++ A (R1)(R1) A (R2)(R1) A (R3)(R2) A (R2)(R1) A (R3)(R1)

19. Proposed system  Proposed GME structure Motion Estimation Frame k A k1 Frame m A mk A m1 Frame m+1 Frame z Chosen to be reference frame …… A (m+1)k A mk

20. Proposed system  Dynamic reference frame 1st frame is the initial reference frame Assigning current frame as new reference frame if  Displaced frame difference between registered current frame and the reference frame it large Reference frame is not like current frame  Relative displacement between current frame and the reference frame is large Overlapping area is too small or where Nr is a parameter between 0 and 1 (Nr=0.1 in practical)

21. Proposed system  Advantages Accuracy  Accurate than short-term and directly long-term estimation Very few memory usage  Estimations are performed frame-to-frame  Sprite building is not necessary

22. Proposed system  GME Reference frame (frame k) Frame z Three step search Block-based partial distortion search Fast gradient method A (m+1)k A mk +

23. Proposed system  Motion model Perspective motion model  8 motion parameters to be determined  Three-step matching 3-level pyramids for frame k and z are built using Gaussian down-sampling filter [ ¼, ½, ¼ ] frame k: reference frame frame z: transformed current frame m+1

24. Proposed system  Block-matching Affine parameters are estimated by solving over- fitting equations Results of block-based motion estimation are used to construct the equations  Parameter estimation Fast gradient descent method by Keller and Averbuch where

25. Proposed system  Two-passed blending to avoid resolution loss First pass: 1st frame as base frame  All frames are projected into 1st frame  Frame with minimal area of projected frame is selected as new base frame Avoiding resolution loss  No real pixel blending applied Second pass: new base frame  All frames are projected into new base frame  Simple temporal average blending With bilinear interpolation

26. Proposed system  Dynamic sprite updating Overcoming illumination variations  Single value in sprite can not represent intensity variations over the time Accumulation of GME error blurring the frame  GME error in a reference frame will inherit into all of frames in the group

27. Proposed system  Studying the generated intensity error an edge pixel a pixel from homogeneous area a pixel from texture area translation in x-direction # of pixel with significant error

28. Proposed system  Distribution of intensity error correlates roughly to the panning motion Errors tends to be clustered in the temporal domain  Errors of homogeneous and texture regions are tend to randomly around zero

29. Proposed system  Sprite updating Selecting frames with significant change in panning direction/speed 051108174206

30. Proposed system  Sprite updating (cont.) Reconstruct next N frame from the sprite Blend the N error frames into a sprite-sized buffer (the sprite update frame) Compute the N error frames Encode and send the sprite update frame to the decoder MPEG4 I-VOP frame

31. Experimental results

32.  Testing Constructing sprite Reconstructing frames from sprite Compute PSNR  Comparison Short-term motion estimation  Estimating between current and previous frame Long-term motion estimation  Estimating between current frame and sprite  No parameters predicting Long-term motion estimation by MPEG-4 VM Long-term motion estimation by Smolic et. al.

33. Experimental results Short-term Long-term

34. Experimental results MPEG-4 VM Proposed method

35. Experimental results  PSNR Proposed MPEG-4 Short-term Long-term Smolic et. al.

36. Experimental results  Average PSNR (dB) Short- term Long- term MPEG-4 VM Smolic et. al. Proposed (affine) Proposed (per- spective) Stefan (150) 18.95519.05820.88920.34722.04622.645 Foreman (150) 27.94128.43228.05726.97328.45828.305 Coast Guard (150) 22.29422.54323.58620.21323.53823.450 Stefan (300) 19.15220.44218.871Failure21.36421.311

37. Experimental results  Selecting threshold Nr Proposed method is better than simple short-term and long-term estimation Short-term 0.1 Long-term

38. Experimental results  Performance of sprite updating SequenceUpdate framesAverage PSNR (dB)Size of updates (kB) stefan-21.133- stefan0,51,108,174,206 * 22.31984.5 stefan0,60,120,180,24022.26579.1 stefan0,51,108,106 * 22.21574.0 stefan0,80,160,24021.99459.5 coast guard-23.538- coast guard0,76 * 24.0857.42 foreman-28.758- foreman0,10,25,64,110 * 30.59011.7 foreman0,30,60,90,12030.71412.1 * Update frames is figured out from the major camera operations of the sequences

39. Conclusions

40.  New global motion estimation method Directly estimation from current frame to a chosen reference frame Combing advantages of short-term and long-term estimation  Error accumulation prevented  Keeping reference frame close to current frame  Sprite updating Encoding & sending sprite update frames  Errors of a group of reconstructed frames Reducing sprite blurring