Copyright © Magnum Semiconductor, Unpublished Introduction to Deinterlacing by Mark Korhonen

Copyright © Magnum Semiconductor, Unpublished 2 Overview: Introduction To Deinterlacing Background Progressive video Interlaced video Deinterlacing Basic Deinterlacing Algorithms Weave Vertical Interpolation (aka Bob) Advanced Deinterlacing Algorithms Diagonal Interpolation Cadence Detection Motion Adaptive Deinterlacing (MADI) Motion Compensated Deinterlacing (MCDI) Deinterlacing Applications

Picture source: Copyright © Magnum Semiconductor, Unpublished 3 Background: Progressive Video A complete video frame is displayed at regular intervals View on computers and digital televisions Example resolutions and frame rates: SD: 720x480p30, 720x576p25 HD: 1280x720p60, 1920x1080p30 Sources: film (movies), animation, progressive cameras Example: 30p = 30 frames/s

Picture source: Copyright © Magnum Semiconductor, Unpublished 4 Background: Interlaced Video Frames of video are sampled at two time intervals Even rows sampled at: 2t Odd rows sampled at: 2t + 1 Terminology: Top field: even rows in the frame Bottom field: odd rows in the frame Field polarity: indicates if a field is a top field or bot. field Example: 60i = 60 fields/s

Copyright © Magnum Semiconductor, Unpublished 5 Background: Interlaced Video Why interlaced video exists Double frame rate w/ same bandwidth = smoother motion Original television formats are all interlaced –NTSC=720x480i60 –PAL=720x576i x1080i60 exists because –Processing requirements for 1920x1080p60 are very high –Motion is smoother than 1920x1080p30 –More detail than 1280x720p60

Copyright © Magnum Semiconductor, Unpublished 6 Background: Deinterlacing Deinterlacing: convert interlaced video to progressive Input: X field/s (e.g. i60) Output: X frames/s (e.g p60) This process generates X “missing” fields/s

Copyright © Magnum Semiconductor, Unpublished 7 Background: Deinterlacing Fields typically used to generate the missing field The current field is the opposite polarity of the field that needs to be generated, but displayed at the same time –E.g. output = bottom field, curr = top field The previous and next fields are the same polarity as the field that needs to be generated –E.g. output = bottom field, prev & next = bottom fields

Picture source: doom9.org Copyright © Magnum Semiconductor, Unpublished 8 Basic Deinterlacing: Weave Weave: use “prev” or “next” as the missing field For still images and progressive content, no missing data Very severe artifacts for moving interlaced content

Copyright © Magnum Semiconductor, Unpublished 9 Basic Deinterlacing: Weave Extension If stationary, average the previous and next fields together as the missing field –Noise reduction (averaging two identical images reduces noise) –Handles fades better if luminance changing every field Output[x,y] = m prev[x,y] + (1-m) next[x,y] m is typically 0.0, 0.5 or 1.0

Copyright © Magnum Semiconductor, Unpublished 10 Basic Deinterlacing: Bob Bob (aka vertical interpolation) Generate the opposite polarity by interpolating along vertical columns in the current field Implementations Line doubling – just use current field –e.g. Output[x,y] = curr[x,y] 2-tap FIR filter – average the line above and below –e.g. Output[x,y] = 0.5 curr[x,y] curr[x,y+1] 8-tap FIR filter – better frequency response

Copyright © Magnum Semiconductor, Unpublished 11 Basic Deinterlacing: Bob Bob Video Quality Looks good for most video content –Still content without a lot of detail –Moving content (hard to see detail on moving objects) Interpolation fails if a lot of detail – causes flickering

Picture source: Computer Desktop Encyclopedia Copyright © Magnum Semiconductor, Unpublished 12 Advanced Deinterlacing: Diagonal Interpolation Problem: Bob isn’t ideal for diagonal edges Solution: Apply FIR filter along a diagonal

Copyright © Magnum Semiconductor, Unpublished 13 Advanced Deinterlacing: Diagonal Interpolation Diagonal Edge Detection Algorithm Try a bunch of different angles and see which is best –Pattern recognition problem: classify edges as a supported angle Determine which angle is most correlated at each pel We can assume that the angle of the edge is wide »Allows using neighbouring pels to improve accuracy

Picture source: Copyright © Magnum Semiconductor, Unpublished 14 Advanced Deinterlacing: Cadence Detection Observation: If interlaced video was generated from a progressive source, we can safely weave it with no artifacts Sample cadences

Copyright © Magnum Semiconductor, Unpublished 15 Advanced Deinterlacing: Cadence Detection Detection Area Size of cadence area can be variable –e.g. entire field, 16x16 block, every pel Detecting entire fields sufficient for ~99% of video Smaller areas only needed if video is a mixed source –moving interlaced text over progressive video e.g. weather warnings on a TV movie –different frame rates of progressive video edited together e.g. source of border was 30p, but source of contents were 24p

Picture source: Copyright © Magnum Semiconductor, Unpublished 16 Advanced Deinterlacing: Cadence Detection Cadence Algorithm 1: detect regular repeated fields For 3:2, every 5th field is a repeat –SAD will be very low every 5th field  called Inverse Telecine Very robust for a lot of content Doesn't work for 2:2 (no repeats), or changing cadences (e.g. slow-motion replays, edited video)

Copyright © Magnum Semiconductor, Unpublished 17 Advanced Deinterlacing: Cadence Detection Cadence Algorithm 2: detect weaving artifacts 1) Weave curr with prev, 2) Weave curr with next Determine if weaving artifacts are less in a field –Pattern recognition problem: classify into three states Progressive – weave with previous Progressive – weave with next Interlaced – perform other deinterlacing

Copyright © Magnum Semiconductor, Unpublished 18 Advanced Deinterlacing: Cadence Detection Cadence Algorithm 2: detect weaving artifacts Tricky for stationary content and at scene changes –Fairly easy to compensate Tricky for vertical motion of detailed video –e.g. pan of venetian blinds Can be tricky for video that looks like it already contains weaving artifacts –Artifacts introduced by video compression can look like weaving artifacts –Certain textures

Copyright © Magnum Semiconductor, Unpublished 19 Advanced Deinterlacing: Motion Adaptive Deinterlacing (MADI) Idea: weave stationary areas, interpolate moving areas Stationary detection = pattern recognition problem Size of stationary area can vary (e.g. entire field, 16x16 block, every pel) –Smaller stationary areas = less flickering but more computation Possible implementations: –SAD of prev and next Watch out for periodic motion –min SAD of last X fields Watch out for how long it takes stationary regions to be detected

Copyright © Magnum Semiconductor, Unpublished 20 Advanced Deinterlacing: Motion Adaptive Deinterlacing (MADI) Basic algorithm: Inputs: –Weave – generated from previous and/or next –Inter – generated from current (diagonal interpolation) Output[x,y] = (K) Weave[x,y] + (1-K) Inter[x,y] –K ~= 1 for stationary/progressive areas (weave) –K ~= 0 for moving areas (interpolation) NOTES: Fades may need special handling –e.g. bias more towards interpolation x and y advance through regions in the video –Could be every pel, 16x16 blocks, or the entire field

Copyright © Magnum Semiconductor, Unpublished 21 Advanced Deinterlacing: Motion Compensated Deinterlacing (MCDI) Idea: If we can’t weave a particular region, could we weave a motion compensated version of this region from curr, prev or next? Comments: Potential for preserving all of the detail Very difficult to do correctly –Especially rotations, zooms, morphing, lighting changes, etc. –Objects can get covered/uncovered –Even small errors can look very bad weaving artifacts, ghosting around edges, weird motion, etc. Motion compensation should be done for every pel –Very computationally demanding

Copyright © Magnum Semiconductor, Unpublished 22 Advanced Deinterlacing: Motion Compensated Deinterlacing (MCDI) Basic Algorithm: Input fields –MC_curr[x,y] = curr[x + dx_c, y + dy_c] –MC_prev[x,y] = prev[x + dx_p, y + dy_p] –MC_next[x,y] = next[x + dx_n, y + dy_n] –MADI[x,y] = pel generated using motion adaptive deinterlacing Output: p + q + r ~= 1 if high confidence in motion compensation p + q + r ~= 0 if low confidence in motion compensation –Confidence in the motion compensation can be based on The smoothness of the motion vector field Estimated SAD (true SAD unknown because the field is missing) Consistency of motion from field to field

Copyright © Magnum Semiconductor, Unpublished 23 Deinterlacing Applications Frame rate conversion from interlaced to interlaced No deinterlacing = motion judder –Need to drop a pair of fields – increases motion judder Deinterlacing = smoother motion –Requires excellent quality deinterlacing Deinterlaced fields will be deinterlaced twice

Copyright © Magnum Semiconductor, Unpublished 24 Deinterlacing Applications Upscaling interlaced video requires deinterlacing Perserves more detail Minimizes aliasing

Copyright © Magnum Semiconductor, Unpublished 25 Deinterlacing Applications If the output frame rate is low, flicker can be severe If input frame rate is high enough –In the output, pick a polarity that only use fields from the input –The other polarity will always be generated (e.g. via interpolation) This works because flicker is caused by alternating rows between a) original and b) interpolated – we have removed the alternating

Copyright © Magnum Semiconductor, Unpublished 26 Appendices

Copyright © Magnum Semiconductor, Unpublished 27 Video Resolutions Common video resolutions SD: 720x480 (NTSC), 720x576 (PAL) HD: 1280x720, 1920x1080 Sub SD: VGA, QVGA, CIF, QCIF, etc. Definitions pels = 1x1 area of video data pixel = 1x1 area on video display E.g. 720x480 video is 720 pels wide, but may be displayed on a television with 1920 pixels

Copyright © Magnum Semiconductor, Unpublished 28 Color Formats RGB vs. YUV Y = greyscale, making deinterlacing decisions primarily based on Y is simpler (but not as accurate) Data in U and V isn't as critical so typically downsampled –4:2:0 vs 4:2:2 vs 4:4:4