Conversion of Standard Broadcast Video Signals for HDTV Compatibility Ph.D. Defense Presentation Elham Shahinfard Advisor: Prof. M. A Sid-Ahmed 16 July 2009

Contributions Proposing a motion adaptive deinterlacing method. It includes: An accurate hierarchical motion detection algorithm A recursive threshold optimization method A motion adaptive interpolation algorithm for estimating the missing lines.

Outline Introduction Background Information & Review of Previous Works Proposed Motion Detection (MD) Method Proposed Motion Adaptive Deinterlacing Method Evaluation of Algorithm Performance and Comparison with Other Methods Summary and Future Works

Problem Definition Digital high definition TV is replacing analog TV all over the world Canada and U.S. have adopted the Advanced Television System Committee fully digital system as their new TV standard Analog TV standards use Interlaced format Deinterlacing for converting interlaced to progressive has attracted attention Vertical lines Horizontal pixels per line Aspect Ratio Progressive/Interlaced 720 1280 16:9 Progressive 1080 1920 Interlace

Video Scanning Format Interlacing

Deinterlacing Estimation De-Interlacing Deinterlacing Objective: To find the best estimation for the missing lines with minimum distortion

Outline Introduction Background Information & Review of Previous Works Proposed Motion Detection (MD) Method Proposed Motion Adaptive Deinterlacing Method Evaluation of Algorithm Performance and Comparison with Other Methods Summary and Future Works

Deinterlacing Categories Spatial deinterlacing (Intra-frame) Temporal deinterlacing (Inter-frame) Hybrid deinterlacing (Inter-frame) Vertical-Temporal (VT) median deinterlacing Motion compensated deinterlacing Motion adaptive deinterlacing

Spatial Deinterlacing Uses only spatial data from current field of video : location of a pixel in the field : spatial displacement vector Examples Line Repetition Line Averaging

Spatial Deinterlacing Example (Line Averaging) Original Progressive Frame Interlaced Field Advantages Simplicity Motion robustness Disadvantages Low quality; blurring effect Deinterlaced Frame

Temporal Deinterlacing Uses only temporal data from previous and/or subsequent fields Examples Field Insertion Bilinear Field Averaging

Temporal Deinterlacing Example (Field Insertion) Original Progressive Frame Interlaced Field Advantages Simplicity Perfect for static regions Disadvantages Severe distortion in dynamic regions Deinterlaced Frame

VT Median Deinterlacing A median filter is used to interpolate the missing lines The median operation is done in both temporal and vertical directions Example: Three tap VT median deinterlacing:

VT Median Deinterlacing Example Original Progressive Frame Interlaced Field Advantages Easy implementation Superior to temporal and spatial methods Disadvantages Low quality Deinterlaced Frame

Motion Compensated Deinterlacing Video sequence is virtually converted to a stationary sequence A motion estimation method estimates the motion The estimated motion is removed consequently A deinterlacing method, which performs well in static regions, is applied to the stationary sequence Motion data is added to the deinterlaced sequence in a later stage

Motion Compensated Deinterlacing Advantages High quality results Perfect for videos with translational motion, such as panning camera Disadvantages Performance is highly dependant to motion estimation results Vulnerable to common motion estimation obstacle, such as: Object deformation Appearance and disappearance of objects Fast motions which goes beyond the search area Sub-pixel motions

Motion Adaptive Deinterlacing Benefits from both interframe and intraframe deinterlacing methods by: Using a motion detector to divide a video sequence into static and dynamic regions Using an interframe deinterlacing in static regions Using an intraframe deinterlacing in dynamic regions Combining the results to obtain best estimation It has proven to be a proper choice for high quality deinterlacing A high quality motion adaptive deinterlacing has been designed in this research

Outline Introduction Background Information & Review of Previous Works Proposed Motion Detection (MD) Method Proposed Motion Adaptive Deinterlacing Method Evaluation of Algorithm Performance and Comparison with Other Methods Summary and Future Works

Design Objectives & Contributions Accurately detecting the presence of motion in video sequence Measuring motion activity level of video sequence with high precision Contributions Proposing an accurate motion detection algorithm which: Uses five consecutive fields of interlaced video Has a hierarchical structure Utilizes two LPF to improve algorithm accuracy

Input Sequence Five consecutive fields of a video sequence are used for motion detection The optimum number of correlated interlaced video fields is 5. Relative position of missing lines in five consecutive fields

Motion Detector Motion detection goal is detecting the possibility of motion Motion direction is not important Increase the accuracy of motion detector. Assumptions: Signal is large and noise is small Low frequency energy in signal is greater than low frequency energy in noise and alias 2D square averaging filters are appropriate choices Improves the consistency of the output Assumption: moving objects are larger than pixels an mxm median filter is an appropriate choice

Hierarchical Block Receives dif1, dif2, dif3 Partitions them into data blocks Calculates the average intensity value for each block

Hierarchical Block Compares the average intensity value of each block with its corresponding data blocks Finds the maximum average intensity value of each three corresponding data blocks Compares the maximum value with a predefined threshold value If less than threshold value, the data block is considered static and its motion value is set to zero If greater than threshold value, the data block is dynamic A dynamic data block will be recursively partitioned to smaller data blocks The recursive procedure may continue up to pixel level The final output is motion value matrix

Hierarchical Block Average Intensity Value before Thresholding Average Intensity Value after Thresholding

Threshold Value Determination Threshold values have been found by experimental tests A video sequence with tractable moving objects is a proper starting point Average intensity values have been monitored for several test sequences to find initial values Initial values have been applied to motion detection methods and recursively optimized for error minimization

Outline Introduction Background Information & Review of Previous Works Proposed Motion Detection Method Proposed Deinterlacing Method Evaluation of Algorithm Performance and Comparison with Other Methods Summary and Future Works

Proposed Motion Adaptive Deinterlacing Improves motion detection consistency by reducing distortion Reduces the possibility of missing motions.

Non-Linear Transformation Converts Motion-Value (MV) to motion possibility value and are predefined threshold values found by recursive error minimization.

Threshold Calculation Pre Conditions: are in the same range as pixel intensity values (0-255 in a general case) Initial setup Initial threshold values: Initial step size: 10 Procedure: Initial values have been applied to the proposed motion adaptive deinterlacing method Deinterlacing error has been calculated and recorded for each initial value Initial values have been changed based on calculated error. Finer step size has been applied to the area with minimum error

Threshold Calculation Same procedure has been applied to several test sequences has proven to be optimum values for a general setup

Interpolation Algorithm : motion possibility value : intensity value of a pixel using Spatial deinterlacing method; Linear interpolation is chosen as spatial deinterlacing method : intensity value of a pixel using Temporal deinterlacing method; Median filtering is chosen as temporal deinterlacing method

Deinterlacing Results Stennis Original Progressive Video Stennis Deinterlaced Video Sflowg Original Progressive Video Sflowg Deinterlaced Video

Implementation Results (a) Grandmom; Original progressive frame (b) Grandmom; Deinterlaced frame (c) Mom; Original progressive frame (d) Mom; Deinterlaced frame

(e) MomDaughter; Original progressive frame (f) MomDaughter; Deinterlaced frame (g) Stennis; Original progressive frame (h) Stennis; Deinterlaced frame

(i) Heart; Original progressive frame (j) Heart; Deinterlaced frame (k) Sflowg; Original progressive frame (l) Sflowg; Deinterlaced frame

(m) Movi; Original progressive frame (n) Movi; Deinterlaced frame (o) Disku; Original progressive frame (p) Disku; Deinterlaced frame

Motion Detection Results (a) Grandmom (b) Mom (c) MomDaughter (d) Stennis

(e) Heart (f) Sflowg (g) Movi (h) Disku

Outline Introduction Background Information & Review of Previous Works Proposed Motion Detection (MD) Method Proposed Motion Adaptive Deinterlacing Method Evaluation of Algorithm Performance and Comparison with Other Methods Summary and Future Works

Performance Evaluation Method

Evaluation Criterion Objective Evaluation Criterion: Peak Signal to Noise Ratio Subjective Evaluation Criterion (According to ITU-R BT.500-11): Mean Score & its associated confidence Interval Grade Impairment level 5 4 3 2 1 Imperceptible Perceptible but not annoying Slightly annoying Annoying Very Annoying

Objective Evaluation Results

Single Frame of Mom Seq. (a) Original progressive frame (b) Deinterlaced by bilinear field averaging (c) Deinterlaced by GA-HDTV method (b) Deinterlaced by proposed MA method

Zoomed on a Moving Region (a) Original progressive frame (b) Deinterlaced by bilinear field averaging (c) Deinterlaced by GA-HDTV method (b) Deinterlaced by proposed MA method

Algorithm Robustness to Frame Rate

Evaluation of Proposed Motion Detection

Subjective Evaluation Results 25 observers have evaluated the algorithm Non-professional random observers Both male and female Ages 15 to 65 Overall mean score is 4.74 Its 95% confidence interval

Outline Introduction: Deinterlacing Problem Statement Review of Existing Methods Proposed Motion Detection (MD) Method Proposed Motion Adaptive Deinterlacing Method Evaluation of Algorithm Performance and Comparison with Other Methods Summary and Future Works

Summary A high accuracy motion detection algorithm was proposed Has a hierarchical structure Uses 5 consecutive video fields for motion detection Is capable of detecting a wide range of motions from slow to fast motions Provides superior PSNR compared to other mthods and improves deinterlacing overall performance by 18% on average A motion adaptive deinterlacing method was proposed Uses motion possibility values to combine line averaging with Vertical-Temporal median filtering and benefits from both Has high performance and obtain high quality deinterlaced video

Recommendations for Future Works Improvement in deinterlacing Study of human eye frequency response to Changes in video contents Changes in motion speed while tracking an object Improvement in motion adaptive deinterlacing Including video content information Automatic recognition of the type of the video for performance improvement Combining video texture information with motion detection results Combining motion detection results with motion estimation Utilizing a motion compensated method instead of temporal method Hardware Implementation (Architecture, Software/Hardware partitioning)