# Image Deblocking Using Local Segmentation Lukasz Kizewski Supervisor:Dr. Peter Tischer Second Examiner:Dr. Andrew Paplinski.

## Presentation on theme: "Image Deblocking Using Local Segmentation Lukasz Kizewski Supervisor:Dr. Peter Tischer Second Examiner:Dr. Andrew Paplinski."— Presentation transcript:

Image Deblocking Using Local Segmentation Lukasz Kizewski Supervisor:Dr. Peter Tischer Second Examiner:Dr. Andrew Paplinski

Presentation Outline Introduction  Lossy Image Compression - JPEG  Discrete Cosine Transform (DCT)  Subbands Present Research  Research gap DCT coefficient study Deblocking Filter Conclusions Further Research

Lossy Image Compression JPEG, MPEG is lossy  Maintaining high image quality and high compression ratios is a major, widespread issue  Used everywhere! Internet, digital photography, digital camcorders, DVDs, Digital TV, video telephony/conference, mobile phones, etc. JPEG/MPEG uses ‘Transform Coding’ technique coupled with data quantization  JPEG/MPEG uses Discrete Cosine Transform (DCT)  DCT is energy-preserving and reversible  Quantization step is the lossy part

Discrete Cosine Transform Image divided into 8x8 pixel blocks DCT applied to each block independently Block decomposed into basis functions  First basis function (0,0) termed ‘DC coefficient’  Remaining 63 basis functions termed ‘AC coefficients’ [1] DC coefficient - Average brightness AC coefficients -White: Add to average -Black: Subtract from average

DCT - Subbands For every 8x8 pixel block, output of DCT is 64 DCT coefficients  Each coefficient corresponds to one basis function  “Image” of 1 DCT coefficient termed a “subband” [1] 2x2 block DCT example

Subbands First 4 (out of 64) subbands of ‘Lenna’ [1]

After performing the DCT, resulting coefficients are quantized  Divided by quantum value, rounded to integer Quantum value dictated by ‘quality’ parameter and quantization table  High-order DCT coefficients more severely quantized (usually to zero) During de-quantization, mid-point of quantization interval is chosen  Usually incorrect DCT Coefficient Quantization Q=10 514 1 5 10

JPEG – Decompression Quality JPEG decoded image at different ‘quality’ parameter settings: Q = 100Q = 50Q = 1

JPEG Image artifacts Incorrectly reconstructing DCT coefficients results in unwanted image artifacts:  Smooth regions are blocky, edges are jagged, discontinuities appear near edges  Aim of project – decrease severity of artifacts Smooth region (shoulder) Staircase effect (edge of hat) Ringing effect (edge of mirror)

Present Research Dozens of filters exist to increase the quality of highly-compressed images  Some filter DCT coefficients (subbands)  Some filter reconstructed pixel values  Others filter both  Most filters target only one type of image artifact Some filters reduce one type of artifact whilst making another more prominent

Research Gap Natural/photographic images have high correlation between neighbouring pixels  Neighbouring pixels are similar in brightness  Property fails when edge in image is encountered Lack of image segmentation in most filters  Results in blurred/smoothed edges Encountering an edge implies more than one segment  Segments should be filtered independently

Filling the Research Gap This project differs in that:  Local segmentation is used Pixels possibly split into 2 groups  Each segment filtered independently  “Do No Harm” policy used – avoid further image quality degradation If filtering model fails, it’s filtered value is disregarded Worst-case scenario – image is not filtered at all

Filling the Research Gap (cont.) Filter can be applied to  DCT Coefficients (filtering subbands)  Pixels (filtering an image)  Pixels may be filtered after subbands are reconstructed as accurately as possible Quality loss occurs at subband level  First filtering step should be reconstructing DCT coefficients better

DCT Coefficient Study Determine each subband’s contribution to overall image quality Selected subbands were not quantized  Simulates subbands being reconstructed perfectly  Subband’s contribution measured by increase in Peak Signal-to-Noise Ratio (PSNR) PSNR: Logarithmically-scaled, mean-squared-error metric NOTE: PSNR value doesn’t always reflect viewer-subjective image quality assessment

DCT Coefficient Study (cont.) Results of study:

Image Deblocking Filter Main goal is to reconstruct DCT coefficients better to reduce severity of image artifacts DC subband filtered  DCT study shows DC subband is best filter candidate  A DCT coefficient is a subband “pixel” Uses 3x3 weighted mask  Mask center is pixel being filtered  Mask scans entire image, filtering each pixel 121 242 121 3x3 filtering mask DC subband of ‘Lenna’

“Do No Harm” Filter employs “Do No Harm” (DNH) policy  If a filtered value is implausible, reject it and leave value unfiltered  Implausible pixel value is one that falls outside the quantization interval Quantization interval: midpoint +/- ½ Quantum  Guarantees image quality cannot degrade further

Sequential Filter Try 1-segment filtering  Replace pixel with mask average  If DNH not triggered, accept Otherwise try 2-segment filtering  Use average-value thresholding to classify pixels in mask  Replace pixel with average of segment to which it belongs to  If DNH not triggered, accept Otherwise trigger DNH and leave unfiltered value 100 15 100105 Weighted average = 50.94 100 15 100105 Blue class average =12.78

Sequential Filter - Results Blocky effect reduced to 8x8 pixels Contrast between blocks minimised Filtered image can be used as input to another filter

DC Subband Expansion Resolution of DC subband is increased 8-fold  To match the resolution of the original image  Inverse-DCT modified to use expanded DC subband Linear-interpolation used to fill the gaps between original DC subband values Applies a “gradient” to DC subband 128x128 pixel image128x128 expanded DC subband DC subband DCTDC expansion

DC Subband Expansion - Results Smooth regions completely void of blocky artifacts Expansion applied to filtered DC subband

What about the edges? ‘Staircase’ and ‘ringing’ effects still visible in all results Sequential deblocking filter inappropriate for AC subband filtering  AC subbands have very low inter-pixel correlation  Filtering AC subbands with this filter introduces ringing  DNH isn’t triggered because AC quantization intervals are very wide

What about the edges? (cont.) AC subbands describe edges, with respect to DC value Modifying AC coefficients adds/removes edges/textures Filtered image (far right) shows a light edge added next to the actual edge – a ringing artifact! AC filtering – topic for further research OriginalReconstructedFiltered-reconstructed

Unsuccessful Additions Always use 2-segment filtering  Contrast between blocks increased  ‘Chessboard’ effect Threshold segmenting  If class representatives are close, treat mask as 1 segment  Finding correct threshold value impossible 3-segment filtering  If 2-segment filtering triggered DNH, 3-segment filtering was attempted  Little-to-no improvement in image quality  Concluded that 2-segment model is sufficient for 95% of cases

Unsuccessful Additions (cont.) Overlapping mask filtering  All pixels’ filtered values in a mask were recorded  Filtered value equal to average of all possible recorded reconstructions (most pixels had up to 9 possible filtered values)  Little-to-no improvement in image quality Applying sequential filter to reconstructed pixel values  “Quantization interval” parameter not known  Guessing interval resulted in: Blurry images – blurred edges Overly-sharpened images – “cardboard cut-out” effect with flat colors

Conclusions Filtering DC subband has been successful in improving overall image quality  As predicted by the DCT coefficient study  Due to high inter-pixel correlation AC subbands must be filtered in some other way  This filter produces ringing artifacts Due to averaging Due to little inter-pixel correlation Due to extremely wide quantization intervals – allowing large change to an AC coefficient’s value  Little or no information left in subband Perhaps try median-filtering, instead of averaging  May introduce blurring of edges

Conclusions (cont.) Knowledge of a subband’s quantization interval aids in segmentation  Maximum quantization error is known  DNH “switches” between 1- or 2-segment filtering Sequential (adaptive) filtering model proved successful  DNH policy ensured no further image quality loss  Sometimes too strict Some blocks “don’t fit” in with surrounding pixels  Conforms to JPEG specifications Critical for video sequence coding (MPEG) – reconstruction errors propagate to subsequent frames

Future Research AC subbands must be filtered in a different manner altogether  Perhaps use DC subband segmentation to drive AC subband filtering Resulting image from filtered, non-expanded DC subband filter can be used as a starting point for a secondary filter  This filter should target edge artifacts Once edges are also filtered, DC expansion could be applied as a third step Try median-filtering, instead of averaging

For More Information… Consult Thesis See Website  http://www.csse.monash.edu.au/~kizewski/ E-mail me  kizewski@mail.csse.monash.edu.au Read referenced material Read filter source code Ask a question [1]: Rabbani, M. and Jones, P. W. (1991). Digital Image Compression Techniques, Vol. TT7 of SPIE Tutorial Texts. [2]: http://www.mat.univie.ac.at/~kriegl/Skripten/CG/node53.html

Thank-You Questions? …and now, for some eye-candy!

Download ppt "Image Deblocking Using Local Segmentation Lukasz Kizewski Supervisor:Dr. Peter Tischer Second Examiner:Dr. Andrew Paplinski."

Similar presentations