1. 11/11/02 IDR Workshop Dealing With Location Uncertainty in Images Hasan F. Ates Princeton University 11/11/02

2. 11/11/02 IDR Workshop Outline: Introduction: –Wavelet transform : a sparse representation for images? –Dependencies of wavelet coefficients around edges Contour + profile approximation –How to exploit edge-directed smoothness of wavelet coefficients? Envelope + phase representation –How to achieve linear phase for wavelet coefficients? Applications –Image coding using profile space approach –Image interpolation Conclusion and future work

3. 11/11/02 IDR Workshop Sparseness of wavelet transform: - Coefs. classified as significant or insignificant. -Location: clusters of insignificant coefs. coded efficiently to reduce bit rate spent on classification map. -Sparseness: Each significant coefficient contributes to total bit rate.

4. 11/11/02 IDR Workshop Dependencies of wavelet coefficients: Interband dependencies: –Zerotrees: for insignificant coefs.  Rigid structure. Intraband dependencies: –Local variance estimation (EQ) –Adaptive nonlinear prediction (RMS). Edge based adaptive modeling:  Edge-directed smoothness needs to be exploited. Problems: 1) Sensitive to mistakes in edge location estimates 2) Side information?

5. 11/11/02 IDR Workshop Shuffling Experiment: Wavelet coefficients shuffled in one subband  Edge structure lost in reconstruction.  Same coding performance.

6. 11/11/02 IDR Workshop Contour+Profile Representation: Cone of influence for : (wavelet and point spread function). Piecewise constant approximation Wavelet values for approximation parametric description for contour:

7. 11/11/02 IDR Workshop Contour+profile (cont’d) : Wavelet values along the contour direction: Wavelet profile Contour + profile model for wavelet values:

8. 11/11/02 IDR Workshop Modeling wavelet coefficients: - Same 1-D profile at each row, sampled at different locations. - Find the profile and the contour that fits the data best (MAP estimation).

9. 11/11/02 IDR Workshop Estimating contour and profile: Discrete wavelet coefficients: step size: (wavelet transform), (redundant) MAP estimation: –AWG noise –Bandlimited profile, high-pass filtered contour; ( )

10. 11/11/02 IDR Workshop Contour optimization: Absolute edge locations by imposing antisymmetric profile. Gradient-descent iterative optimization Initial estimate crucial: –Wavelet local maxima connected by smooth contours –Edge locations updated by estimating zero crossings, and point of antisymmetry for the wavelet coefficients. –COI region depends on wavelet filter length, and other contours in the vicinity. –If initial contour and profile estimate captures less than 50% of energy within COI, then discard.

11. 11/11/02 IDR Workshop Contour+profile (cont’d) : Modeling assumptions: –Small contour curvature.  Effects of filtering in vertical direction ignored. Justified for locally linear edges. –Isolated edge: no other high-freq. patterns within COI  Not applicable for texture and multiple neighboring edges.  Edge estimates at coarser scales less reliable than estimates at finer scales. –AWGN model for residuals –Bandlimited profile: a more localized description is better  bandwidth: if too big, noise components enter if too small, fails to capture variations

12. 11/11/02 IDR Workshop Contour Estimation Using Previous Subband Estimate contour from previous subband -Profile estimate defined by orthogonal projection to a subspace (profile space) in current subband. (Redundant)

13. 11/11/02 IDR Workshop Phase Location Fourier phase: 1.Aliasing: non-integer displacements don’t give linear phase shifts (if not bandlimited). 2. FFT  Global transform; no localized phase information. Searching for smooth and localized phase:

14. 11/11/02 IDR Workshop Phase descriptions: 3. Superposition of compact isolated signals  Fourier phase doesn’t reflect locations. 4. Signal smoothness not captured by Fourier phase. The need for a smooth phase description that isn’t affected by aliasing and that provides location information of compact signals.

15. 11/11/02 IDR Workshop Phase descriptions (cont’d): The signal given in terms of two smooth (preferably bandlimited) components: Corresponding phase description: Design problem: choice of F (linear, nonlinear), properties of m1(t), m2(t).

16. 11/11/02 IDR Workshop Envelope+Phase Description: Signal described as an amplitude and phase modulated sinusoid.

17. 11/11/02 IDR Workshop Envelope+Phase (cont’d): m1, m2 have smaller (effective) bandwidth  Effects of aliasing reduced. Modulation freq: choose W to minimize m2(t). Linear phase:  Special case (FT symmetric w.r.t. W). Location uncertainty  linear shifts in phase

18. 11/11/02 IDR Workshop Envelope+Phase (cont’d): Compact m1, m2  env.+phase descriptions are additive for compact isolated signals. h[k], m1[k], m2[k] have equal lengths  # of dimensions doubled (redundant). fewer degrees of freedom allowed for m1, m2  linear phase approximation less successful.

19. 11/11/02 IDR Workshop Envelope+Phase in 2-D: Wavelet coefs. for jump discontinuity  compact; aliased due to downsampling in wavelet transform. Ideal edge in 2-D: wavelet coefs. in vertical subband; effects of vertical low-pass filter ignored.  Coefficients at each row k given by a common envelope+ phase representation and 1-D contour function

20. 11/11/02 IDR Workshop Envelope+Phase in 2-D: m1, m2 shaped by point spread function and wavelet filter. Design wavelet filters to minimize m2 (achieve linear phase).

21. 11/11/02 IDR Workshop Constructing envelope representation: Method of relaxation: achieves smooth m1, m2. – Start with (# dimensions) k=1, – Choose m1, m2 that is closest to the projection of previous estimates onto a smooth subspace of dim k, – Increase k and repeat until m1, m2 converges to smooth functions.

22. 11/11/02 IDR Workshop Filterbanks for phase descriptions: Redundant description: not suitable for coding. m1, m2 sampled at half-rate  filterbanks:

23. 11/11/02 IDR Workshop Image Coding:

24. 11/11/02 IDR Workshop Profile space approximation: Edge structure captured Over 90% energy in a single basis vector.

25. 11/11/02 IDR Workshop Image Interpolation: Image pixels sampled from same edge profile. Wavelet filtering in horizontal direction.

26. 11/11/02 IDR Workshop Model-based Interpolation: High-frequency enhancement based on profile model. Baseline interpolation : bilinear. Interpolation given by the least square solution of: such that

27. 11/11/02 IDR Workshop Reliability of model measured with percentage error Use the model when For multiple edges, texture, etc., no improvement. Diagonal edges: weighted average of vertical and horizontal models used. Interpolation (cont’d):

28. 11/11/02 IDR Workshop Interpolation (cont’d): Smooth edge contours. Up to 6dB improvement

29. 11/11/02 IDR Workshop Conclusion: Stability: Edge-directed adaptation too restrictive.  More general measures of similarity among neighboring wavelet coefficients.  Edge contour: can be defined as a probability distribution on the possible set of locations  Modeling phase based on m1, m2  Edge profiles at different directions Contour estimation: Simpler, computationally less expensive strategies.

30. 11/11/02 IDR Workshop Conclusion: Extending the model to multiple edges, texture, etc.  Related to the model being too restrictive.  More localized descriptions need to be used.