QUASI MAXIMUM LIKELIHOOD BLIND DECONVOLUTION QUASI MAXIMUM LIKELIHOOD BLIND DECONVOLUTION Alexander Bronstein

BIBLIOGRAPHY 2 2 A. Bronstein, M. Bronstein, and M. Zibulevsky, "Quasi maximum likelihood blind deconvolution: super- ans sub-Gaussianity vs. asymptotic stability", submitted to IEEE Trans. Sig. Proc. A. Bronstein, M. Bronstein, M. Zibulevsky, and Y. Y. Zeevi, "Quasi maximum likelihood blind deconvolution: asymptotic performance analysis", submitted to IEEE Trans. Information Theory. A. Bronstein, M. Bronstein, and M. Zibulevsky, "Relative optimization for blind deconvolution", submitted to IEEE Trans. Sig. Proc. A. Bronstein, M. Bronstein, M. Zibulevsky, and Y. Y. Zeevi, "Quasi maximum likelihood blind deconvolution of images acquired through scattering media", Submitted to ISBI04. A. Bronstein, M. Bronstein, M. Zibulevsky, and Y. Y. Zeevi, "Quasi maximum likelihood blind deconvolution of images using optimal sparse representations", CCIT Report No. 455 (EE No. 1399), Dept. of Electrical Engineering, Technion, Israel, December A. Bronstein, M. Bronstein, and M. Zibulevsky, "Blind deconvolution with relative Newton method", CCIT Report No. 444 (EE No. 1385), Dept. of Electrical Engineering, Technion, Israel, October 2003.

AGENDA 3 3 Introduction QML blind deconvolution Asymptotic analysis Relative Newton Generalizations Problem formulation Applications

BLIND DECONVOLUTION PROBLEM 4 4 source signal convolution kernel observed signal sensor noise signal WH CONVOLUTION MODELDECONVOLUTION restoration kernel source estimate arbitrary scaling factor arbitrary delay

APPLICATIONS 5 5 Acoustics, speech processing  DEREVERBERATION Optics, image processing, biomedical imaging  DEBLURRING Communications  CHANNEL EQUALIZATION Control  SYSTEM IDENTIFICATION Statistics, finances  ARMA ESTIMATION

AGENDA 6 6 Introduction QML blind deconvolution Asymptotic analysis Relative Newton Generalizations ML vs. QML The choice of φ(s) Equivariance Gradient and Hessian

ML BLIND DECONVOLUTION is i.i.d. with probability density function 2 has no zeros on the unit circle, i.e. ASSUMPTIONS MAXIMUM-LIKELIHOOD BLIND DECONVOLUTION: 3 No noise (precisely: no noise model) 4 is zero-mean

QUASI ML BLIND DECONVOLUTION 8 8 The true source PDF in usually unknown Many times is non-log-concave and not well-suited for optimization Substitute with some model function PROBLEMS OF MAXIMUM LIKELIHOOD QUASI MAXIMUM LIKELIHOOD BLIND DECONVOLUTION

THE CHOICE OF  (s) 9 9 SUPER-GAUSSIANSUB-GAUSSIAN

EQUIVARIANCE 10 QML estimator of given the observation Theorem: The QML estimator is equivariant, i.e., for every invertible kernel, it holds where stands for the impulse response of the inverse of. ANALYSIS OF

GRADIENT & HESSIAN OF 11 GRADIENT where is the mirror operator. HESSIAN

AGENDA 12 Introduction QML blind deconvolution Asymptotic analysis Relative Newton Generalizations Asymptotic Hessian structure Asymptotic error covariance Cramér-Rao bounds Superefficiency Examples

ASYMPTOTIC HESSIAN AT THE SOLUTION POINT 13 For a sufficiently large sample size, the Hessian becomes At the solution point, where

ASYMPTOTIC ERROR COVARIANCE 14 Estimation kernel from the data Exact restoration kernel The scaling factor has to obey

ASYMPTOTIC ERROR COVARIANCE 15 Estimation error From second-order Taylor expansion, equivariance

ASYMPTOTIC ERROR COVARIANCE 16 Asymptotically ( ), Separable structure:

ASYMPTOTIC ERROR COVARIANCE 17 The estimation error covariance matrix asymptotically separates to

ASYMPTOTIC ERROR COVARIANCE 18 Asymptotic gradient covariance matrices where

ASYMPTOTIC ERROR COVARIANCE 19 Asymptotic signal-to-interference ratio (SIR) estimate: Asymptotic estimation error covariance:

CRAMER-RAO LOWER BOUNDS 20 True ML estimator: The distribution-dependent parameters simplify to Asymptotic error covariance simplifies to where

CRAMER-RAO LOWER BOUNDS 21 Asymptotic SIR estimate simplifies to

SUPEREFFICIENCY 22 Let the source be sparse, i.e., Let be the smoothed absolute value with smoothing parameter In the limit

SUPEREFFICIENCY 23 Similar results are obtained for uniformly-distributed source with Can be extended for sources with PDF vanishing outside some interval.

ASYMPTOTIC STABILITY 24 The QML estimator is said to be asymptotically stable if is a local minimizer of in the limit. Theorem: The QML estimator is asymptotically stable if the following conditions hold: and is asymptotically unstable if one of the following conditions hold:

EXAMPLE 25 Generalized Laplace distribution

STABILITY OF THE SUPER-GAUSSIAN ESTIMATOR 26 SUPER-GAUSSIAN SUB-GAUSSIAN

STABILITY OF THE SUB-GAUSSIAN ESTIMATOR 27 SUPER-GAUSSIAN SUB-GAUSSIAN

PERFORMANCE OF THE SUPER-GAUSSIAN ESTIMATOR 28 SUPER-GAUSSIAN

PERFORMANCE OF THE SUB-GAUSSIAN ESTIMATOR 29 SUB-GAUSSIAN

AGENDA 30 Introduction QML blind deconvolution Asymptotic analysis Relative Newton Generalizations Relative optimization Relative Newton Fast Relative Newton

RELATIVE OPTIMIZATION (RO) 31 0 Start with and 1 For until convergence 4 Update source estimate 2 Start with 3 Find such that 5 End For Restoration kernel estimate: Source estimate:

RELATIVE OPTIMIZATION (RO) 32 Observation: The k-th step of the relative optimization algorithm depends only on Proposition: The sequence of target function values produced by the relative optimization algorithm is monotonically decreasing, i.e.,

RELATIVE NEWTON 33 Relative Newton = use one Newton step in the RO algorithm Near the solution point Newton system separates to

FAST RELATIVE NEWTON 34 Fast relative Newton = use one Newton step with approximate Hessian in the RO algorithm + regularized approximate Newton system solution. Approximate Hessian evaluation = order of gradient evaluation

AGENDA 35 Introduction QML blind deconvolution Asymptotic analysis Relative Newton Generalizations

GENERALIZATIONS 36 IIR KERNELS BLOCK PROCESSING  ONLINE DECONVOLUTION MULTI-CHANNEL DECONVOLUTION  BSS+BD DECONVOLUTION OF IMAGES + USE OF SPARSE REPRESENTATIONS

GENERALIZATIONS: IIR KERNELS IIR FIR

GENERALIZATIONS: ONLINE PROCESSING Fast Relative Newton (block)

GENERALIZATIONS: DECONVOLUTION OF IMAGES SOURCEOBSERVATIONRESTORATION

GENERALIZATIONS: DECONVOLUTION OF IMAGES Fast relative Newton Newton Fast relative Newton