1. 1 Regularization with Singular Energies: Error Estimation and Numerics Martin Burger Institut für Numerische und Angewandte Mathematik Westfälische Wilhelms Universität Münster martin.burger@uni-muenster.de TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AA A AA

2. Regularization with Singular Energies Oberwolfach, Januar 20072 Stan Osher, Jinjun Xu, Guy Gilboa (UCLA) Lin He (Linz / UCLA) Klaus Frick, Otmar Scherzer (Innsbruck) Don Goldfarb, Wotao Yin (Columbia) Collaborations

3. Regularization with Singular Energies Oberwolfach, Januar 20073 Classical regularization schemes for inverse problems and image smoothing are based on Hilbert spaces and quadratic energy functionals Example: Tikhonov regularization for linear operator equations Introduction ¸ 2 k A u ¡ f k 2 + 1 2 k L u k 2 ! m i n u

4. Regularization with Singular Energies Oberwolfach, Januar 20074 These energy functionals are strictly convex and differentiable – standard tools from analysis and computation (Newton methods etc.) can be used Disadvantage: possible oversmoothing, seen from first-order optimality condition Tikhonov yields Hence u is in the range of (L*L) -1 A* Introduction L ¤ L u = ¡ ¸ A ¤ ( A u f )

5. Regularization with Singular Energies Oberwolfach, Januar 20075 Classical inverse problem: integral equation of the first kind, regularization in L 2 (L = Id), A = Fredholm integral operator with kernel k Smoothness of regularized solution is determined by smoothness of kernel For typical convolution kernels like Gaussians, u is analytic ! Introduction u = ¸ ZZ k ( y ; x )( ¡ k ( y ; z ) u ( z ) + f ( z )) d y d z

6. Regularization with Singular Energies Oberwolfach, Januar 20076 Classical image smoothing: data in L 2 (A = Id), L = gradient (H 1 -Seminorm) On a reasonable domain, standard elliptic regularity implies Reconstruction contains no edges, blurs the image (with Green kernel) Image Smoothing ¡ ¢ u + ¸ u = ¸f u 2 H 2 ( ­ ), ! C ( ­ )

7. Regularization with Singular Energies Oberwolfach, Januar 20077 Let A be an operator on (basis repre- sentation of a Hilbert space operator, wavelet) Penalization by squared norm (L = Id) Optimality condition for components of u Decay of components determined by A*. Even if data are generated by sparse signal (finite number of nonzeros), reconstruction is not sparse ! Sparse Reconstructions ? ` 2 ( Z ) u k = ¸ ( A ¤ ( ¡ A u + f )) k

8. Regularization with Singular Energies Oberwolfach, Januar 20078 Error estimates for ill-posed problems can be obtained only under stronger conditions (source conditions) cf. Groetsch, Engl-Hanke-Neubauer, Colton-Kress, Natterer. Engl-Kunisch-Neubauer. Equivalent to u being minimizer of Tikhonov functional with data For many inverse problems unrealistic due to extreme smoothness assumptions Error estimates 9 w:u = A ¤ w

9. Regularization with Singular Energies Oberwolfach, Januar 20079 Condition can be weakened to cf. Neubauer et al (algebraic), Hohage (logarithmic), Mathe-Pereverzyev (general). Advantage: more realistic conditions Disadvantage: Estimates get worse with f Error estimates 9 v:u = f ( A ¤ A ) v

10. Regularization with Singular Energies Oberwolfach, Januar 200710 Let A be the identity on Nonlinear Penalization by Optimality condition for components of u If r k is smooth and strictly convex, then Taylor expansion yields Singular Energies ` 2 ( Z ) P r k ( u k ) r 00 k ( f k ) u k + ¸ u k ¼ r 00 k ( f k ) f k + ¸f k r 0 k ( u k ) + ¸ u k = ¸f k

11. Regularization with Singular Energies Oberwolfach, Januar 200711 Example becomes more interesting for singular (nonsmooth) energy Take Then optimality condition becomes Singular Energies r k ( t ) = j t j s i gn ( u k ) + ¸ u k = ¸f k

12. Regularization with Singular Energies Oberwolfach, Januar 200712 Result is well-known soft-thresholding of wavelets Donoho et al, Chambolle et al Yields a sparse signal Singular Energies u k = 8 < : f k ¡ 1 ¸ f k > 1 ¸ f k + 1 ¸ f k < ¡ 1 ¸ 0 e l se

13. Regularization with Singular Energies Oberwolfach, Januar 200713 Image smoothing: try nonlinear energy for penalization Optimality condition is nonlinear PDE If r is strictly convex usual smoothing behaviour If r is not convex problem not well-posed Try singular case at the borderline Singular Energies Z r ( r u ) ¡ r ¢ (( r r )( r u )) + ¸ u = ¸f

14. Regularization with Singular Energies Oberwolfach, Januar 200714 Simplest choice yields total variation method Total variation methods are popular in imaging (and inverse problems), since - they keep sharp edges - eliminate oscillations (noise) - create new nice mathematics Total Variation Methods r ( p ) = j p j

15. Regularization with Singular Energies Oberwolfach, Januar 200715 ROF model for denoising Rudin-Osher Fatemi 89/92, Acar-Vogel 93, Chambolle-Lions 96, Vogel 95/96, Scherzer-Dobson 96, Chavent-Kunisch 98, Meyer 01,… ROF Model

16. Regularization with Singular Energies Oberwolfach, Januar 200716 Optimality condition for ROF denoising Dual variable p enters ! Subgradient of convex functional ROF Model p + ¸ u = ¸f ; p 2 @ j u j TV @ J ( u ) = f p 2 X ¤ j 8 v 2 X : J ( u ) ¡ h p ; v ¡ u i · J ( v ) g

17. Regularization with Singular Energies Oberwolfach, Januar 200717 ROF Model Reconstruction (code by Jinjun Xu) cleannoisy ROF

18. Regularization with Singular Energies Oberwolfach, Januar 200718 ROF model denoises cartoon images resp. computes the cartoon of an arbitrary image ROF Model

19. Regularization with Singular Energies Oberwolfach, Januar 200719 From Master Thesis of Markus Bachmayr, 2007 Numerical Differentiation with TV

20. Regularization with Singular Energies Oberwolfach, Januar 200720 Methods with singular energies offer great potential, but still have some shortcomings - difficult to analyze and to obtain error estimates - systematic errors (clean images not reconstructed perfectly) - computational challenges - some extensions to complicated imaging tasks are not well understood (e.g. inpainting) Singular energies

21. Regularization with Singular Energies Oberwolfach, Januar 200721 General problem leads to optimality condition First of all „dual smoothing“, subgradient p is in the range of A* Singular energies ¸ 2 k A u ¡ f k 2 + J ( u ) ! m i n u p + ¸ A ¤ A u = ¸ A ¤ f ; p 2 @ J ( u )

22. Regularization with Singular Energies Oberwolfach, Januar 200722 For smooth and strictly convex energies, the subdifferential is a singleton Dual smoothing directly results in a primal one ! For singular energies, subdifferentials are not usually multivalued. The consequence is a possibility to break the primal smoothing Singular energies @ J ( u ) = f J 0 ( u ) g

23. Regularization with Singular Energies Oberwolfach, Januar 200723 First question for error estimation: estimate difference of u (minimizer of ROF) and f in terms of Estimate in the L 2 norm is standard, but does not yield information about edges Estimate in the BV-norm too ambitious: even arbitrarily small difference in edge location can yield BV-norm of order one ! Error Estimation

24. Regularization with Singular Energies Oberwolfach, Januar 200724 We need a better error measure, stronger than L 2, weaker than BV Possible choice: Bregman distance Bregman 67 Real distance for a strictly convex differentiable functional – not symmetric Symmetric version Error Estimation

25. Regularization with Singular Energies Oberwolfach, Januar 200725 Bregman distances reduce to known measures for standard energies Example 1: Subgradient = Gradient = u Bregman distance becomes Error Estimation J ( u ) = 1 2 k u k 2 D J ( u ; v ) = 1 2 k u ¡ v k 2

26. Regularization with Singular Energies Oberwolfach, Januar 200726 Bregman distances reduce to known measures for standard energies Example 2: - Subgradient = Gradient = log u Bregman distance becomes Kullback-Leibler divergence (relative Entropy) Error Estimation J ( u ) = Z u l ogu Z u D J ( u ; v ) = Z u l og u v + Z ( v ¡ u )

27. Regularization with Singular Energies Oberwolfach, Januar 200727 Total variation is neither symmetric nor differentiable Define generalized Bregman distance for each subgradient Symmetric version Kiwiel 97, Chen-Teboulle 97 Error Estimation

28. Regularization with Singular Energies Oberwolfach, Januar 200728 For energies homogeneous of degree one, we have Bregman distance becomes Error Estimation

29. Regularization with Singular Energies Oberwolfach, Januar 200729 Bregman distance for singular energies is not a strict distance, can be zero for In particular d TV is zero for contrast change Resmerita-Scherzer 06 Bregman distance is still not negative (convexity) Bregman distance can provide information about edges Error Estimation

30. Regularization with Singular Energies Oberwolfach, Januar 200730 Let v be piecewise constant with white background and color values on regions Then we obtain subgradients of the form with signed distance function and Error Estimation

31. Regularization with Singular Energies Oberwolfach, Januar 200731 Bregman distances given by In the limit we obtain for being piecewise continuous Error Estimation

32. Regularization with Singular Energies Oberwolfach, Januar 200732 For estimate in terms of we need smoothness condition on data Optimality condition for ROF Error Estimation

33. Regularization with Singular Energies Oberwolfach, Januar 200733 Subtract q Estimate for Bregman distance, mb-Osher 04 Error Estimation

34. Regularization with Singular Energies Oberwolfach, Januar 200734 In practice we have to deal with noisy data f (perturbation of some exact data g) Estimate for Bregman distance Error Estimation

35. Regularization with Singular Energies Oberwolfach, Januar 200735 Optimal choice of the penalization parameter i.e. of the order of the noise variance Error Estimation

36. Regularization with Singular Energies Oberwolfach, Januar 200736 Direct extension to deconvolution / linear inverse problems under standard source condition mb-Osher 04 Extension: stronger estimates under stronger conditions, Resmerita 05 Nonlinear inverse problems, Resmerita-Scherzer 06 Error Estimation ¸ 2 k A u ¡ f k 2 + j u j TV ! m i n u 2 BV

37. Regularization with Singular Energies Oberwolfach, Januar 200737 Extension to other fitting functionals (relative entropy, log-likelihood functionals for different noise models) Extension to anisotropic TV (Interpretation of subgradients) Extension to geometric problems (segmentation by Chan-Vese, Mumford-Shah): use exact relaxation in BV with bound constraints Chan-Esedoglu-Nikolova 04 Error Estimation: Future tasks

38. Regularization with Singular Energies Oberwolfach, Januar 200738 Natural choice: primal discretization with piecewise constant functions on grid Problem 1: Numerical analysis (characterization of discrete subgradients) Problem 2: Discrete problems are the same for any anisotropic version of the total variation Discretization

39. Regularization with Singular Energies Oberwolfach, Januar 200739 In multiple dimensions, nonconvergence of the primal discretization for the isotropic TV (p=2) can be shown Convergence of anisotropic TV (p=1) on rectangular aligned grids Fitzpatrick-Keeling 1997 Discretization

40. Regularization with Singular Energies Oberwolfach, Januar 200740 Alternative: perform primal-dual discretization for optimality system (variational inequality) with convex set Primal-Dual Discretization

41. Regularization with Singular Energies Oberwolfach, Januar 200741 Discretization Discretized convex set with appropriate elements (piecewise linear in 1D, Raviart- Thomas in multi-D) Primal-Dual Discretization

42. Regularization with Singular Energies Oberwolfach, Januar 200742 In 1 D primal, primal-dual, and dual discretization are equivalent Error estimate for Bregman distance by analogous techniques Note that only the natural condition is needed to show Primal / Primal-Dual Discretization

43. Regularization with Singular Energies Oberwolfach, Januar 200743 In multi-D similar estimates, additional work since projection of subgradient is not discrete subgradient. Primal-dual discretization equivalent to discretized dual minimization (Chambolle 03, Kunisch-Hintermüller 04). Can be used for existence of discrete solution, stability of p Mb 07 ? Primal / Primal-Dual Discretization

44. Regularization with Singular Energies Oberwolfach, Januar 200744 For most imaging applications Cartesian grids are used. Primal dual discretization can be reinterpreted as a finite difference scheme in this setup. Value of image intensity corresponds to color in a pixel of width h around the grid point. Raviart-Thomas elements on Cartesian grids particularly easy. First component piecewise linear in x, pw constant in y,z, etc. Leads to simple finite difference scheme with staggered grid Cartesian Grids

45. Regularization with Singular Energies Oberwolfach, Januar 200745 ROF minimization has a systematic error, total variation of the reconstruction is smaller than total variation of clean image. Image features left in residual f-u g, clean f, noisy u, ROFf-u Iterative Refinement & ISS

46. Regularization with Singular Energies Oberwolfach, Januar 200746 Idea: add the residual („noise“) back to the image to pronounce the features decreased to much. Then do ROF again. Iterative procedure Osher-mb-Goldfarb-Xu-Yin 04 Iterative Refinement & ISS

47. Regularization with Singular Energies Oberwolfach, Januar 200747 Improves reconstructions significantly Iterative Refinement & ISS

48. Regularization with Singular Energies Oberwolfach, Januar 200748 Iterative Refinement & ISS

49. Regularization with Singular Energies Oberwolfach, Januar 200749 Simple observation from optimality condition Consequently, iterative refinement equivalent to Bregman iteration Iterative Refinement & ISS

50. Regularization with Singular Energies Oberwolfach, Januar 200750 Choice of parameter less important, can be kept small (oversmoothing). Regularizing effect comes from appropriate stopping. Quantitative stopping rules available, or „stop when you are happy“ – S.O. Limit to zero can be studied. Yields gradient flow for the dual variable („inverse scale space“) mb-Gilboa-Osher-Xu 06, mb-Frick-Osher-Scherzer 06 Iterative Refinement & ISS

51. Regularization with Singular Energies Oberwolfach, Januar 200751 Non-quadratic fidelity is possible, some caution needed for L 1 fidelity He-mb-Osher 05, mb-Frick-Osher-Scherzer 06 Error estimation in Bregman distance mb-He-Resmerita 07 Iterative Refinement & ISS

52. Regularization with Singular Energies Oberwolfach, Januar 200752 MRI Data Siemens Magnetom Avanto 1.5 T Scanner He, Chang, Osher, Fang, Speier 06 PenalizationTV + Wavelet Iterative Refinement

53. Regularization with Singular Energies Oberwolfach, Januar 200753 MRI Data Siemens Magnetom Avanto 1.5 T Scanner He, Chang, Osher, Fang, Speier 06 Iterative Refinement

54. Regularization with Singular Energies Oberwolfach, Januar 200754 MRI Data Siemens Magnetom Avanto 1.5 T Scanner He, Chang, Osher, Fang, Speier 06 Iterative Refinement

55. Regularization with Singular Energies Oberwolfach, Januar 200755 Smoothing of surfaces obtained as level sets 3D Ultrasound, Kretz / GE Med. Surface Smoothing

56. Regularization with Singular Energies Oberwolfach, Januar 200756 Inverse Scale Space

57. Regularization with Singular Energies Oberwolfach, Januar 200757 Application to other regularization techniques, e.g. wavelet thresholding is straightforward Starting from soft shrinkage, iterated refinement yields firm shrinkage, inverse scale space becomes hard shrinkage Osher-Xu 06 Bregman distance natural sparsity measure, source condition just requires sparse signal, number of nonzero components is smoothness measure in error estimates Iterative Refinement & ISS

58. Regularization with Singular Energies Oberwolfach, Januar 200758 Download and Contact Papers and Talks: www.math.uni-muenster.de/u/burger e-mail: martin.burger@uni-muenster.de