1. 1 Image Completion using Global Optimization Presented by Tingfan Wu

2. 2 The Image Inpainting Problem

3. 3 Outline Introduction Introduction History of Inpainting History of Inpainting Camps – Greedy & Global Opt. Camps – Greedy & Global Opt. Model and Algorithm Model and Algorithm Markov Random Fields (MRF) & Inpainting Markov Random Fields (MRF) & Inpainting Belief Propagation (BP) Belief Propagation (BP) Priority BP Priority BP Results Results Structural Propagation Structural Propagation

4. 4 Method Type Inpainting Bertalmio et al. SIGGRAPH 2000 Statistical Example Based Texture Synth./Patch Greedy Criminisi et al. 2003 Global Optimization Priority BP [This paper] Structural Propagation [Sun et. al, SIGGRAPH 05] PDE Bertalmio et al. 2001 Priority Texture Synth. Need User Guidance

5. 5 Exampled Based Method — Jigsaw Puzzle Patches Not Available

6. 6 Method Type Inpainting Bertalmio et al. SIGGRAPH 2000 Statistical Example Based Texture Synth./Patch Greedy Criminisi et al. 2003 Global Optimization Priority BP [This paper] Structural Propagation [Sun et. al, SIGGRAPH 05] PDE Bertalmio et al. 2001 Priority Texture Synth. Need User Guidance

7. 7 Greedy v.s Global Optmization Greedy Method Cannot go back  Global Optimization Ooop s Refine Globally

8. 8 Outline Introduction Introduction History of Inpainting History of Inpainting Camps – Greedy & Global Opt. Camps – Greedy & Global Opt. Model and Algorithm Model and Algorithm Markov Random Fields (MRF) & Inpainting Markov Random Fields (MRF) & Inpainting Belief Propagation (BP) Belief Propagation (BP) Priority BP Priority BP Results Results Structural Propagation Structural Propagation

9. 9 Random Fields / Belief Network Good Project Writer? (High Project grade) Good Test Taker? (High test score) Smart Student? (High GPA) Good Employee (No Observation yet) Edge: Dependency RF ： Random Variables on Graph RF ： Random Variables on Graph Node : Random Var. (Hidden State) Node : Random Var. (Hidden State) Belief : from Neighbors, and Observation Belief : from Neighbors, and Observation Random Variable (Observation)

10. 10 Story about MRF (Bayesian) Belief Network (DAG) (Bayesian) Belief Network (DAG) Markov Random Fields (Undirected, Loopy) Markov Random Fields (Undirected, Loopy) Special Case: Special Case: 1D - Hidden Markov Model (HMM) 1D - Hidden Markov Model (HMM) Office Helper Wizard Hidden Markov Model (HMM)

11. 11 Inpainting as MRF optimization Node : Grid on target region, overlapped patches Node : Grid on target region, overlapped patches Edge : A node depends only on its neighbors Edge : A node depends only on its neighbors Optimal labeling (hidden state) that minimizing mismatch energy Optimal labeling (hidden state) that minimizing mismatch energy

12. 12 MRF Potential Functions Mismatch (Energy) between.. V p (X p ) : Source Image vs. New Label V p (X p ) : Source Image vs. New Label V pq (X p, X q ) : Adjacent Labels V pq (X p, X q ) : Adjacent Labels Sum of Square Distances (SSD) in Overlapping Region Sum of Square Distances (SSD) in Overlapping Region

13. 13 Global Optimizatoin min

14. 14 Outline Introduction Introduction History of Inpainting History of Inpainting Camps – Greedy & Global Opt. Camps – Greedy & Global Opt. Model and Algorithm Model and Algorithm Markov Random Fields (MRF) & Inpainting Markov Random Fields (MRF) & Inpainting Belief Propagation (BP) Belief Propagation (BP) Priority BP Priority BP Results Results Structural Propagation Structural Propagation

15. 15 Belief Propagation(1/3) Undirected and Loopy Propagate forward and backward Good Project Writer? (High Project grade) Good Test Taker? (High test score) Smart Student? (High GPA) Good Employee (No Observation yet)

16. 16 Belief Propagation(2/3) Message Forwarding Message Forwarding Iterative algorithm until converge Iterative algorithm until converge X q X p Candidates at Node Q Candidates at Node P Neighbors (P) O(|Candidate| 2 )

17. 17 Belief Propagation(3/3)

18. 18 Priority BP BP too slow: BP too slow: Huge #candidates  Time msg = O(|Candidates| 2 ) Huge #candidates  Time msg = O(|Candidates| 2 ) Huge #Pairs  Cannot cache pairwise SSDs. Huge #Pairs  Cannot cache pairwise SSDs. Observations Observations Non-Informative messages in unfilled regions Non-Informative messages in unfilled regions Solution to some nodes is obvious (fewer candidates.) Solution to some nodes is obvious (fewer candidates.)

19. 19 Human Wisdom Nobody start from here Start from non- ambiguous part And Search for Brown feather +green grass Candidates

20. 20 Priority BP Observations Observations needless messages in unfilled regions needless messages in unfilled regions Solution to some nodes is obvious (fewer candidates.) Solution to some nodes is obvious (fewer candidates.) Solution: Enhanced BP: Solution: Enhanced BP: Easy nodes goes first (priority message scheduling) Easy nodes goes first (priority message scheduling) Keep only highly possible candidates (maintain a Active Set) Keep only highly possible candidates (maintain a Active Set)

21. 21 Priority & Pruning ? ? ? ? ? ? ? ? High Priority prune a lot Pruning may miss correct label Low Priority Candidates sorted by relative belief Discard Blue Points

22. 22 #Candidates after Pruning Active Set (Darker means smaller) Histogram of #candidatesSimilar candidates

23. 23 A closer look at Priority BP Priority Calculation Priority Calculation Priority : 1/(#significant candidate) Priority : 1/(#significant candidate) Pruning (on the fly ) Pruning (on the fly ) Discard Low Confidence Candidates Discard Low Confidence Candidates Similar patches  One representative (by clustering) Similar patches  One representative (by clustering) Result Result More Confident  More Pruning More Confident  More Pruning Confident node helps increase neighbor ’ s confidence. Confident node helps increase neighbor ’ s confidence. Warning: Warning: PBP and Pruning must be used together PBP and Pruning must be used together

24. 24 Extensions (Optional) Adding constraints by modifying distance function Adding constraints by modifying distance function Spatial Coherence : fill target region with large chunks. Spatial Coherence : fill target region with large chunks.  Good for texture synthesis Patch blending with weights confidence Patch blending with weights confidence Multi-scale inpainting. Multi-scale inpainting. Create pseudo source image at fine scale Create pseudo source image at fine scale Recover both coarse and fine texture Recover both coarse and fine texture Fast SSD by FFT. Fast SSD by FFT. _

25. 25 Outline Introduction Introduction History of Inpainting History of Inpainting Camps – Greedy & Global Opt. Camps – Greedy & Global Opt. Model and Algorithm Model and Algorithm Markov Random Fields (MRF) & Inpainting Markov Random Fields (MRF) & Inpainting Belief Propagation (BP) Belief Propagation (BP) Priority BP Priority BP Results Results Conclusion Conclusion Structural Propagation Structural Propagation

26. 26 Darker pixels  higher priority Automatically start from salient parts. Results-Inpainting(1/3)

27. 27 Results-Inpainting(2/3)

28. 28 Results-Inpainting(3/3) Up to 2minutes / image (256x170) on P4-2.4G Up to 2minutes / image (256x170) on P4-2.4G

29. 29 More : Texture Synthesis Interpolation as well as extrapolation Interpolation as well as extrapolation

30. 30

31. 31 Conclusion Priority BP Priority BP {Confident node first} + {candidate pruning} {Confident node first} + {candidate pruning} Generic – applicable to other MRF problems. Generic – applicable to other MRF problems. Speed up Speed up MRF for Inpainting MRF for Inpainting Global optimization Global optimization avoid visually inconsistence by greedy avoid visually inconsistence by greedy Priority BP for Inpainting Priority BP for Inpainting Automatically start from salient point. Automatically start from salient point.

32. 32 Sometimes … Image contains hard high-level structure Image contains hard high-level structure Hard for computers Hard for computers Interactive completion guided by human. Interactive completion guided by human.

33. 33 Potential Func. For Structural Propagation User input a guideline by human region. User input a guideline by human region. Potential Function respect distance between lines Potential Function respect distance between lines Jian Sun et al, SIGGRAPH 2005

34. 34 Video Link:Microsoft Research Link:Microsoft ResearchMicrosoft ResearchMicrosoft Research