1. Creating images the 2-D way Jean-François Lalonde April 20, 2010

2. Creating images (3-D)

3. Creating images (2-D + 3-D)

4. Inserting objects into images

5. Inserting objects into images(2-D + 3-D) [Debevec, 98]

6. Inserting objects in images Realistic renderings Expensive and impractical Highly detailed geometry Highly detailed materials Very expensive [Debevec, 98]

7. Alternative: Clip art Easy, intuitive, cheap Not realistic

8. Creating images (2-D) Photo-realisticCartoon Expensive and impracticalCheap and intuitive Image-based renderingClip ArtPhoto Clip Art ? ?

9. Photoshop-ing Composite by David Dewey

10. Inserting objects into images

11. Challenges object orientation scene illumination Insert THIS object: impossible!

12. The use of data Insert SOME object: much easier!

13. The Google model Database Sort the objects QueryResults

14. 2-D image vs 3-D scene

15. Outline Phase I: Database annotation Phase II: Object insertion Name: person Subgroup: person, standing Height: 1.5m Local context: in shadow Illumination: sunny, bright day, no cloud Segmentation quality: excellent, >40 points Upsampling blur: low

16. Data source: LabelMe Online (http://labelme.csail.mit.edu), user- contributedhttp://labelme.csail.mit.edu 170,000 objects in 40,000 images Polygons and names [Russell et al., 2005]

17. Data organization Top-level categories (chosen manually, 16 total) Second-level categories (from annotations or clustering)

18. Annotating the objects

19. Camera parameters Assume flat ground plane all objects on ground camera roll is negligible (consider pitch only) Camera parameters: height and orientation

20. Human height distribution 1.7 +/- 0.085 m (National Center for Health Statistics) Car height distribution 1.5 +/- 0.19 m (automatically learned) Camera parameters

21. Object heights Database image Pixel heightsReal heights

22. ObjectEstimated average height (m) Car1.51 Man1.80 Woman1.67 Parking meter1.36 Fire hydrant0.87 Estimated object heights 1.0 m Car 0.5 m ManWoman Parking meter Fire hydrant 1.5 m

23. Geometry is not enough

24. Illumination context Database image Environment map rough approximation Exact environment map is impossible Approximations [Khan et al., 06]

25. Illumination context Database imageP(pixel|class)CIE L*a*b* histograms Automatic Photo Popup Hoiem et al., SIGGRAPH 05 Automatic Photo Popup Hoiem et al., SIGGRAPH 05

26. Illumination nearest-neighbors

27. Other criteria: local context

28. Other criteria: segmentation LabelMe contributors not always reliable Segmentation quality 38 points / polygon4 points / polygon

29. Other criteria: blur Resolution: avoid up-sampling x3 up-sampling

30. Recap Phase I: Database annotation Phase II: Object insertion 3-D heightSegmentationBlur Object properties (used for sorting the database) LabelIllumination contextClusterLocal contextCamera

31. Lets insert an object! Poor user-provided segmentations Noticeable seams

32. Seams InputDestination image Result Visible seam! [Perez et al., 2003]

33. Poisson blending: idea InputDestination Result Enforce boundary color (seamless result) Enforce same gradient than input

34. Why gradients? 1-D example Regular blending bright dark

35. 1-D example Blending derivatives Original signalsDerivatives Reintegration results

36. 1-D example Gradient domainIntensity domain ?

37. 2-D: not so easy +1 +2 -3 4 02 5 -2 Non integrable: sum over a loop 0 Actually happens all the time in practice

38. 2-D: some notation Finite differences

39. 2-D: a (possible) solution Least-squares solution:?

40. Solution: Poisson equation Popular because: Solution is obtained by solving a linear system of equations Can be solved (somewhat) efficiently \ in matlab FFT Multi-grid solvers (approximate, but really fast!) 2-D: a (popular) solution

41. Results & limitations Image editing Some limitations Images need to be very well aligned Differences in background bleed through Images from [Perez et al., 2003]

42. Poisson blending: improvements Drag-and-Drop Pasting [Jia et al., 2006] User-selected boundary Poisson blending Refined boundary Poisson blending Images from [Jia et al., 2006] Color bleeding

43. Still not right! Not so sensitive to shadow direction [Cavanagh, 2005]

44. Shadow transfer ++ == Database imageShadow estimateRefined shadow Object aloneShadow aloneObject with shadow

45. User interface

46. Street accident

47. Bridge

48. Painting

49. Alley

50. Failure cases Shadow transfer Porous objects

51. Failure cases Best matching objects

52. Pros & cons ProsCons 3-D More control (camera, geometry, lighting...) Complex! 2-D + 3-DRealistic Complex! Need access to scene 2-D (photoshop) Realistic Complex! Need to sort through images 2-D (automatic) Easy, intuitiveGeneric objects

53. Thanks! jlalonde@cs.cmu.edu