1. 3D LayoutCRF Derek Hoiem Carsten Rother John Winn

2. 2 Goal 1: Object Description Object Description: Bounding Box Viewpoint Color Pose Subclass

3. 3 Goal 2: Object Segmentation

4. 4 Combine object-level and pixel-level reasoning Key Idea

5. 5 Recognition Requires Object-Level Reasoning Position Shape/Size Viewpoint/Pose Style/Color

6. 6 Recognition Requires Object-Level Reasoning

7. 7 Solution: Window Detector? 45 degree range of viewpoints Minor scale/position variation

8. 8 What if we have a really good model?

9. 9 Recognition Requires Part-Level Reasoning Propose good global model

10. 10 Recognition Requires Part-Level Reasoning Propose good global model Occlusions

11. 11 Context Requires Both Object and Part-Level Info Size relationships require object model

12. 12 Context Requires Both Object and Part-Level Info Surface relationships require occlusion info Visibly sitting on ground Not visibly sitting on ground

13. 13 Our Object/Part Model T i = { h j  object parts bounding box, viewpoint, color model, instance cost } part consistency occlusions TmTm h1h1 h2h2 h3h3 h4h4 h5h5 h6h6 h7h7 h8h8 h9h9 h 10 h 11 hnhn x … … Extension from [Winn Shotton 2006] T1T1 …

14. 14 Modeling Viewpoint Parameterized by Bounding Box and Corner

15. 15 Assigning Parts from Model Training Image F L Training Annotation Assigned Parts 3D Parts Model

16. 16 Part Assignment Consistency

17. 17 Relabeling Allowing slight deformations, relabel training data Training Image Original Labels New Labels

18. 18 Eight Viewpoint/Scale Ranges Height Range Appearance (but not location) constant within each range

19. 19 Eight Viewpoint Ranges Left – Back 1Left - Back 2Front-Left 1Front - Left 2 Back - Right 1Back - Right 2Right – Front 1Right - Front 2 Mirrored

20. 20 Modeling Part Appearance Template patches (normalized xcorr) Intensity / Color Image Edges (DT)

21. 21 Modeling Part Appearance Randomized decision trees –25 trees, 250 leaf nodes Once: –Learn structure on 50,000 object / 50,000 background pixels For each appearance model: –Learn parameters on all pixels (850 LabelMe images)

22. 22 Inference Input Image

23. 23 Inference Input Image Proposals One per appearance model Objects proposed by connected components

24. 24 Proposal Stage Model h i  object parts part consistency occlusions h1h1 h2h2 h3h3 h4h4 h5h5 h6h6 h7h7 h8h8 h9h9 h 10 h 11 hnhn x … … CRF Inference (TRW-BP)

25. 25 Inference Refinement One per proposal Incorporate viewpoint, size information Proposals Input Image

26. 26 Refinement Stage Model T i = { h i  object parts bounding box, viewpoint } part consistency occlusions T1T1 h1h1 h2h2 h3h3 h4h4 h5h5 h6h6 h7h7 h8h8 h9h9 h 10 h 11 hnhn x … …

27. 27 Inference Refinement Proposals Arbitration Includes color model, instance penalty (graph cuts) Input Image

28. 28 Preliminary Results on UIUC Trained on 20, tested on rest Quantitatively comparable to best

29. 29 Preliminary Results on UIUC Without Instance Cost With Instance Cost T1T1 h1h1 h2h2 h3h3 h4h4 h5h5 h6h6 h7h7 h8h8 h9h9 h 10 h 11 hnhn x … …

30. 30 Preliminary Results on PASCAL’06 25 images –One proposal (viewpoint within 45 degrees, scale of 26-38 pixels)

31. 31 Preliminary Results on PASCAL’06

32. 32 Preliminary Results on PASCAL’06

33. 33 Preliminary Results on PASCAL’06 Without Color Model With Color Model

34. 34 Conclusion Combined object-level and pixel-level reasoning –Object-level: Position/Size, Viewpoint, Color –Pixel-level: Part appearance, Occlusion reasoning Good preliminary results

35. 35 To Do Further refinement of color model (e.g. could have color/texture-based instance cost) Obtain specific 3D model from detection and texture-map onto it (tbd by Carsten ) Quantitative evaluation on Pascal –Evaluate color model, instance penalty, etc. –Compare to John’s UIUC results