1. CSP Examples Tamara Berg CS 560 Artificial Intelligence Many slides throughout the course adapted from Svetlana Lazebnik, Dan Klein, Stuart Russell, Andrew Moore, Percy Liang, Luke Zettlemoyer

2. Announcements/Reminders HW1 due date extended to Sunday, Sept 13, 11:59pm on classroom server – submit one solution per group and email Ric (poirson@cs.unc.edu) with the location of your directory and names of your group memberspoirson@cs.unc.edu

3. Review from last class

4. Backtracking search In CSP’s, variable assignments are commutative – For example, [WA = red then NT = green] is the same as [NT = green then WA = red] We only need to consider assignments to a single variable at each level (i.e., we fix the order of assignments) Depth-first search for CSPs with single-variable assignments is called backtracking search

5. Example

9. Red?Green?Blue? x xx x

10. Backtracking search algorithm

11. Improving Backtracking Efficiency Ordering – How should we order variables for assignment? – How should we order values from the domains?

12. Backtracking search algorithm

13. Improving Backtracking Efficiency Ordering – How should we order variables for assignment? – How should we order values from the domains? Filtering – Can we detect inevitable failures early?

14. Early detection of failure Apply inference to reduce the space of possible assignments and detect failure early

15. Early detection of failure: Forward checking Keep track of remaining legal values for unassigned variables Terminate search when any variable has no legal values

16. Simplest form of constraint propagation makes each pair of variables consistent: – X  Y is consistent iff for every value of X there is some allowed value of Y Arc consistency Consistent!

17. Simplest form of constraint propagation makes each pair of variables consistent: – X  Y is consistent iff for every value of X there is some allowed value of Y Arc consistency

18. Simplest form of propagation makes each pair of variables consistent: – X  Y is consistent iff for every value of X there is some allowed value of Y – When checking X  Y, throw out any values of X for which there isn’t an allowed value of Y If X loses a value, all pairs Z  X need to be rechecked Arc consistency

19. Simplest form of propagation makes each pair of variables consistent: – X  Y is consistent iff for every value of X there is some allowed value of Y – When checking X  Y, throw out any values of X for which there isn’t an allowed value of Y Arc consistency detects failure earlier than forward checking Can be run before or after each assignment Arc consistency

20. Improving Backtracking Efficiency Ordering – How should we order variables for assignment? – How should we order values from the domains? Filtering – Can we detect inevitable failures early? Structure – Can we exploit problem structure?

21. Tasmania and the mainland are separate structures. We can solve them independently and then combine their solutions. More generally, we can decompose the CSP into separate connected components (sub-problems of the CSP). Then the solution is the union of the sub- problem solutions. This can be much more efficient than solving the entire CSP as one problem.

22. Algorithm for tree-structured CSPs Choose one variable as root, order variables from root to leaves such that every node's parent precedes it in the ordering http://cs188ai.wikia.com/wiki/Tree_Structure_CSPs

23. Algorithm for tree-structured CSPs Choose one variable as root, order variables from root to leaves such that every node's parent precedes it in the ordering Backward removal phase: check arc consistency starting from the rightmost node and going backwards http://cs188ai.wikia.com/wiki/Tree_Structure_CSPs

24. Algorithm for tree-structured CSPs Choose one variable as root, order variables from root to leaves such that every node's parent precedes it in the ordering Backward removal phase: check arc consistency starting from the rightmost node and going backwards Forward assignment phase: select an element from the domain of each variable going left to right. We are guaranteed that there will be a valid assignment because each arc is arc consistent http://cs188ai.wikia.com/wiki/Tree_Structure_CSPs

25. Alternatives to backtracking

26. Local Search Methods Local Search: Improve what you have until you can’t make it better Generally much faster and more memory efficient than back-tracking search (but not necessarily complete)

27. Local search for CSPs Start with an initial assignment of variables to values Allow states with unsatisfied constraints Attempt to improve states by reassigning variable values Hill-climbing search: – In each iteration, randomly select any conflicted variable and choose value that violates the fewest constraints – I.e., attempt to greedily minimize total number of violated constraints h = number of conflicts

28. Local search for CSPs Start with “complete” states, i.e., all variables assigned Allow states with unsatisfied constraints Attempt to improve states by reassigning variable values Hill-climbing search: – In each iteration, randomly select any conflicted variable and choose value that violates the fewest constraints – I.e., attempt to greedily minimize total number of violated constraints – Problem: local minima h = 1

31. Origin of CSPs

32. CSP in computer vision: Line drawing interpretation An example polyhedron: David WaltzDavid Waltz, 1975 Desired output: Goal: Label lines in drawing as convex (+), concave (-), or boundary (>). How could we set this up as a CSP? Variables? Domains? Constraints?

33. CSP in computer vision: Line drawing interpretation Four vertex types: Constraints imposed by each vertex type: David WaltzDavid Waltz, 1975

35. CSPs in recent research!

36. Inferring Temporal Order of Images from 3d Structure Grant Schindler, Frank Dellaert, Sing Bing Kang IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2007. http://www.cc.gatech.edu/~phlosoft/

37. The World Changes Over Time How can we reason about time in structure from motion problems?

38. Ultimate Goal Produce time-varying 3d models of cities from historical photographs Requires: Building 3d models Reasoning about time

39. Temporal Inference Problem 1. When was each photograph taken? 2. When did each building first appear? 3. When was each building removed? Set of Photographs: Set of Objects: Buildings

40. Temporal Inference Problem (a, b)Time Interval tTime Temporal Inference Problem Given observations Z and scene geometry X, what are the temporal parameters T? Images Structure from Motion (x,y,z)Position (x,y,z)Translation (y,p,r)Rotation f Focal Length Structure Cameras Correspondence s 3D

41. Overview 3D Images + Time Structure + Time Intervals 1891 1934 2007 Buildings 1885 - 19531906 - 20091968 - 2009 4D Correspondences Images

42. Overview 3D Images + Time Structure + Time Intervals 1891 1934 2007 Buildings 1885 - 19531906 - 20091968 - 2009 4D Correspondences Images

43. Overview 3D Images + Time Structure + Time Intervals 1891 1934 2007 Buildings 1885 - 19531906 - 20091968 - 2009 4D Correspondences Images

44. Overview 3D Images + Time Structure + Time Intervals 1891 1934 2007 Buildings 1885 - 19531906 - 20091968 - 2009 4D Correspondences Image s 1937 20012009

45. 3D Reconstruction Images 3D Point Cloud 1891 1934 2007 1885 - 19531906 - 20091968 - 2009 Structure from Motion Bundler Software by Noah Snavely (Snavely et al SIGGRAPH 2006)

46. Scene Reconstruction Automatically estimate – position, orientation, and focal length of cameras – 3D positions of feature points Feature detection Pairwise feature matching Pairwise feature matching Incremental structure from motion Correspondence estimation

47. Input Photos

48. Feature detection Detect features points on objects [Lowe, IJCV 2004]

49. Pairwise feature matching Match feature points between each pair of images

50. Correspondence estimation Link up pairwise matches to form matches across several images Image 1Image 2Image 3Image 4

51. Structure from motion Camera 1 Camera 2 Camera 3 R 1,t 1 R 2,t 2 R 3,t 3 p1p1 p4p4 p3p3 p2p2 p5p5 p6p6 p7p7 minimize f (R, T, P)f (R, T, P)

52. Incremental structure from motion See: e.g. http://www.cs.cornell.edu/projects/bigsfm/ for more infohttp://www.cs.cornell.edu/projects/bigsfm/

53. Grouping Points into Buildings 3D Point Cloud 3D Buildings Building Geometry 1. Convex hull of each group 2. Fit ground plane to camera centers 3. Extend convex hulls to ground 1891 1934 2007 1885 - 19531906 - 20091968 - 2009 Point Grouping Group points according to both: a. Distance < threshold in 3D b. Observed simultaneously in 1+ images

54. 3D Reconstruction: Points vs. Objects Lower Manhattan 454 images 83,860 points960 Buildings 1928-2010

55. Reasoning About Time: From Constraint Satisfaction… Key Idea: Visibility of 3D points constrains image ordering Variables? Domains? Constraints? Disadvantage: Only relative image ordering

56. Visibility Reasoning Goal: reorder images (columns) to have as few violations as possible observed missing occluded Columns: images Rows: points Violates constraints: Satisfies constraints:

57. Temporal Ordering via CSP Goal: reorder images (columns) to have as few violations as possible Backtracking search: intractable (n! solutions for n images) Local search: start with random ordering of columns, swap columns or groups of columns to reduce the number of conflicts

58. Results...

59. Temporal Ordering via CSP Goal: reorder images (columns) to have as few violations as possible Local search: start with random ordering of columns, swap columns or groups of columns to reduce the number of conflicts Can also reorder the rows to group together points that appear and disappear at the same time – that gives you buildings

60. Results: Time-varying 3d model

61. ... Results

62. Conclusions Authors demonstrated that constraint satisfaction problems provide a powerful framework in which to solve temporal ordering problems Presented the first known method for solving this ordering problem. Remaining challenges? – Dealing with noise in classification of points as observed/missing/occluded.