Presentation is loading. Please wait.

Presentation is loading. Please wait.

Efficient Selection Of Disambiguating Actions for Stereo Vision Ronald Parr Duke University Joint work with Monika Schaeffer (Duke University)

Similar presentations


Presentation on theme: "Efficient Selection Of Disambiguating Actions for Stereo Vision Ronald Parr Duke University Joint work with Monika Schaeffer (Duke University)"— Presentation transcript:

1 Efficient Selection Of Disambiguating Actions for Stereo Vision Ronald Parr Duke University Joint work with Monika Schaeffer (Duke University)

2 Traditional Stereo These points match really well A benchmark stereo pair (Middlebury) Lots of texture Small disparity range How realistic is this?

3 What Robots See Huge disparity range Large areas with little or no texture Let’s go down this hallway without hitting a wall! ????? LSRC hallway

4 Why not Use a Laser Range Finder? ● Weight ● Cost $$ ● 2D or $$$ ● (lack of) stealth ● Power consumption ● Low data bandwidth ● Calibrated moving parts ● Sensor can drive robot design

5 Alternatives ● Sonar  ● 3D laser – Slow – $100K ● Rotated/Rotating 2D laser – Retains nearly all disadvantages of 2D laser – Information per sweep: ~100 Kbits

6 Motivation Accurate Cheap Light Stealthy Expensive Bulky Impractical Calibrated mechanics Fails where Robots need accuracy Laser range findersTraditional Stereo Benefits Problems

7 Motivation Accurate Cheap Light Stealthy Expensive Bulky Impractical Calibrated mechanics Fails where Robots need accuracy Active Stereo Vision (stereo + laser pointer) Benefits Problems

8 Active Stereo Vision ● Take base stereo pair of images ● Take stereo pair(s) with addition of laser line (only crude calibration needed) ● Image subtraction isolates laser line ● Line disambiguates pixel matchings between pair The laser line divides problem into two independent stereo problems.

9 Using the laser A real stereo pair of images An artificial stereo pair of images Test set: Lain m x n x d: 600 x 900 x 160

10 Using the laser Shine laser Calculate laser lines using ground truth image Line in right image = line in ground Brighter in ground = farther right in left image

11 Using the laser Extract laser lines and update matchings Update matchings These points match These segments match These points match These segments match

12 Our Prototype

13 But what about… ● Bulk, size & cost? – Prototype is much larger than necessary – P/T head need not be high quality (calibration not needed) ● Stealth & speed? – Only use laser when/where necessary – Plan laser aims to reduce entropy – This is our sensor planning problem

14 How is Stereo Different? ● Extremely large event space – Millions of pixels/image – Hundreds of values for each pixels ● Cost of inference is high ● (naïve) one step lookahead is impossible ● Our main result: Can determine aim point for the laser that maximizes expected entropy reduction (information gain) in same asymptotic complexity as one run of stereo

15 Doing Stereo ● Bobick & Intille present stereo as a shortest path problem ● Construct the Disparity Space Image (DSI) ● Find the shortest path in linear time using dynamic programming ● Path through DSI = Stereo Matching for a scanline ● Costs: ● Assume n pixels/scanline (thousands) ● Max disparity level of d (hundreds) ● O(nd) per scanline

16 Constructing the DSI The DSI takes on three equivalent forms: – A dxn image containing information about the quality of matchings for a scanline – An dxnx3 graphical structure where paths through the graph represent valid pixel matchings for the scanline – An n-state HMM with O(d) possible values per state.

17 Constructing the DSI As an image: – A pixel in the right scanline is a column in disparity space. – A pixel in the left scanline is a diagonal in disparity space. – The left and right values are run through a cost function to get the matching score.

18 Constructing the DSI As an image: – A pixel in the right scanline is a column in disparity space. – A pixel in the left scanline is a diagonal in disparity space. – The left and right values are run through a cost function to get the matching score. – Not shown: Occlusion penalties

19 M L R M L R M L R M L R M L R M L R M L R M L R M L R M L R M L R M L R M L R M L R M L R M L R M L R M L R M L R M L R x = ix = i+1 cost: D L cost: s(i+1,j) d = j-1 d = j d = j+1 cost: D R Constructing the DSI As a graph: – Each pixel in DSI image corresponds to three nodes representing the state of that pixel. – Transitions from pixel (i,j) ● M, R, L to M of (i+1,j) ● M, L to L of (i, j-1) ● M, R to R of (i+1, j+1)

20 M L R M L R M L R M L R M L R M L R M L R M L R M L R M L R M L R M L R M L R M L R M L R M L R M L R M L R M L R M L R SiSi S i+1 Constructing the DSI As an HMM: – Ms and Rs within a column i are mutually exclusive, jointly exhaustive. Considered possible values to state S i – Ls in a column encode a more complicated set of transitions from Ms in the column to Ms in the next column

21 Finding the Shortest Path ● DSI is a highly structured DAG – We define the set of predecessor nodes, Γ - – Graph traversed from bottom to top, left to right. ● Shortest path can be found in linear time with dynamic programming. For node c,

22 Query Selection ● To maximize expected benefits of laser aims, we need a distribution over outcomes ● Arc costs considered unnormalized log probabilities ● Forward/backward algorithm to calculate node probabilities. For node c: Calculated backwards. Γ + is the successor set.

23 Query Selection ● Stereo matching = Path through DSI ● Path entropy through DSI  measure of our confusion over the best path ● Query strategy: Maximize expected reduction in entropy

24 Query Selection Use this observation from Anderson & Moore : For entropy H(x), path space , and queries Q t : IG(Q t ) = H(  ) - H(  |Q t ) symmetry of mutual information: IG(Q t ) = H(Q t ) - H(Q t |  ) Markov property: IG(Q t ) = H(Q t ) - H(Q t |S t ) IG(Q t ) = H(S t ) - H(S t |Q t ) Expected entropy after query Q t Linear time!

25 Updating the DSI ● If the laser is detected in both images, we split the DSI into two independent sections. ● Paths are funnelled through M node. left scanline right scanline DSI There are no valid paths through these dead zones that match our observation.

26 Updating the DSI ● If the laser is detected in both images, we split the DSI into two independent sections. ● Many subtle details (ask later…) Each side is now independent of the other.

27 Real World Implementation

28 ● Took roughly one base and 200 lasered 1000x650 px images ● Used all 200 images to establish “ground truth” ● Recalled nearest laser aim to query to simulate real time aiming Original Right ImageOur Ground Truth Doorknob Hinge Copier

29 Disparity map with no lasers entropy Entropy

30 After two laser aims Entropy

31 After nine laser aims Entropy

32 Results: Path Entropy

33 Results: Pixel Error

34 Results on existing images We also ran the algorithm on two sets of existing images, the Middlebury Benchmark set “cones”, and some artificially generated airport security camera style images with little texture. We used ground truth to generate fake laser lines. cones security cam

35 Results on Security Camera

36 Results on Cones

37 Conclusion ● Computational properties – O(nd) complexity – No asymptotic penalty for planning laser actions ● Practical benefits of hybrid system – Small – Inexpensive – Selective use of laser – Accuracy increases with laser use

38 Conclusion ● Results – Shown to work on both fake and real world images – Far more accurate than stereo alone – Better than random or equally spaced aims Questions? Thanks to: Carlo Tomasi, NSF, SAIC, IAI, Sloan Foundation.

39 Updating the DSI ● If the laser is detected in both images, we split the DSI into two independent sections. ● Paths are funnelled through M node. Each side is now independent of the other.

40 Updating the DSI ● The ordering constraint is an assumption that keeps the stereo algorithm linear. ● It does not necessarily hold in the real world. ● The laser sometimes picks up on this. left scanline right scanline DSI

41 Updating the DSI ● The ordering constraint is an assumption that keeps the stereo algorithm linear. ● It does not necessarily hold in the real world. ● The laser sometimes picks up on this. left scanline right scanline DSI Detectable because violations occur in previously established dead zones.

42 Updating the DSI ● The ordering constraint is an assumption that keeps the stereo algorithm linear. ● It does not necessarily hold in the real world. ● The laser sometimes picks up on this. left scanline right scanline DSI Detectable because violations occur in previously established dead zones.

43 Updating the DSI ● Pixels in one image do not necessarily map one to one with pixels in the other image. ● The borders of dead zones must be left possible, though improbable left scanline right scanline DSI

44 Query Selection ● We could also calculate the expected path entropy reduction in linear time using dynamic programming... h(c) = p(c) + Σ(p(b)log(p(c))+h(b)) bє Γ - (c) Run forward to get the total path entropy, run in both directions to get path entropy though each node.


Download ppt "Efficient Selection Of Disambiguating Actions for Stereo Vision Ronald Parr Duke University Joint work with Monika Schaeffer (Duke University)"

Similar presentations


Ads by Google