1 Computational Vision CSCI 363, Fall 2012 Lecture 18 Stereopsis III

2 Summary of Marr-Poggio Horizontal lines: constant Right eye position. Vertical lines: Constant left eye position. Diagonal lines: Constant disparity.

3 Limitations of the Marr-Poggio Algorithm Problems with Marr-Poggio algorithm: Does not specify what features are being matched. Does not make use of zero crossings or multiple spatial scales. Does not make use of vergence eye movements. Importance of Vergence Eye Movements: Humans can only fuse images with disparities of about +/- 10 min of arc. This range is called Panum's fusional area. If we need to fuse the images of an object that is nearer or farther than this, we make vergence eye movements to change the plane of fixation. Humans make many vergence eye movements as we scan the environment.

4 Marr-Poggio-Grimson Basic idea: Use zero crossings after filtering with Large leads to widely spread zero crossings that are easier to match but may give less accurate estimates of disparity. Small leads to more finely tuned disparities. Start with large filters, and find matches for the zero crossings. We can use the estimates of disparity with the large filters to guide vergence eye movements to align features for comparing finer disparities with the smaller filters.

5 Zero crossings for different filter sizes

6 Range of disparities to search Suppose we filter an image with a operator for which the width of the inner portion is w. Given a zero crossing at some position in the filtered image, the probability of another zero crossing within +/- w/2 of the first one is 5%. Therefore, if there is a zero crossing at a given position in the left image and a zero crossing in the right image within the range of +/- w/2 of the one in the left image, the probability that these two are a correct match is 95%. +w/2 -w/2 L R d

7 Determining correctness If the correct disparity is in the range of +/- w/2, then we will find matches for nearly all the zero crossings in the left image (95%). If the correct disparity is not in the range, then the probability of a match is 40%. Therefore, if we are only finding matches for 40% of the zero crossings, we are not looking at the correct range of disparities. We can correct this by making vergence eye movements.

8 Increasing the range of disparities examined The actual algorithm uses a range of +/- w. This leads to more false matches. For a range of +/- w, the likelihood of a false match is at most 50%. So 50% of the time our matches will be correct. If we have a false match, we usually have one crossed disparity and one uncrossed disparity matches. We can use neighboring points to disambiguate the false matches.

9 Disambiguating false matches To determine the correct matches, we match both from the left image to the right and from the right image to the left. L1L1 L2L2 R1R1 R2R2 L 2 can only match R 2 R 1 can only match L 1 These determine the matches uniquely. It can be shown that if more than 70% of the zero crossings have a match within +/- w, then the disparity range is correct. If not, use a vergence eye movement and try again.

10 The full algorithm A.To determine initial correspondence: 1. Find zero crossings using a operator with a central width, w. 2. For each horizontal slice: 2.1 Find the nearest neighbors in the right image for each zero- crossing in the left image. 2.2 Find the nearest neighbors in the left image for each zero- crossing in the right image. 2.3 For each pair of zero-crossings that are closest neighbors of one another, find the distance,  initial, of the right fragment from the left. If  initial is less than the matching tolerance, m, consider the zero crossings matched with disparity  initial.

11 Finishing the algorithm B. To determine final correspondence: 1.Find the zero-crossings using a operator of width w/2. 2.For each horizontal slice: 2.1 For each zero crossing in the left image: Find the nearest zero crossing in the left image that matched when the operator was width w Offset the zero crossing by a distance of  initial, the disparity of the nearest matching zero crossing found at the lower resolution with operator width w. 2.2 Find the nearest neighbors in the right image for the zero crossing in the left image 2.3 Find the nearest neighbors in the left image for the zero crossing in the right image. 2.4 For each pair of zero crossings that are closest neighbors to one another, find the distance,  new, that the right is separated from the left. If  new is less than the matching tolerance, m/2, then consider the zero crossings matched with a total disparity of  final =  new +  initial.

12 Do Humans Use the Same Constraints as Marr-Poggio? 1.Similarity: We probably use this one. Humans cannot fuse a white dot with a black dot. 2.Epipolar: The brain probably uses some version of this. If one image is shifted upward (or downward), people cannot fuse the two images. 3.Uniqueness: We probably don't rely on this. There are examples of images where can fuse two features in one image with a single feature in the other.

13 Violations of Uniqueness constraint Panum's limiting case: Matching one line with two Braddick's demonstration: Matching one point with two repeatedly Stereo algorithms can deal with Braddick's demonstration with slight modifications.

14 The Continuity Constraint The brain probably uses some form of continuity constraint. Evidence: There is a limit to how quickly disparity can change from one location to the next, and still produce stereo fusion. For a plane that is steeply slanted in depth, people lose the ability to see the slant and just see the step edge.

15 Does the Brain use Zero Crossings? Many machine vision algorithms extract edges first (e.g. with zero crossings) and then compute the disparities of matched edges. They use edges for matching because they correspond to important physical features (e.g. object boundaries). We also know that people can localize the positions of edges very accurately. This accurate localization is required for stereo vision. However, it is not clear what primitives the human brain matches when computing stereo disparity. The information is combined in V1, so it is after the center- surround operators (like the laplacian operators). Zero crossing information would be available.

16 Zero crossings are not enough Stereo Pairs Luminance Profile Convolved images Left Right

17 Perception vs. Computation Perceived Depth Computed depth of zero crossings Positions of peaks and troughs Computed depth using peaks and troughs

18 Some V1 cells are tuned to disparity Tuned Excitatory Tuned Inhibitory Some cells are narrowly tuned for disparity. Most prefer a disparity near zero.

19 Near and Far cells Some cells are broadly tuned for disparity, preferring either near objects or far objects.