Perception and VR MONT 104S, Fall 2008 Lecture 8 Seeing Depth

How can we see in depth? Problem: When light is focused on the retina to form an image, we go from 3 dimensions in the world to 2 dimensions on the image. We have lost the depth dimension. Question: How, then, can we see depth? Answers: Monocular cues to depth: from texture, motion, shading, etc. Binocular stereopsis: This uses the slightly different images on the two retinas to compute relative depth.

Monocular cues to Depth When we view a scene with one eye, we still interpret the 3D scene fairly accurately. The brain uses cues to the distance to estimate how far away things are. Some of these are listed here: Height in the visual field Size of familiar objects (and relative object size) Texture gradients Linear perspective Shading and shadows Relative Motion (motion parallax)

Height in the Visual Field In normal images, things that are farther away often have images that are higher in the visual field. The visual system uses this height in the visual field as a cue to distance. We then adjust our interpretation of the size of the object based on this distance. (Note: familiar object size also gives a cue to depth here).

Relative object size Recall that with perspective projection, the image size of a nearby object is larger than for a distant object. Therefore, 2 similar objects of different sizes may be perceived at different distances. Demo: http://psych.hanover.edu/krantz/art/rel_size.html This cue works better when the objects are familiar and have known sizes (see previous slide).

Texture Gradients When we view a textured surface, the nearby texture is larger and coarser and the more distant texture is more fine grained. The texture forms a gradient from course to fine. This is a consequence of perspective projection. Demo: http://psych.hanover.edu/krantz/art/texture.html

Linear Perspective Because the size of images decreases with distance, the gap between 2 parallel lines will become smaller as the lines recede into the distance. This is known as linear perspective. Our visual system uses this convergence of lines as a cue to distance. Demo: http://psych.hanover.edu/krantz/art/linear.html

Shape from Shading Shading in an image can give cues to the 3D shape of an object. The brain appears to assume the light source is above the scene. In these images, when the shadow is on the bottom, we see bumps. When it is on the top, we see indentations.

Motion Parallax As we learned in the motion lecture, for a moving observer, the images of nearby objects move faster than the images of distant objects. The difference in image speed, known as motion parallax, can signal depth differences. Demo: http://psych.hanover.edu/krantz/MotionParallax/MotionParallax.html Structure from motion: Relative motion in the image can also be used to determine the 3D shape of objects. Demo: http://www.viperlib.org/ (General depth images, page 7, rotating cylinder).

Moving Shadows Combining motion and shadows can lead to some interesting perceptions of motion in depth. Demo: http://www.viperlib.org/ General Depth Images, page 1. The only difference in these images is the position of the shadow.

Binocular Stereo The fact that our two eyes see the world from different positions allows the brain to use differences in the image to compute depth. This is known as stereopsis or stereo vision. A 3D Mars Rover When viewed with a red filter over the left eye and a cyan filter over the right eye, this image shows a 3 dimensional view of Mars.

Binocular Stereo The image in each of our two eyes is slightly different. Images in the plane of fixation fall on corresponding locations on the retina. Images in front of the plane of fixation are shifted outward on each retina. They have crossed disparity. Images behind the plane of fixation are shifted inward on the retina. They have uncrossed disparity.

Crossed and uncrossed disparity Disparity is the amount that the image locations of a given feature differ in the two eyes. 1 uncrossed (negative) disparity plane of fixation 2 crossed (positive) disparity

Stereo processing To determine depth from stereo disparity: Extract the "features" from the left and right images For each feature in the left image, find the corresponding feature in the right image. Measure the disparity between the two images of the feature. Use the disparity to compute the 3D location of the feature.

Exam Review Topics There is a list of exam review topics at: http://mathcs.holycross.edu/~croyden/mont104S/notes/exam1_review.html Expect short answer questions on any of the topics listed. Examples: Why is vision considered an "ill-posed" problem? List 2 assumptions made by the visual system to deal with this problem. What is the difference between rods and cones?

More Sample Questions 3. Describe the Craik-Obrien-Cornsweet illusion. What does it tell us about how the brain processes changes in contrast? 4. Diagram the neural circuit for direction selectivity. Briefly explain how it works. 5. Why is it that we can match any perceived color by a combination of red, blue and green lights? 6. List 3 monocular cues to depth. Briefly explain each one.

Problem 1. Compute the 2D image positions for the following 3D points. Assume the image plane is 2 cm behind the pinhole (center of projection). All units are given in cm. Point 1: (5, 10, 50) Point 2: (16, 24, 8) Point 3: (25, 45, 50)

Problem 2. The receptive field shown is partially in light and partially in dark. What is the change in firing rate of the neuron with this receptive field for this light stimulus? Assume that the partially covered squares are 1/2 covered.