1. Robot Vision

2. Methods for Digital Image Processing

3. Every picture tells a story
Goal of computer vision is to write computer programs that can interpret images

4. Image Image : a two-dimensional array of pixels
The indices [i, j] of pixels : integer values that specify the rows and columns in pixel values

5. Gray level image vs. binary image

6. Can do amazing things like:
Human Vision
Can do amazing things like:
Recognize people and objects
Navigate through obstacles
Understand mood in the scene
Imagine stories
But still is not perfect:
Suffers from Illusions
Ignores many details
Ambiguous description of the world
Doesn’t care about accuracy of world

7. Computer Vision
What we see
What a computer sees

8. Computer Vision
What we see
What a computer sees

9. Components of a Computer Vision System
Camera
Lighting
Scene Interpretation
Computer
Scene

10. Microsoft Kinect
IR LED Emitter
IR Camera
RGB Camera

11. Face detection

12. Face detection Many digital cameras detect faces Canon, Sony, Fuji, …
Why would this be useful? Main reason is focus. Also enables “smart” cropping.
Many digital cameras detect faces
Canon, Sony, Fuji, …

13. Smile detection?
Sony Cyber-shot® T70 Digital Still Camera

14. Face Recognition
Principle Components Analysis (PCA)

15. Vision-based biometrics
“How the Afghan Girl was Identified by Her Iris Patterns” Read the story
wikipedia

16. Robots
Today’s robots perform complex tasks with amazing precision and speed
Why then have they not moved from the structure of the factory floor into the “real” world? What is the limiting factor?

17. Definition of Robot vision
Robot vision may be defined as the process of extracting, characterizing, and interpreting information from images of a three dimensional world

18. Common reasons for failure of vision systems
Small changes in the environment can result in significant variations in image data
Changes in contrast
Unexpected occlusion of features

19. What Skills Do Robots Need?
Identification: What/who is that?
Object detection, recognition
Movement: How do I move safely?
Obstacle avoidance, homing
Manipulation: How do I change that?
Interacting with objects/environment
Navigation: Where am I?
Mapping, localization

20. Visual Skills: Identification
Recognizing face/body/structure: Who/what do I see?
Use shape, color, pattern, other static attributes to distinguish from background, other hypotheses
Gesture/activity: What is it doing?
From low-level motion detection & tracking to categorizing high-level temporal patterns
Feedback between static and dynamic

21. Visual Skills: Movement
Steering, foot placement or landing spot for entire vehicle
MAKRO sewer
shape pattern
Demeter region
boundary detection

22. Visual Skills: Manipulation
Moving other things
Grasping: Door opener (KTH)
Pushing, digging, cranes
KTH robot &
typical handle
Clodbusters push a box
cooperatively

23. Visual Skills: Navigation
Building a map
Localization/place recognition
Where are you in the map?
Laser-based wall map (CMU)
Minerva’s ceiling map

28. Binary Image Creation
Popularly used in industrial robotics

29. Bit per Pixel

30. Color models Color models for images, RGB, CMY Color models for video,
YIQ, YUV (YCbCr)
Relationship between color models :

31. Simplified diagram of camera to CPU interface

32. Interfacing Digital Cameras to CPU
Digital camera sensors are very complex units.
In many respects they are themselves similar to an embedded controller chip.
Some sensors buffer camera data and allow slow reading via handshake (ideal for slow microprocessors)
Most sensors send full image as a stream after start signal
(CPU must be fast enough to read or use hardware buffer)

33. Idea • Use FIFO as image data buffer
• FIFO is similar to dual-ported RAM, it is required since there is no synchronization between camera and CPU
• Interrupt service routine then reads FIFO until empty

34. Vision Sensors
Single Perspective Camera

35. Vision Sensors
Multiple Perspective Cameras (e.g. Stereo Camera Pair)

36. Vision Sensors
Multiple Perspective Cameras (e.g. Stereo Camera Pair)

37. There are several good approaches to detect objects:
Model-based vision.
1) We can have stored models of line- drawings of objects (from many possible angles, and at many different possible scales!), and then compare those with all possible combinations of edges in the image.
Notice that this is a very computationally intensive and expensive process.

38. Motion vision. 2) We can take advantage of motion.
If we look at an image at two consecutive time-steps, and we move the camera in between, each continuous solid objects (which obeys physical laws) will move as one.
This gives us a hint for finding objects, by subtracting two images from each other.
But notice that this also depends on knowing well:
how we moved the camera relative to the scene (direction, distance),
and that nothing was moving in the scene at the time.

39. Binocular stereopsis
3) We can use stereo (i.e., binocular stereopsis, two eyes/cameras/points of view).
Just like with motion vision above, but without having to actually move,
we get two images,
we subtract them from each other,
if we know what the disparity between them should be, (i.e., if we know how the two cameras are organized/positioned relative to each other), we can find the information like in motion vision.

40. Clever Special Tricks that work:
to do object recognition, it is possible to simplify the vision problem in various ways:
1) Use color; look for specifically and uniquely colored objects, and recognize them that way (such as stop signs, for example)
2) Use a small image plane; instead of a full x 512 pixel array, we can reduce our view to much less.
Of course there is much less information in the image, but if we are clever, and know what to expect, we can process what we see quickly and usefully.

41. Smart Tricks continued:
3) Use other, simpler and faster, sensors, and combine those with vision.
IR cameras isolate people by body-temperature.
Grippers allow us to touch and move objects, after which we can be sure they exist.
4) Use information about the environment;
if you know you will be driving on the road which has white lines, look specifically for those lines at the right places in the image.
This is how first and still fastest road and highway robotic driving is done.