1. An Introduction to Real-time Machine Vision in Mechatronics
Dr. Onur TOKER

2. Outline RT Machine Vision ? Mechatronics ?
Review of previous experiments
Image sensors (CMOS versus CCD)
CMUCam, and cwCAM
Interfacing a CCD camera to an 8-bit uC
Difficulties in real-time machine vision
Conclusion
Dr. Onur TOKER

3. RT Machine Vision ? Mechatronics ?
Machine vision is the ability of a computation machine to "see."
Visual object tracking
Object recognition
Automated inspection, sorting
Pattern recognition, etc.
RT: There is no strict real-time system. There are systems with very short event response latency times.
Dr. Onur TOKER

4. Experiment #1 1-D tracking system Analog video camera & PCI grabber
VB 6 & VFW based
Simple algorithm
PID control
Pentium 2/350MHz
Dr. Onur TOKER

5. Experiment #2 Line following Wireless video camera and ToyCar
Processing on a remote PC
VC++ & DirectX based
Simple algorithm
Pentium 3/1GHz
Dr. Onur TOKER

6. Experiment #3 Intel 8051 Very primitive machine vision
Rapid prototyping board
LDR sensor
MOSFET driver
Dr. Onur TOKER

7. Experiment #3
Dr. Onur TOKER

8. Prototyping / Final product
Final design EPROM based minimum size PCB
Prototyping board Serial download, EEPROM based, 9V battery
Dr. Onur TOKER

9. Phototransistor based sensors
A Student Project
Line following robot
Phototransistor based sensors
Dr. Onur TOKER

10. BOE-BOT kit Simple kit PBASIC Not very flexible Very small RAM
IR LEDs &
photo transistors
Dr. Onur TOKER

11. BOE-BOT demo
Dr. Onur TOKER

12. (Tracking by stereo machine vision)
Other demos
CMUCam demo
(Color tracking)
WAM demo (MIT 1995)
(Tracking by stereo machine vision)
Dr. Onur TOKER

13. Image sensor types Charged coupled devices (CCD)
Charge injection devices (CID)
CMOS Active Pixel Sensors (CMOS)
They all convert incident light (photons) into electronic charge (electrons) by a photo-conversion process.
Color sensors can be made by coating each individual pixel with a filter color (e.g. red, green, and blue).
Beyond that point, everything is different.
Dr. Onur TOKER

14. CUMCam uses such a sensor 2nd PCB has a Scenix uC
CMOS image sensors
Digital output
Easy to interface
A CMOS sensor (OV7620)
CUMCam uses such a sensor nd PCB has a Scenix uC
DALSA CMOS Sensor
Dr. Onur TOKER

15. CCD image sensors Analog output Difficult to interface
Require several support chips
DALSA CCD Sensor
Dr. Onur TOKER

16. CMOS versus CCD Under same lightning, same distance,
comparable budget,
CCD image is better.
CMOS sensor
640x480 mode
CCD sensor
640x480 (NTSC output)
Dr. Onur TOKER

17. CMUCam architecture CMUCam SX28 CMOS sensor uC uC/DSP
Serial I/O
“User device” issues high level commands
SX28 does the processing (Limited built-in functions)
SX28 replies
Dr. Onur TOKER

18. What is wrong with CMUCam ?
Serial I/O (Low bandwidth)
Low frame rate (Max. 17fps)
CMOS sensor
Processing done by SX28
Limited to built in functions
Not much flexibility
Instead of FPGA, uses SX28
Very compact design
Dr. Onur TOKER

19. Proposed cwCam architecture
uC/DSP
CCD camera
FPGA
uC/DSP
Video ADC
uC/DSP
Co-operating windowing approach (Discussed later)
Parallel processors
Parallel application specific digital architectures in the FPGA
ASIC CPU cores in FPGA
Dr. Onur TOKER

20. Machine Vision with an Analog Industrial camera
NTSC/30fps or PAL/25fps
Even/odd field interlacing: 60fips/50fips rate
31ms VSYNC, 4.7us HSYNC for NTSC
Needs a high speed ADC (AD9048 is 35 MHz)
Most 8-bit uCs are too slow for this task
Scenix SX28AC/DP 13.3 ns instruction cycle
FPGA for accurate and high resolution capture
Dr. Onur TOKER

21. Digitized video signal
One field
One frame
VSYNC
Dr. Onur TOKER

22. A single field
VSYNC
Several HSYNCs
Dr. Onur TOKER

23. Conclusion:Use 10MHz ADC
Video ADC speed ?
HSYNC ???
VSYNC
Conclusion:Use 10MHz ADC
Dr. Onur TOKER

24. Scenix SX28AC/DP 13.3 ns instruction cycle (75MHz clock)
10MHz video sampling = 100 ns loop time
1 Branch=3 cycles
4 instruction loop OK, but int. RAM too small
8051 too slow !
PIC16F877 too slow !
USE AN FPGA !
Dr. Onur TOKER

25. Our FPGAs (Prototyping boards)
Spartan II FPGA 50 Kgate
8MB RAM
8051
Dr. Onur TOKER

26. Actual photo of AD9048 used in our video digitizer
Our ADC (AD9048)
Actual photo of AD9048 used in our video digitizer
35MSPS, 8-bit Flash ADC, Bipolar, 550mW, DIP 28 available
AD9203, 40MSPS, 10-bit,CMOS, 74mW, No DIP available
Dr. Onur TOKER

27. Cortex-I approach Bederson, 1992
Logarithmic structured space variant pixel geometry
Based on human vision system
For real-time machine vision, reduce data to < 1500 pixels
Dr. Onur TOKER

28. Co-operating windowing (1)
Nassif & Capson, 1997
2 Watch windows (200x20)
1 Peripheral window (40x40 … 200x200)
1 Foveal window (20x20)
Object tracking at 113Hz
Dr. Onur TOKER

29. Co-operating windowing (2)
Dr. Onur TOKER

30. Where we are at cwCAM ?
AD9048 Video ADC board design completed (PCB layout !)
AD9048 interfaced to 8051 prototyping board and tested
Logic design is being done by Xilinx ISE software
Mixed VHDL and graphical logic designs
Tedious and long task
cwCam
CCD camera
FPGA
Video ADC
Dr. Onur TOKER

31. PUMA robot arm and dual camera set
Human Vision ?
PUMA robot arm and dual camera set
HMD and Dual monitor support
Dr. Onur TOKER

32. Conclusion
Real time machine vision requires innovative use of software and hardware techniques.
Cortex-I (Human Eye), Co-operating windowing, etc.
Innovative use of FPGAs and uC/DSPs.
High frame rate CCD sensors.
Optimum designs likely to be an application specific one.
cwCAM is based on co-operating windowing approach and innovative hardware/software techniques.
Dr. Onur TOKER

33. QUESTIONS ?
Dr. Onur TOKER