1. Real-time Image Processing System
Peter Chun
Dr. Lev Kirischian
Dr. Vadim Geurkov

2. Overview Image Processor & Data Acquisition device
(CMOS digital camera)
Image Processor &
hardware controllers
(FPGA)
VGA interface
(DAC)
Communication Interface
(USB, FireWire, Parallel Port)
Memory
(SRAM)
16/09/2018
Peter Chun

3. Schematic
16/09/2018
Peter Chun

4. Topics on the table Controllers on FPGA
Digital Camera (OV5017 and M4088)
SRAM (IDT71V416: 256K X 16-bit)
VGA (DAC board – THS8134)
16/09/2018
Peter Chun

5. Topics on the table (continue…)
Real-time Image Processing system
Step1 : VGA verification
Step2 : SRAM WRITE/READ verification using verified VGA
Step3: Digital camera data acquisition on verified VGA using verified SRAM
Static one-frame
Continuous video capture
Video capture with pixel update
Step by Step Guide : How to construct one
16/09/2018
Peter Chun

6. VGA Hardware Interface DB15 Number of pins you need: Pin Descriptions6
Pin Descriptions
Hsync: Horizontal Sync
Vsync: Vertical Sync
R: red
G: green
B: blue
GND: ground
16/09/2018
Peter Chun

7. Hardware Interface: DB152 3 4 5 6 7 8 9 10 11 12 13 14 15
16/09/2018
Peter Chun

8. VGA (continue…) VGA arrangement 16/09/2018 Peter Chun (0, 0)
(0,0)
(0, 0)
Horizontal
Synchronization
Vertical
Synchronization
(640,480)
(640, 480)
16/09/2018
Peter Chun

9. Timing Requirements Horizontal 16/09/2018 Peter Chun Red, Green, Blue
Horizontal Sync (Hsync) B C D E A
PARAMERTES A B C D E
TIME
31.77us
3.77us
1.89us
25.17us
0.94us
16/09/2018
Peter Chun

10. Timing Requirements Vertical 16/09/2018 Peter Chun R Red, Green, Blue
Horizontal Sync (Hsync) Q S P O
PARAMERTES O P Q R S
TIME
16.67ms
64us
1.02ms
15.25ms
0.35ms
16/09/2018
Peter Chun

11. VGA (continue…)
Example
20MHz
16/09/2018
Peter Chun

12. SRAM Signals Control signals Data signals
we, oe, cs, address(length..0), bhe, ble
Data signals
Data(depth..0)
NOTE:
“depth” is how many bits can you access at one time.
“length” is how many of data can you store, which is equivalent to 2length.
16/09/2018
Peter Chun

13. SRAM (continue…)
read
16/09/2018
Peter Chun

14. SRAM (continue…)
write
16/09/2018
Peter Chun

15. SRAM (continue…)
Example
40MHz
16/09/2018
Peter Chun

16. Digital Camera Hardware Overview 16/09/2018 Peter Chun
Optical Sensor Array
Row
Column
Amplifier for
Gain Control
A/D converter
hsync
vsync
pclk
data[max..0]
Digital Pixel Process
Analog pixel region
Digital pixel region
16/09/2018
Peter Chun

17. Digital Camera (continue…)
Reference Signals
Hsync
indicates the duration of active horizontal pixel insertion. It remains high during valid line and reset while not valid.
Vsync
represents the duration of active frame. When valid, it remains low and asserts high briefly to indicate the start of new frame.
Pclk
establishes the availability of the pixels. Either at the rising or falling edge of pclk, the pixel data is ready to be outputted.
16/09/2018
Peter Chun

18. Digital Camera (continue…)
SetUp
Exposure control, AGC (auto gain control), Gamma correction, Frame rate, resolution control
Method
Memory Access
I2C protocol
16/09/2018
Peter Chun

19. Digital Camera (continue…)
Memory Access 1
initial 2
ready 3
done 1
initial 2
ready 3
done
=100ns
=100ns
=50ns
=50ns
=50ns
=20ns
16/09/2018
Peter Chun

20. SetUp Example 1 1 2 2 3 3 initial initial ready ready done done
case y is
-- Initial state after resets
when initial =>
if(loop_num = num_mul) then
loop_num <= 0;
y<=HWCTL_initial;
else
loop_num <= loop_num + 1;
end if;
-- Horizontal Window size control
when HWCTL_initial =>
y<=HWCTL_ready;
when HWCTL_ready =>
y<=HWCTL_done;
when HWCTL_done =>
y<=VWCTL_initial;
process(y)
begin
-- initial and reset values
oeb <= '0';
csb <= '1';
web <= '1';
a <= "0000";
cam_data <= " ";
if y = initial then
elsif y = HWCTL_initial then
a <= "0110";
oeb <= '1';
csb <= '0';
web <= '0';
elsif y = HWCTL_ready then
elsif y = HWCTL_done then 1
initial 1
initial 2
ready 2
ready 3
done 3
done
16/09/2018
Peter Chun

21. Digital Design Tips Multiple processes
For reference signals (pclk, vsync, href)
Filtering unstable states of the signals
16/09/2018
Peter Chun

22. Digital Design Tips (continues…)
Address space division
Address setup time exceeds synchronous clock events
Long integer or std logic vector need attention
Flip image
Use decrement of address space
16/09/2018
Peter Chun

23. Digital Camera (continue…)
Pixel Timing
16/09/2018
Peter Chun

24. Digital Camera (continue…)
Horizontal Timing
16/09/2018
Peter Chun

25. Digital Camera (continue…)
Vertical Timing
16/09/2018
Peter Chun

26. Design evolution Frame based Pixel based One memory Internal memory
16/09/2018
Peter Chun

27. Examples
40MHz
66MHz
16/09/2018
Peter Chun

28. Edge Detection
Gradient based
16/09/2018
Peter Chun

29. Edge Detection (continue…)
An edge pixel is described using two important features
Edge strength, which is equal to the magnitude of the gradient
Edge direction, which is equal to the angle of the gradient.
16/09/2018
Peter Chun

30. Edge Detection (continue…)
Robert’s cross Operator
385 pixels a b c d
Max(|d-a|, |b-c|)
,where
288 pixels
16/09/2018
Peter Chun

31. Robert’s Cross Operation
385 pixels a1 b1 c1 d1 e1 f1
…………………………… x1 y1 z1 1 -1 a2 b2 c2 d2 e2 f2
…………………………… x2 y2 z2
288 pixels
……………………………………………………………………………………… or 1 -1 A B C D E F
…………………………… X Y Z
16/09/2018
Peter Chun

32. Example Real-time image processing system with edge detection
16/09/2018
Peter Chun