1. 3. ISP Hardware Design & Verification
System Architecture for ISP Hardware
FPGA Based ISP Simulator
FPGA Based ISP Simulator GUI Componets
Design & Verification for ISP Functional Blocks

2. System Architecture for ISP Hardware
Total 12 functional blocks
- Sytem clock : pixel clock
Input : 8-bit bayer
Output : 16-bit 4:2:2 YCbCr
Functional block control through
I2C
Function selector generates
“enable” singal to each
fucnctional block of ISP
Operational frequency for blocks :
More than 100 MHz
Hardware architectue for ISP sytem

3. Synthesis Results of ISP Hareware Design
- Op. freq. : MHz
- Critical path : AWB
Item
Value
Total logic elements
10,338
Total registers
7,431
Total memory bits
316,896
Setup time of the ISP functional block measured in slow model.
Resource usage

4. FPGA Based ISP Simulator
Test Enveronment :
8M CIS PC
DE2-70 FPGA board
HyImage V4P board
HyImage interface board :
Communication bet. PC & FPGA
Operation speed :
In case of full resolution :
7~8 Frames per sec.
In case of down scaling :
10 Frame per sec.
FPGA Based Real-time ISP Simulator

5. FPGA Based ISP Simulator
PCB board for FPGA-PC conncetion
PCB Board Design for FPGA-PC Connection
gpio0, gpio1
FPGA board GPIO
Required of PCB design for FPGA board I/O with a fixed power pins(3.3V)
Connection information for PCB mode

6. FPGA Based ISP Simulator GUI① ② ③ ④ ⑤ ⑥
No.
Main
function
Details ①
Down scaling
- Full/down scaling selection ②
Color interpolation
5x5 adaptive interpolation
3x3 bilinear interpolation ③
Option
Selection for Gamma value
Setting weight for Edge enhancement ④
Auto focus
Selection for operation ⑤
AE_Option
- Setting for in-door/out-door ⑥
ISP Function
Activate each functional block
ISP Simulator GUI

7. FPGA Based ISP Simulator GUI
Gamma value setting window
Gamma value setting window - possible from 0.1 through 0.9
Edge enhancement weight value setting window
Edge enhancement setting window
- Value setting: 1 ~ 5
- Increase by 0.5

8. ISP Functional Block Design & Verification – Color Interpolation
Applied Algorithm
3ⅹ3 bilinear color interpolation
Process
Line buffer : 3 line data store
3x3 data generator : 3x3 data matrix generation
Bilinear operator : bilinear interpolation
⇒ Applying total 4-level pipeline

9. ISP Functional Block Design & Verification – Color Interpolation
Program Source Analysis
Hardware file configuration
inter_3_3.v
buffer_3_3_8bit.v : 3x3 buffer
dpsram3200x8_2.v : 3x3 buffer memory simulation file generation (Modelsim file)
buffer_3_3_8bit.v
dpsram3200x8_2.v
image_gen.v : 8bits bayer data generation fil
inter_tb.v : Test-bench top file
Simulation File Generation(Matlab file)
rgbtogbrg.m : bayer pattern file generation from image-file input
interpolation_3_3.m : Result image generation after performing 3x3 interpolation
image_out.m : Converting hardware-based 3x3 interpolation results to a corresponding image file

10. ISP Functional Block Design & Verification - Color interpolation
5x5 adaptive color interpolation
Hardware file composition
color_inter_new.v : color interpolation top file
buffer_5_5.v : 5 line buffer
dpsram3200x16.v : memory for 5-line buffer
pattern_gen.v : generation pattern value according to bayer pattern
param.h : Store 8 masks
Simulation file composition
4 higher level files
inter_tb.v : Top test-bench file
image_gen.v : bayer pattern input generation

11. ISP Functional Block Design & Verification -Color interpolation
Simulation file configuration
Matlab file
rgbtogbrg.m : generate bayer pattern file from image file input
interpolation_5_5.m : Result image generation after 5x5 interpolation
image_out.m : Converting hardware based 5x5 interpolation results to image file

12. ISP Functional Block Design & Verification – Edge enhancement
Applied algorithm
Unsharp mask filter
Performing process
Buffer 3x3
Store 3-line data
3x3 data generator
Generate 3x3 data matrix
Sum
3-line summation
Divider
Get average data
Add multiplication result of weight and subtracted value from original pixel data to the original pixel
Total 19-stage pipeline

13. ISP Functional Block Design & Verification - Auto White Balance
Gray World Algorithm
Hardware file composition
AWB.v : AWB top file
divider.vhd : division calculation
Simulation file composition
awb_lee3.v : AWB top file
awb_lee3_tb.v : Test bench top file
divider.vhd: 21bit / 11bit
dpsram3200x16.v : memory for 5-line buffer
im_64x64.txt : Text image file
RGB_image_gen.v : RGB pattern input generation
RGB_image_write.v

14. ISP Functional Block Design & Verification - Auto White Balance
Simulation file composition
Matlab file
* im2text.m : convert an image file to a text file
* text2im.m : convert a text file to an image file
* compare_result.m : result comparison

15. ISP Functional Block Design & Verification -Auto White Balance
GWA AWB 하드웨어 구조
Applying 1-stage pipeline
Consists of 5 sub-blocks
Signal generator : generate a frame completion signal
FIFO : Perform delay for Divider block
Accumulator : perform total summation R, G, B pixels
Shifter : perform average of R, G ,B
Divider : divide R, G, B average total by each R, G, B average
Source analysis
See reference files

16. ISP Functional Block Design & Verification – Edge enhancement
Program source analysis
Hardware file composition
Edge_enhance.v : edge_enhance.top file
divider.vhd : division calculation
dpsram3200x32.v : Line buffer memory
FIFO.v : FIFO process
Simulation file composition(Modelsim file)
buffer3_3.v : 3 line buffer
edge_en_tb.v : edge enhance top file
FIFO.v dpsram3200x32.v edge_enhance.v
divider.vhd YCbCr_image_gen.v : YCbCr image generation
Reg_Nbits.v YCbCr_image_write.v
Simulation file composition(Matlab file)
im2textYCbCr.m : convert into test file after converting image file to YCbCr
textYCbCr2im.m : convert text file to YCbCr image file
compare_result.m : Result comparison

17. ISP Functional Block Design & Verification – Auto Focus
Applied algorithm
Tenengrad (calculating focusing value)
Global search(Movement of focusing position)
Performing process
3 line Buffer : store 3-line Y data
3x3 data generator : generate 3x3 data matrix
Focus value calculating : calculate focus value per frame
Main operator : setting the moving range for actuator step
I2C Controller : activate actuator step movement
그림 틀

18. ISP Funcional Block Design & Verification – Auto Focus
Program source analysis
Hardware file composition
global_se.v : autofocus.top file
focus_val.v : calculate focus value
I2C_AF.v : Moving Motor
dpsram3200x16: Line buffer memory
af_mem.v : memory access
Simulation file composition(Modelsim file)
Imposible to derive program by Modelsim (register control)

19. ISP Functional Block Design & Verification기 – Gamma correction
Applied algorithm
Piece-wise linear Gamma-correction
Performing process
Top block for gamma correction

20. ISP Functional Block Design & Verification – Gamma correction
Performing process
Step decision
Gamma correction performing by various
Linear partitions, not by gamma curve
𝑦= 𝑦 𝑛−1 + 𝑦 𝑛 − 𝑦 𝑛−1 𝑥 𝑛 − 𝑥 𝑛−1 ×(𝑥− 𝑥 𝑛−1 )
y ′ =( x n − x n−1 ) y n−1 +( y n − y n−1 )( 𝑥 𝑛 − 𝑥 𝑛−1 )
𝑦= 𝑦 ′ 𝑥 𝑛− 𝑥 𝑛−1
Calculating 8-step output pixel using Gamma table
Piece-wise linear
gamma correction

21. 4. Embedded System Design & Verification
Software ISP Simulator
Software ISP Simulator GUI
Hardware ISP 설계 및 검증

22. Block diagram of an ISP Embedded System
SoCBase 1.0 Based Design
Process : ARM926EJS
FPGA : Virtex4LV80
Design block :
- ISP top
·I2C controller
·ISP functional block control
·ISP function blocks
- Dualport SRAM controller
- VGA controller
Block diagram of an ISP Embedded System

23. Design Details
System spec.
Design Tool

24. SoCBase 1.0
1. Single-processor based SoC development application platform
2. Provided with total 26 IPs
· AMBA bus
· Memory controller
· External unit controller
SoC base 1.0

25. Hardware Design & Verification
Dual port SRAM controller
dual port SRAM controller design for high-speed store/output of image data
Use dual port FIFO
x 32 bits
Generate SRAM control data from Write/read data controller
Required in case for output of high-resolution image
Dual port SRAM controller Stucture

26. Hardware Design & Verification
VGA controller
Ics_307
block
Vga data
Ctroller
Top_VGA
iCLK_25
Ics_sclk
ics_data
ics_strobe
oVGA_R[7:0]
oVGA_G[7:0]
oVGA_B[7:0]
oVGA_HS
oVGA_VS
oVGA_BLANK
oVGA_SYNK
oVGA_CLOCK
iRST
Ics_307 chip
Clk_108
Clock generation through control of ICS 307 chip input signal
Clock generation of 25MHz for 640x480 resolution output
VGA controller usage example : Refer to HBE-SoC-IPD Users Guide p.250
ICS 307 chip manual : HBE-SoC-IPD Users Guide p.265 참조

27. Dual port memory controller
SRAM-VGA Simulation
RGB
Text Input
Image generator
Dual port memory controller
VGA controller
SRAM model
Text Output
24bits RGB output
YCbCr
Color conversion
Source file location :
sram_vga_test folder
Input file : ycbcr_in.txt
Output file : image_out.txt
SRAM model generation
Confirm image through Matlab

28. SRAM-VGA Simulation Simulation result Down scaling
Original image(3200x2408)
Resulting image(800x600)

29. Pin Configuration

30. ISP embedded system verification environment
Simulator
Access slave address through Multi-ICE
Control ISP functional blocks through embedded software
VGA
ARM
FPGA
UART
CIS
ISP embedded system verification environment

31. Future Study PCB Connection bet. CIS-IPD board
Need to remove noise of output images
SDRAM memory based dualport memory controller design
Need to increase output resolution
ISP functional block software/hardware partition
SystemC based
RT level simulation
Use Seamless CVE based virtual core