Distortion Correction ECE 6276 Project Review Team 5: Basit Memon Foti Kacani Jason Haedt Jin Joo Lee Peter Karasev

2 ECE 6276 Final Project Team 5 7/14/2009 Outline Motivation Components Component Optimization Current Results Plans for Catapult C Schedule

3 ECE 6276 Final Project Team 5 7/14/2009 Objective Given a distorted image with known size and known lens distortion parameter, generate an undistorted image.

4 ECE 6276 Final Project Team 5 7/14/2009 Motivation – Why? The formation of undistorted images can be described by a series of matrix multiplications Distortion makes it very difficult to calibrate a camera to measure geometry (depth, size, orientation, etc) Many applications in image processing and computer vision like structure estimation, image mosaicing, and ultimately vision-based control.

5 ECE 6276 Final Project Team 5 7/14/2009 Motivation contd.. Application: Measure motion and geometry Problem: Known geometry in the scene is warped, relationship between 3D and 2D points is nonlinear. Solution: Undo the distortion, so x 2D = A * X 3D

6 ECE 6276 Final Project Team 5 7/14/2009 Literature Review (I) K.T. Gribbon, C.T. Johnston, and D.G. Bailey, “A Real-time FPGA Implementation of a Barrel Distortion Correction Algorithm with Bilinear Interpolation” Focus on reducing hardware complexity. Uses LUTs to store mapping data. No quantitative results provided. Logic resource utilization on RC-100 is 51%

7 ECE 6276 Final Project Team 5 7/14/2009 Literature Review (II) Qiang, L.; Allinson, N.M.;” FPGA Implementation of Pipelined Architecture for Optical Imaging Distortion Correction”, Signal Processing Systems Design and Implementation, SIPS '06.Signal Processing Systems Design and Implementation, SIPS '06. Same algorithm as previous one Implementation on a Xilinx FPGA XCS uses 75% of the hardware multipliers. Residual error of the undistorted image was 1.5% of the distorted image.

8 ECE 6276 Final Project Team 5 7/14/2009 Literature Review (III) Hany Farid & Alin C. Popescu, “Blind Removal of Lens Distortion”, Journal of the Optical Society of America 2001 For removal of distortion in absence of any calibration data. Uses polyspectral analysis to detect higher order correlations in frequency domain which are proportional to the distortion. Computationally intensive. No quantitative results. Accuracy is not comparable to those based on known distortion parameters.

9 ECE 6276 Final Project Team 5 7/14/2009 Components to Achieve Objective Component 1: Matlab forward distortion function –Verifies correctness of undistortion algorithm Component 2: Data ordering test bench –Order the C++ input stream from MATLAB generated data Component 3: Undistortion lookup table or lookup function –Initial prototype in MATLAB Component 4: Least squares interpolation lookup function –Initial prototype in MATLAB –Compare different techniques such as LUT vs NEAREST NEIGHBOR Component 5: Verification structure –Compare original C++ result to undisorted image in MATLAB MATLAB  C++  Catapult C  MATLAB

10 ECE 6276 Final Project Team 5 7/14/2009 Component Optimization Questions What size buffers do we need to compare against previous frames? For undistortion function can we compute them dynamically or do we need a pre-defined LUT? For finding fast least squares / linear system solver, compare speed cost vs. nearest neighbor and effect on output error

11 ECE 6276 Final Project Team 5 7/14/2009 Matlab Demo of Algorithm - Original

12 ECE 6276 Final Project Team 5 7/14/2009 Matlab Demo of Algorithm - Distorted

13 ECE 6276 Final Project Team 5 7/14/2009 Matlab Demo of Algorithm - Recovered

14 ECE 6276 Final Project Team 5 7/14/2009 Design Goals Limited Buffers 8 bits per sub-pixel (24 bits total) Resolution (up to 640x480) Concerned with geometry Area, Throughput, Latency Target a low-cost implementation that handles consumer video application pixel clocks of 165 MHz (Apprx 6 ns cycle time).

15 ECE 6276 Final Project Team 5 7/14/2009 Test Vectors The Line Test Original ImageDistorted Image Recovered Image

16 ECE 6276 Final Project Team 5 7/14/2009 Plans for Catapult C Code Use C/C++ and Algorithmic C data types to describe synthesizable hardware Architectures (type of hardware interface (streaming buffers) ) Constraints (Throughput, area, latency) RTL generation and verification Optimizations -Pipelining -Parallelism -Loop Unrolling -Scheduling -Streaming buffers/ Read & Write

17 ECE 6276 Final Project Team 5 7/14/2009 Project Timeline

18 ECE 6276 Final Project Team 5 7/14/2009 Project Risks Risk 1: Slowness of interpolation plan: the key goal of undistortion is “geometrical accuracy” which hold even if noise is injected. To mitigate this risk we have a forward mapping algorithm that is fast compared to any interpolation method but at the cost of missing pixels near the edges. Risk 2: Not enough storage space to keep a lookup table of coordinates. To mitigate this risk we could compute coordinates on the fly at the cost of math operations. Risk 3: Cannot find least squares method for FPGA. To mitigate this risk we could do nearest neighbor interpolation.

19 ECE 6276 Final Project Team 5 7/14/2009 Current Status Straight lines are recovered with minimal injection of missing points or noise through our algorithm. Add more quantitative results on the geometrical accuracy of recovered images than presented in previous results. Generalize the coordinate mapping to make the implementation more robust to varying distortion models and hence supporting more cameras “on the fly.”

20 ECE 6276 Final Project Team 5 7/14/2009 References Yi Ma, Stefano Soatto, et al., “An Invitation to 3-D Vision”Yi MaStefano Soatto Richard Hartley, Andrew Zisserman, “Multiple View Geometry in Computer Vision”Richard HartleyAndrew Zisserman Edward M. Mikhail, James S. Bethel, J. Chris McGlone, “Introduction to Modern Photogrammetry”James S. BethelJ. Chris McGlone

21 ECE 6276 Final Project Team 5 7/14/2009 Questions? ?