Presentation on theme: "Stereovision Image Noise John Morris Department of Computer Science, Tamaki Campus The University of Auckland Iolanthe at 14 knots on Cockburn Sound, Perth."— Presentation transcript:
Stereovision Image Noise John Morris Department of Computer Science, Tamaki Campus The University of Auckland Iolanthe at 14 knots on Cockburn Sound, Perth
2 Correlation methods - summary Simplest code Poorest performance Adaptive windows don’t help much! Medium speed Window size is an important factor Simple hardware realization but Expensive in resource use Handle random noise only Window is just treated as vector of pixels No spatial information used Occlusions ignored
3 Dynamic Programming Stereo Attempts to find the ‘best path’ (sequence of disparity values) Can recognize occlusions! Averages noise over a scanline Essentially local Always moves ‘forward’ in a scanline Solution generated by backtracking through predecessor array Doesn’t adjust values in backtrack Uses the ordering constraint Readily adapted to allow for gain and offset changes Perspective distortion ‘Stubborn’ Incorporates a penalty for occlusions Tends to make ‘streaks’ in disparity images..\resources\DP_Gonzalez.pdf..\resources\DP_Gonzalez.pdf Can be improved by using neighbouring scan lines Requires fewer scan line buffers than correlation window
4 Dynamic Programming Result Note the horizontal streaks! It’s like a bureaucrat: once a DP algorithm ‘decides’ to adopt a disparity value – it doesn’t want to change its mind! Adding inter-scanline constraints (using a neighbouring scanline) generally improves this!
5 Dynamic Programming Performance Better matching than correlation methods ‘Global’ along scan lines Recognizes occlusions Uses ordering constraint Uses spatial information Time complexity O (n 2 d) Faster than correlation methods in software Uses memory (typical of DP algorithms) For n=10 3, d=10 2, t = c 10 8 c = ~ 30 cycles / pixel on 3GHz machine Not enough for real time in software!
Dynamic Programming Basic idea Given a very large problem Solve small problems first, Save solutions to the small problems, Use them to solve larger problem Continue until large problem is solved Dynamic programming uses storage space (for solutions to the small problems) to gain speed 6
Dynamic programming stereo Small problems are the costs of a ‘path’ to a certain pixel Costs are based on Difference between pixel intensities Cost for an occlusion Sometimes - Discontinuity cost Assume that disparity changes (depth changes) are relatively rare Alternatively – assume most surfaces are smooth 7
Dynamic programming stereo 8 8 Monocular Right Pixels Left scan line Monocular Left pixels Right scan line Start End Optimum path marks matching pixels Diagonal path – smooth surface – no change in disparity Jumps – changes in disparity This path would indicate all matches at zero disparity
Dynamic programming stereo - Costs 9 9 Monocular Left Pixels Left scan line Monocular Right pixels Right scan line Start End DP algorithm calculates cumulative costs for all possible paths – using costs for previous pixel to estimate costs for the current pixel Maximum disparity Compute costs for paths in this region Cost stored in this location represents cost of best path to this point The minimum cost here is the end of the best path COSTS ARRAY
Dynamic programming stereo - Predecessors 10 Monocular Left Pixels Left scan line Monocular Right pixels Right scan line Start End Keep a second array of predecessors Maximum disparity Store here best predecessor for a path ending here The minimum cost here is the end of the best path PREDECESSORS ARRAY
Dynamic Programming 11 Two arrays used: Costs and Predecessors As best cost C dx to reach (d,x) is computed, the best predecessor is stored in d,x
Dynamic Programming - general 12 When the end of the scanline is reached, the minimum cost is selected and its predecessor used to build the optimum profile (red arrows)
Dynamic programming stereo 13 Monocular Left Pixels Left scan line Monocular Right pixels Right scan line Start End As cost for each pixel at each disparity is determined, save the predecessor Use the predecessors to back track Need w predecessors – Biggest resource cost! Predecessor array stores disparity of best previous point Maximum disparity Compute costs for matches in this region
Symmetric Dynamic Programming Stereo 15 Compute costs for the Cyclopean image centred between O L and O R Use visibility constraints - If disparity changes by n pixels, must be n MR or ML points between the two B points Only need to save state in predecessor array.. not disparity!
Symmetric Dynamic Programming Stereo Conventional Stereo Match pixel in L image with pixels to R of it in R image SDPS For pixel position, x, in Cyclopaean image, match pixels at x L +d/2 with x R -d/2 Even x positions have even disparity, odd x position have odd disparity Calculate in two phases: odd and even Resultant image is 2w pixels wide Backtrack fills in extra pixels in B regions 16
SDPS Cost equations Binocular points C(x,d,B) = dI(x,d) + min(C(x-1,d-1,ML), C(x-2,d,B), C(x-2,d,MR) ) (1) Costs to reach monocularly visible states are: C(x,d,ML) = occ_term + min(C(x-1,d-1,ML), C(x-2,d,B), C(x-2,d,MR) ) (2) C(x,d,MR) = occ term + min(C(x-1,d+1,MR),C(x-1,d+1,B) ) (3) Pixel mismatch is absolute difference dI(x,d) = |g L -g R | Any other dissimilarity measure could be used 17
SDPS Occlusion cost is fixed constant Results are not very sensitive to the actual value Choose about three times pixel intensity variance. ie signal noise Predecessor array only stores previous state ML, B or MR – 2 bits only 18
SDPS transitions 19 This pixel, at disparity d 0, can only have 3 states – ML, B, MR Arrows indicate costs used to evaluate cost to reach each state. Visibility constraints allow only these transitions. MR ML Note half integral x positions – SDPS generates 2w points/line
20 System Architecture Interface LVDS/ CameraLink Corrected Images Depth Map Line Buffers Distortion Removal Image Alignment Host Higher order Interpretation L Camera R Camera Control Signals FPGA PC Stereo Matching Disparity Depth
SDPS hardware – Gidel FPGA card 21
SDPS hardware – Disparity calculator block 22 Small and compact – easily replicated to provide large disparity range
SDPS hardware – proposed system 23
FPGA hardware Field Programmable Gate Arrays Soft hardware ‘Programmed’ in a high level language – VHDL ‘Compiled’ programs soft loaded onto FPGA No cost to change the circuit! Tradeoffs Flexible Pipeline or inherent parallelism easily achieved Compare GPU Slower than state of the art CPUs – n00 MHz vs n GHz Memory may be a problem 24
25 FPGA Hardware FPGA = Field Programmable Gate Array ‘Circuit’ is stored in static RAM cells Changed as easily as reloading a new program
26 FPGA Hardware Why is programmability important? or Why not design a custom ASIC? Optical systems don’t have the flexibility of a human eye Lenses fabricated from rigid materials Not possible to make a ‘one system fits all’ system Optical configurations must be designed for each application Field of view Resolution required Physical constraints … Processing hardware has to be adapted to the optical configuration If we design an ASIC, it will only work for one application!!
Other correspondence algorithms Belief propagation DP Stereo extended in all directions Labels each pixel with a state – its disparity Minimizes an energy function Two terms Data term – pixel difference Smoothness term – penalizes changes (in disparity) Iterates until convergence (or cutoff) Exchanging messages with neighbouring pixels 27
SDPS Results 28 Corrected Left ImageCorrected Right Image Disparity MapOcclusion Map
Belief propagation Best general performance Simple BP being overtaken by segmentation-based algorithms Memory intensive Needs to store costs for each label for each pixel in a region Compare last two pixels in scan line for SDPS Iterates – so slow 29
30 Algorithm Comparison Symmetric DP stereo Graph cut Symmetric BP SCSM CSM Ground Truth