1. Image Noise John Morris Department of Computer Science, Tamaki Campus The University of Auckland

2. 2 Stereo Image Noise Sources Signal noise Electromagnetic interference eg cross-talk Quantum behaviour of electronic devices eg resistor shot-noise Quantization : digitization of real-valued signals Geometric sources Discrete pixel sensors with finite area Occlusions Perspective distortion Opto-Electronic sources Sensitivity variations between cameras Different ‘dark noise’ levels Real lenses Depth-of-focus Optical Surface properties   Single camera sources  Stereo (2-camera) sources Note that we use the term ‘noise’ for all problem sources!

3. 3 Stereo Image Noise Sources  Optical Surface Properties  Lambertian scatterers  A “perfect” scatterer scatters light uniformly in all directions  Most correspondence algorithms assume perfect (Lambertian) scatterers  This means that surface patches will appear with the same intensity – independent of viewing angle..  Simple matching Intensities should be the same (perturbed by random noise!)  Reflectors

4. 4 Stereo Image Noise Sources  Ordering Constraint  Often used to simplify matching algorithms  Particularly dynamic programming  Imaged points appear in the same order in both images  Violated by ‘poles’ – narrow objects in front of planes

5. 5 Electronic Noise  Antennae (Receivers)  Wires act as antennae for EM waves  ‘Wire’ includes discrete wires but also  Tracks on circuit boards  Interconnects on chips  Transmitters  Any wire with a changing current emits EM waves

6. 6 Electronic Noise  Digital circuits  very rapid transitions (switching events)  High frequency signals  Crosstalk  One wire is influenced by neighbouring wires Ideal digital signal ‘Instaneous’ rise or fall≡ infinite frequency  perfect radiator Real digital signal ‘Fast’ rise or fall  high frequency  very good radiator Signal driven into purple wire Signal picked up on green wire EM coupling

7. 7 Electronic noise  Quantum effects  Resistor ‘shot’ noise  Resistive element is composed of discrete atoms  Always in motion for all T > 0 o K (absolute zero)  Noise as effective resistance changes  Moving atoms ‘collide’ with electrons moving to form the current  Random fluctuations in current or  Noise as effective resistance changes  Similar effects in all current carrying or producing devices Transistors Capacitors Inductors, etc e-e- e-e-

8. 8 Electronic noise  Digitization noise  Analogue signal  Taking all possible values At least at a macroscopic level!  Digital signal  Represented by a range of integers 0.. 255 (8 bit signal) 0.. 4095 (12 bit signal) -2048.. 2047 (12 bit signed signal)  A to D converter  Decides to which integer value to map a real value  Discretization  Values which differed (in real domain) become the same (in integer domain)

9. 9 Opto-electronic noise  Cameras have different gain settings  Amplifiers are not ‘matched’ perfectly  Sensors have different ‘dark current’ characteristics  All sensors produce some electrons (current) with no light  Quantum ‘tunneling’ out of the sensor device Stereo Problem Light Intensity Current Different slopes Gains differ Different offsets Dark currents differ

10. 10 Electronic Noise Summary

11. 11 Geometric noise  ‘Pixelisation’ of images  Sensor is divided into discrete regions – pixels  ‘Edges’ in images don’t conveniently fall onto pixel boundaries Red object Blue object Real image has blurred edges

12. 12 Geometric noise  Occlusions  Points visible from one camera only  Points which it is impossible to match  Perspective distortion  Field of view in one camera differs from other  Left and right images contain different numbers of pixels  Impossible to match all pixels correctly Stereo Problem B ML B MRMR

13. 13 Perspective problem Case 1 – ‘Fronto-planar’ object Object with face  to image planes  to optic axes extent, a In image plane, A L = A R = az / f Case 2 – Angled object A L > A R Effect exacerbated by Increased baseline Increased angle of object Verged axes,   

14. 14 Optical effects  Real lenses  Finite depth of field  Image is ‘in focus’ over finite region only Due to deviations of real (thick) lens from ideal shape  Depth of field increased by decreasing aperture Always work with a small aperture! f-stop Ratio of focal length to aperture ‘f-stop’ = f / a ‘f’ = 1.4 – aperture wide open f’ = 22 – aperture closed down a f Mechanical aperture Image plane Real lens For accurate work (objects in focus over a wide range) Use more light to permit narrower aperture

15. 15 Optical effects  Depth of focus  Finite depth of field  Image is ‘in focus’ over finite region only Due to deviations of real (thick) lens from ideal shape  Depth of field increased by decreasing aperture Always work with a small aperture! f-stop Ratio of focal length to aperture ‘f-stop’ = f / a ‘f’ = 1.4 – aperture wide open f’ = 22 – aperture closed down a f=22 Mechanical aperture Image plane Real lens For accurate work (objects in focus over a wide range) Use more light to permit narrower aperture f=1.4 Depth of field

16. 16 Lambertian Scatterers  ‘Perfect scattering’  Intensity of scattered light is the same in all directions  Observers at all angles see the same intensity  Characteristics of a scattering surface  Optically ‘rough’  Surface roughness ≈  = wavelength of incident light  700-400 nm for visible light

17. 17 Reflectors  Perfect reflector  100% of incident intensity reflected  Angle of incidence = angle of reflection   i =  r  All reflected intensity seen at one angle  Properties of a reflecting surface  Optically smooth  Surface roughness  ii rr ii rr

18. 18 Smooth surface Real Surfaces  Reflector  Most of incident intensity reflected  Bright ‘spot’ seen over a small angle  Properties of a reflecting surface  Optically smooth  Surface roughness   Specular reflections are the result of partly reflecting surfaces  Water surfaces  Polished glass, plastic, … ii rr

19. 19 Real Surfaces  Scattering surface  Reasonably uniform distribution of intensity with angle  Intensity of the same patch varies with angle  Intensity mis-match between L and R images  Systematic mis-match between views of the same patch  One is always less intense than the other  Shape of reflectance curve varies with surfaces  Random mis-match between different patches  Reflectance curves can be  dependent also  Different views of the same patch appear to have different colours

20. 20 Real Surfaces  Grating effects  In extreme cases  Regular surface variations with dimensions   Grating effects  Spectrum generation  Small variations in view angle  large colour variations  Systematic mis-match between different patches

21. 21 Ordering constraint  If M appears to the left of N in the left image, it should appear to the left of N in the right image  Violated by ‘poles’

22. 22 Stereo Correspondence is an ill-posed problem! There are always multiple solutions! or

23. 23 right - scanline signal left scan-line signal signal-based corresponding areas Corresponding signals and possible surface profiles Single surface reconstruction Extreme disjoint variant

24. 24 Effect of Noise but … What happens if we use ‘noise-free’ images? L Image - ‘corridor’ set Synthetic (ray traced)  Precise ‘ground truth’ is available

25. 25 Noise-free Image Matching Mismatch I L (x)-I R (x-d x ) Intensity Disparity (from ground truth) Examine one scan line – line 152 Mismatch is the matching error R image pixels shifted by known d (from ground truth)  Zero difference with L pixel?  Even ‘noise-free’ images have matching errors!

26. 26 Real Image Matching Mismatch I L (x)-I R (x-d x ) Intensity Disparity (from ground truth) Tsukuba – line 173 Occlusion ?

27. 27 Noise-free Image Matching Mismatch I L (x)-I R (x-d x ) Intensity Disparity (from ground truth) Examine one scan line – line 152 Edge effects No change in d,  no occlusion!

28. 28 Distribution of signal differences Pixel-wise correspondences – ‘Tsukuba’ pair (line 173) Grey-coded signal differences

29. 29 Matching in the presence of noise  Correlation algorithms  Match over a window  Correlation functions: SSD (sum squared differences), SAD (sum absolute differences)  Average matching errors  Handles random intensity fluctuations Electronic, quantization, … noise  Doesn’t handle Occlusions Pixelisation Intensity anomalies in one window compared to the other Gain and dark noise variations Perspective Tries to assign all pixels in a window to the same disparity Non-Lambertian surfaces & specular effects  Normalised correlation functions  Normalize pixel intensities to mean within a window  Some tolerance for gain and dark noise changes Often not worth the additional computational effort! Better: Correct the images at source! Align amplifiers, calibrate electronics, …

30. 30 Correlation algorithms  Generally poor matching  Mismatches (small windows) or  Blurred edges due to large windows  Simple (almost trivial!) code  Adaptive windows improve performance 

31. 31 Correlation-Based Methods  Matching Performance  The success of correlation-based methods depends on whether the image window in one image exhibits a distinctive structure that occurs infrequently in the search region of the other image.  How to choose the size of the window, w  h ?  Too small a window may not capture enough image structure and may be too noise sensitive  many false matches  Too large a window makes matching less sensitive to noise (desired) but also decreases precision (blurs disparity map)  An adaptive searching window has been proposed

32. 32 Correlation Methods Input – Ground truth 3x3 window Too noisy! 7x7 window Sharp edges are blurred! Adaptive window Sharp edges and less noise

33. 33 Correlation-Based Methods

34. 34 Correlation algorithms  Generally poor matching  Mismatches (small windows) or  Blurred edges due to large windows  Simple (almost trivial!) code for each pixel position,w for each disparity, d C wd =  (I L w+j -I R w+j-d ) 2 Choose min(C wd ) over d  d for position w  Fast enough?

35. 35 Correlation algorithms  Not fast enough for small images  For n  n image, d disparity values and h  w window – O (n 2 dhw)  For small (300x300) images and low depth accuracy (5%  d = 20 )  If n = 3x10 2, d = 20, h=w=9, then t(n) = c  81  2 x 10  9 x 10 4 ≈ 1.5 x 10 7  For a 3GHz processor, cycles / pixel ≈ 3  10 9 / 1.5 x 10 7 ≈ 200  ??? Enough to  Compute indices,  Fetch two pixels,  Subtract  Square or absolute difference  Find minimum from d costs  Repeat for adaptive window? but ……

36. 36 Correlation algorithms  Moderate resolution images  For moderate (1Mpixel) images and reasonable depth accuracy (1%  d = 100 )  If n = 10 3, d = 100, h=w=9, then t(n) = c  81  10 2  10 6 ≈ 10 10  For a 3GHz processor, cycles / pixel ≈ 3  10 9 / 10 10 ≈ 0.3  Inherently parallel  C wd  (I L w+j -I R w+j-d ) 2  Hardware acceleration needed!!  MMX (Intel Graphics Processing Extensions) helps  Optimized for simple arithmetic on pixels but  MMX pipeline needs to be filled efficiently  Max 16 operations in parallel  Excellent for 4  4 windows  Usually 9  9 needed!

37. 37 Correlation algorithms  Easy to implement in hardware  Match for each disparity at the same time  Add minimum circuit  Simple circuit..  But O (hw) computations for each disparity  Relatively large

38. 38 SAD Algorithm Block Diagram Note: High ‘fan-in’ for SAD calculator block Strains FPGA routing resources Large buffers needed for scanlines Consequence of window size Note: High ‘fan-in’ for SAD calculator block Strains FPGA routing resources Large buffers needed for scanlines Consequence of window size

39. 39 Correlation methods  Simplest code  Poorest performance  Adaptive windows don’t help much!  Medium speed  Window size is an important factor  Simple hardware realization but  Expensive in resource use  Handle random noise only  Window is just treated as vector of pixels  No spatial information used  Occlusions ignored

40. 40 Dynamic Programming Stereo  Attempts to find the ‘best path’ (sequence of disparity values)  Can recognize occlusions!  Averages noise over a scanline  Essentially local  Always moves ‘forward’ in a scanline  Solution generated by backtracking through predecessor array Doesn’t adjust values in backtrack  Uses the ordering constraint  Readily adapted to  allow for gain and offset changes  Perspective distortion  ‘Stubborn’  Incorporates a penalty for occlusions  Tends to make ‘streaks’ in disparity images..\resources\DP_Gonzalez.pdf..\resources\DP_Gonzalez.pdf  Can be improved by using neighbouring scan lines  Requires fewer scan line buffers than correlation window

41. 41 Dynamic Programming Result Note the horizontal streaks! It’s like a bureaucrat: once a DP algorithm ‘decides’ to adopt a disparity value – it doesn’t want to change its mind! Adding inter-scanline constraints (using a neighbouring scanline) generally improves this!

42. 42 Dynamic Programming Performance  Better matching than correlation methods  ‘Global’ along scan lines  Recognizes occlusions  Uses ordering constraint  Uses spatial information  Time complexity O (n 2 d)  Faster than correlation methods in software  Uses memory (typical of DP algorithms)  For n=10 3, d=10 2, t = c 10 8  c = ~ 30 cycles / pixel on 3GHz machine  Not enough for real time in software!

43. 43 Real time stereo  Real time stereo vision  Implemented Gimel’farb’s Symmetric Dynamic Programming Stereo in FPGA hardware  Real time precise stereo vision  Faster, smaller hardware circuit  Real time 3D maps  1% depth accuracy with 2 scan line latency at 25 frames/sec System block diagram: lens distortion removal, misalignment correction and depth calculator Output is stream of depth values: a 3D movie!

44. 44 Real time 3D data acquisition  Possible Applications  Collision avoidance for robots  Recognition via 3D models  Fast model acquisition Imaging technology not scanning !  Recognition of humans without markers  Tracking objects Recognizing orientation, alignment  Process monitoring eg Resin flow in flexible (‘bag’) moulds  Motion capture – robot training System block diagram: lens distortion removal, misalignment correction and depth calculator Output is stream of depth values: a 3D movie!

45. 45 System Block Diagram Cameras Distortion & Alignment DP Stereo – Forward Pass DP Stereo – Backward Pass

46. 46 Symmetric Dynamic Programming Stereo  Don’t try to reconstruct an ‘image’ corresponding to the left (or right) image  Turn yourself into Cyclops .. and reconstruct a Cyclopaean view  Optical centre half-way along baseline  Key advantage: Label visibility states: MR, B, ML  Disparity changes are associated with visibility state changes

47. 47 SDPS Disparity Calculator Block Small 2 x min3 1 x min2 1 x abs difference 3 x adders Mostly 8-14 bits Since it’s small, it can be replicated many times  Large disparity range  High depth resolution

48. 48 FPGA Stereo System Firewire Cables Firewire Physical Layer ASIC Firewire Link Layer ASIC FPGA Altera Stratix Parallel Host Interface FPGA Prog Cable

49. 49 An Artificial Vision System

50. 50 Artificial System Requirements  Highly Parallel Computation  Calculations are not complex but  There are a lot of them in megapixel+ ( >10 6 ) images!  High Resolution Images  Depth is calculated from the disparity  If it’s only a few pixels, then depth accuracy is low  Basic equation (canonical configuration only!) Depth, z = b f d p Baseline Focal Length Disparity Pixel size

51. 51 Artificial System Requirements  Depth resolution is critical!  A cricket* player can catch a 100mm ball travelling at 100km/h  High Resolution Images Needed  Disparities are large numbers of pixels  Small depth variations can be measured but  High resolution images increase the demand for processing power! *Strange game played in former British colonies in which a batsmen defends 3 small sticks in the centre of a large field against a bowler who tries to knock them down!

52. 52 Artificial System Requirements  Conventional processors do not have sufficient processing power  but Moore’s Law says  Wait 18 months and the power will have doubled but  The changes that give you twice the power also give your twice as many pixels in a row and four times as many in an image! Specialized highly parallel hardware is the only solution!

53. 53 Processing Power Solution

54. 54 FPGA Hardware  FPGA = Field Programmable Gate Array  ‘Soft’ hardware  Connections and logic functions are ‘programmed’ in much the same way as a conventional von Neuman processor  Creating a new circuit is about as difficult as writing a programme!  High order parallelism is easy  Replicate the circuit n times As easy as writing a for loop!

55. 55 FPGA Hardware  FPGA = Field Programmable Gate Array  ‘Circuit’ is stored in static RAM cells  Changed as easily as reloading a new program

56. 56 FPGA Hardware  Why is programmability important? or  Why not design a custom ASIC?  Optical systems don’t have the flexibility of a human eye  Lenses fabricated from rigid materials  Not possible to make a ‘one system fits all’ system  Optical configurations must be designed for each application  Field of view  Resolution required  Physical constraints  …  Processing hardware has to be adapted to the optical configuration  If we design an ASIC, it will only work for one application!!

57. 57 Correspondence or Matching

58. 58 Stereo Correspondence Can you find all the matching points in these two images? “Of course! It’s easy!” The best computer matching algorithms get 5% or more of the points completely wrong! …and take a long time to do it! They’re not candidates for real time systems!!

59. 59 Stereo Correspondence  High performance matching algorithms are global in nature  Optimize over large image regions using energy minimization schemes  Global algorithms are inherently slow  Iterate many times over small regions to find optimal solutions

60. 60 Correspondence Algorithms  Good matching performance, global, low speed  Graph-cut, belief-propagation, …  High speed, simple, local, high parallelism, lowest performance  Correlation  High speed, moderate complexity, parallel, medium performance Dynamic programming algorithms

61. 61 Depth Accuracy

62. 62 Stereo Configuration  Canonical configuration – Two cameras with parallel optical axes  Rays are drawn through each pixel in the image  Ray intersections represent points imaged onto the centre of each pixel Points along these lines have the same disparity  but To obtain depth information, a point must be seen by both cameras, ie it must be in the Common Field of View Depth resolution

63. 63 a Stereo Camera Configuration  Now, consider an object of extent, a  To be completely measured, it must lie in the Common Field of View but  place it as close to the camera as you can so that you can obtain the best accuracy, say at D ?Now increase b to increase the accuracy at D !But you must increase D so that the object stays within the CFoV!  Detailed analysis leads to an optimum value of b  a b D a

64. 64 Increasing the baseline % good matches Baseline, b Images: ‘corridor’ set (ray-traced) Matching algorithms: P2P, SAD Increasing the baseline decreases performance!!

65. 65 Increasing the baseline Standard Deviation Examine the distribution of errors Images: ‘corridor’ set (ray-traced) Matching algorithms: P2P, SAD Increasing the baseline decreases performance!! Baseline, b

66. 66 Increased Baseline  Decreased Performance  Statistical  Higher disparity range  increased probability of matching incorrectly - you’ve simply got more choices!  Perspective  Scene objects are not fronto-planar  Angled to camera axes  subtend different numbers of pixels in L and R images  Scattering  Perfect scattering (Lambertian) surface assumption  OK at small angular differences  increasing failure at higher angles  Occlusions  Number of hidden regions increases as angular difference increases  increasing number of ‘monocular’ points for which there is no 3D information!

67. 67 Evolution  Human eyes ‘verge’ on an object to estimate its distance, ie the eyes fix on the object in the field of view Configuration commonly used in stereo systems Configuration discovered by evolution millions of years ago Note immediately that the CFoV is much larger!

68. 68 Look at the optical configuration!  If we increase f, then D min returns to the critical value! Original f Increase f

69. 69 Depth Accuracy - Verging axes, increased f Now the depth accuracy has increased dramatically! Note that at large f, the CFoV does not extend very far!

70. 70 Summary

71. 71 Summary: Real time stereo  General data acquisition is:  Non contact  Adaptable to many environments  Passive  Not susceptible to interference from other sensors  Rapid  Acquires complete scenes in each shot  Imaging technology is well established  Cost effective, robust, reliable  3D data enhances recognition  Full capabilities of 2D imaging system + Depth data  With hardware acceleration  3D scene views available for  Control Monitoring in real time  Rapid response  rapid throughput Host computer is free to process complex control algorithms  Intelligent Vision Processing Systems which can mimic human vision system capabilities!

72. 72 Our Solution

73. 73 System Architecture Interface LVDS/ CameraLink Corrected Images Depth Map Line Buffers Distortion Removal Image Alignment Host Higher order Interpretation L Camera R Camera Control Signals FPGA PC Stereo Matching Disparity  Depth

74. 74 Distortion removal  Image of a rectangular grid from camera with simple zoom lens  Lines should be straight!  Store displacements of actual image from ideal points in LUT  Removal algorithm  For each ideal pixel position  Get displacement to real image  Calculate intensity of ideal pixel (bilinear interpolation)

75. 75 Distortion Removal  Fundamental Idea  Calculation of undistorted pixel position  Simple but slow  Not suitable for real time but  It’s the same for every image!  So, calculate once!  Create a look up table containing ideal  actual displacements for each pixel u d = u ud  (1+  1 r 2 +  2 r 4 +..) r 2 = (u ud +v ud ) 2

76. 76 Distortion Removal  Creating the LUT  One entry (dx,dy) per pixel  For a 1 Mpixel image needs 8 Mpixels!  Each entry is a float – (dx,dy) requires 8 bytes  However, distortion is a smooth curve  Store one entry per n pixels  Trials show that n=64 is OK for severely distorted image  LUT row contains 2 10 / 2 6 = 2 4 = 16 entries  Total LUT is 256 entries  Displacement for pixel j,k  du jk = (j mod 64) *  u j/64,k/64   u j/64,k/64 is stored in LUT  Simple, fast circuit Since the algorithm runs along scan lines, this multiplication is done by repeated addition

77. 77 Alignment correction  In general, cameras will not be perfectly aligned in canonical configuration  Also, may be using verging axes to improve depth resolution  Calculate locations of epipolar lines once!  Add displacements to LUT for distortion!

78. 78 Real time 3D data acquisition  Real time stereo vision  Implemented Gimel’farb’s Symmetric Dynamic Programming Stereo in FPGA hardware  Real time precise stereo vision  Faster, smaller hardware circuit  Real time 3D maps  1% depth accuracy with 2 scan line latency at 25 frames/se System block diagram: lens distortion removal, misalignment correction and depth calculator Output is stream of depth values: a 3D movie!

79. 79 Previous work: pros & cons Conventional approach: energy minimisation combining image dissimilarity, surface curvature and occlusions Exact minimisation with dynamic programming :  global 1D optimum matching under ordering constraints; can account for local photometric (offset or contrast) deviations and occlusions; fast processing;  no inter-scan-line constraints; random deviations on textureless regions; error propagation along scan-lines Approximate minimisation with Min-Cut techniques:  2D surface curvature constraints (MRF); a provably close approximate solution of an NP-hard problem; can account for local occlusions;   cannot account for local or global photometric deviations; high computational complexity Heuristic approximate minimisation with Belief Propagation  2D surface curvature constraints (MRF); can account for local occlusions;  cannot account for local or global photometric deviations; high computational complexity

80. 80 Conventional approaches: basic problems No account for intrinsic ill-posed nature of stereo problems Search for a single surface giving the best correspondence between stereo images but the single surface assumption is too restrictive in practice Heuristic or empirical weights of energy terms dramatically affect matching accuracy Large images and large disparity ranges lead to high computational cost of min-cut or belief propagation algorithms

81. 81 Distribution of signal differences Pixel-wise correspondences for a “Tsukuba” stereo pair (scan-line y = 173) Grey-coded signal differences

82. 82 Concurrent Stereo Matching: Main ideas  Human ‘stroke-wise’ analysis of a 3D scene Eyes browse from low to high frequency regions, from sharp points to smooth areas rather than scan line-by- line (Torralba, 2003)  Appropriate (likely) correspondence rather than best matching  Separation of noise estimation and signal matching from selection of surfaces and occlusion handling Stereo matching should avoid the ‘best match’ or signal difference minimisation almost universally used now in favour of a likely match based on a local signal noise model

83. 83 Modular structure of CSM Step 1: Estimate the image noise model ( allow it to be spatially variant ) Segment based on noise Select candidate 3D volumes Step 2: Fit constrained surfaces to the candidate volumes Could use K-Mean, SUSAN, etc Could be surface optimisation

84. 84 Noise Map Scaled (Amplitudex6) Noise Map White regions have higher noise - almost always appears in occluded regions Technique A: use a fast, efficient stereo matching technique (SDPS) to produce a disparity map – use mismatches as noise estimates

85. 85 Noise-Driven Segmentation Colour Mean Shift Segmentation Colour-position clustering in a 5D feature space: 3D-colour model L*u*v and 2D-lattice coordinates The noise map is considered to be the extra, sixth dimension Convert an image into data tokens Choose initial search window locations Compute the ‘mean shift’ window location for each initial position Merge windows that end up on the same ‘peak’ or mode Cluster data over the merged windows After noise-driven segmentation: occluded regions are segmented into small isolated blocks

86. 86 CSM: candidate 3D volumes and surface fitting Black regions contain likely matching points in the ‘slice’ for each disparity, d Surface fitting shrinks or expands each segmented region from slice to slice (based on counts of candidate points) d= 5 d=8 d= 6 d=10d=9 d=12d=11d=14d=13 d= 4 d= 7 d= 3

87. 87 CSM: candidate 3D volumes and surface fitting Ideal disparity slices CSM surface fitting d=5d=8d=6d=10d=9 d=12d=11d=14d=13Disparity map d=5d=8d=6 d=10 d=9 d=12d=11d=14d=13CSM Disparity map

88. 88 Symmetric CSM: candidate 3D volumes and surface fitting d=5 d=8 d=6 d=10d=9 d=12 d=11 d=14d=13 d=7 d=3 d=4

89. 89 Symmetric CSM: candidate 3D volumes and surface fitting Ideal disparity slices SCSM surface fitting d=5d=8d=6 d=10 d=9 d=12d=11d=14d=13Disparity map d=5d=8d=6d=10d=9 d=12d=11d=14d=13SCSM Disparity map

90. 90 Algorithm Comparison Symmetric DP stereo Graph cut Symmetric BP SCSM CSM Ground Truth

91. 91 Middlebury Benchmark (MB) Algorithms TsukubaSawtooth alluntexdiscalluntexdisc MB-SBPO 0.97 1 0.28 3 5.45 3 0.19 1 0.00 1 2.09 3 CSM 1.15 3 0.80 12 1.86 2 0.98 13 0.62 25 1.69 2 SCSM 0.97 1 0.74 11 1.80 1 0.96 12 0.60 24 1.57 1 * MB-SBPO – symmetric belief propagation algorithm ( best-performing Middlebury benchmark) Algorithms VenusMap alluntexdiscalldisc MB-SBPO 0.16 3 0.02 3 2.77 1 0.16 1 2.20 1 CSM 1.18 12 1.04 10 1.48 3 3.08 34 7.34 18 SCSM 1.15 11 0.91 9 1.38 2 3.05 33 7.03 17 rank among 40 algorithms

92. 92 Conclusions  Stereo matching is an ill-posed problem,  reconstruction of actual 3D optical surfaces is impractical  More reasonable goal: mimic human binocular stereo vision  Conventional constrained best matching does not explicitly account for a multiplicity of equivalent matches, for noise in both images of a stereo pair and for local contrast or offset image distortions  Concurrent stereo matching gives promising results because it separates the problem into search for all the candidate volumes with equivalent good matches (allowing for the estimated noise) and search for surfaces fitting to the volumes  Even the simplest implementation of the new approach competes with the best-performing conventional algorithms  Sloping surfaces challenge our CSM algorithm – watch this space!

93. 93 IVCNZ’2006  For a conference with a different style, consider IVCNZ’2006 (Image and Vision Computing, New Zealand)  Great Barrier Island, New Zealand Gateway to the Hauraki Gulf and Auckland 40 mins by light plane from Auckland, 3 hours by ferry Full range of accommodation options: Hotel style, cabins, …, even tents! Book early and you can sail there

94. 94 Great Barrier Island

95. 95 Stereo: Correspondence Problem  Stereo Pair  Images from identical cameras separated by some distance to produce two distinct views of a scene xLxL xRxR Disparity = x L - x R  1 z Corresponding Regions Left ImageRight Image