1. Shapelets Correlated with Surface Normals Produce Surfaces Peter Kovesi School of Computer Science & Software Engineering The University of Western Australia

2. Shape from Shading and Shape from Texture attempt to Calculate the surface normal direction at each point in the image, and then Infer the surface shape from the surface normals.

3. Surface normals are usually calculated in terms of slant and tilt N slant  Slant is the angle between the viewing direction and the surface normal surface eye

4. Tilt is the angle of the projection of the surface normal on a plane perpendicular to the viewing direction viewer plane perpendicular to viewing direction N slant  tilt  projection of surface normal

5. Tilt often has an ambiguity of  Problem:

6. Tilt often has an ambiguity of  Problem:

7. Regularization schemes are often used to deal with noise. However, they typically involve the minimization of an Energy Term that has mixed units surface gradient error (highly nonlinear with surface angle) bending energy shading error

8. Problem: Regularization schemes are often used to deal with noise. However, they typically involve the minimization of an Energy Term that has mixed units surface gradient error (highly nonlinear with surface angle) bending energy The weighting parameters have to act as unit conversion operators. The appropriate values will be dependent on scale. This can be problematic if the regularization is being applied within a multi-scale framework. Discontinuities have to be automatically detected and treated specially. shading error

9. Alternative approach: Project the surface gradients onto a set of integrable basis functions and reconstruct the surface from these. Frankot and Chellappa [1988] first introduced this approach using Fourier basis functions. Features of work presented here: Localized, non-orthogonal basis functions are used. Correlations with basis functions are performed with respect to slant and tilt rather than in terms of gradient in x and y. Ambiguities of  in tilt can be considered. Low quality reconstructions are possible with slant information alone.

10. Decomposing and Reconstructing Signals with Basis Functions signal filtered at scale 2 filtered at scale 4 filtered at scale 6 Reconstruction obtained by summing band-passed signals over 6 scales band-pass filters/wavelets

11. We only have the gradient of the surface…

12. Differentiation is linear surface band-pass filter surface gradient

13. We only have the gradient of the surface… Differentiation is linear A band-passed version of the signal can be obtained from a correlation performed between the gradient of the signal and integral of the band-pass filter. surface band-pass filter Integral of band-pass filter (shapelet gradient) surface gradient

14. Choice of Shapelet The gradient of shapelet function must satisfy the admissibility condition of zero mean. Must ensure preservation of phase information in the signal. The bank of shapelet filters should provide a uniform coverage of the signal spectrum so that it is faithfully reconstructed. These requirements are satisfied by a Gaussian

15. Note that correlating the gradient of one shapelet filter with the gradient of the signal will correspond to extracting a band of frequencies from the signal gradient.

16. The signal gradient spectrum differs from the original signal spectrum in that its amplitudes are scaled by the frequency.

17. Note that correlating the gradient of one shapelet filter with the gradient of the signal will correspond to extracting a band of frequencies from the signal gradient. The signal gradient spectrum differs from the original signal spectrum in that its amplitudes are scaled by the frequency. The sum of the shapelet gradient transfer functions should form a curve that is inversely proportional to frequency to counteract the scaling of the gradient spectrum. The net result will then be an even coverage of the original signal spectrum.

18. Shapelet bank formed by five Gaussians with height proportional to scale. Transfer functions of the gradients of the shapelets above. The sum forms a curve that is approximately inversely proportional to frequency.

19. A bank of shapelets will not form an orthogonal basis set This does not matter if we have a strongly redundant basis set. (Achieved by using a continuous wavelet transform.) A non-orthogonal basis set can be considered to be the sum of several orthogonal basis sets. - We get accurate reconstruction up to a scale factor

20. The scaling of the reconstruction due to the redundancy of the basis set can be determined and corrected if required. Overcomplete basis sets have the advantage that small changes in local features result in correspondingly small changes in the basis coefficients. Overcompleteness also provides robustness to noise.

21. In Summary : Correlate the gradients/normals of the surface with the gradients/normals of a set of basis functions. These correlation results correspond to bandpassed versions of the original signal. Reconstruct surface by summing correlation results. Perform correlation in terms of slant and tilt separately - slant and tilt are calculated by different means. - slant and tilt have differing degrees of uncertainty.

22. Correlation of Slant Slant lies in the range 0 -   which only provides gradient magnitude information. The slant correlation at scale i is formed by With no tilt information this correlation measure matches positive and negative gradients equally. (No distinction between concave or convex shape) gradient of surface gradient of shapelet at scale i

23. Correlation of Tilt If gradient magnitudes match and tilts match correlation is +ve If gradient magnitudes match and tilts differ by  correlation is -ve If tilts differ by  /2 correlation is 0 Use the cosine of the tilt difference as a correlation measure tilt of surface tilt of shapelet at scale i

24. Overall Correlation Reconstruction

25. synthetic test surface needle diagram of surface normals

26. reconstruction over 2 scales reconstruction over 4 scalesreconstruction over 6 scales

27. Shapelet reconstruction using the gradients of the range data Range data from OSU database

28. Slant and tilt with Gaussian noise of 0.3 radians standard deviation Shapelet reconstruction Terzopoulos multigrid reconstruction Frankot Chellappa reconstruction

29. Allowing for Tilt Ambiguity of  When tilts are aligned value is 1 When tilts are orthogonal value is 0 When tilts are opposite value is 1 … results in rectification of the band-pass components of the surface

30. Reconstruction with a tilt ambiguity of  reconstruction of ramps surface bumps surface reconstruction

31. Reconstruction with slant data alone Slant-only reconstruction of bumpsSlant-only reconstruction of ramps Slant-only reconstruction of sinc Sinc function

32. Slant-only reconstruction of face …after segmentation and negation

33. Shapelet reconstruction Real data: Surface normals from texture

35. Sand ridges: Slant set to  /4 at all points Tilt set to 0 for white regions,  for black regions Shapelet reconstruction

36. Shape from Occluding Contours Manually marked contours Surface normals at occluding contours are perpendicular to viewing direction. This is a powerful cue to shape.

37. At all points slant is set to 0, except along contours where slant is set to , and tilt is set to be perpendicular to contour. This gradient field is not integrable. A crude approximation of the surface normals in the scene…

38. Shapelet reconstruction

40. Manually drawn contours

45. Conclusions Reconstructions from surface normals using shapelets is simple and the process is robust to noise. The use of basis functions implicitly imposes a continuity constraint in the reconstruction. Being able to treat slant and tilt separately permits data with tilt ambiguity, or no tilt data, to be considered. Much can be made from limited amounts of surface normal data. This allows some some aspects of structure to be deduced from line drawings.