Materials II Lavanya Sharan March 2nd, 2011

Computational thinking about materials Physics-basedPseudo physics-based

Computational thinking about materials Physics-based Precise, detailed, complex models. Pseudo physics-based Quick, dirty, hacky models.

Computational thinking about materials Physics-based Precise, detailed, complex models. BRDF, BTF, BSSRDF etc. Pseudo physics-based Quick, dirty, hacky models. Histogram statistics, contrast tricks etc. Image sources: VPfaCGP Fig. 10.3, Fleming & Buelthoff (2005)

Computational thinking about materials Physics-based Precise, detailed, complex models. BRDF, BTF, BSSRDF etc. Useful for recreating appearance. Pseudo physics-based Quick, dirty, hacky models. Histogram statistics, contrast tricks etc. Useful for conveying appearance. Image sources: Henrik Wann Jensen, Motoyoshi et al. (2007)

Computational thinking about materials Physics-based Large, unwieldy models for real world stimuli. Perceptual testing limited to toy worlds. Pseudo physics-based Simple techniques to manipulate real world stimuli. Perceptual testing possible on real world imagery. Image sources: Boyaci et al. (2004), Doerschner et al. (2007), Sharan et al. (2008)

Computational thinking about materials Physics-based Precise, detailed, complex models. BRDF, BTF, BSSRDF etc. Useful for recreating appearance. Large, unwieldy models for real world stimuli. Perceptual testing limited to toy worlds. ‘Inverse optics’, ‘estimation’-based approaches Pseudo physics-based Quick, dirty, hacky models. Histogram statistics, contrast tricks etc. Useful for conveying appearance. Simple techniques to manipulate real world stimuli. Perceptual testing possible on real world imagery. ‘Image statistics’, ‘classification’-based approaches

Today Physics-basedPseudo physics-based Khan et al., SIGGRAPH 2006 Liu et al., CVPR 2010

Image-based material editing Khan et al. (2006) Goal: Change the material of an object in a photograph. Given: A single HDR image + alpha matte to identify the object + specification of final material properties. Input Output

Image-based material editing Why is this hard? Hugely under constrained problem. Doing inverse optics from a single, unknown image is very difficult. BRDF estimation techniques make lots of assumptions to get around this problem. Input Output

Image-based material editing What are the unknowns here? 3-D shape of object (we only have the silhouette information in input) Illumination on object Current reflectance properties of object (we only know the final reflectance properties in input) In essence, nothing is known at input.

Image-based material editing Khan et al.’s clever solution. Somehow figure out 3-D shape and illumination, and use this to create object in new material. Make gross assumptions that violate physics but look perceptually okay. (Inverse optics folks will need smelling salts right about now.)

Image-based material editing Recovering shape. Depth map = Luminance values Compress luminance, smooth gradients to improve results. Key insight: Bas-relief ambiguity helps, as do subsequent operations.

Image-based material editing Recovering shape. If the final result is shown from the same viewpoint as before and as static image, can get away with a lot.

Image-based material editing Recovering illumination. Environment map = Background image pixels. As we don’t have a metallic sphere in the scene (ideal way), let’s approximate what the metallic sphere would have seen, i.e., the scene minus the object. Key insight: As long as local consistency exists, we are not very good at estimating illumination.

Image-based material editing Recovering illumination. If the final result is shown in the same environment, this is a good approximation.

Image-based material editing Wait, what about specular highlights? Detect specular highlights, and paste them back in! Works because there is no presumed change in viewpoint, illumination or geometry.

Image-based material editing Totally fake but very believable translucency. Texture map to create translucency. Use estimate of geometry to warp the (filled-in) background pixels and blur. Voila! Original Result

Image-based material editing Can manipulate gloss easily (using known hack). Luminance remapping to change perceived gloss. Original Specular resultDiffuse result

Image-based material editing Change textures entirely. Texture remapping using estimate of geometry. Original Result

Image-based material editing Can do boring BRDF editing too. Given BRDF specification, can render using approximations of illumination and shape. Original Result 1Result 1I

Image-based material editing Contributions and criticisms. + Editing single, unknown real world photograph. + Producing very good results with pseudo-physics. + These hacks tell us something interesting about material perception. - At the expense of physical correctness. - Cannot explicitly change viewpoint, illumination or shape of the object in original image.

Image-based material editing Contributions and criticisms. + Editing single, unknown real world photograph. + Producing very good results with pseudo-physics. + These hacks tell us something interesting about material perception. - At the expense of physical correctness. - Cannot explicitly change viewpoint, illumination or shape of the object in original image. Pseudo physics-based Physics-based Image copyright: Ben Heine, Image source: thepagansphinx.blogspot.com

Real world material recognition Liu et al. (2010) Goal: Categorize the material in a given photograph. Given: A single uncalibrated image + matte + a training database of 10 material categories.

Real world material recognition Why is this hard? Hugely under constrained problem. Doing inverse optics from a single, unknown image is very difficult. BRDF estimation techniques make lots of assumptions to get around this problem. If you only need high-level categories, why not use object/texture recognition approaches? Stone PaperFabric

Real world material recognition What are the unknowns here? 3-D shape of object (we only have the silhouette information in input) Illumination on object Reflectance properties of object In essence, nothing is known at input.

Real world material recognition Liu et al.’s solution. Use learning. Sharan et al. created a dataset that was challenging even for humans. Let’s use that. Try a bunch of standard features, see if they work. Use a standard bag-of-words approach.

Real world material recognition Previous work. Use learning. Use a 3-D texture database that is sub- optimal for material recognition problem. Try a bunch of standard features, see if they work. Use a standard bag-of-words approach. Choice of database matters, a lot. (It changes the problem, and changes the difficulty of the problem.)

Real world material recognition Database matters.

Real world material recognition Use standard features + some new ones.

Micro-texture Shape Reflectance Color & texture

Real world material recognition Use standard LDA model (with some tweaks).

Real world material recognition Beat state-of-art because of stronger features and model. Varma-Zisserman (23.8%)Liu et al. (44.6%) Varma-Zisserman with Liu et al. features (37.4%)

Real world material recognition Trails human performance by a huge margin. Human performance (92.3%)Liu et al. (44.6%) Reproduces Sharan et al. perceptual result on Mechanical Turk.

Real world material recognition Contributions and criticisms. + Categorizing single, uncalibrated real world photograph. + Better results than state-of-art. + Addresses the problem of material categorization, rather than recognizing specific 3-D surface. - Cannot explain human performance. - Database too challenging, tailored for humans not machines.

Real world material recognition Contributions and criticisms. + Categorizing single, uncalibrated real world photograph. + Better results than state-of-art. + Addresses the problem of material categorization, rather than recognizing specific 3-D surface. - Cannot explain human performance. - Database too challenging, tailored for humans not machines. Mid & high-level vision theories needed Low-level explanation insufficient Image copyright: Ben Heine, Image source: thepagansphinx.blogspot.com Explaining material perception