Inverse Volume Rendering with Material Dictionaries Ioannis Gkioulekas1 Shuang Zhao2 Kavita Bala2 Todd Zickler1 Anat Levin3 1Harvard 2Cornell 3Weizmann

Translucency is everywhere food skin jewelry architecture We deal with translucency in many aspects of our everyday life. Our food and skin are translucent; and so are objects as diverse as jewelry and even buildings.

Subsurface scattering outgoing direction incident direction isotropic extinction coefficient σt(λ) scattering coefficient σs(λ) radiative transfer equation phase function p(θ, λ) Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions. Chandrasekhar 1960, Ishimaru 1978

Parameters are important extinction coefficient σt(λ) scattering coefficient σs(λ) phase function p(θ, λ) Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions.

Parameters are non-trivial ≈ ≠ Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions.

Light transport is complicated rendering: Monte-Carlo techniques accurate (exact at convergence) efficient (relatively) rendering (σt σs p(θ)) Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions. Veach 1997

Light transport is complicated inverse rendering: fancy term for image-based parameter estimation no standard way to solve it rendering (σt σs p(θ)) Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions. inverse rendering Bal 2009

Inverse rendering approximations single-scattering approximation [Narashimhan et al. 2006] [Mukaigawa et al. 2010] thin materials, diluted liquids only approximate isolation (σt σs p(θ)) Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions. inverse rendering

Inverse rendering approximations diffusion approximation [Jensen et al. 2001] [Papas et al. 2013] … … … … optically thick materials parameter ambiguity (σt σs p(θ)) Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions. inverse rendering Wyman et al. 1989, Jensen et al. 2001, Papas et al. 2013

Inverse rendering approximations parameterization of phase function [Chen et al. 2006] Henyey-Greenstein lobes [Donner et al. 2008] [Fuchs et al. 2007] [Goesele et al. 2004] [Gu et al. 2008] single-parameter family: [Hawkins et al. 2005] [Holroyd et al. 2011] g∈ −1,1 [McCormick et al. 1981] not general enough [Pine et al. 1990] The most commonly used phase function in computer graphics is the Henyey-Greenstein family. These are parabola-like lobes, controlled by a single parameter g. For different values of this parameter, we can have backward scattering, isotropic, and forward scattering lobes. Thinking of the phase function as a spherical distribution, the parameter g is equal to the first moment mu1, also commonly referred to as the average cosine [Prahl et al. 1993] [Gkioulekas et al. 2013] [Wang et al. 2008]

Our contributions dictionary-based material representation operator-theoretic framework for inverse volume rendering without approximations 𝜕𝐸 𝜕 𝑤 𝑞 =2 𝑤 𝑞 𝑆 𝐼−𝐾 −1 𝐿 𝑖 −𝜇 𝑆 𝐼−𝐾 −1 𝐾 𝑞 𝐼−𝐾 −1 𝐿 𝑖 physical setup implementation and accurate measurement of broader classes of materials The most commonly used phase function in computer graphics is the Henyey-Greenstein family. These are parabola-like lobes, controlled by a single parameter g. For different values of this parameter, we can have backward scattering, isotropic, and forward scattering lobes. Thinking of the phase function as a spherical distribution, the parameter g is equal to the first moment mu1, also commonly referred to as the average cosine

Basic idea appearance matching collect images μ1 μ2 μ3 μ4 μ5 μ6 μ7 μ8 μ9 find material that matches them m = (σt σs p(θ)) min Σ ǁ μs - render(m, scenes) ǁ2 Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions. m

Dictionary parameterization dictionary of phase functions p9 p7 p10 p11 p6 p8 p5 p1 p2 p3 p4 D = {p1, p2, …, pQ} express arbitrary phase function in terms of dictionary p p = Σ wq pq similarly for all parameters Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions.

Dictionary parameterization dictionary of materials D = {m1, m2, …, mQ} express arbitrary material in terms of dictionary m = Σ wq mq similarly for all parameters Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions.

Basic idea appearance matching collect images μ1 μ2 μ3 μ4 μ5 μ6 μ7 μ8 μ9 find material that matches them m = (σt σs p(θ)) m = (D, w) min Σ ǁ μs - render( m w , scenes) ǁ2 Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions. m w

Temporally resolved light transport propagation step τ = c Δt τ τ τ τ propagation probabilities: phase function p(θ) scattering coefficient σs extinction coefficient σt continue straight: π1 = 1 - τ σt absorption event: π2 = τ (σt - σs) Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions. volume event scattering event: π3 = τ σs (p(θ))

Temporally resolved light transport linear algebra of single propagation step τ = c Δt Lt+Δt = K Lt Lt+Δt = (π1I + π3 P) Tτ Lt single-step matrix K straight: function of material: m = (σt σs p(θ)) Lt+Δt,str = Tτ Lt π1 = 1 - τ σt absorption: Lt+Δt,abs = 0 π2 = τ (σt - σs) lightfield L scattering: Lt+Δt,sca = P Tτ Lt π3 = τ σs Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions.

Temporally resolved light transport linear algebra of multiple L1 = K Li propagation steps τ = c Δt L2 = K2 Li L3 = K3 Li Lt+Δt = K Lt L4 = K4 Li single-step matrix K L5 = K5 Li L = Σ Kn Li = (I - K)-1 Li L6 = K6 Li lightfield L t = 4 Δt t = 6 Δt t = 2 Δt t = 7 Δt t = 5 Δt t = 3 Δt t = 8 Δt t = 9 Δt t = Δt t = 0 rendering matrix R L7 = K7 Li Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions. L8 = K8 Li

Dictionary parameterization dictionary of materials D = {m1, m2, …, mQ} express arbitrary material in terms of dictionary m = Σ wq mq single-step matrix K becomes K = Σ wq Kq rendering matrix R becomes Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions. R = (I - Σ wq Kq)-1

Basic idea appearance matching collect images μ1 μ2 μ3 μ4 μ5 μ6 μ7 μ8 μ9 find material that matches them m = (D, w) min Σ ǁ μs - Ss (I - Σ wq Kq)-1 Li,s) ǁ2 min Σ ǁ μs - render( w , scenes) ǁ2 Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions. w

Optimization appearance matching loss function E(w) = Σ ǁ μs - Ss (I - Σ wq Kq)-1 Li,s) ǁ2 differentiable! 𝜕E w 𝜕 w q = 2wq Σ(Ss (I - Σ wq Kq)-1 Li,s - μs) Ss (I - Σ wq Kq)-1 Kq (I - Σ wq Kq)-1 Li,s after some simplification 𝜕E w 𝜕 w q = 2wq Σ(Ss R(w) Li,s - μs) Ss R(w) Kq R(w) Li,s Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions. use your favorite gradient descent variant to do inverse rendering

The world is continuous linear algebra of temporally resolved light transport Lt+Δt = K Lt single-step matrix K L = R Li = (I - K)-1 Li rendering matrix R lightfield L light transport can be described using matrix-vector products (continuous function) Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions.

The world is continuous operator theory of temporally resolved light transport Lt+Δt = K Lt single-step operator K L = R Li = (I - K)-1 Li rendering operator R lightfield L matrix-vector products become (Monte-Carlo) rendering (continuous function) Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions. Arvo 1995

Optimization (operator version) operator version of gradient is identical 𝜕E w 𝜕 w q = 2wq Σ(Ss R(w) Li,s - μs) Ss R(w) Kq R(w) Li,s three cascaded Monte-Carlo operations noisy estimates use stochastic gradient descent Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions. R(w) Kq R(w)

Basic idea appearance matching collect images μ1 μ2 μ3 μ4 μ5 μ6 μ7 μ8 μ9 find material that matches them m = (D, w) min Σ ǁ μs - Ss (I - Σ wq Kq)-1 Li,s) ǁ2 Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions. w

Acquisition setup front lighting material sample backlighting hyperspectral camera Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions.

Acquisition setup front lighting material sample backlighting hyperspectral camera top rotation stage bottom rotation stage Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions.

Acquisition setup frontlighting material sample front lighting backlighting hyperspectral camera hyperspectral camera top rotation stage top rotation stage bottom rotation stage Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions.

Acquisition setup MEMS light switch blue (480 nm) laser green (535 nm) laser red (635 nm) laser RGB combiner Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions. light generation

Acquisition setup Some acquisition statistics: 3 lighting angles (for both front- and back-lighting) 3 viewing angles LDR images at 18 exposures 3 wavelengths total 54 HDR measurements per material 75 minutes capture per material Translucent appearance is caused by scattering of light inside material volumes. This process is described by the radiative transfer equation, and is controlled by three material-dependent parameters. Let’s take a closer look: As light travels through a medium, the extinction coefficient controls the distance before a volume events occurs. Then, at each volume event, light may get absorbed with some probability determined by the absorption coefficient. Otherwise, light is scattered, meaning that travels in a different direction, as determined by the phase function. In general these parameters are functions of wavelength, but we will be ignoring this dependence dealing only with “grayscale” materials. We will be focusing on the phase function in this talk. The phase function is a probability distribution on the sphere of directions, and is often assumed to be spherically symmetric. We will also use this assumption, and therefore use polar plots to represent them: For light incoming from the left, the plot shows the probability that light will get scattered at each outgoing direction. We show for reference an isotropic phase function, meaning one that scatters equally in all directions.

Validation materials reference nanodispersions particles of known molecular type dispersed in liquid media very precise size distributions (NIST Traceable Standards) ground-truth materials: parameters predicted exactly by Mie theory polystyrene monodispersions aluminum oxide polydispersions Frisvad et al. 2007

Validation materials comparison of predicted and measured parameters polystyrene 1 polystyrene 2 polystyrene 3 aluminum oxide all parameters estimated within 4% error

Validation materials comparison of captured and rendered images in novel geometries captured rendered rendered with HG profiles polystyrene 1 images under unseen geometries predicted within 5% RMS error

Validation materials comparison of captured and rendered images in novel geometries captured rendered rendered with HG profiles polystyrene 2 images under unseen geometries predicted within 5% RMS error

Validation materials comparison of captured and rendered images in novel geometries captured rendered rendered with HG profiles polystyrene 3 images under unseen geometries predicted within 5% RMS error

Validation materials comparison of captured and rendered images in novel geometries captured rendered rendered with HG profiles aluminum oxide images under unseen geometries predicted within 5% RMS error

Measured materials highly scattering liquids highly absorbing liquids hand cream curacao highly scattering liquids olive oil shampoo robitussin mixed soap whole milk highly absorbing liquids solids milk soap wine liquid clay mustard coffee reduced milk

Measured materials whole milk reduced milk mustard shampoo hand cream liquid clay milk soap mixed soap glycerine soap robitussin coffee olive oil curacao wine

Measured materials

Measured materials

Measured materials

Discussion more interesting materials: heterogeneous volumes, polarization-dependent behavior, birefringent and fluorescing materials use of our optimization framework with other setups: alternative lighting (basis, adaptive, high-frequency), geometries, or imaging (transient imaging) simpler optimization and setup: trade-offs between accuracy, generality, mobility, and usability

Take-home messages dictionary-based material representation operator-theoretic framework for inverse volume rendering without approximations 𝜕𝐸 𝜕 𝑤 𝑞 =2 𝑤 𝑞 𝑆 𝐼−𝐾 −1 𝐿 𝑖 −𝜇 𝑆 𝐼−𝐾 −1 𝐾 𝑞 𝐼−𝐾 −1 𝐿 𝑖 physical setup implementation and accurate measurement of broader classes of materials The most commonly used phase function in computer graphics is the Henyey-Greenstein family. These are parabola-like lobes, controlled by a single parameter g. For different values of this parameter, we can have backward scattering, isotropic, and forward scattering lobes. Thinking of the phase function as a spherical distribution, the parameter g is equal to the first moment mu1, also commonly referred to as the average cosine

Acknowledgments Shuang Zhao Todd Zickler Kavita Bala Anat Levin

Thank you for your attention