Download presentation

Presentation is loading. Please wait.

Published byKylie Hoover Modified over 4 years ago

1
Simulation Photoréaliste de l'Éclairage en Synthèse d'Images Nicolas Holzschuch Habilitation à Diriger des Recherches Université Joseph Fourier Grenoble I

2
Overview Motivations Previous work Contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

3
Overview Motivations Previous work Contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

4
Photorealistic Image Synthesis Starting from the 3D model of a scene… © C. Soler, 2007. Model by L. Boissieux.

5
Photorealistic Image Synthesis …we get a picture of this virtual world © C. Soler, 2007. Model by L. Boissieux.

6
Motivations Why are we doing this? –Many applications: Virtual prototyping Cultural heritage Video games Special effects –Good looking pictures –Multi-disciplinar activity (CS, physics, math)

7
Motivations Why are we doing this? –Many applications: Virtual prototyping Cultural heritage Video games Special effects –Good looking pictures –Multi-disciplinar activity (CS, physics, math) Model from DaimlerChrysler / RealReflect IST project / picture by C. Soler

8
Motivations Why are we doing this? –Many applications: Virtual prototyping Cultural heritage Video games Special effects –Good looking pictures –Multi-disciplinar activity (CS, physics, math) © P. Müller

9
Motivations Why are we doing this? –Many applications: Virtual prototyping Cultural heritage Video games Special effects –Good looking pictures –Multi-disciplinar activity (CS, physics, math) © Valve, 2006

10
Motivations Why are we doing this? –Many applications: Virtual prototyping Cultural heritage Video games Special effects –Good looking pictures –Multi-disciplinar activity (CS, physics, math) © Lucas Digital Ltd., 2002

11
Motivations Why are we doing this? –Many applications: Virtual prototyping Cultural heritage Video games Special effects –Good looking pictures –Multi-disciplinar activity (CS, physics, math) Why is it difficult?

12
Motivations Why are we doing this? Why is it difficult? –Interactions between light and matter Complex local effects (reflections) –Self-dependent –Long distance interactions –Differences of scale/frequency –Time/quality compromise –Blurry effects are costlier –Too many computations?

13
Overview Motivations Timeline Selected contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

14
Timeline

15
Timeline: positions

16
Timeline: research themes

17
Timeline: projects and contracts

18
Overview Motivations Timeline Selected contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

19
Overview Motivations Timeline Selected contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

20
Wavelet Radiosity: credits 1993-1996: iMAGIS –Under the supervision of C. Puech, F. Sillion, G.Drettakis 1997-2000: ISA/LORIA –Working with J.-C. Paul, L. Alonso, F. Cuny, X. Cavin, C. Winkler, H. Barthélémy, S. Petitjean… 2000-2004: iMAGIS/GRAVIR –Working with L. Alonso (at ISA), C. Damez, F. Sillion…

21
Wavelet radiosity Extension of Hierarchical Radiosity –[Gortler et al., 1993] Uses wavelets for basis functions –Haar wavelets (pw. constant) equivalent to HR M 2, M 3 wavelets : linear, quadratic functions Higher order wavelets should work better –Better representation for smooth functions Experimental study: no, they dont! –[Wilmott & Heckbert, 1997]

22
Wavelet radiosity Everyone stopped working on WR –…except us at ISA/LORIA Point is: it should really work better –So why not? Experimental studies Everyone had used a na ï ve approach: –Just changing the basis function –But higher order wavelets are costlier n 2 for each patch n 4 for each interaction

23
Wavelet radiosity Na ï ve implementation doesn t work Must change the approach Adapt the algorithm: –Meshing strategy –No link storage + shooting –Refinement oracle –Visibility tests

24
Why didnt it work?

25
Wavelets need quad/triangular patches

26
Why didnt it work? Patches are then refined The initial triangulation binds the algorithm And each patch is n 2 times costlier

27
Extended mesh Keep the original mesh Adapt the radiosity algorithm [Holzschuch, Cuny, Alonso, EGWR 2000]

28
Extended mesh Separation between function and geometry

29
Link storage and consequences Biggest issue is memory cost –n 4 for each interaction –Solution: dont store them! –[ Stamminger, Schirmacher, Slusallek & Seidel, EG 1998] But think further: –Gathering/shooting –Gathering works better with HR –…except if you dont store the interactions We should use shooting! –…then it makes more sense not to store links!

30
Higher-Order Wavelets Other changes (refinement oracle, visibility) This time it works: –Smaller memory costs –Faster computations [Cuny, Holzschuch, Alonso, EG 2000]

31
Memory costs Observed memory costs for the different wavelet bases (rendering time follows similar curve) Consistent with [Willmott & Heckbert 1997]!

32
Higher order wavelets II Efficient global illumination algorithm –Uses higher-order wavelets –Robust, works on real-world scenes –Memory efficient So, is everything perfect? –Actually, no. –Shadow boundaries are a big issue

33
Shadow boundaries Discontinuity: the oracle keeps refining Each patch is n 2 times more costly –We lose all the ground we gained elsewhere

34
Discontinuity meshing Well-known method: –Shadow boundary position is known –Cut the mesh according to these discontinuities –Refine the resulting mesh Impossible with wavelet radiosity: –Wavelets only defined over quads/triangles –…but we already have a workaround!

35
DM and Higher-Order Wavelets [Holzschuch and Alonso, EGSR 2004] Discontinuities are introduced only if necessary –Use regular subdivision as much as possible Modified refinement oracle: –If refinement could have been avoided using DM –Then Subdivide patch using discontinuity –Embed the new patches inside quads –Regular subdivision over the new quads

36
DM and Higher-Order Wavelets M2 wavelets, point light source, no DM [Holzschuch and Alonso, EGSR 2004]

37
DM and Higher-Order Wavelets M2 wavelets, point light source, with DM [Holzschuch and Alonso, EGSR 2004]

38
DM and Higher-Order Wavelets M3 wavelets, area light source, with DM [Holzschuch and Alonso, EGSR 2004]

39
Wavelet Radiosity: summary Compact representation of illumination Applications: –Cooperation with Nancy School of Architecture –We tested our algorithms on real-world scenes –Great for robustness Research: –Real scientific problems came up –We had to solve them, resulting in nice papers Transfer: –The software was transferred to a start-up –VSP-Technology, headed by F. Cuny

40
Wavelet radiosity: looking back Shadow boundaries is an interesting issue Without DM: –90 % of computation time for shadows With DM: –90 % of implementation time for shadows Only important for (direct lighting+directly visible) Indirect lighting can use low quality version Waste of time?

41
Overview Motivations Timeline Contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

42
Frequency Analysis: credits Joint research work with MIT/CSAIL At MIT/CSAIL: F. Durand, E. Chan At ARTIS/GRAVIR: C. Soler, F. Sillion Financed by an INRIA Équipe Associée [Durand, Holzschuch, Soler, Chan, Sillion, SIGGRAPH 2005]

43
Illumination effects Blurry reflections: From [Ramamoorthi and Hanrahan 2001]

44
Illumination effects Shadow boundaries: © U. Assarsson 2005. Point light source Area light source

45
Illumination effects Indirect lighting is usually blurry: Complete lighting © C. Soler 2005.

46
Illumination effects Indirect lighting is usually blurry: Indirect lighting only Direct lighting only © C. Soler 2005.

47
Frequency aspects of light transport Blurriness = frequency content –Sharp variations: high frequency –Smooth variations: low frequency All effects are expressed as frequency content: –Diffuse shading: low frequency –Shadows: introduce high frequencies –Indirect lighting: tends to be low frequency

48
Problem statement How does light interaction in a scene explain the frequency content? Theoretical framework: –Understand the frequency spectrum of the radiance function –From equations of light transport

49
Unified framework: Spatial frequency (e.g. shadows, textures) Angular frequency (e.g. blurry highlight)

50
Disclaimer: not Fourier optics We do not consider wave optics, interference, diffraction Only geometrical optics From [Hecht]

51
Disclaimer: not Fourier optics We do not consider wave optics, interference, diffraction Only geometrical optics From [Hecht]

52
Overview Previous work Our approach: –Local light field –Transformations on local light field Case studies: –Diffuse soft shadows –Adaptive shading sampling Conclusions and future directions

53
Overview Previous work Our approach: –Local light field –Transformations on local light field Case studies: –Diffuse soft shadows –Adaptive shading sampling Conclusions and future directions

54
Previous work Vast body of literature: –Light field sampling –Perceptually-based rendering –Wavelets for Computer Graphics –Irradiance caching –Photon mapping –… We focus on frequency analysis in graphics: –Light field sampling –Reflection as a convolution

55
Light field sampling [Chai et al. 00, Isaksen et al. 00, Stewart et al. 03] –Light field spectrum as a function of object distance –No BRDF, occlusion ignored From [Chai et al. 2000]

56
Signal processing for reflection [Ramamoorthi & Hanrahan 01, Basri & Jacobs 03] Reflection on a curved surface is a convolution Direction only From [Ramamoorthi and Hanrahan 2001]

57
Overview Previous work Our approach: –Local light field –Transformations on local light field Case studies: –Diffuse soft shadows –Adaptive shading sampling Conclusions and future directions

58
Our approach Light sources are input signal Interactions are filters/transforms –Transport –Visibility –BRDF –Etc.

59
Our approach Light sources are input signal Interactions are filters/transforms –Transport –Visibility –BRDF –Etc. Light source signal

60
Our approach Light sources are input signal Interactions are filters/transforms –Transport –Visibility –BRDF –Etc. Light source signal Transport

61
Our approach Light sources are input signal Interactions are filters/transforms –Transport –Visibility –BRDF –Etc. Light source signal Signal 1 Transport

62
Our approach Light sources are input signal Interactions are filters/transforms –Transport –Visibility –BRDF –Etc. Light source signal Transport Occlusion Signal 2

63
Our approach Light sources are input signal Interactions are filters/transforms –Transport –Visibility –BRDF –Etc. Light source signal Transport Occlusion Signal 3 Transport

64
Our approach Light sources are input signal Interactions are filters/transforms –Transport –Visibility –BRDF –Etc. Light source signal Transport Occlusion Signal 4 Transport BRDF

65
Local light field 4D light field, around a central ray Central ray

66
Local light field 4D light field, around a central ray We study its spectrum during transport

67
Local light field 4D light field, around a central ray We study its spectrum during transport

68
Local light field 4D light field, around a central ray We study its spectrum during transport

69
Local light field We give explanations in 2D –Local light field is therefore 2D See SIGGRAPH paper for extension to 3D

70
Local light field parameterization Space and angle space angle Central ray

71
Local light field representation Density plot: Area light source Space Angle

72
Local light field Fourier spectrum We are interested in the Fourier spectrum of the local light field Also represented as a density plot

73
Local light field Fourier spectrum Spatial frequency Angular frequency Spectrum of an area light source:

74
Basics of Fourier analysis Spectrum corresponds to blurriness: –Sharpest feature has size 1/F max Convolution theorem: –Multiplication convolution Classical spectra: –Box sinc –Dirac constant

75
Overview Previous work Our approach: –Local light field –Transformations on local light field Case studies: –Diffuse soft shadows –Adaptive shading sampling Conclusions and future directions

76
Overview Previous work Our approach: –Local light field –Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions

77
Example scene Blockers Light source Receiver

78
Overview Previous work Our approach: –Local light field –Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions

79
Transport in free space Shear Space Angle

80
Transport in free space Shear Space Angle Shear Spatial frequency Angular freq. Spatial frequency Angular freq.

82
Transport Shear This is consistent with light field spectra [Chai et al. 00, Isaksen et al. 00] From [Chai et al. 2000]

83
Overview Previous work Our approach: –Local light field –Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions

84
Occlusion: multiplication Occlusion is a multiplication in ray space –Convolution in Fourier space Creates new spatial frequency content –Related to the spectrum of the blockers

86
Overview Previous work Our approach: –Local light field –Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions

88
Overview Previous work Our approach: –Local light field –Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions

89
BRDF integration Outgoing light: –Integration of incoming light times BRDF Outgoing light Incoming light BRDF

90
BRDF integration Ray-space: convolution –Outgoing light: convolution of incoming light and BRDF –For rotationally-invariant BRDFs Fourier domain: multiplication –Outgoing spectrum: multiplication of incoming spectrum and BRDF spectrum

91
BRDF in Fourier: multiplication = BRDF is bandwidth-limiting in angle

92
Example: diffuse BRDF Convolve by constant: –multiply by Dirac –Only spatial frequencies remain =

93
Overview Previous work Our approach: –Local light field –Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions

94
Curved receiver Reduce to the case of a planar surface: –Unroll the curved receiver Equivalent to changing angular content: –Linear effect on angular dimension –No effect on spatial dimension Shear in the angular dimension

96
Transforms: summary Radiance/FourierEffect TransportShear OcclusionMultiplication/ConvolutionAdds spatial frequencies BRDFConvolution/MultiplicationRemoves angular frequencies CurvatureShear

97
More effects in paper Cosine term (multiplication/convolution) Fresnel term (multiplication/convolution) Texture mapping (multiplication/convolution) Central incidence angle (scaling) Separable BRDF Spatially varying BRDF (semi-convolution) …and extension to 3D

98
Overview Previous work Our approach: –Local light field –Transformations on local light field Case studies: –Diffuse soft shadows –Adaptive shading sampling Conclusions and future directions

99
Overview Previous work Our approach: –Local light field –Transformations on local light field Case studies: –Diffuse soft shadows –Adaptive shading sampling Conclusions and future directions

100
Case study: diffuse soft shadows Decreasing blockers size: –First high-frequencies increase –Then only low frequency –Non-monotonic behavior

101
Diffuse soft shadows (2) Occlusion : convolution in Fourier –creates high frequencies –Blockers scaled down spectrum scaled up Fourier space v (angle) x (space) Fourier space v (angle) x (space) blocker spectrum

102
Diffuse soft shadows (3) Transport after occlusion: –Spatial frequencies moved to angular dimension Diffuse reflector: –Angular frequencies are cancelled

103
Diffuse soft shadows (3) Transport after occlusion: –Spatial frequencies moved to angular dimension Diffuse reflector: –Angular frequencies are cancelled

104
Frequency Analysis: summary Framework for frequency analysis of light transport Gives high frequencies locations Occlusion creates high frequencies Transport + reflection removes HFs High frequencies mostly if: –Small transport (quasi-contact) –Specular reflection (or highly glossy) –Point light source

105
Overview Motivations Timeline Contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

106
GPU-based algorithms: credits Specular reflections: D. Roger Real-time soft shadows: L. Atty, M. Lapierre, C. Hansen, J.-M. Hasenfratz, F. Sillion Contact shadows: M. Malmer, F. Malmer, U. Assarsson Global Illumination: J. Kontkannen, E. Turquin, F. Sillion

107
GPU-based algorithms GPUs are highly powerful machines –Widely available –Used in the industry Video games, cinema But they also have their limitations –Memory accesses, per-pixel computations… They work better on local effects –Direct lighting, shadow boundaries… 2003: AS CNRS Real-Time Rendering

108
Looking back… Radiosity: –Lots of work for direct lighting+directly visible High frequency effects in illumination: –Small transport –Shadow boundaries –Specular reflections –…mostly local or semi-local effects Involve a small subset of the scene for each pixel

109
Think globally Our hard points: –Direct lighting+directly visible –Local or semi-local effects … all correspond to the forte of GPUs Main idea: –use GPUs for HF effects+directly visible –Something else (CPU) for low frequency effects

110
GPU-based algorithms Using GPUs for local/semi-local effects: –Specular reflections –Shadows (soft and hard) –Contact shadows Global illumination in static scenes: –GPU for direct lighting, CPU for indirect

111
Specular reflections Vertex based reflections For each vertex in the scene –Compute its reflected position –Find light paths of extremal length –Using gradient descent [Roger and Holzschuch, EG 2006]

112
Specular reflections Our methodEnv. mapping Reference [Roger and Holzschuch, EG 2006]

113
Real-time soft shadows Shadows: important issue for realism Real-time hard shadows: easy with GPU Soft shadows: still an open problem EG 2003: State-of-the-Art Report –Hasenfratz, Lapierre, Holzschuch, Sillion

114
Soft Shadow Maps Discretize occluders using shadow map Compute soft shadow for each surfel –Easy because surfel axis-aligned with LS Add the contributions of each surfel [Atty, Holzschuch, Lapierre, Hasenfratz, Hansen, Sillion, CGF 2006]

115
Contact shadows Shadows cast on nearby objects: [Malmer, Malmer, Assarsson, Holzschuch, JGT 2007]

116
Contact shadows Equivalent to ambient occlusion –Important for realism –Low-cost effect, but with high impact Low frequency effect Several techniques for storing AO field –Moving with the object –Casting shadows on close objects –Indexed by direction: k*n 2 storage –Post-processing to extract AO values

117
Contact shadows Our solution:

118
Contact shadows Store raw AO values, un-processed 3D grid: n 3 storage No post-processing required Why is it interesting? –Low frequency effect: small values of n –Raw values stored: no constant –Similar memory costs, faster rendering

119
Global Illumination in static scenes Direct lighting using GPU Indirect lighting: –Wavelet representation –precomputed Global Transport Operator [Kontkanen, Turquin, Holzschuch, Sillion, EGSR 2006]

120
Global Illumination in static scenes Wavelet basis for radiance Local Transport Operator –expressing a single light bounce –precomputed Global Transport Operator –Built from LTO, using Neumann series For each frame: –Compute direct lighting, projected on WT basis –Apply GTO to direct lighting –Result is indirect lighting

121
Global Illumination in static scenes DirectIndirect Global

122
GPU-based algorithms: summary GPU techniques for local/semi-local effects: –Specular reflections –Soft shadows –Contact shadows Interactive global illumination –GPU for direct lighting, CPU for indirect –Limited to static scenes

123
Overview Motivations Previous work Contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

124
Photorealistic Image Synthesis Contributions: –Wavelet radiosity –Frequency analysis –GPU-based algorithms Several applications and contracts: –VSP-Tech, SIMULGEN, Eden Games…

125
Future work: Real Time Global Illumination Frequency analysis in real-time –Used to guide computations GPU for local and semi-local HF effects –Direct lighting, shadows –Specular/glossy reflections –Occlusion and reflection fields CPU for other effects –Indirect lighting –Low Frequency effects Lots of applications and contacts: –Video games (GENAC II) –Movies

126
Future work: Image-Based Rendering We want more realistic pictures Requires more realistic models –Hard to create from scratch –Image-Based Modeling for acquisition Opens several new directions of research: –Easy model acquisition –Real-time rendering of the model –Re-lighting of the model Need for cooperation: –2 ANR projects submitted this year Lots of potential applications

127
Thank you!

128
Acknowledgements Everyone I worked with In addition to those already mentioned: –iMAGIS, 1993-1996: A. Lamouret, F. Faure, JC Lombardo, JD Gascuel, N. Tsingos, M. Desbrun, S. Rivière, R. Orti, JC Vedel, A. Verroust, MP Cani, A. Opalach,… –UCT, 1996-1997: E. Blake, S. Nirenstein, D. Cook, A. Secchia,… –ISA, 1997-2000: S. Lazard, S. Merzouk, K. Tombre, H. Everett, MO Berger, A. Tabbone… –iMAGIS, 2000-2003: J. Thollot, X. Decoret, J. Turbet, F. Neyret, L. Reveret, G. Debunne, S. Grabli… –ARTIS, 2004-2007: A. Martinet, L. Boissieux, V. Ostromoukhov, E. Eisemann, P. Barla, L. Baboud, H. Bezerra, A. Bousseau, A. Orzan, T. Stein, K. Smith, F. Moulin, D. Vanderhaeghe…

Similar presentations

OK

3D Graphics Processor Architecture Victor Moya. PhD Project Research on architecture improvements for future Graphic Processor Units (GPUs). Research.

3D Graphics Processor Architecture Victor Moya. PhD Project Research on architecture improvements for future Graphic Processor Units (GPUs). Research.

© 2018 SlidePlayer.com Inc.

All rights reserved.

To make this website work, we log user data and share it with processors. To use this website, you must agree to our Privacy Policy, including cookie policy.

Ads by Google