1. Optimal Illumination for Image and Video Relighting Francesc Moreno-Noguer Shree K. Nayar Peter N. Belhumeur Department of Computer Science – Columbia University European Conference on CVMP November 2005, London, UK

2. Objective

6. Relighting Static Objects

31. ?

32. ? xxxxxx xxxxx xxxxx xxxxx x x x

33. xxxxxx xxxxx xxxxx xxxxx x x x

34. Relighting Moving Objects

59. ?

60. ? xxxxxx xxxxx xxxxx xxxxx x x x

61. xxxxxx xxxxx xxxxx xxxxx x x x

62. xxxxxx xxxxx xxxxx xxxxx x x x

83. = I = L Relighting Moving Objects

84. = I align = L Relighting Moving Objects

85. L new Relighting Moving Objects = I align = L

86. L new ? I new Relighting Moving Objects = I align = L

87. Relighting Moving Objects I new = I align pinv (L) L new

88. +++ = Relighting Moving Objects

89. = I new = I align pinv (L) L new +++ =

90. What is the optimal lighting?

91. L opt ?

92. Sources of error in video relighting Sub-basis error Ground Truth

93. Sources of error in video relighting Sub-basis error Ground Truth Rendered 3 basis Error 3 basis

94. Sources of error in video relighting Sub-basis error Ground Truth Rendered 3 basis Error 3 basis

95. Sources of error in video relighting Sub-basis error Ground Truth Rendered 3 basis Error 3 basis Rendered 6 basis Error 6 basis

96. Sources of error in video relighting Sub-basis error Ground Truth Rendered 3 basis Error 3 basis Rendered 6 basis Error 6 basis

97. Sources of error in video relighting Alignment error Reference frame

98. Sources of error in video relighting Alignment error Reference frame Frame t+1

99. Sources of error in video relighting Alignment error Reference frame Alignment errors Frame t+1

100. Sources of error in video relighting Alignment error Reference frame Alignment errors Frame t+1 Frame t+2

101. Sources of error in video relighting Alignment error Reference frame Alignment errors Frame t+1 Frame t+2

102. Sources of error in video relighting Alignment error Reference frame Alignment errors Frame t+1 Frame t+2 Frame t+3

103. Sources of error in video relighting Alignment error Reference frame Alignment errors Frame t+1 Frame t+2 Frame t+3

104. Sources of error in video relighting Alignment error Reference frame Alignment errors Frame t+1 Frame t+2 Frame t+3 Frame t+4

105. Sources of error in video relighting Alignment error Reference frame Alignment errors Frame t+1 Frame t+2 Frame t+3 Frame t+4

106. Sources of error in video relighting Alignment error Reference frame Alignment errors Frame t+1 Frame t+2 Frame t+3 Frame t+4 Frame t+5

107. Sources of error in video relighting Alignment error Reference frame Alignment errors Frame t+1 Frame t+2 Frame t+3 Frame t+4 Frame t+5

108. Sources of error in video relighting Orientation error

109. Sources of error in video relighting Orientation error

110. Sources of error in video relighting Reference frame Orientation error

111. Sources of error in video relighting Reference frame Perfectly aligned frame Orientation error

112. Sources of error in video relighting Reference frame Error after perfect alignment Perfectly aligned frame Orientation error

113. Sources of error in video relighting Sub-basis error Alignment error Orientation error

114. Criterion for selecting the OLB Minimize the sub-basis error BUT Keep the number of reference images to a minimum

115. Object independent lighting bases Spherical Harmonic basis (SHLB)

116. Object independent lighting bases Spherical Harmonic basis (SHLB) Fourier basis (FLB)

117. Object independent lighting bases Fourier basis (FLB) Haar basis (HaLB) Spherical Harmonic basis (SHLB)

118. Object dependent lighting basis Optimal Lighting Basis (OLB) Computed as a simple calibration procedure before acquisition Initial acquisition of the images of the still object under single light sources Compute SVD over the matrix of images Computed as a simple calibration procedure before acquisition Initial acquisition of the images of the still object under single light sources Compute SVD over the matrix of images... A=A=

119. A = UDW T Object dependent lighting basis

120. A = UDW T... A=A= Object dependent lighting basis

121. A = UDW T... A=A= U=U= Object dependent lighting basis

122. A = UDW T... A=A= U=U= W = T Object dependent lighting basis

123. A = UDW T... A=A= U=U= W = T L opt = Object dependent lighting basis

124. Synthetic results Optimal Lighting Basis (OLB) Spherical Hamonic (SHLB)

125. Synthetic results Optimal Lighting Basis (OLB) Spherical Hamonic (SHLB) Optimal Lighting Basis (OLB) Fourier (FLB) Haar (HaLB)

126. Synthetic results – Static objects Ground Truth

127. Synthetic results – Static objects Fourier 16 basis Error Fourier 16 basis Ground Truth

128. Synthetic results – Static objects Fourier 16 basisOLB 16 basis Error Fourier 16 basis Error OLB 16 basis Ground Truth

129. Synthetic results – Static objects Ground Truth

130. Synthetic results – Static objects Haar 3 basis Error Haar 3 basis Ground Truth

131. Synthetic results – Static objects Haar 3 basisOLB 3 basis Error Haar 3 basis Error OLB 3 basis Ground Truth

132. Synthetic results – Static objects Ground Truth

133. Synthetic results – Static objects Sph. Harm. 7 basis Error Sph. Harm. 7 basis Ground Truth

134. Synthetic results – Static objects Sph. Harm. 7 basis OLB 7 basis Error Sph. Harm. 7 basis Error OLB 7 basis Ground Truth

135. Synthetic results – Static objects Ground Truth

136. Synthetic results – Static objects Sph. Harm. 3 basis Error Sph. Harm. 3 basis Ground Truth

137. Synthetic results – Static objects Sph. Harm. 3 basisOLB 3 basis Error Sph. Harm. 3 basis Error OLB 3 basis Ground Truth

138. Synthetic results – Static objects

139. Dragon Example:

140. Synthetic results – Static objects 8 OLB = 8 x 1.6 SHLB 13 SHLB Dragon Example:

141. Synthetic results – Static objects 8 OLB = 8 x 1.6 SHLB 13 SHLB 8 OLB = 8 x 1.9 SHLB 15 FLB Dragon Example:

142. Synthetic results – Static objects Dragon Example: 8 OLB = 8 x 1.6 SHLB 13 SHLB 8 OLB = 8 x 1.9 SHLB 15 FLB 8 OLB = 8 x 3.2 SHLB 25 HaLB

143. Frame #50 Synthetic results – Moving objects Frame #1 OLB vs. SHLB

144. Frame #50 Synthetic results – Moving objects Frame #1 9 OLB OLB vs. SHLB

145. Frame #50 Synthetic results – Moving objects Frame #1 9 OLB 16 SHLB OLB vs. SHLB

146. Frame #50 Synthetic results – Moving objects Frame #1 OLB vs. Fourier & Haar

147. Frame #50 Synthetic results – Moving objects Frame #1 3 OLB OLB vs. Fourier & Haar

148. Frame #50 Synthetic results – Moving objects Frame #1 3 OLB 5 FLB OLB vs. Fourier & Haar

149. Frame #50 Synthetic results – Moving objects Frame #1 3 OLB 5 FLB OLB vs. Fourier & Haar 16 HaLB

150. Real Experiments - Setup

151. Real Experiments - Procedure OLB computation 1.1. Static scene, single lights 1.2. SVD  OLB Video adquisition 2. Moving scene, OLB Postprocessing 3.1. Alignment 3.2. Relighting

152. Real Experiments – Tennis Ball

154. Real Experiments – Face

157. Real Experiments – Room corner

161. Conclusions Proposed an “object-dependent” lighting basis appropriate for relighting tasks. Minimizes the number of reference images needed for relighting. Effective method for video relighting. Proposed an “object-dependent” lighting basis appropriate for relighting tasks. Minimizes the number of reference images needed for relighting. Effective method for video relighting.

162. Future work Use faster cameras. Use “motion-dependent” relighting bases. Use faster cameras. Use “motion-dependent” relighting bases. Frame t+1 Frame t+2 Frame t+3 Frame t+4 Frame t+5 Frame t Slow motion More accurate results

163. Future work Use faster cameras. Use “motion-dependent” relighting bases. Use faster cameras. Use “motion-dependent” relighting bases. Fast motion Frame t+1 Frame t+2 Frame t+3 Frame t+4 Frame t+5 Frame t Slow motion Reduce alignment & orientation errors More accurate results

164. Acknowledgments Ko Nishino for his useful comments. Anne Fleming for her help in the edition of the videos. All the people from the “Laboratory for the Study of Visual Appearance” and “Columbia Automated Vision Environment's” at Columbia University. Cyberware for the 3D models. Funding Sponsors: Spanish Ministry of Science & Technology National Science Foundation Ko Nishino for his useful comments. Anne Fleming for her help in the edition of the videos. All the people from the “Laboratory for the Study of Visual Appearance” and “Columbia Automated Vision Environment's” at Columbia University. Cyberware for the 3D models. Funding Sponsors: Spanish Ministry of Science & Technology National Science Foundation