1. CS 395: Adv. Computer Graphics Introduction to Image-Space Methods Watt & Watt: Chapter 14 + readings Jack Tumblin jet@cs.northwestern.edu

2. Image Space Methods A ‘Digital Image’ = a data structure for displayed imagesA ‘Digital Image’ = a data structure for displayed images ?What can it do, besides display??What can it do, besides display? what can we change? what can we change? –Position; offset, stretch, squeeze, smear, cut,… Try It! (Java Applets) http://www.mrl.nyu.edu/~hertzman/nudge/ Try It! (Java Applets) Try It! (Java Applets) –Combine: Add, Subtract/ Overlay / Compositing –Separate: matte or ‘segmenting’ –Elaborate: make ‘more of the same’ picture

3. ‘Real’ Images: Lens-Focused Light 3D  2D Light Intensity Map I(x,y)3D  2D Light Intensity Map I(x,y) Angle( ,  )  Position(x,y) Angle( ,  )  Position(x,y) ‘Blurring’—sharpness set by focus, lens quality‘Blurring’—sharpness set by focus, lens quality I(x,y) Image Plane Intensity Angle( ,  ) Position(x,y) Viewed Scene

4. ‘Digital’ Images: 2D Grid of Numbers NO intrinsic meaning, but...NO intrinsic meaning, but... Widely assumed to representWidely assumed to represent –Point Samples of a “smoothed” 2D intensity surface –Uniform sampling pattern (but not always) x y (!weasel-word!)

5. Real vs. Digital Real Image == 2D Light Intensity map: I(x,y)Real Image == 2D Light Intensity map: I(x,y) Digital Image == 2D grid of numbers: I(m,n)Digital Image == 2D grid of numbers: I(m,n) I(x,y) x,y I(m,n) m,n (pixels) (pixels)

6. Digital Images As Vectors ‘Stack up’ pixel values: VERY LONG vector‘Stack up’ pixel values: VERY LONG vector –1 digital image == 1 point in N-dim. Space –Nearby points == Similar images –All possible digital images: a grid of N-D points –All possible real images: Unbounded dimensionsUnbounded dimensions no intensity quantizing!no intensity quantizing!I(m,n)m,n I 00 I 01 I 02 … I 10 I 11 I 12 … I = I 00 I 01 I 02 I 03 I 04 I 05

7. Digital Images As Vectors Sensible element-by-element operations:Sensible element-by-element operations: –Add, subtract, scale two images: 0.5 + (I 1 - I 2 ) = out 0.5 + (I 1 - I 2 ) = out - = I1I1I1I1 I2I2I2I2out

8. Digital Images As Vectors Sensible element-by-element operations:Sensible element-by-element operations: –  - Weighted sum of two images: 0 <  < 1 + =  I1 I1 I1 I1 (1-  ) I 2 out

9. Digital Images As Vectors Compositing:  : Use an  image  : Use an  image as ‘opacity’: + = .* I 1 (1-  ).* I 2 out

10. ! NOT EQUIVALENT ! ? Digital image: pixel-by-pixel operations ? ? Real Image: point-by-point operations ? Source of much frustration & error! Digital Image == Real Image Digital Image == Real Image Also see Alvy Ray Smith’s article entitled: “A pixel is NOT a little square, a pixel is NOT a little square, a pixel is NOT a little square, a pixel is NOT a little square!” found here. a pixel is NOT a little square!” found here.found here.found here.

11. Real Digital –Image Intensity –Image Spectral Distribution (‘color’) –Image Geometry (Position,Angle, Distortion, …) –Image Resolution (‘sharpness’ or ‘focus’) Real Images: Smooth, Real Images: Smooth, Continuous, Continuous, Variable, Variable, Complete CompleteUnlimited… Digital Images: Digital Images: Sampled, Sampled, Discrete, Discrete, Quantized, Quantized, Fixed Fixed Limited … Limited …                      

12. Conversion Method? Real Image == 2D Light Intensity map: I(x,y)Real Image == 2D Light Intensity map: I(x,y) Digital Image == 2D grid of numbers: I(m,n)Digital Image == 2D grid of numbers: I(m,n)I(x,y)x,y ????I(m,n)m,n

13. Digital  Real : ‘Reconstruction’ A pixel sets strength of a real Basis function (aka ‘reconstruction filter’s impulse response’) I(m,n) m,n 0 0.5 -0.5 1.0 I(x,y) x,y ‘Box’

14. Digital  Real : ‘Reconstruction’ A pixel sets strength of a real Basis function (aka ‘reconstruction filter’s impulse response’) I(m,n) m,n 0 1.0 1.0 I(x,y) x,y ‘Linear’

15. Digital  Real : ‘Reconstruction’ A pixel sets strength of a real Basis function (aka ‘reconstruction filter’s impulse response’) I(m,n) m,n 0 1.0 1.0 I(x,y) x,y 2.0 ‘Cubic’

16. Real  Digital: ‘Sampling’ If I(x,y) is ‘smooth enough’ already,If I(x,y) is ‘smooth enough’ already, ?Why not just grab I(x,y) values at (m,n) ??Why not just grab I(x,y) values at (m,n) ?I(x,y)x,y      I(m,n) m,n

17. ‘Smooth enough’ to Sample? Because ‘smooth enough’ is undefined;Because ‘smooth enough’ is undefined; Samples may hit spurious peaks, valleysSamples may hit spurious peaks, valleys I(x,y) x,y I(m,n) m,n BAD!

18. Real  Digital : ‘Pre-filter’ ‘Smoothing’ == Linear Pre-filter: Pixel = local weighted average of real image around pixel sample point I(x,y) x,y m,n 0 1.0 1.0 2.0 ‘Cubic’weights Weight1.0

19. Pre-Filter functions Pre-filters == anti-aliasing in Comp GraphicsPre-filters == anti-aliasing in Comp Graphics SHOULD be done bySHOULD be done by continuous integration, USUALLY done byUSUALLY done by super-sampling super-sampling ? Quality Measures ? 0 1.0 1.0 2.0 ‘Bi-cubic’filter 0 1.0 1.0 ‘Bi-linear’filter 0 0.5 -0.5 1.0 ‘Box’filter

20. ‘Smoothness Measures’ Sine wave magic:Sine wave magic: 1)ALL (continuous) functions are a (possibly infinite) weighted sum of sinusoids: f(x) = A 0 sin(w 0 x) + B 0 cos(w 0 x)+ A 1 sin(w 1 x) + B 1 cos(w 0 x)+… 2)For ALL FILTERS (box, linear, cubic,…, anything…) sinusoid in  scaled, shifted sinusoid out 3)High-Frequency Sinusoids hold ALL the sampling trouble!

21. 1-D Aliasing Example Real image: I(x,y) =sin(2  fx) f=3.0 Real Aliasing: when bad conversion makes scrambles the frequency of sinusoids

22. 1-D Aliasing Example Real image: I(x,y) =sin(2  fx) f=3.0 Sample at: x(n) = n / 36.0 (~12 samples per cycle) Sample Real

23. Aliasing Example Real image: I(x,y) =sin(2  fx) f=3.0 Sample at: x(n) = n / 36.0 (~12 samples per cycle) Reconstruct: (use ‘box’ filter)  approximates f=3.0 Sample Real Display

24. Aliasing Example Real image: I(x,y) =sin(2  fx) f=3.0 Sample at: x(n) = n / 36.0 (~12 samples per cycle) Reconstruct: (use ‘box’ filter)  approximates f=3.0 Another Real Image: I(x,y) = sin(2  fx) where f= 39.0 Sample Real Display Real

25. Aliasing Example Real image: I(x,y) =sin(2  fx) f=3.0 Sample at: x(n) = n / 36.0 (~12 samples per cycle) Reconstruct: (use ‘box’ filter)  approximates f=3.0 Another Real Image: I(x,y) = sin(2  fx) where f= 39.0 Sample at: x(n) = n / 36.0 Sample Real Display Sample Real

26. Aliasing Example Real image: I(x,y) =sin(2  fx) f=3.0 Sample at: x(n) = n / 36.0 (~12 samples per cycle) Reconstruct: (use ‘box’ filter)  approximates f=3.0 Another Real Image: I(x,y) = sin(2  fx) where f= 39.0 Sample at: x(n) = n / 36.0 Reconstruct: (use ‘box’ filter)  approximates f=3.0 !! Sample Real Display Sample Real Display

27. Aliasing Example Real image: I(x,y) =sin(2  fx) f=3.0 Sample at: x(n) = n / 36.0 (~12 samples per cycle) Reconstruct: (use ‘box’ filter)  approximates f=3.0 Another Real Image: I(x,y) = sin(2  fx) where f= 39.0 Sample at: x(n) = n / 36.0 Reconstruct: (use ‘box’ filter)  approximates f=3.0 Sample Real Display Sample Real Display

28. Aliasing Example Real image: I(x,y) =sin(2  fx) f=3.0 Sample at: x(n) = n / 36.0 (~12 samples per cycle) Reconstruct: (use ‘box’ filter)  approximates f=3.0 Another Real Image: I(x,y) = sin(2  fx) where f= 39.0 Sample at: x(n) = n / 36*4 = n / 144.0 Reconstruct: (use ‘box’ filter)  approximates f=39.0 Sample Real Display

29. Conclusion: Image-space manipulations must explicitly address real-digital (discrete-continuous) conversions to avoid aliasing artifacts.Image-space manipulations must explicitly address real-digital (discrete-continuous) conversions to avoid aliasing artifacts. Digital images are compact, indirect, ambiguous descriptions of light distributionsDigital images are compact, indirect, ambiguous descriptions of light distributions

30. Image Space Methods Outline:Outline: –Definitions, limitations (What’s a ‘Digital Image’? Can they really represent ANY image? (no). –The Image Vector –What can we do with Digital Images? Intensity math, editingIntensity math, editing Image Comparisons / Similarity MeasuresImage Comparisons / Similarity Measures Pattern finding, matching, learning, …Pattern finding, matching, learning, … Spatial math, editing*Spatial math, editing* –*Pitfall: Real  Digital Images: Discrete  Continuous Representations –Conversions ‘Reconstruction’: digital  real each pixel scales a basis function‘Reconstruction’: digital  real each pixel scales a basis function ‘Sampling’: real  digitaleach found by basis function (wieghted sum of real image neighborhood)‘Sampling’: real  digitaleach found by basis function (wieghted sum of real image neighborhood)

31. Image Space Methods Outline (cont’d);Outline (cont’d); –‘Aliasing Artifacts’ == Bad Conversions Somewhere: not enough smoothing –Sine Wave Magic! Fourier’s Big Idea 1: all functions are a unique weighted sum of sinusoids! (square wave example)Fourier’s Big Idea 1: all functions are a unique weighted sum of sinusoids! (square wave example) Fourier’s Big Idea 2: no weighted-average filter can change a sinusoid’s shape! (sine wave in  sine-wave out (but it might be shifted & scaled))Fourier’s Big Idea 2: no weighted-average filter can change a sinusoid’s shape! (sine wave in  sine-wave out (but it might be shifted & scaled)) –Infinite sharpness? Infinitely high-frequ Why does it matter? We can’t reconstruct a sine-wave changing > ½ sample rate.Why does it matter? We can’t reconstruct a sine-wave changing > ½ sample rate. Math jargon; sinewaves are eigenfunctions of the space of all linear functionals.Math jargon; sinewaves are eigenfunctions of the space of all linear functionals. Suppose you plot the amount of shift & amount of scale for all possible sinusoids: Fourier Transform!Suppose you plot the amount of shift & amount of scale for all possible sinusoids: Fourier Transform!

32. Aliasing Example Real image: I(x,y) =sin(2  fx) f=3.0 Sample at: x(n) = n / 36.0 (~12 samples per cycle) Reconstruct: (use ‘box’ filter)  approximates f=3.0 Another Real Image: I(x,y) = sin(2  fx) where f= 39.0 Sample at: x(n) = n / 36.0 Reconstruct: (use ‘box’ filter)  approximates f=3.0 Sample Real Display Sample Real Display

33. SAFETY COPY

34. Reconstruction: Digital  Real How to ‘Read Between the Pixels’

35. Pre-Filter: Real  Digital

36. Image Space Methods Intro: What can we do with images?Intro: What can we do with images? Re-sampling:Continuous & Discrete FiltersRe-sampling:Continuous & Discrete Filters –Interpolation: What’s ‘in-between’ a pixel? –Antialiasing: What makes a good sampling? –Math Rigor: the Fourier Transform –Examples: box, triangle, Mitchell-Netravali Compositing: TransparencyCompositing: Transparency Un-Compositing: Matte SeparationsUn-Compositing: Matte Separations Further extensions:Further extensions: –Environment Matte –Polynomial Texture Map –Texture Elaboration, Image Inpainting(Sigg2000)