2. Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon

3. Overview  Sampling in Television and Film  The problems of aliasing  Filtering requirements  Conversion between differing formats  Problems that can occur  How we can mitigate some of the problems and maintain or improve quality

4. Sampling Theory  Harry Nyquist – 1889 to 1976  “The number of independent pulses that can be put through a telegraph channel per unit time is limited to twice the bandwidth of the channel”

5. Sampling Theory  Harry Nyquist – 1889 to 1976  “The number of independent pulses that can be put through a telegraph channel per unit time is limited to twice the bandwidth of the channel”  Later Nyquist-Shannon  “Exact reconstruction of a continuous-time baseband signal from its samples is possible if the signal is bandlimited and the sampling frequency is greater than twice the signal bandwidth”

6. Sampling Theory Frequency Amplitude Fs

7. Sampling Theory  Audio:  20kHz bandwidth, Fs = 44.1kHz, 48kHz Frequency Amplitude Fs

8. Sampling Theory  Audio:  20kHz bandwidth, Fs = 44.1kHz, 48kHz  Video:  5.75MHz bandwidth, Fs = 13.5MHz  30MHz bandwidth, Fs = 74.25MHz Frequency Amplitude Fs

9. Aliasing Frequency Amplitude Fs Nyquist Frequency

10. Aliasing  Frequencies above Fs/2 are “reflected” into the lower portion of the spectrum and become entangled with the low-frequency signals Frequency Amplitude Fs Nyquist Frequency

11. Aliasing  Frequencies above Fs/2 are “reflected” into the lower portion of the spectrum and become entangled with the low-frequency signals  These signals CANNOT be removed afterwards Frequency Amplitude Fs Nyquist Frequency

12. Aliasing  Frequencies above Fs/2 are “reflected” into the lower portion of the spectrum and become entangled with the low-frequency signals  These signals CANNOT be removed afterwards  Filtering BEFORE sampling is needed Frequency Amplitude Fs Nyquist Frequency

13. Image Sampling Horizontal - pixels Vertical - lines Temporal – frames

14. Image Sampling  Horizontal resolution  Sampling rate of 720, 1280, 1920 or 2048 samples/picture width Resulting resolution of 360, 640, 960 or 1024 cycles/pw

15. Image Sampling  Horizontal resolution  Sampling rate of 720, 1280, 1920 or 2048 samples/picture width Resulting resolution of 360, 640, 960 or 1024 cycles/pw  Vertical resolution  Sampling rate of 480, 576, 720, 1080 samples/picture height Resulting resolution of 240, 288, 360 or 540 cycles/ph

16. Image Sampling  Horizontal resolution  Sampling rate of 720, 1280, 1920 or 2048 samples/picture width Resulting resolution of 360, 640, 960 or 1024 cycles/pw  Vertical resolution  Sampling rate of 480, 576, 720, 1080 samples/picture height Resulting resolution of 240, 288, 360 or 540 cycles/ph  Temporal resolution  Sampling rate of 24, 25, 30, 50, 60... samples/second Resulting resolution of 12, 15, 25, 30 cycles/sec

17. Re-sampling  Image size changes are common

18. Re-sampling  Image size changes are common  Simple example of interpolating a 1080 picture to 480: Input resolution is 540 cycles/ph Output resolution is 240 cycles/ph (division by 2.25) Vertical Frequency Amplitude 1080 Vertical Frequency Amplitude 480 Filter Potential Alias

19. Re-sampling  Interpolation is only one part of the problem  Filtering is needed to control the signal spectrum and avoid the introduction of aliases  Simple interpolators are generally poor filters

20. Re-sampling  Interpolation is only one part of the problem  Filtering is needed to control the signal spectrum and avoid the introduction of aliases  Simple interpolators are generally poor filters  Alias terms are “folded” about the Nyquist point  Inverted in frequency, inverted “movement”  Highly noticeable to the human eye, which references its own internal 3D model

21. Re-sampling  Interpolation is only one part of the problem  Filtering is needed to control the signal spectrum and avoid the introduction of aliases  Simple interpolators are generally poor filters  Alias terms are “folded” about the Nyquist point  Inverted in frequency, inverted “movement”  Highly noticeable to the human eye, which references its own internal 3D model  Alias terms left in the image will be shifted again in any subsequent operations  Potentially cumulative problems

22. 3D Sampling Horizontal - pixels Vertical - lines Temporal – frames Restricted by practical limitations Linked by aspect ratio and pixel shape

23. Spatio-Temporal Sampling Spatial - lines Temporal – frames Temporal Frequency Spatial Frequency Frame Rate No of Lines Spectrum Potential alias

24. Spatio-Temporal Sampling  Filtering:  Spatial – optical LPF and lens MTF Spatial - lines Temporal – frames Temporal Frequency Spatial Frequency Frame Rate No of Lines Spectrum Potential alias

25. Spatio-Temporal Sampling  Filtering:  Spatial – optical LPF and lens MTF  Temporal – integration time of sensor system Spatial - lines Temporal – frames Temporal Frequency Spatial Frequency Frame Rate No of Lines Spectrum Potential alias

26. Spatio-Temporal Sub-Sampling Temporal – frames Temporal Frequency Spatial Frequency Frame Rate No of Lines Spectrum Spatial - lines  Where is the filter? Potential alias

27. Up-conversion Temporal Frequency Spatial Frequency Frame Rate No of Lines ? Spectrum Temporal Vertical Horizontal

28. Up-conversion Temporal Frequency Spatial Frequency Frame Rate No of Lines  Adaptive filtering ? Spectrum Temporal Vertical Horizontal

29. Up-conversion Temporal Frequency Spatial Frequency Frame Rate No of Lines  Adaptive filtering  Motion compensation ? Spectrum Temporal Vertical Horizontal

30. Format Interchange Temporal Frequency Spatial Frequency 1080p 720p Film 1080i 480i 1080p (24) 15c/s30c/s0c/s 0c/ph 250c/ph 500c/ph

31. Format Interchange Temporal Frequency Spatial Frequency  Conversion between formats requires care 1080p 720p Film 1080i 480i 1080p (24) 15c/s30c/s0c/s 0c/ph 250c/ph 500c/ph

32. Format Interchange Temporal Frequency Spatial Frequency  Conversion between formats requires care  Mixing formats such as film and video is to be avoided 1080p 720p Film 1080i 480i 1080p (24) 15c/s30c/s0c/s 0c/ph 250c/ph 500c/ph

33. Format Interchange Temporal Frequency Spatial Frequency  Conversion between formats requires care  Mixing formats such as film and video is to be avoided  1080p down- conversion might raise new challenges 1080p 720p Film 1080i 480i 1080p (24) 15c/s30c/s0c/s 0c/ph 250c/ph 500c/ph

34. Over-sampling  Commonly applied to audio – eg 96kHz down to 48kHz  Allows the use of a high performance digital filter: Frequency Amplitude 96 Filter Frequency Amplitude 48

35. Over-sampling  Commonly applied to audio – eg 96kHz down to 48kHz  Allows the use of a high performance digital filter:

36. Over-sampling  Commonly applied to audio – eg 96kHz down to 48kHz  Allows the use of a high performance digital filter:  1080p allows similar gains for outputs of 720p and 1080i  Good temporal filtering must introduce delay

37. Over-sampling  Commonly applied to audio – eg 96kHz down to 48kHz  Allows the use of a high performance digital filter:  1080p allows similar gains for outputs of 720p and 1080i  Good temporal filtering must introduce delay  Film sampling at >1080 lines/ph also allows controlled down-sampling

38. Conclusion  Spatio-temporal quincunx sub-sampling (aka interlace) is likely to be with us for some time

39. Conclusion  Spatio-temporal quincunx sub-sampling (aka interlace) is likely to be with us for some time  Modern cameras and processing can stress the format unless care is taken

40. Conclusion  Spatio-temporal quincunx sub-sampling (aka interlace) is likely to be with us for some time  Modern cameras and processing can stress the format unless care is taken  Imprinted alias is difficult (or impossible) to remove  Camera integration is an important filter for interlace

41. Conclusion  Spatio-temporal quincunx sub-sampling (aka interlace) is likely to be with us for some time  Modern cameras and processing can stress the format unless care is taken  Imprinted alias is difficult (or impossible) to remove  Camera integration is an important filter for interlace  Poor anti-alias filtering leads to additional compression concatenation artefacts

42. Conclusion  Spatio-temporal quincunx sub-sampling (aka interlace) is likely to be with us for some time  Modern cameras and processing can stress the format unless care is taken  Imprinted alias is difficult (or impossible) to remove  Camera integration is an important filter for interlace  Poor anti-alias filtering leads to additional compression concatenation artefacts  1080p down-conversion could make the stress worse