1. Chenlei Guo Liming Zhang Image Processing 2010
A Novel Multiresolution Spatiotemporal Saliency Detection Model and Its Applications in Image and Video Compression
Chenlei Guo
Liming Zhang
Image Processing 2010

2. Outline
Introduction
Phase Spectrum of Quaternion Fourier Transform (PQFT)
Detect Proto-Objects in the Spatiotemporal Saliency Map
Hierarchical Selectivity (HS)
Experiment Result
Applications in Image and Video Coding
Conclusions and Discussions

3. Introduction Most traditional object detectors need training
Graph-based visual saliency detection can be very powerful but it demands a very high computational cost
Most of the models only consider static images

4. Phase Spectrum of Quaternion Fourier Transform(PQFT) (1/3)
Locations with less periodicity or less homogeneity create ”pop out” proto objects in the reconstruction of the image’s phase spectrum
An early saliency detection model : PFT
Why phase?
When the waveform is a positive or negative pulse, its reconstruction contains the largest spikes at the jump edge of the input
pulse. This is because many varying sinusoidal components locate there. In contrast, when the input is a single sinu- soidal
component of constant frequency, there is no distinct spike in the reconstruction. Less periodicity or less homo- geneity of a location,
in comparison with its entire wave- form, creates more ”pop out”.
The same rule can be applied to two-dimension signals like images as well. [12] pointed out that the amplitude spectrum specifies
how much of each sinusoidal component is present and the phase information specifies where each of the sinusoidal components resides
within the image. The location with less periodicity or less homogeneity in vertical or horizonal orientation creates the ”pop out” proto objects
in the reconstruction of the image, which indicates where the object candidates are located.
Early work=> PFT model + 4 channels = PQFT

5. Quaternion Representation (2/3)
Define the input image captured at time t as F(t)
r(t), g(t), b(t) are color channels of F(t)

6. Calculate the Saliency Map By PQFT (3/3)
2-D gaussian filter

7. Detect Proto-Objects (1/3)
𝑂𝐶𝐴 𝑖 : the 𝑖 𝑡ℎ object candidate area
The search stops when is satisfied

8. Alpha (2/3)

9. Gamma (3/3)

10. How PQFT Select Visual Resolution
PQFT simulates the human vision system(HVS)
PQFT 可以套用在不同的resolution上
PQFT會模擬人類的視覺系統，在面對不同的resolution時，就像是人類眼睛在不同的距離看東西時的情形一樣

11. Hierarchical Selectivity
Set hierarchical level

12. Psychological Patterns
Experiment Results
Video Sequence
Natural Images
Psychological Patterns

13. Video Sequence (1/3)

14. Video Sequence (2/3)

15. Video Sequence (3/3)

16. Natural Image

17. Evaluation Method - ROC
True Positive Rate(TPR), False Positive Rate(FPR)
Receiver Operating Characteristic (ROC)
ROC curve = TPR/FPR
ROC area = area beneath ROC curve
The larger ROC area is, the better the prediction power of a saliency map.
SR => spectral residual (use the SR of the amplitude spectrum to calculate the image’s saliency map)
PFT => phase spectrum of Fourier transform
STB => saliency tool box (some models been proposed to simulate the behavior of eyes)
NVT => Neuromorphic Vision C++ Toolkit (a bottom-up model used to simulate the human’s virtual attention)
Evaluate saliency map 準不準的一個方法
找很多人來看data set裡面的圖片，叫他們分出target(1)跟background(0)，存成binary map
跟我們的saliency map做比較
TPR => 人找到的target中有幾%的points剛好是saliency map中標示為fixation
FPR => 人找到的background中有幾%的points是saliency map中標示為fixation

18. Psychological Patterns (1/3)
找顏色跟方向

19. Psychological Patterns (2/3)

20. Psychological Patterns (3/3)
找不對稱=找不一樣的那個

21. Applications in Image and Video Coding
Multiresolution Wavelet Domain Foveation Model (MWDF)
Evaluate the performance of the HS-MWDF model in Image and video compression

22. Multiresolution Wavelet Domain Foveation Model (MWDF)
JPEG 2000 has included the region-of-interest(RoI) coding in drafts
A better way to find RoI:use Hierarchical Selectivity

23. Multiresolution Wavelet Domain Foveation Model (MWDF)

24. The Performance of HS-MWDF in Image Compression
We use HS-MWDF model as a front end before standard compression (JPEG 2000)
Set
N fov => we only use the first n OCAs found by PQFT
Auto fov => let the program itself decide the number of foveas
n fov => we only use the first n OCAs found by PQFT
Auto fov => let the program itself decide the number

26. The Performance of HS-MWDF in Video Compression

27. Conclusion and Discussion
Extend PFT model to PQFT model
PQFT model is independent of parameters and prior knowledge, and is fast enough to meet real-time requirements
Develop a model called HS-MWDF as a front end before the image/video encoder
Problems:
Can’t deal with closure patterns well
Only considers bottom-up information
Insert the model into the image/video encoders

28. References