1. Enhanced-alignment Measure for Binary Foreground Map Evaluation
Deng-Ping Fan, Cheng Gong, Yang Cao,
Bo Ren, Ming-Ming Cheng, Ali Borji
Nankai University
So, good afternoon everyone.
The title of presentation is Enhanced-alignment measure for binary foreground map evaluation.
My name is Deng-Ping Fan from Nankai University.

2. Binary Foreground Map
The binary foreground map consists of values of either 0 or 1.
1 denotes foreground, 0 for background.
(a) Image
(b) Binary foreground map 1
So, what is the binary foreground map.
Actually, it consists of values of either 0 or 1.

3. Application
Object Segmentation, Foreground/background detection, Saliency, etc.
(a) Image
(b) GT
There are many tasks generate the binary foreground map.
Such as Object segmentation, Foreground/background detection, Saliency and so on.

4. VS How to Evaluate Object Segmentation (a) Image (b) GT
However,
one of the most important things is that how to evaluate the similarity between ground-truth and the segmentation result.
(d) MDF (CVPR’15)
(f) Noise

5. Previous Work Intersection-over-Union (IoU)
The most widely-used measure is intersection-over-union, called IoU.
Such as, for this input image,

6. Previous Work A Intersection-over-Union (IoU)
The red box ‘A’ is the ground-truth.

7. Previous Work B Intersection-over-Union (IoU)
The yellow box ‘B’ is the detection result.

8. Previous Work A B Intersection-over-Union (IoU)
Then we compute their overlap by the following formulation:

9. Previous Work 𝐼𝑜𝑈= 𝐴∩𝐵 𝐴∪𝐵 = A∩B A ∪ B Intersection-over-Union (IoU)
Intersection divided by the union.

10. Previous Work Contour Mapping (CM)[1]
Another early measure is contour mapping.
[1] Movahedi and Elder. Design and perceptual validation of performance measures for salient object segmentation. CVPRW, 2010.

11. Previous Work Contour Mapping (CM)[1]
They assigned each pair of point and then compute their similarity.
[1] Movahedi and Elder. Design and perceptual validation of performance measures for salient object segmentation. CVPRW, 2010.

12. Previous Work 𝐼𝑜𝑈= 𝐹1 2−𝐹1 Contour Mapping (CM)[1]
Weighted 𝐹 𝛽 -measure (Fbw)[2]
Introducing weight to the 𝐹 𝛽 -measure (related to IoU) framework.
𝐼𝑜𝑈= 𝐹1 2−𝐹1
One of the famous work presented in CVPR 2014 is Fbw. It provides good evaluation compare with previous work.
[1] Movahedi and Elder. Design and perceptual validation of performance measures for salient object segmentation. CVPRW, 2010.
[2] Margolin et al. How to evaluate foreground maps? CVPR, 2014.

13. Previous Work Contour Mapping (CM)[1] Weighted 𝐹 𝛽 -measure (Fbw)[2]
Introducing weight to the 𝐹 𝛽 -measure (related to IoU) framework.
Visual Quality (VQ)[3]
Considering psychological function based on the IoU.
Recently, the method of Visual Quality have further improved the evaluation performance
based on IoU by considering the psychological function.
[1] Movahedi and Elder. Design and perceptual validation of performance measures for salient object segmentation. CVPRW, 2010.
[2] Margolin et al. How to evaluate foreground maps? CVPR, 2014.
[3] Shi et al. Visual quality evaluation of image object segmentation: Subjective assessment and objective measure. TIP, 2015.

14. Previous Work Contour Mapping (CM)[1] Weighted 𝐹 𝛽 -measure (Fbw)[2]
Introducing weight to the 𝐹 𝛽 -measure (related to IoU) framework.
Visual Quality (VQ)[3]
Considering psychological function based on the IoU.
Structure measure (S-measure)[4]
Mainly focus on the non-binary maps evaluation.
[1] Movahedi and Elder. Design and perceptual validation of performance measures for salient object segmentation. CVPRW, 2010.
[2] Margolin et al. How to evaluate foreground maps? CVPR, 2014.
[3] Shi et al. Visual quality evaluation of image object segmentation: Subjective assessment and objective measure. TIP, 2015.
[4] Fan et al. Structure-measure: A New Way to Evaluate Foreground Maps. ICCV, 2017.
Last year, the structure measure achieve the best performance due to considering the structure similarity.
However, this measure is mainly designed for non-binary map evaluation.
So in the past few years, we have seen a lot of progress in the problem of foreground map evaluation.
However, another important thing is that are they really provide reliable evaluation result?
The answer is no.

15. Problem (a) Image (b) GT (c) Foreground map (d) Noise
The are two examples, the column (b) is the ground-truth.
We use abovementioned measure to evaluate the similarity of column (c) and (d).

16. > Problem (b) Noise (c) Foreground map (a) GT
Almost all of current measure (e.g., IoU, CM, Fbw, VQ) prefer the Noise map.
What surprised us is that almost all of measure prefer the noise map.

17. > Problem (b) Noise (c) Foreground map (a) GT
Almost all of current measure (e.g., IoU, CM, Fbw, VQ) prefer the Noise map.
They are either edge-based(local details) or region-based (global information).
The reason is that they are either local-based or global-based measure.

18. > Problem (b) Noise (c) Foreground map (a) GT
Almost all of current measure (e.g., IoU, CM, Fbw, VQ) prefer the Noise map.
They are either edge-based(local details) or region-based (global information).
None of them consider both local and global simultaneously.
None of them consider both local and global information simultaneously.
So this motivate us to do this work.

19. Motivation COGNIVISION 1.Global information can be captured by the
eye movement.
COGNIVISION
2. Local details
recorded by focusing
the special image region.
Cognitive vision studies have shown that human vision is highly sensitive to both global information and local details in scenes.
Human can obtain the global information by their eye movement and record the local details by focusing on region of interest.

20. Example 1.Global information (a) Image (b) GT (c) Noise (d) Map1
Here, we show an example to explain the global information.
There are two maps: noise map in (c) and segmentation map in (d).

21. Example 1.Global information (a) Image (b) GT (c) Noise (d) Map1
Based on the global information, we will compare their body similarity.

22. > Example 1.Global information (a) Image (b) GT (d) Map1 (c) Noise
And rank Map1 higher than noise map.
(d) Map1
(c) Noise

23. Example 1.Global information & 2.Local Details (a) Image (b) GT
(c) Map1
(d) Map2
Taking the both global and local detail information into account for these example.

24. Example 1.Global information & 2.Local Details (a) Image (b) GT
(c) Map1
(d) Map2
We first evaluate the similarity based on main body of deer.

25. Example 1.Global information & 2.Local Details (a) Image (b) GT
(c) Map1
(d) Map2
Follow by the details such as legs.

26. > Example 1.Global information & 2.Local Details (a) Image (b) GT
So, we will rank map2 higher than map1.
(c) Map2
(d) Map1

27. Enhanced-alignment measure
Enhanced-alignment measure = alignment term with enhanced function = +
Enhanced
function
Alignment
term
So, what is Enhanced-alignment measure?
Actually, it is very simple.
It is just combination of alignment term with one enhanced function.
It is an evaluation measure, we hope it can solve this challenging problem.

28. E-measure: Alignment term
1.Global information
Firstly, we compute the global mean of the input map to capture global information.
The first part of our measure is the alignment term.
The alignment term should consider the global and local information.
We first compute the global mean of the map to present their global information.

29. E-measure: Alignment term
1.Global information
Easy, fast to implement and use
𝜇 𝐼 ∙𝐴= 1 𝑀∗𝑁 𝑗=1 𝑀 𝑖=1 N 𝑋 𝑖𝑗 ∙𝐴=
(a) Map 𝑋 𝑖𝑗
(b) Global mean
So, it is an easy, and fast to implement and use.

30. E-measure: Alignment term
1.Global information
Firstly, we compute the global mean of the input map to capture global information.
𝜇 𝐼 ∙𝐴= 1 𝑀∗𝑁 𝑗=1 𝑀 𝑖=1 N 𝑋 𝑖𝑗 ∙𝐴=
(a) Map 𝑋 𝑖𝑗
(b) Global mean
2.Local details
Then, we treat each pixel in the map as the local details. (e.g., 𝑋 12 =6)
In the second step of our alignment term, we need to consider the local information.

31. E-measure: Alignment term
1.Global information
Firstly, we compute the global mean of the input map to capture global information.
𝜇 𝐼 ∙𝐴= 1 𝑀∗𝑁 𝑗=1 𝑀 𝑖=1 N 𝑋 𝑖𝑗 ∙𝐴=
(a) Map 𝑋 𝑖𝑗
(b) Global mean
2.Local details
Treating each pixel in the map as the local details. (e.g., 𝑋 12 =6)
Here, we treat each pixel of the map as the local details.

32. E-measure: Alignment term
1.Global information
Firstly, we compute the global mean of the input map to capture global information.
𝜇 𝐼 ∙𝐴= 1 𝑀∗𝑁 𝑗=1 𝑀 𝑖=1 N 𝑋 𝑖𝑗 ∙𝐴=
(a) Map 𝑋 𝑖𝑗
(b) Global mean
2.Local details
Treating each pixel in the map as the local details. (e.g., 𝑋 12 =6)
3.Combine global information with local details
Introducing a bias matrix which can be treated as the signal centering by removing
the mean from the signal.
The final step is to combine the global and local information together.

33. E-measure: Alignment term
1.Global information
Firstly, we compute the global mean of the input map to capture global information.
𝜇 𝐼 ∙𝐴= 1 𝑀∗𝑁 𝑗=1 𝑀 𝑖=1 N 𝑋 𝑖𝑗 ∙𝐴=
(a) Map 𝑋 𝑖𝑗
(b) Global mean
2.Local details
Treating each pixel in the map as the local details. (e.g., 𝑋 12 =6)
3.Combine global information with local details
Introducing a bias matrix which can be treated as the signal centering by removing
the mean from the signal.
𝜑 𝑖𝑗 = 𝑋 𝑖𝑗 − 𝜇 𝐼 ∙𝐴 = − =
We introduce a bias matrix to formulate it.
It can be treated as the signal centering by removing the mean from the signal.
(c) Bias matrix

34. E-measure: Alignment term
1.Global information
Firstly, we compute the global mean of the input map to capture global information.
𝜇 𝐼 ∙𝐴= 1 𝑀∗𝑁 𝑗=1 𝑀 𝑖=1 N 𝑋 𝑖𝑗 ∙𝐴=
(a) Map 𝑋 𝑖𝑗
(b) Global mean
2.Local details
Treating each pixel in the map as the local details. (e.g., 𝑋 12 =6)
3.Combine global information with local details
Introducing a bias matrix which can be treated as the signal centering by removing
the mean from the signal.
(c) Bias matrix

35. E-measure: Alignment term
Bias matrix
After obtained the bias matrix of GT and the FM
We need to compare their similarity.

36. E-measure: Alignment term
4.Alignment matrix
Alignment matrix [1][2]
Following the two famous work, we adopt the ‘alignment matrix‘ to evaluate their similarity.
[1] Fan et al. Structure-measure: A New Way to Evaluate Foreground Maps. ICCV, 2017.
[2] Wang et al. Image quality assessment: from error visibility to structural similarity. TIP, 2004

37. E-measure: Alignment term
4.Alignment matrix
Alignment matrix [1][2]
This is the alignment matrix.
[1] Fan et al. Structure-measure: A New Way to Evaluate Foreground Maps. ICCV, 2017.
[2] Wang et al. Image quality assessment: from error visibility to structural similarity. TIP, 2004

38. E-measure: Alignment term
4.Alignment matrix
If the two regions are aligned,
[1] Fan et al. Structure-measure: A New Way to Evaluate Foreground Maps. ICCV, 2017.
[2] Wang et al. Image quality assessment: from error visibility to structural similarity. TIP, 2004

39. E-measure: Alignment term
4.Alignment matrix
the region of alignment matrix will receive a high score in these region.
[1] Fan et al. Structure-measure: A New Way to Evaluate Foreground Maps. ICCV, 2017.
[2] Wang et al. Image quality assessment: from error visibility to structural similarity. TIP, 2004

40. E-measure: Alignment term
4.Alignment matrix
Also, for unaligned region, it will produce low score.
[1] Fan et al. Structure-measure: A New Way to Evaluate Foreground Maps. ICCV, 2017.
[2] Wang et al. Image quality assessment: from error visibility to structural similarity. TIP, 2004

41. E-measure: Enhanced function
5. Enhanced alignment matrix
Finally, we use an function to enhanced the alignment,
Also, this function can be treated as an normalized term.

42. VS Experiments Meta-Measure 1: Application Ranking Measure ranking
Next, I will show our experiments
We use the meta-measure to evaluate the performance of the measure.
The first one is application ranking. We input a series of segmentation results to application, it will generate a standard ranking.
Then we compare the measure ranking with the application ranking. A good measure will result the same ranking.
Measure ranking

43. Experiments Meta-Measure 2: SOTA vs. Generic Maps
The second meta-measure is that an evaluation measure
should assign higher scores to maps in (c) obtained by the SOTA models than generic maps without any meaningful contents.

44. Experiments Meta-Measure 2: SOTA vs. Generic Maps
Meta-Measure 3: SOTA vs. Random Noise
The third meta-measure is that an evaluation measure should prefer the map generated by a SOTA model over the random noise map on average.

45. Experiments Meta-Measure 4: Human Ranking
The fourth meta-measure is to test the ranking correlation between an evaluation measure and the human ranking.

46. Results 9.08%-19.65% improvement.
Salient Object Segmentation Results on 4 popular datasets
9.08%-19.65% improvement.
So, here are the salient object segmentation results on 4 popular datasets.
We observed about 9%-19% improvement compare with previous measures.
It demonstrates that our measure is pretty good.

47. Example
From this example, we can see that the global and local information is very
useful for the foreground map evaluation.
Only our measure rank these maps correctly.

48. Conclusion Code & dataset: http://dpfan.net/e-measure
Evaluation measure
Reliable
Good speed
Intuitive
Easy to use
So, in conclusion we have presented an Enhanced alignment measure, called E-measure
Which servers as an evaluation measure for the problem of binary foreground maps.
It’s reliable and has a good speed and easy to use.
Our code and dataset will be available in our website.
Thank your for your attention.
Code & dataset: