Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, Cheng Gong, Yang Cao, Bo Ren, Ming-Ming Cheng, Ali Borji Nankai University So, good afternoon everyone. The title of presentation is Enhanced-alignment measure for binary foreground map evaluation. My name is Deng-Ping Fan from Nankai University. http://mmcheng.net/e-measure

Binary Foreground Map The binary foreground map consists of values of either 0 or 1. 1 denotes foreground, 0 for background. (a) Image (b) Binary foreground map 1 So, what is the binary foreground map. Actually, it consists of values of either 0 or 1.

Application Object Segmentation, Foreground/background detection, Saliency, etc. (a) Image (b) GT There are many tasks generate the binary foreground map. Such as Object segmentation, Foreground/background detection, Saliency and so on.

VS How to Evaluate Object Segmentation (a) Image (b) GT However, one of the most important things is that how to evaluate the similarity between ground-truth and the segmentation result. (d) MDF (CVPR’15) (f) Noise

Previous Work Intersection-over-Union (IoU) The most widely-used measure is intersection-over-union, called IoU. Such as, for this input image,

Previous Work A Intersection-over-Union (IoU) The red box ‘A’ is the ground-truth.

Previous Work B Intersection-over-Union (IoU) The yellow box ‘B’ is the detection result.

Previous Work A B Intersection-over-Union (IoU) Then we compute their overlap by the following formulation:

Previous Work 𝐼𝑜𝑈= 𝐴∩𝐵 𝐴∪𝐵 = A∩B A ∪ B Intersection-over-Union (IoU) Intersection divided by the union.

Previous Work Contour Mapping (CM)[1] Another early measure is contour mapping. [1] Movahedi and Elder. Design and perceptual validation of performance measures for salient object segmentation. CVPRW, 2010.

Previous Work Contour Mapping (CM)[1] They assigned each pair of point and then compute their similarity. [1] Movahedi and Elder. Design and perceptual validation of performance measures for salient object segmentation. CVPRW, 2010.

Previous Work 𝐼𝑜𝑈= 𝐹1 2−𝐹1 Contour Mapping (CM)[1] Weighted 𝐹 𝛽 -measure (Fbw)[2] Introducing weight to the 𝐹 𝛽 -measure (related to IoU) framework. 𝐼𝑜𝑈= 𝐹1 2−𝐹1 One of the famous work presented in CVPR 2014 is Fbw. It provides good evaluation compare with previous work. [1] Movahedi and Elder. Design and perceptual validation of performance measures for salient object segmentation. CVPRW, 2010. [2] Margolin et al. How to evaluate foreground maps? CVPR, 2014.

Previous Work Contour Mapping (CM)[1] Weighted 𝐹 𝛽 -measure (Fbw)[2] Introducing weight to the 𝐹 𝛽 -measure (related to IoU) framework. Visual Quality (VQ)[3] Considering psychological function based on the IoU. Recently, the method of Visual Quality have further improved the evaluation performance based on IoU by considering the psychological function. [1] Movahedi and Elder. Design and perceptual validation of performance measures for salient object segmentation. CVPRW, 2010. [2] Margolin et al. How to evaluate foreground maps? CVPR, 2014. [3] Shi et al. Visual quality evaluation of image object segmentation: Subjective assessment and objective measure. TIP, 2015.

Previous Work Contour Mapping (CM)[1] Weighted 𝐹 𝛽 -measure (Fbw)[2] Introducing weight to the 𝐹 𝛽 -measure (related to IoU) framework. Visual Quality (VQ)[3] Considering psychological function based on the IoU. Structure measure (S-measure)[4] Mainly focus on the non-binary maps evaluation. [1] Movahedi and Elder. Design and perceptual validation of performance measures for salient object segmentation. CVPRW, 2010. [2] Margolin et al. How to evaluate foreground maps? CVPR, 2014. [3] Shi et al. Visual quality evaluation of image object segmentation: Subjective assessment and objective measure. TIP, 2015. [4] Fan et al. Structure-measure: A New Way to Evaluate Foreground Maps. ICCV, 2017. Last year, the structure measure achieve the best performance due to considering the structure similarity. However, this measure is mainly designed for non-binary map evaluation. So in the past few years, we have seen a lot of progress in the problem of foreground map evaluation. However, another important thing is that are they really provide reliable evaluation result? The answer is no.

Problem (a) Image (b) GT (c) Foreground map (d) Noise The are two examples, the column (b) is the ground-truth. We use abovementioned measure to evaluate the similarity of column (c) and (d).

> Problem (b) Noise (c) Foreground map (a) GT Almost all of current measure (e.g., IoU, CM, Fbw, VQ) prefer the Noise map. What surprised us is that almost all of measure prefer the noise map.

> Problem (b) Noise (c) Foreground map (a) GT Almost all of current measure (e.g., IoU, CM, Fbw, VQ) prefer the Noise map. They are either edge-based(local details) or region-based (global information). The reason is that they are either local-based or global-based measure.

> Problem (b) Noise (c) Foreground map (a) GT Almost all of current measure (e.g., IoU, CM, Fbw, VQ) prefer the Noise map. They are either edge-based(local details) or region-based (global information). None of them consider both local and global simultaneously. None of them consider both local and global information simultaneously. So this motivate us to do this work.

Motivation COGNIVISION 1.Global information can be captured by the eye movement. COGNIVISION 2. Local details recorded by focusing the special image region. Cognitive vision studies have shown that human vision is highly sensitive to both global information and local details in scenes. Human can obtain the global information by their eye movement and record the local details by focusing on region of interest. http://knowledgelotus.info/human-brain-facts/

Example 1.Global information (a) Image (b) GT (c) Noise (d) Map1 Here, we show an example to explain the global information. There are two maps: noise map in (c) and segmentation map in (d).

Example 1.Global information (a) Image (b) GT (c) Noise (d) Map1 Based on the global information, we will compare their body similarity.

> Example 1.Global information (a) Image (b) GT (d) Map1 (c) Noise And rank Map1 higher than noise map. (d) Map1 (c) Noise

Example 1.Global information & 2.Local Details (a) Image (b) GT (c) Map1 (d) Map2 Taking the both global and local detail information into account for these example.

Example 1.Global information & 2.Local Details (a) Image (b) GT (c) Map1 (d) Map2 We first evaluate the similarity based on main body of deer.

Example 1.Global information & 2.Local Details (a) Image (b) GT (c) Map1 (d) Map2 Follow by the details such as legs.

> Example 1.Global information & 2.Local Details (a) Image (b) GT So, we will rank map2 higher than map1. (c) Map2 (d) Map1

Enhanced-alignment measure Enhanced-alignment measure = alignment term with enhanced function = + Enhanced function Alignment term So, what is Enhanced-alignment measure? Actually, it is very simple. It is just combination of alignment term with one enhanced function. It is an evaluation measure, we hope it can solve this challenging problem.

E-measure: Alignment term 1.Global information Firstly, we compute the global mean of the input map to capture global information. The first part of our measure is the alignment term. The alignment term should consider the global and local information. We first compute the global mean of the map to present their global information.

E-measure: Alignment term 1.Global information Easy, fast to implement and use 𝜇 𝐼 ∙𝐴= 1 𝑀∗𝑁 𝑗=1 𝑀 𝑖=1 N 𝑋 𝑖𝑗 ∙𝐴= (a) Map 𝑋 𝑖𝑗 (b) Global mean So, it is an easy, and fast to implement and use.

E-measure: Alignment term 1.Global information Firstly, we compute the global mean of the input map to capture global information. 𝜇 𝐼 ∙𝐴= 1 𝑀∗𝑁 𝑗=1 𝑀 𝑖=1 N 𝑋 𝑖𝑗 ∙𝐴= (a) Map 𝑋 𝑖𝑗 (b) Global mean 2.Local details Then, we treat each pixel in the map as the local details. (e.g., 𝑋 12 =6) In the second step of our alignment term, we need to consider the local information.

E-measure: Alignment term 1.Global information Firstly, we compute the global mean of the input map to capture global information. 𝜇 𝐼 ∙𝐴= 1 𝑀∗𝑁 𝑗=1 𝑀 𝑖=1 N 𝑋 𝑖𝑗 ∙𝐴= (a) Map 𝑋 𝑖𝑗 (b) Global mean 2.Local details Treating each pixel in the map as the local details. (e.g., 𝑋 12 =6) Here, we treat each pixel of the map as the local details.

E-measure: Alignment term 1.Global information Firstly, we compute the global mean of the input map to capture global information. 𝜇 𝐼 ∙𝐴= 1 𝑀∗𝑁 𝑗=1 𝑀 𝑖=1 N 𝑋 𝑖𝑗 ∙𝐴= (a) Map 𝑋 𝑖𝑗 (b) Global mean 2.Local details Treating each pixel in the map as the local details. (e.g., 𝑋 12 =6) 3.Combine global information with local details Introducing a bias matrix which can be treated as the signal centering by removing the mean from the signal. The final step is to combine the global and local information together.

E-measure: Alignment term 1.Global information Firstly, we compute the global mean of the input map to capture global information. 𝜇 𝐼 ∙𝐴= 1 𝑀∗𝑁 𝑗=1 𝑀 𝑖=1 N 𝑋 𝑖𝑗 ∙𝐴= (a) Map 𝑋 𝑖𝑗 (b) Global mean 2.Local details Treating each pixel in the map as the local details. (e.g., 𝑋 12 =6) 3.Combine global information with local details Introducing a bias matrix which can be treated as the signal centering by removing the mean from the signal. 𝜑 𝑖𝑗 = 𝑋 𝑖𝑗 − 𝜇 𝐼 ∙𝐴 = − = We introduce a bias matrix to formulate it. It can be treated as the signal centering by removing the mean from the signal. (c) Bias matrix

E-measure: Alignment term 1.Global information Firstly, we compute the global mean of the input map to capture global information. 𝜇 𝐼 ∙𝐴= 1 𝑀∗𝑁 𝑗=1 𝑀 𝑖=1 N 𝑋 𝑖𝑗 ∙𝐴= (a) Map 𝑋 𝑖𝑗 (b) Global mean 2.Local details Treating each pixel in the map as the local details. (e.g., 𝑋 12 =6) 3.Combine global information with local details Introducing a bias matrix which can be treated as the signal centering by removing the mean from the signal. (c) Bias matrix

E-measure: Alignment term Bias matrix After obtained the bias matrix of GT and the FM We need to compare their similarity.

E-measure: Alignment term 4.Alignment matrix Alignment matrix [1][2] Following the two famous work, we adopt the ‘alignment matrix‘ to evaluate their similarity. [1] Fan et al. Structure-measure: A New Way to Evaluate Foreground Maps. ICCV, 2017. [2] Wang et al. Image quality assessment: from error visibility to structural similarity. TIP, 2004

E-measure: Alignment term 4.Alignment matrix Alignment matrix [1][2] This is the alignment matrix. [1] Fan et al. Structure-measure: A New Way to Evaluate Foreground Maps. ICCV, 2017. [2] Wang et al. Image quality assessment: from error visibility to structural similarity. TIP, 2004

E-measure: Alignment term 4.Alignment matrix If the two regions are aligned, [1] Fan et al. Structure-measure: A New Way to Evaluate Foreground Maps. ICCV, 2017. [2] Wang et al. Image quality assessment: from error visibility to structural similarity. TIP, 2004

E-measure: Alignment term 4.Alignment matrix the region of alignment matrix will receive a high score in these region. [1] Fan et al. Structure-measure: A New Way to Evaluate Foreground Maps. ICCV, 2017. [2] Wang et al. Image quality assessment: from error visibility to structural similarity. TIP, 2004

E-measure: Alignment term 4.Alignment matrix Also, for unaligned region, it will produce low score. [1] Fan et al. Structure-measure: A New Way to Evaluate Foreground Maps. ICCV, 2017. [2] Wang et al. Image quality assessment: from error visibility to structural similarity. TIP, 2004

E-measure: Enhanced function 5. Enhanced alignment matrix Finally, we use an function to enhanced the alignment, Also, this function can be treated as an normalized term.

VS Experiments Meta-Measure 1: Application Ranking Measure ranking Next, I will show our experiments We use the meta-measure to evaluate the performance of the measure. The first one is application ranking. We input a series of segmentation results to application, it will generate a standard ranking. Then we compare the measure ranking with the application ranking. A good measure will result the same ranking. Measure ranking

Experiments Meta-Measure 2: SOTA vs. Generic Maps The second meta-measure is that an evaluation measure should assign higher scores to maps in (c) obtained by the SOTA models than generic maps without any meaningful contents.

Experiments Meta-Measure 2: SOTA vs. Generic Maps Meta-Measure 3: SOTA vs. Random Noise The third meta-measure is that an evaluation measure should prefer the map generated by a SOTA model over the random noise map on average.

Experiments Meta-Measure 4: Human Ranking The fourth meta-measure is to test the ranking correlation between an evaluation measure and the human ranking.

Results 9.08%-19.65% improvement. Salient Object Segmentation Results on 4 popular datasets 9.08%-19.65% improvement. So, here are the salient object segmentation results on 4 popular datasets. We observed about 9%-19% improvement compare with previous measures. It demonstrates that our measure is pretty good.

Example From this example, we can see that the global and local information is very useful for the foreground map evaluation. Only our measure rank these maps correctly.

Conclusion Code & dataset: http://dpfan.net/e-measure Evaluation measure Reliable Good speed Intuitive Easy to use So, in conclusion we have presented an Enhanced alignment measure, called E-measure Which servers as an evaluation measure for the problem of binary foreground maps. It’s reliable and has a good speed and easy to use. Our code and dataset will be available in our website. Thank your for your attention. Code & dataset: http://dpfan.net/e-measure