1. James C. Tilton Code 606.3 Computational & Information Sciences and Technology Office NASA Goddard Space Flight Center August 15, 2013 update National Aeronautics and Space Administration www.nasa.gov

2. HSeg Background HSeg performs image segmentation through a form of best merge region growing. It is based on the basic hierarchical step-wise optimization approach (HSWO) described in: J.-M. Beaulieu and M. Goldberg, “Hierarchy in picture segmentation: A stepwise optimal approach,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 11, no. 2, pp. 150-163, Feb., 1989. HSWO finds individual closed-connected region objects. 2 15 August 2013USGS Global Croplands Working Group Meeting

3. HSeg Background (cont’d) HSeg modifies HSWO by also aggregating spectrally similar but spatially separated region objects into groups of region objects – which we call region classes. Here is the HSeg Flowchart: S wght, ranging from 0 to 1, controls the relative importance of merges between adjacent regions versus nonadjacent regions 3 15 August 2013USGS Global Croplands Working Group Meeting

4. HSeg Background (cont’d) 4 HSeg and HSWO both produce a hierarchical set of image segmentations with the following characteristics: A set of segmentations that 1.consist of segmentations at different levels of detail, in which 2.the coarser segmentations can be produced from merges of regions from the finer segmentations, and 3.the region boundaries are maintained at the full image spatial resolution The HSeg algorithm is fully described in: James C. Tilton, Yuliya Tarabalka, Paul M. Montesano and Emanuel Gofman, “Best Merge Region Growing Segmentation with Integrated Non-Adjacent Region Object Aggregation,” IEEE Transactions on Geoscience and Remote Sensing, Vol. 50, No. 11, Nov. 2012, pp. 4454-4467. 15 August 2013USGS Global Croplands Working Group Meeting

5. RHSeg Flowchart 5 15 August 2013USGS Global Croplands Working Group Meeting High level flowchart of RHSeg approach. L r is determined as the number times the input image must be subdivided to achieve a small enough image size for efficient processing with HSeg. Flowchart of rhseg(L,X) function. ). N min is equal to ¼ the number of pixels in the subimage processed at the deepest level of recursion. See Nov. 2012 TGRS paper for description of “Processing window artifact elimination.”

6. HSeg/RHSeg – an Ideal Starting Point for Object-Based Image Analysis Pixel-based analysis is not fully effective in extracting the information content from high spatial resolution image data. Object-Based Image Analysis (OBIA) addresses the need to move beyond pixel-based analysis. OBIA relies on image segmentation as a starting point. The image segmentations produced by HSeg are ideal because of: i. their high spatial fidelity, ii. HSeg’s automatic grouping of spatially connected region objects into region classes, and iii. HSeg’s automatic production of a hierarchical set of image segmentations. HSeg is unique in its tight intertwining of region growing segmentation with region object classification. This tight intertwining is what gives HSeg segmentation qualities (i)and (ii) above. HSeg’s nearest competitor, eCognition, does not do this –and also does not automatically produce hierarchical segmentations. 6 15 August 2013USGS Global Croplands Working Group Meeting

7. HSWO Segmentation (a conventional region growing approach): A true color rendition of a 768x768 pixel section of Ikonos data from the Patterson Park/Inner Harbor area of Baltimore, MD. HSWO Segmentation with 4,541 region objects. (global dissimilarity = 0.423) 7 15 August 2013USGS Global Croplands Working Group Meeting

8. HSEG/RHSEG Segmentation: HSeg Segmentation (S wght = 0.5) with 10 region classes and 10,466 region objects. (global dissimilarity = 0.426) RHSeg Segmentation (S wght = 0.5, L r = 3) with 10 region classes and 14,198 region objects. (global dissimilarity = 0.414) 8 15 August 2013USGS Global Croplands Working Group Meeting

9. HSEG/RHSEG Segmentation – more detailed level: HSeg Segmentation (S wght = 0.5) with 28 region classes and 13,973 region objects. (global dissimilarity = 0.354) RHSeg Segmentation (S wght = 0.5, L r = 3) with 23 region classes and 25,847 region objects. (global dissimilarity = 0.335) 9 15 August 2013USGS Global Croplands Working Group Meeting

10. Problem: Large homogeneous regions with gradual gradients aren’t readily formed. 10 Observation: The boundaries between HSeg (or HSWO) subregions of large homogeneous regions do not correspond to any visibly apparent boundary – There is no “edge” between these subregions. Idea: Can edge information be utilized to influence the HSWO/HSeg region growing process to encourage the merging together of large homogeneous regions with gradual gradients? 15 August 2013USGS Global Croplands Working Group Meeting

11. Edge Detection Edge detection aims to identify image points at which the image values change abruptly. The Canny edge detector is considered to be the “state-of-the-art” in edge detection. The Canny edge detector uses a multi-stage process to form a binary edge image. For the purpose of incorporating edge information into HSWO/HSeg, we don’t need such a complicated edge detector – and it would be better to have relative edge information instead of an all or nothing binary edge image. Some simpler choices are the Sobel, Prewitt, Roberts Cross and Frei-Chen edge difference operators. I prefer the Frei-Chen operator because it is the only one that (i) is sensitive to diagonal edges as well as vertical and horizontal edges, and (ii) is normalized to give numeric results in a consistent range (0.0 to 1.0). 11 15 August 2013USGS Global Croplands Working Group Meeting

12. Frei-Chen Edge Difference Operator 12 15 August 2013USGS Global Croplands Working Group Meeting

13. Frei-Chen Edge Difference Operator Result: A true color rendition of a 768x768 pixel section of Ikonos data from the Patterson Park/Inner Harbor area of Baltimore, MD. Frei-Chen Edge Difference Operator Result, maximum over spectral bands, thresholded at 0.07. 13 15 August 2013USGS Global Croplands Working Group Meeting

14. Incorporating Edge Information into HSWO/HSeg/RHSeg 14 Edge information is incorporated at three different stages: 1.An initialization stage in which the edge information directs a fast first-merge region growing process (Muerle-Allen)to quickly merge connected areas with low edge values, and 2.The normal HSWO/HSeg best merge region growing stage in which the edge information influences the best merge decisions. 3.In performing processing window artifact elimination in RHSeg. J. L. Muerle, D. C. Allen, “Experimental Evaluation of Techniques for Automatic Segmentation of Objects in a Complex Scene,” in G. C. Cheng, et al. (Eds.), Pictorial Pattern Recognition, Thompson, Washington, DC, pp. 3-13, 1968. 15 August 2013USGS Global Croplands Working Group Meeting

15. Incorporating Edge Information into HSWO/HSeg/RHSeg 15 New (or modified) HSWO/HSeg/RHSeg parameters: -edge_threshold:Threshold for initial neighborhood merges based on edge information -edge_wght:Relative weight for edge information in calculating neighboring region dissimilarity -seam_threshold_factor:Factor multiplied times “edge_threshold” in determining across seam region merges The normal HSWO/HSeg neighboring region dissimilarity is multiplied by an “edge_factor” ranging from (1.0 – edge_wght) to 1.0 based on edge information from the compared regions. 15 August 2013USGS Global Croplands Working Group Meeting

16. Region Edge Information: 16 Each region maintains a list of edge values for each pixel on the boundary with each neighboring region. In computing the edge_factor, the average edge value is computed along the boundary from each side of the boundary, and the “edge_result” used to compute the edge_factor is the maximum of those two average edge values. Alternative: Set “edge_result” equal to the average edge value computed along both sides of the region boundary. 15 August 2013USGS Global Croplands Working Group Meeting

17. Edge_factor Computation: 17 Let MinEV be the minimum value of the input edge image. Let MaxEV be the maximum value of the input edge image. Given the edge_result from comparing two neighboring regions (from the previous slide), the edge_factor is computed as follows: edge_scaled = (edge_result – MinEV)/(MaxEV – MinEV) edge_factor = (1.0 – edge_wght) + edge_scaled*edge_wght This edge_factor is multiplied times the region dissimilarity value normally computed by HSWO/HSeg. Alternative: Raise “edge_scaled” to a power “p” before using it to compute “edge_factor” 15 August 2013USGS Global Croplands Working Group Meeting

18. Processing Window Artifact Elimination with Edge Information: 18 Find all straight vertical and horizontal boundaries longer than 4 pixels. Compute the average of the edge_value of all pixels on each side of these “long” straight vertical and horizontal boundaries Force the merge of the pairs of regions across these “long” straight vertical and horizontal boundaries if the average edge_value is <= edge_threshold*seam_threshold_factor NOTE: This process replaces the much more complicated previous approach for processing window artifact elimination (described in the Nov. 2012 TGRS paper). 15 August 2013USGS Global Croplands Working Group Meeting

19. Comparison of HSeg/RHSeg with and without Edge Information: HSeg Segmentation (S wght = 1.0, edge_threshold = 0.07, edge_wght=1.0) with 224 region classes and 11,686 region objects. (global dissimilarity = 0.374). RHSeg Segmentation (S wght = 1.0, edge_threshold = 0.07, edge_wght=1.0, s_t_f = 1.4) with 225 region classes and 22,309 region objects. (global dissimilarity = 0.342). 19 HSeg Segmentation (S wght = 0.5) with 223 region classes and 15,580 region objects. (global dissimilarity = 0.319). 15 August 2013USGS Global Croplands Working Group Meeting

20. Comparison of HSeg/RHSeg with and without Edge Information: HSeg Segmentation (S wght = 1.0, edge_threshold = 0.07, edge_wght=1.0) with 28 region classes and 11,016 region objects. (global dissimilarity = 0.407). RHSeg Segmentation (S wght = 1.0, edge_threshold = 0.07, edge_wght=1.0, s_t_f = 1.4) with 28 region classes and 18,838 region objects. (global dissimilarity = 0.378). 20 HSeg Segmentation (S wght = 0.5) with 28 region classes and 13,973 region objects. (global dissimilarity = 0.354). 15 August 2013USGS Global Croplands Working Group Meeting

21. Comparison of HSeg/RHSeg with and without Edge Information: HSeg Segmentation (S wght = 1.0, edge_threshold = 0.07, edge_wght=1.0) with 16 region classes and 10,267 region objects. (global dissimilarity = 0.429). RHSeg Segmentation (S wght = 1.0, edge_threshold = 0.07, edge_wght=1.0, s_t_f = 1.4) with 19 region classes and 17,575 region objects. (global dissimilarity = 0.391). 21 HSeg Segmentation (S wght = 0.5) with 15 region classes and 12,381 region objects. (global dissimilarity = 0.389). 15 August 2013USGS Global Croplands Working Group Meeting

22. Comparison of HSeg/RHSeg with and without Edge Information: HSeg Segmentation (S wght = 1.0, edge_threshold = 0.07, edge_wght=1.0) with 7 region classes and 7,107 region objects. (global dissimilarity = 0.518). RHSeg Segmentation (S wght = 1.0, edge_threshold = 0.07, edge_wght=1.0, s_t_f = 1.4) with 11 region classes and 13,680 region objects. (global dissimilarity = 0.426). 22 HSeg Segmentation (S wght = 0.5) with 10 region classes and 10,466 region objects. (global dissimilarity = 0.426). 15 August 2013USGS Global Croplands Working Group Meeting

23. Comparison of HSeg/RHSeg with and without Edge Information: HSeg Segmentation (S wght = 1.0, edge_threshold = 0.07, edge_wght=1.0) with 4 region classes and 3670 region objects. (global dissimilarity = 0.726). RHSeg Segmentation (S wght = 1.0, edge_threshold = 0.07, edge_wght=1.0, s_t_f = 1.4) with 5 region classes and 4589 region objects. (global dissimilarity = 0.657). 23 HSeg Segmentation (S wght = 0.5) with 5 region classes and 3876 region objects. (global dissimilarity = 0.616). 15 August 2013USGS Global Croplands Working Group Meeting

24. Comparison of HSeg/RHSeg with and without Edge Information: HSeg Segmentation (S wght = 1.0, edge_threshold = 0.07, edge_wght=1.0) with 3 region classes and 2227 region objects. (global dissimilarity = 0.882). RHSeg Segmentation (S wght = 1.0, edge_threshold = 0.07, edge_wght=1.0, s_t_f = 1.4) with 3 region classes and 2384 region objects. (global dissimilarity = 0.890). 24 HSeg Segmentation (S wght = 0.5) with 3 region classes and 3565 region objects. (global dissimilarity = 0.867). 15 August 2013USGS Global Croplands Working Group Meeting

25. Processing Window Artifact Elimination: effect of seam_threshold_factor RHSeg Segmentation (Swght = 1.0, edge_threshold = 0.07, edge_wght=1.0, s_t_f = 1.2) with 225 region classes and 22,727 region objects. (global dissimilarity = 0.338). RHSeg Segmentation (S wght = 1.0, edge_threshold = 0.07, edge_wght=1.0, s_t_f = 1.4) with 225 region classes and 22,309 region objects. (global dissimilarity = 0.342). 25 RHSeg Segmentation (Swght = 1.0, edge_threshold = 0.07, edge_wght=1.0, s_t_f = 0.0) with 222 region classes and 22,661 region objects. (global dissimilarity = 0.338). 15 August 2013USGS Global Croplands Working Group Meeting

26. Power of scaled edge value: Frei-Chen edge value, scaled from 0 to 1, raised to power 2 Frei-Chen edge value, scaled from 0 to 1, raised to power 4. 26 Frei-Chen edge value, scaled from 0 to 1, raised to power 1. 15 August 2013USGS Global Croplands Working Group Meeting

27. Incorporating Edge Information into HSWO/HSeg/RHSeg 27 Timing Comparisons for a 768x768 section of an Ikonos image: Version 1.60 (no edge information)- HSeg with S wght = 0.5: 2 minutes, 31 seconds Version 1.70 (no edge information) – HSeg with S wght = 0.5: 2 minutes, 27 seconds Version 1.70 (no edge information) – RHSeg with S wght = 0.5: 4 minutes, 47 seconds Version 1.70 (with edge information) – HSeg with S wght = 1.0: 2 minutes, 49 seconds Version 1.70 (with edge information) – RHSeg with S wght = 1.0: 3 minutes, 52 seconds 15 August 2013USGS Global Croplands Working Group Meeting

28. SignOffPage 28 15 August 2013USGS Global Croplands Working Group Meeting