1. 1 Context-Inclusive Approach to Speed-up Function Evaluation for Statistical Queries: An Extended Abstract Vijay Gandhi, James Kang, Shashi Shekhar University of Minnesota, USA Junchang Ju, Eric D. Kolaczyk, Sucharita Gopal Boston University, USA ICDM Workshop on Spatial and Spatio-Temporal Data Mining December 2006

2. 2 Overview Motivation Problem Statement Challenges Related Work Contribution Validation Conclusion & Future Work

3. 3 Motivation Landcover Change  Loss of land - 217 square miles of Louisiana’s coastal lands were transformed to water after Hurricanes Katrina and Rita.  Deforestation – Brazil lost 150,000 sq. km. of forest between May 2000 and August 2006  Urban Sprawl Mississippi River Delta, Louisiana (Red represents land loss between 2004 and 2005. Courtesy: USGS) Deforestation, Ariquemes, Brazil (Courtesy: Global Change Program, University of Michigan) Urban Sprawl in Atlanta (Red indicates expansion between 1976 and 1992)

4. 4 Multiscale Multigranular Image Classification (MSMG) Input: Class hierarchy, Likelihood of specific classes ConiferHardwoodBrush Grass Likelihood of specific-classes Land-use Class Hierarchy Output: Classified images at multiple scales Scale: 2x2 Scale: 4x4 Scale: 64x64... Scale: 1x1

5. 5 Problem Statement Given:  A set of hierarchical class labels  Probability densities of each specific class at (2 n x 2 n ) pixels Find:  Class labels for every pixel at coarser scales Objective:  Best quality measure of each non-specific class using the function i.e., Expectation Maximization (EM) Constraints:  Function evaluation is expensive  Coarser scales are defined implicitly in powers of 2  2x2, 4x4, …, 2 n-1 x 2 n-1

6. 6 Algorithm: Expectation Maximization Given:  Class hierarchy,  Likelihood of specific classes Find:  Best Class for a region (e.g. 2x2 region) Likelihood of a specific class = sum of corresponding likelihood Likelihood of non-specific class (EM): 1. Initialize the proportion of each corresponding specific class 2. Multiply each likelihood by corresponding specific class proportion 3. Add the likelihood at corresponding pixel 4. Divide the value in step 1 by corresponding value in Step 2 5. Average the likelihood for each specific class 6. Repeat Step 2 to Step 5 until required accuracy Likelihood of classes C1 and C2 at a 2x2 region C C1C2 Class hierarchy L ij (C1) L ij (C2) 0.40.5 0.4 0.20.6 0.2 Example

7. 7 Execution Trace: Expectation Maximization Given:  Class hierarchy,  Likelihood of specific classes 0.20.6 0.2 0.40.5 0.4 Find: Best Class for the 2x2 region Likelihood of C1 = ∑ L ij (C1) = 1.6; C2 = ∑ L ij (C2) = 1.8 Likelihood of C: 1. Iteration 1: EM(p1 n, p2 n ) 2. Multiply: L1 ij (C1) = L ij (C1) * p1 n ; L2 ij (C2) = L ij (C2) * p2 n 3. Add: L ij = L1 ij (C1) + L2 ij (C2) 4. Divide: L1 ij (C1) = L1 ij (C1)/L ij ; L2 ij (C2) = L2 ij (C2)/L ij 5. Average: p1 n+1 = Avg(L1 ij (C1)); p2 n+1 = Avg(L2 ij (C2)) 0.10.3 0.1 0.20.25 0.2 EM(0.5, 0.5) 0.30.55 0.3 0.330.54 0.33 0.660.45 0.66 0.439, 0.560 Likelihood of classes C1 and C2 at a 2x2 region C C1C2 Class hierarchy L ij (C1) L ij (C2)

8. 8 6. Compute error = sqrt((p1 n+1 -p1 n ) 2 +(p2 n+1 -p2 n ) 2 )  if(error < Limiting Factor)  Return (p1 n+1, p2 n+1 )  else  EM(p1 n+1, p2 n+1 ) Execution Trace: Example 0.085 > 0.07 EM(0.439,0.560) 0.20.6 0.2 0.40.5 0.4 Likelihood of classes C1 and C2 at a 2x2 region C C1C2 Class hierarchy L ij (C1) L ij (C2) 0.0570.171 0.057 0.2860.357 0.286 Likelihood of C = 0.456 + 1.286 = 1.742  Winner = Maximum Likelihood (C, C1, C2) = C2 Likelihood of C1, ∑ L ij (C1) = 1.6; C2, ∑ L ij (C2) = 1.8 Limiting Factor = 0.07  Iteration 2: EM(0.439, 0.560), error = 0.078  Iteration 3: EM(0.3831, 0.6155), error = 0.074  Iteration 4: EM(0.33, 0.0027), error = 0.069  Final proportions: p1 = 0.285, p2 = 0.715  Likelihood of C = (∑ Lij(C1) * p1 ) + (∑ Lij(C1) * p2 )

9. 9 Best Class at a region   = candidate models  e.g. Forest, Vegetation, Conifer  = observations  Likelihood of specific classes corresponding to M within the region  = likelihood (Quality Measure) of M  For non-specific classes, calculated using the function i.e. EM  = Penalty function  Used for non-specific classes MSMG Classification - Formulation

10. 10 Related Work Multi-resolution Image Classification Formal Statistical Method Other [Irons, Markham, Raptis] Context-Exclusive [Kolaczyk et al.] Context-Inclusive

11. 11 Context-Exclusive Approach Instance Tree  Each candidate model is analyzed independently until convergence  The candidate model with maximum likelihood is selected Instance Tree Context-Exclusive Approach: 1. Select the best specific class, Brush 2. Vegetation is evaluated until convergence (46) 3. Forest is evaluated until convergence (34) 4. Non-Forest is evaluated until convergence (3) 5. Select the best class (Non-Forest) 1. 2. 3. 4. 1 2 3 4 Land-use Class Hierarchy Total iterations: 46 + 34 + 3 = 83

12. 12 Limitations of Context-Exclusive Approach Computational Scalability  For 512 x 512 pixels - 7 hours of CPU time Where is the computational bottleneck?  80% of total execution time is spent in computing maximum likelihood  Number of function calls is dependent on the number of pixels, and spatial scale CPU Time for example datasets As spatial scale increases, the computation time increases exponentially

13. 13 Contributions Context Inclusive Approach Instance Tree is evaluated with context  Each candidate model is analyzed until it is better than the current best  Uses a instance-level syntax tree Context-Inclusive Approach: 1. Select the best specific class, Brush 2. Vegetation is evaluated until convergence (46) 3. Forest is evaluated (4) 4. Non-Forest is evaluated (1) 5. Non-Forest is the best-so-far 1. 2. 3. 4. 1 2 3 4 Land-use Class Hierarchy Total iterations: 46 + 4 + 1 = 51

14. 14 Context-Exclusive vs. Context-Inclusive Algorithm 1 Context-Exclusive Approach 1: Function ContextExclusive(set Cand) 2: Select the best specific class 3: for each candidate model c Cand do 4: repeat 5: Refine quality measure for each candidate model c Cand 6: until EM converges 7: end for 8: Select candidate model with the maximum quality measure 9: return c 1: Function ContextInclusive(set Cand) 2: Select the best specific class 3: for each remaining candidate model c Cand do 4: repeat 5: Refine quality measure for each candidate model c Cand 6: until EM converges OR quality measure exceeds best so far 7: end for 8: Select candidate model that is best so far 9: return c Algorithm 2 Context-Inclusive Approach

15. 15 Convergence Test Convergence  Until ABS(Quality Measure i+1 – Quality Measure i ) < Limiting Factor Impact  As Limiting Factor decreases, Computation cost increases for Context-Exclusive  As Limiting Factor decreases, precision of Quality Measure increases for Context-Exclusive Tradeoff  Precision of Quality Measure vs. Computation cost  Tradeoff is controlled by Limiting Factor

16. 16 Experimental Design Input: Synthetic dataset and Real dataset Language: MATLAB Platform: UltraSparc III 1.1 GHz, 1 GB RAM Measurements: Number of Iterations, CPU Time, Accuracy Image Classification Benchmark Datasets Limiting Factor Measurements Experimental Design Experimental Questions:  How does change in the limiting factor affect the Context- Exclusive approach?  How does Context-Exclusive compare to Context-Inclusive approach? Candidates: Context-Exclusive, Context-Inclusive Compare Classifications Classification Accuracy

17. 17 Experiments – Dataset 1 Synthetic Dataset 128 x 128 pixels, 7 Classes Input: Class hierarchy, Likelihood of specific classes ConiferHardwoodBrush Grass Likelihood of specific-classes Land-use Class Hierarchy Output: Classified images at multiple scales Scale: 2x2 Scale: 4x4 Scale: 64x64... Scale: 1x1

18. 18 Experiments – Dataset 2 Real Dataset, Plymouth County, Massachusetts 128 x 128 pixels, 12 Classes Input: Class hierarchy, Likelihood of specific classes Land-use Class Hierarchy Output: Classified images at multiple scales Scale: 2x2 Scale: 4x4 Scale: 64x64... BarrenBrushPitch Pine Bogs … Scale: 1x1

19. 19 Accuracy of Limiting Factor = 0.01 relative to Limiting Factor of 0.00001  Above 99% for change in Limiting Factor to 0.01 Number of Iterations CPU Time Number of Iterations, CPU Time  Reduced the CPU time by 58% for change in limiting factor value from 0.00001 to 0.01 How does change in the Limiting Factor affect the Context-Exclusive approach?

20. 20 How does Context-Exclusive Compare to Context-Inclusive? Number of Iterations (Limiting Factor: 0.00001)  Reduced by 67% for Dataset 1  Reduced by 61% for Dataset 2 Dataset 1Dataset 2

21. 21 Accuracy (Limiting Factor = 0.00001)  Above 98% for Context-Inclusive Number of Iterations (Limiting Factor = 0.00001)  Reduced by 53% for Dataset 1 and 47% for Dataset 2 How does Context-Exclusive Compare to Context-Inclusive? Dataset 1Dataset 2

22. 22 Conclusion & Future Work Context-Inclusive approach for function evaluation Insight into Limiting Factor Experimental results supporting contributions Other methods may be explored:  Other type of context: Spatial Correlation between regions  Bottom-up strategy instead of top-down approach

23. 23 Number of Iterations. Example 2 Quad: 4703, Scale: 2x2 1 2 3 4 ClassCECI Vegetation68 Forest82 Non-Forest41 Savings: 9 EM Iterations 1. 2. 3. 4.

24. 24 Number of Iterations. Example 3 Quad: 10855, Scale: 2x2 ClassCECI Vegetation34 Forest192 Non-Forest31 Savings: 19 EM Iterations 1. 2. 3. 4. 1 2 3 4

25. 25 Execution Trace: Example Given: Class hierarchy, Likelihood of specific classes 0.20.6 0.2 0.40.5 0.4 Likelihood of classes C1 and C2 at a 2x2 region C C1C2 Class hierarchy L ij (C1) L ij (C2) Find: Best Class for the 2x2 region Likelihood of C1 = ∑ L ij (C1) = 1.6; C2 = ∑ L ij (C2) = 1.8 Likelihood of C: EM(mc1 n, mc2 n )  Multiply: L1 ij (C1) = L ij (C1) * mc1; L2 ij (C2) = L ij (C2) * mc2  Add: L ij = L1 ij (C1) + L2 ij (C2)  Divide: L1 ij (C1) = L1 ij (C1)/L ij ; L2 ij (C2) = L2 ij (C2)/L ij  Average: mc1 n+1 = Avg(L1 ij (C1)) mc2 n+1 = Avg(L2 ij (C2))  Compare:  if(sqrt((mc1 n+1 -mc1 n ) 2 +(mc2 n+1 -mc2 n ) 2 )) < Limiting Factor  Return  else  EM(mc1 n+1, mc2 n+1 ) 0.10.3 0.1 0.20.25 0.2 EM(0.5, 0.5) 0.30.55 0.3 0.330.54 0.33 0.660.45 0.66 0.439, 0.560 0.085 > 0.07 Limiting Factor = 0.07 EM(0.439, 0.56)

26. 26 Summary of Changes Done  Problem statement re-defined  Context-Inclusive Algorithm changed  Change in Convergence Test  Change of example graph to show Context Inclusive approach  Rearrangement of Experiments  Gray scale images for input To do  Illustration with real numbers

27. 27 Forest Non-Forest Vegetation L1 Context-Exclusive Approach Instance Tree  Each candidate model is analyzed independently until convergence  The candidate model with maximum likelihood is selected Instance Tree Iterations Quality Measure Context-Exclusive Approach: 1. Vegetation is evaluated until convergence, L1 2. Forest is evaluated until convergence, L2 3. Non-Forest is evaluated until convergence, L3 L2 L3

28. 28 Contributions Context Inclusive Approach Instance Tree is evaluated with context  Each candidate model is analyzed until it is better than the current best  Uses a instance-level syntax tree Context-Inclusive Approach: 1. Vegetation is evaluated until convergence, L1 2. Forest is evaluated until L2 3. Non-Forest is evaluated until L3 Forest Non-Forest Vegetation Iterations Quality Measure L1 L2 L3

29. 29 Experimental Questions How does change in the limiting factor affect the Context-Exclusive approach?  Number of Iterations  CPU Time  Accuracy How does Context-Exclusive compare to Context-Inclusive approach?  Number of Iterations  CPU Time  Accuracy

30. 30 Experiments – Dataset 1  Synthetic dataset  128 x 128 pixels, 7 classes Input Conifer Hardwood Brush Grass

31. 31 Experiments – Dataset 1 Output: Images at multiple spatial and granular scale Conifer Scale: 2x2 Scale: 32 x 32 Scale: 16x16 Scale: 4x4Scale: 8x8 Scale: 64 x 64

32. 32 Experiments – Dataset 2 Real dataset, Plymouth County, Massachusetts 128 x 128 pixels, 12 classes BarrenBrushPitch Pine White PineOakGrassWaterBogs

33. 33 Experiments – Dataset 2 Output at multiple scales Scale: 2x2 Scale: 32 x 32 Scale: 16x16 Scale: 4x4Scale: 8x8 Scale: 64 x 64