1. Saliency-Assisted Navigation of Very Large Landscape Images Cheuk Yiu IpAmitabh Varshney

2. Very Large Landscape Images Stitch images to create multi-gigapixel very large images But WHERE should we start looking? Image Acquisition: Gigapan MS HDView ToG 2007 Image Stitching: Kazhdan et al ToG 2008, 2010 Summa et al ToG 2010

3. Visual Knowledge Discovery Visual knowledge discovery Identify what is interesting Visualize them ChallengesContributions Visual Scalability:Sliding-Window Saliency Information Scalability:Anomaly Detection Data Scalability:Parallel filtering, Saliency Storage Validation:Validate against Web Community Tags

4. Results Preview

5. Visual Scalability

6. Information Scalability Design effective algorithms to process large images The SMALL unique regions in the large images contain the MOST information Identify informative regions from repetitive scene elements

7. Data Scalability Very large images represent a large amount of data 5 Gpix RGBA = 20GB uncompressed Multicore and manycore parallel processing Requires efficient algorithms O(n) and out-of-core GPU methods

8. Overview Sliding-Window Saliency Map Detection Anomalous Regions Interactive Exploration

9. Traditional Multiscale Image Saliency Detects “Pop-out” spots from the scene Inspired by human visual system Pre-attentive vision Find multiscale contrasting regions Intensity, Color Opponencies ( I, RG, BY ) 1.Convolve ( I, RG, BY ) with Difference of Gaussians (DoG) filter (σ is stdev) 2.Repeat on downsampled images for multiscales Image Saliency Itti et al. PAMI,1998 Bruce et al. IJCV,2009 Goferman et al. CVPR 2010 Work on small images, very accurate but slow.

10. Multiscale Aggregation Works well on small images If we have many more scales … Large regions dominate small regions Wait… we don’t want to miss the small regions Traditional multiscale saliency is insufficient

11. Our Sliding-Window Aggregation We see different things at different zoom levels One saliency map per level Only aggregate up to 4x Use a sliding-window across scales Why 4x? Eye resolution difference ~5x All (σ - 256σ) σ – 4σ 4σ – 16σ 16σ – 64σ

12. There are still too many regions… 18,000+ regions in 1.3Gpix (5 hours if a user spends 1s on each) Regions are enlarged for visibility There are many contrasting repetitive elements

13. Information Discovery Identify the informative regions from the salient regions Compare regions to find the most different ones Detect the anomalous regions and outliers Visual Data Analysis Mesh and Volume Saliency (Lee et al. ToG 2005, Kim et al. TVCG 2006) Video Summarization (Daniel et al. Vis 2003) Flow and Information Theory (Janicke et al. TVCG 2010) Molecular Dynamics Layout (Patro et al. Biovis 2011)

14. Image Region Descriptors Represent salient regions by histograms (rotational invariance) Global Colors RGB, HSV, CIELAB: Not discriminative Local Edges: Too discriminative Histograms of colors in 8x8 moving windows work well(MPEG-7 CSD) Compare histograms, p, q, by the Euclidean distance

15. Uniqueness, U(p), is the average distance of p to its k-Nearest- Neighbors. Repeating regions have a low U(p) Distinct regions have a high U(p) Spatial data structures (kD-trees) accelerate the retrieval k-Nearest-Neighbors Anomaly Detection

16. Where are they … ? Top 3% (500) of the most distinct regions. Most of the repeating region are eliminated. Can you see the remaining regions?

17. Visualizing the Detected Regions Problem: Small regions of interests are NOT visible Adaptively enlarge regions Determine the scale and colors by the region’s rank of uniqueness Increase when zooming out Decrease when zooming in (Formula in paper)

18. Automatic Exploration Explore the regions in descending order of their uniqueness k-NN anomaly detection step provides uniqueness ordering

19. Interactive Refinement 1.Locate similar undesired regions 2.Select a representative 3.Move the slider to adjust the coverage 4.Delete the selection The spatial data structure indexes the regions and provides fast retrieval

20. After User Refinement The remaining 300 regions after 3 refinement interactions

21. Data Scalability GPU Out-of-core saliency computation Break the image into tiles Parallel Gaussian filtering on GPU Filter overlapping boundary tiles to maintain continuity Saliency map storage Fit and store ellipses of the salient regions Do not store an extra image Tiled Image Viewer View dependent mipmap image tiles loading and prefetching for smooth pan and zoom

22. Royal Gorge Bridge (1.4 Gpix)

23. Cacti (4.0 GPix)

24. Mount Whitney (5.0 GPix)

25. Gigapan Community Tags Grimsel Pass Royal Gorge Bridge

26. Gigapan Community Tags Cacti Mount Whitney

27. Limitations Buffelgrass after fire The “Original” cactus Tags with semantic information Domain knowledge necessary Why are they tagged ?

28. Performance ImageGigaPan TagsDetected AllNon Semantic Grimsel Pass353225 (78%) Royal Gorge253024 (80%) Cacti241715 (88%) Whitney11 10 (90%) Each GPix takes 2.5 1 hours to preprocess (1 NVIDIA GeForce GTX 285 GPU and 1 CPU) Each interaction takes 10 ms

29. Conclusions First step on visual knowledge discovery on very large landscape images Visual Scalability:Sliding-Window Saliency Information Scalability:Anomaly Detection Data Scalability:Parallel filtering, Saliency Storage Interactive Navigation

30. Future work There are a lot of very large images Astronomy Microscopy Product inspection Urban Scenes 1. Domain specific descriptors 2. Fast discovery of locally distinct regions. 3. Accurate Identification of globally unique regions.

31. Acknowledgements National Science Foundation: CCF 05-41120, CMMI 08-35572, CNS 09-59979 NVIDIA CUDA Center of Excellence Program Derek Juba, Sujal Bista, Rob Patro, Icaro da Cunha, Yang Yang, Adil Yalcin, and the reviewers for improving this paper and presentation The Vis paper award committees Thank you!

32. Questions ? Please see our websites for the paper and video: Cheuk Yiu Ip www.cs.umd.edu/~ipcy/ GVIL Research Highlights www.cs.umd.edu/gvil/