Interactive Discovery and Semantic Labeling of Patterns in Spatial Data Thomas Funkhouser, Adam Finkelstein, David Blei, and Christiane Fellbaum Princeton University FODAVA Review Meeting December 2009

Motivation Lots of 3D data is available with spatial patterns that reveal semantic information Archeology Climate Simulation Molecular Biology

Example: Lidar Scans of Cities Ottawa

Goal Discover spatial patterns in 3D data to assist semantic segmentation and labeling

Related Work Specific Objects [Chen and Chen 07] Buildings, trees, etc. Markov Random Fields [Anguelov et al. 05] Point labeling Segment, label [Secord and Zakhor 07] [Carlberg et al. 09] [Golovinskiy et al. 09]

Current Approach Supervised learning: Training data  Locate  Segment  Describe  Build classifier New data  Locate  Segment  Describe  Apply classifier Training Area

Current Approach Supervised learning: Problems  Training data only useful if matches new data  Trainer prescribes semantic classes  Trainer must label enough training data to cover all possible new data Training Area

Problems I Raw data may be difficult to segment automatically into semantic objects

Problems II Local spatial patterns may not be descriptive enough to assign semantic labels What is this?

Problems III Spatial patterns/features for objects of same type may be different in different data sets

Problems IV Semantic objects of interest may be different for different users What areas of city have too few street lights? What is spacing between fire hydrants? Where should trees be planted? Where could a terrorist could hide a bomb? Where do people park?

Another Possible Approach Active learning: Off-line  Locate  Segment  Describe On-line  System builds classifier by requesting user labels

Another Possible Approach Active learning: Problems  Computer drives training process – tries to learn user’s semantic model  Usually classes are pre-specified  Discrete sequence of visual recognition tasks – if jump from example to example

Our Approach User-driven learning: Off-line  Locate  Segment  Describe On-line  User interactively adjusts segments and labels on data  System builds clusters, classifiers, and provides visual feedback

Our Approach User-driven learning: Advantages:  System can guide user towards most useful input with visualization  User drives process – can focus on what he/she cares about  User can create/remove classes during labeling process  Continuous visual recognition, easier since camera is controlled by user

Main Challenge User-driven learning: Integrate off-line analysis with unsupervised learning with interactively updated probabilistic inference model while providing interactive visual feedback User Class labels

First Steps

Outline Introduction User-driven learning Specific research issues Segmentation Shape description Pattern discovery Visualization Wrap up

Outline Introduction User-driven learning Specific research issues  Segmentation Shape description Pattern discovery Visualization Wrap up

Segmentation Problem: Clustering points into semantic objects

Segmentation Current approach: Hierarchical clustering to find candidate clusters Min-cut separation of foreground from clutter SegmentationProximity graphInput data

Segmentation Current approach: Hierarchical clustering to find candidate clusters Min-cut separation of foreground from clutter

Segmentation Current results:

Segmentation Current results:

Segmentation Future challenges: 1)Provide interactive, adaptive segmentation tools 2)Integrate segmentation with recognition 3)Integrate segmentation with inference

Outline Introduction User-driven learning Specific research issues Segmentation  Shape description Pattern discovery Visualization Wrap up

Shape Description Problem: Describe cluster of points by a feature vector that discriminates its semantic class

Shape Description Current approach: Shape features (volume, eccentricity, …) Shape descriptors (spin images, shape contexts, …) Contextual cues (distances to other objects) Spin Image

Shape Description Current results: Training AreaTesting Area

Shape Description Current results:

Shape Description Proposed approach: Data-adaptive dictionaries -- adaptable filters designed to discriminate user selected object types Blue = positive, Red = negative Feature vector Adaptable filters Query Shape

Shape Description Future challenges: Computational representation for adaptable descriptors Efficient adaptation of descriptors as user labels examples Interactive user guidance in refinement of descriptors

Outline Introduction User-driven learning Specific research issues Segmentation Shape description  Pattern discovery Visualization Wrap up

Pattern Discovery Goal: Recognize spatial patterns and use them to segment and label clusters of points

Pattern Discovery Current approach: Learn probabilistic representation of symmetries and use it to predict labels Input dataMarked lampposts Symmetry transform (probabilistic model of translational symmetry)

Pattern Discovery Current results: Adding probabilistic symmetry as a feature helps recognition (by a little) Symmetry

Pattern Discovery Future challenges: 1)Better representations for symmetries and spatial relationships 2)Integrate symmetries and spatial relationships into probabilistic inference model 3)Interactive specification and visualization of symmetries and spatial patterns Symmetry transform (probabilistic model of translational symmetry)

Outline Introduction User-driven learning Specific research issues Segmentation Shape description Pattern discovery  Visualization Wrap up

Visualization Goals: Provide interactive displays to help user understand …  Input data  Specified segments and labels  Inferred segments and labels  Value of further input  Computational models

Visualization Proposed approach: Multiple views  3D space  Feature space  Symmetry space Symmetry Space 3D Space Feature Space

Visualization Future challenges: Provide methods to …  Integrate multiple views  Represent uncertainty  Guide user input  Reduce clutter

Other Applications Molecular Biology Paleontology Climate Archeology

Wrap Up Goal: Segment and label patterns in 3D data Approach: user-driven learning User interactively guides segmentation and labeling System learns model and provides visual feedback Research challenges: user-driven … Segmentation Shape description Pattern discovery Inference Visualization

Acknowledgments Students: Former: Alex Golovinskiy Current: Aleksey Boyko, Vladimir Kim Other funding sources: Former: AFRL, URGENT Current: NSF, Google