Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial Visual Object Recognition Bastian Leibe & Computer Vision Laboratory ETH Zurich Chicago, Kristen Grauman Department of Computer Sciences University of Texas in Austin

Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial 2 K. Grauman, B. Leibe ? ??? Identification vs. Categorization

Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial 3 K. Grauman, B. Leibe Object Categorization How to recognize ANY car How to recognize ANY cow

Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial What could be done with recognition algorithms? Medical image analysis Navigation, driver safety Autonomous robots Situated search Content-based retrieval and analysis for images and videos There is a wide range of applications, including…

Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial 5 K. Grauman, B. Leibe Object Categorization Task Description  “Given a small number of training images of a category, recognize a-priori unknown instances of that category and assign the correct category label.” Which categories are feasible visually?  Extensively studied in Cognitive Psychology, e.g. [Brown’58] German shepherd animaldogliving being “Fido”

Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial 6 K. Grauman, B. Leibe Visual Object Categories Basic Level Categories in human categorization [Rosch 76, Lakoff 87]  The highest level at which category members have similar perceived shape  The highest level at which a single mental image reflects the entire category  The level at which human subjects are usually fastest at identifying category members  The first level named and understood by children  The highest level at which a person uses similar motor actions for interaction with category members

Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial 7 K. Grauman, B. Leibe Visual Object Categories Basic-level categories in humans seem to be defined predominantly visually. There is evidence that humans (usually) start with basic-level categorization before doing identification.  Basic-level categorization is easier and faster for humans than object identification!  Most promising starting point for visual classification Basic level Individual level Abstract levels “Fido” dog animal quadruped German shepherd Doberman catcow … … … … … …

Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial 8 K. Grauman, B. Leibe Other Types of Categories Functional Categories  e.g. chairs = “something you can sit on”

Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial 9 K. Grauman, B. Leibe Other Types of Categories Ad-hoc categories  e.g. “something you can find in an office environment”

Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial 10 K. Grauman, B. Leibe Different levels of recognition  Which object class is in the image?  Obj/Img classification  Where is it in the image?  Detection/Localization  Where exactly ― which pixels?  F/G segmentation Levels of Object Categorization “cow” “motorbike” “car”

Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial Challenges: robustness IlluminationObject pose Clutter Viewpoint Intra-class appearance Occlusions K. Grauman, B. Leibe

Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial 12 K. Grauman, B. Leibe Challenges: robustness Detection in Crowded Scenes  Learn object variability –Changes in appearance, scale, and articulation  Compensate for clutter, overlap, and occlusion

Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial Challenges: context and human experience K. Grauman, B. Leibe

Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial Challenges: context and human experience Context cues Dynamics Video credit: J. Davis Image credit: D. Hoeim

Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial Challenges: scale, efficiency Thousands to millions of pixels in an image Estimated 30 Gigapixels of image/video content generated per second About half of the cerebral cortex in primates is devoted to processing visual information [Felleman and van Essen 1991] 3,000-30,000 human recognizable object categories 30+ degrees of freedom in the pose of articulated objects (humans) Billions of images indexed by Google Image Search 18 billion+ prints produced from digital camera images in million camera phones sold in 2005 K. Grauman, B. Leibe

Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial Challenges: learning with minimal supervision More Less Cropped to object, parts and classes labeled Classes labeled, some clutter Unlabeled, multiple objects K. Grauman, B. Leibe

Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial Rough evolution of focus in recognition research 1980s Currently 1990s to early 2000s K. Grauman, B. Leibe

Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial This tutorial Intended for broad AAAI audience  Assuming basic familiarity with machine learning, linear algebra, probability  Not assuming significant vision background Our goals  Describe main approaches to recognition  Highlight past successes and future challenges  Provide the pointers (to literature and tools) that would allow you to take advantage of existing techniques in your research Questions welcome 18 K. Grauman, B. Leibe

Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial 19 K. Grauman, B. Leibe Outline 1. Detection with Global Appearance & Sliding Windows 2. Local Invariant Features: Detection & Description 3. Specific Object Recognition with Local Features ― Coffee Break ― 4. Visual Words: Indexing, Bags of Words Categorization 5. Matching Local Features 6. Part-Based Models for Categorization 7. Current Challenges and Research Directions