Learning Patterns of Activity Eric Grimson Paul Viola Trevor Darrell

Far Field Visual Analysis Monitor a distributed setting Track users as they move throughout a site Learn common patterns of activity Recognize participants and their actions Recognize location of myself relative to world Detect unusual events To show you an example of these kinds of capabilities, we are going to describe an approach to meet these goals.

Examples of Tracking Moving Objects Examples of this system tracking multiple moving objects in scenes around our laboratory are shown in the following movie sequences.

Multi-camera Coordination This means we can stitch together events from different cameras, as shown here in these montage views composed of three different cameras. Notice how well the trajectories of the moving objects line up as they move from one view to another.

Mapping Patterns to Groundplane And we can automatically warp the information to show any desired view -- for example, a bird’s eye view of the scene, showing the movement of the objects on the ground plane.

Detect Regularities & Anomalies in Events? Note only can we detect motions of individual people, we can also automatically classify actions in a site. If we go back to our outdoor setting, here are a set of trajectories taken over a period of a day. Each image represents the view for a one hour interval, and the colored patterns are the tracks of the detected moving objects, with color encoding direction, and brightness encoding speed. Our goal is to create computer systems that can automatically classify these trajectories into common events, and to record information about the frequency and execution of those events.

Example Track Patterns Running continuously for over 3 years during snow, wind, rain, dark of night, … have processed 1 Billion images one can observe patterns over space and over time have a machine learning method that detects patterns automatically Classifies patterns into most probable clusters Associates statistics of occurrence with each cluster Learns to identify outliers that don’t fit a cluster Can use shape, movement, color or other features to cluster We have built such a system, based on a large set of data we have gathered from our site.

Automatic Activity Classification Here is an example of the system learning to classify patterns of activity. We have asked the system to take all of the motions observed over a day, and break it up into the most likely classes of motion. At the first level, it does this by separating out motion based on direction. At the second level, it further divides based on size of object. Further levels subdivide based on shape, speed and location of the object. All of this is learned automatically by the system, with no input from the programmer other than the motion sequences.

Example Categories of Patterns Video of sorted activities These patterns can then be collected into classes, giving us information on how often different events occur. This is shown in these selected classes.

Analyzing Event Sequences Histogram of activity over a single day 12am 6am 12pm 6pm 12pm people (1993 total with .1% FP) Resulting classifier 12am 6am 12pm 6pm 12pm groups of people (712 total with 2.2% FP) 12am 6am 12pm 6pm 12pm clutter/lighting effects (647 total with 10.5% FP) As an example of how we can use these classes, here is a set of data taken from a weeks worth of observations. The system has automatically classified the moving objects into four classes, which roughly correspond to people, vehicles, groups of people, and everything else. Shown on the right are histograms of when each of these classes are observed as a function of time of day. One can clearly see morning and evening rush hour, the lunchtime break, the fact that people tend to drive alone to work, but walk in groups to lunch, and so on. 12am 6am 12pm 6pm 12pm cars (1564 total with 3.4% FP)

Example Application

Example Application

Analyzing Individual Motions Classify types of locomotion using templates of frequency and phase variations Identify individuals based on gait Jumping Walking Skipping Running Crawling Quad1

…and this works for many problems Recognizing activities in and around buildings Detecting unusual events Gathering statistics on patterns of events around a site Eldercare monitoring Retail marketing and analysis So as you can see, we can already demonstrate many of the capabilities we need. What has us excited, however, is the potential opportunities. We think that these systems can allow us to support customized viewing of sporting or cultural events. But the same system could also be used to monitor an elderly person living alone, allowing their remote children to know if they are well, or if they are in difficulty and need assistance. And the system could be used to support health care, such as monitoring the progression of diseases such as Parkinson’s. So we believe that such a system will change the way in which we deal with dynamic information -- letting us capture events in much the same way we now capture images with a digital camera.