1. Learning Patterns of Activity
Eric Grimson
Paul Viola
Trevor Darrell

2. Far Field Visual Analysis
Monitor a distributed setting
Track users as they move throughout a site
Learn common patterns of activity
Recognize participants and their actions
Recognize location of myself relative to world
Detect unusual events
To show you an example of these kinds of capabilities, we are going to describe an approach to meet these goals.

3. Examples of Tracking Moving Objects
Examples of this system tracking multiple moving objects in scenes around our laboratory are shown in the following movie sequences.

4. Multi-camera Coordination
This means we can stitch together events from different cameras, as shown here in these montage views composed of three different cameras. Notice how well the trajectories of the moving objects line up as they move from one view to another.

5. Mapping Patterns to Groundplane
And we can automatically warp the information to show any desired view -- for example, a bird’s eye view of the scene, showing the movement of the objects on the ground plane.

6. Detect Regularities & Anomalies in Events?
Note only can we detect motions of individual people, we can also automatically classify actions in a site.
If we go back to our outdoor setting, here are a set of trajectories taken over a period of a day. Each image represents the view for a one hour interval, and the colored patterns are the tracks of the detected moving objects, with color encoding direction, and brightness encoding speed. Our goal is to create computer systems that can automatically classify these trajectories into common events, and to record information about the frequency and execution of those events.

7. Example Track Patterns
Running continuously for over 3 years
during snow, wind, rain, dark of night, …
have processed 1 Billion images
one can observe patterns over space and over time
have a machine learning method that detects patterns automatically
Classifies patterns into most probable clusters
Associates statistics of occurrence with each cluster
Learns to identify outliers that don’t fit a cluster
Can use shape, movement, color or other features to cluster
We have built such a system, based on a large set of data we have gathered from our site.

8. Automatic Activity Classification
Here is an example of the system learning to classify patterns of activity. We have asked the system to take all of the motions observed over a day, and break it up into the most likely classes of motion.
At the first level, it does this by separating out motion based on direction.
At the second level, it further divides based on size of object.
Further levels subdivide based on shape, speed and location of the object.
All of this is learned automatically by the system, with no input from the programmer other than the motion sequences.

9. Example Categories of Patterns
Video of sorted activities
These patterns can then be collected into classes, giving us information on how often different events occur.
This is shown in these selected classes.

10. Analyzing Event Sequences
Histogram of activity over a single day
12am am pm pm pm
people
(1993 total with
.1% FP)
Resulting
classifier
12am am pm pm pm
groups
of people
(712 total with
2.2% FP)
12am am pm pm pm
clutter/lighting
effects
(647 total with
10.5% FP)
As an example of how we can use these classes, here is a set of data taken from a weeks worth of observations. The system has automatically classified the moving objects into four classes, which roughly correspond to people, vehicles, groups of people, and everything else.
Shown on the right are histograms of when each of these classes are observed as a function of time of day. One can clearly see morning and evening rush hour, the lunchtime break, the fact that people tend to drive alone to work, but walk in groups to lunch, and so on.
12am am pm pm pm
cars
(1564 total
with
3.4% FP)

11. Example Application

12. Example Application

13. Analyzing Individual Motions
Classify types of locomotion using templates of frequency and phase variations
Identify individuals based on gait
Jumping
Walking
Skipping
Running
Crawling
Quad1

14. …and this works for many problems
Recognizing activities in and around buildings
Detecting unusual events
Gathering statistics on patterns of events around a site
Eldercare monitoring
Retail marketing and analysis
So as you can see, we can already demonstrate many of the capabilities we need.
What has us excited, however, is the potential opportunities. We think that these systems can allow us to support customized viewing of sporting or cultural events.
But the same system could also be used to monitor an elderly person living alone, allowing their remote children to know if they are well, or if they are in difficulty and need assistance.
And the system could be used to support health care, such as monitoring the progression of diseases such as Parkinson’s.
So we believe that such a system will change the way in which we deal with dynamic information -- letting us capture events in much the same way we now capture images with a digital camera.