2. Activity Monitoring October 19-20, 1999 DARPADARPA Bob Bolles, Brian Burns, Marty Fischler, Ravi Gopalan, Marsha Jo Hannah, Dave Scott SRI International Rama Chellappa, Yiannis Aloimonos, Doug Ayers, Ross Cutler, Larry Davis, Azriel Rosenfeld, Chandra Shekhar University of Maryland

3. 2 October 19-20, 1999 Application Challenge Develop techniques for dramatically increasing the productivity of an aerial video analyst.

4. 3 October 19-20, 1999 High-Level Approach Sensor Multiplexing to Monitor Several Sites “Simultaneously” and Semi-automatically

5. 4 October 19-20, 1999 Technical Goal for Activity Monitoring Develop techniques to monitor sites, such as cantonment areas and tree lines, from an airborne platform and identify tactically significant activities involving people and vehicles. Sample Activities: people entering a forbidden area people congregating near an embassy vehicles convoying along a road people readying a missile for launch

6. 5 October 19-20, 1999 Technical Challenges for Activity Monitoring Representation of activities Recognition of activities from a moving platform Moving object classification Activity A large tactical vehicle exiting a hide site (along a tree line). People are often visible guiding the vehicle out. Starting search Looking for people Detect person(s) Looking for large vehicle All people leave the FOV Exit of large vehicle detected Detect small vehicle Activity Template Zoom to a NFOV & aim close to tree line Move to new point along tree line Detect large vehicle

7. 6 October 19-20, 1999 Approach Task specification Retrieve or sketch a site model (roads, buildings,…) Specify the task (what, where, when, & reports/alarms) Automatic monitoring Scan the appropriate area Stabilize the video (MTS -- Sarnoff) Register the video to the site model (PVR -- Harris) Detect and track moving objects Characterize & classify the tracked objects Recognize activities Report tactically significant events AMIS -- Activity Monitoring Integrated Systesm

8. 7 October 19-20, 1999 Site Model Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events Powers Road Mosby Road Motorpool Berm “Residence” Area

9. 8 October 19-20, 1999 Task Specification Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events Drivers jog to their vehicles Vehicles drive away Drivers jog to their vehicles Motorpool Residence Area

10. 9 October 19-20, 1999 Scan Area Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events Motorpool Residence Area Sensor Field of View

11. 10 October 19-20, 1999 Stabilize Video Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events Raw Video

12. 11 October 19-20, 1999 Stabilize Video Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events Stabilized Video

13. 12 October 19-20, 1999 Register Video Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events Desired field of view Actual field of view

14. 13 October 19-20, 1999 Track Objects Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events

15. 14 October 19-20, 1999 Characterize Objects Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events Object Properties Size, velocity, … Articulation -- periodicity (for animate/inanimate) Could it be parallax? Color, shape, … Location in the site

16. 15 October 19-20, 1999 Report Events Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events People moving down Powers Road Vehicles leaving motorpool area People approaching motorpool area People entered motorpool area Alert: Battle Group Pullout

17. 16 October 19-20, 1999 Primary Contributions Representation and recognition of activities (in the context of a site model) –augmented finite state machines –dynamic belief networks Moving object classification components –parallax analysis –animate/inanimate classification –velocity, size,...

18. 17 October 19-20, 1999 Introduction to Live Flight Experiments

19. 18 October 19-20, 1999 Activity Monitoring 1. Battle group pullout 2. Battle group return 3. People exiting woods near berm 4. People crossing the road Berm “Residence” Area Activities Motorpool

20. 19 October 19-20, 1999 Activity Templates Event Primitives –Approaching/Leaving –Gaining-Ground/ Losing-Ground –Entering/Exiting –Moving-inside-region –Temporal durations Combinations –Boolean operations –Sequences –Graphs Starting search Looking for people Detect person(s) Looking for large vehicle All people leave the FOV Exit of large vehicle detected Detect small vehicle Activity Template Zoom to a NFOV & aim close to tree line Move to new point along tree line Detect large vehicle

21. 20 October 19-20, 1999 Site Model Sketching

22. 21 October 19-20, 1999 Video Registration Image World

23. 22 October 19-20, 1999 Activity Analysis in World Coordinates Image World

24. 23 October 19-20, 1999 Moving Object Detection Raw video fields Raw differences AND’d differences Image N

25. 24 October 19-20, 1999 Parallax Versus Independent Motion

26. 25 October 19-20, 1999 Animate/Inanimate Periodicity analysis

27. 26 October 19-20, 1999 Align and scale objects Compute similarity matrix S Template fit peaks of S Track objects Autocorrelate S Periodicity Analysis for Classifying Objects as Animate or Inanimate

28. 27 October 19-20, 1999 Parallax Detection Flagged as being locally consistent with “motion parallax”

29. 28 October 19-20, 1999 AM’s Windows

30. 29 October 19-20, 1999 Stabilization Params Metadata MTS-Ground Multiple Target Surveillance Precision Video Registration Raw Video (analog) CAGS-Ground CAGS-Air Ground Station Activity Monitoring Air-Ground Partition for 1999

31. 30 October 19-20, 1999 Battle Group Pullout 1. Battle group pullout 2. Battle group return 3. People exiting woods near berm 4. People crossing the road Activities Drivers jog to their vehicles Vehicles drive away

32. 31 October 19-20, 1999 Battle Group Return Vehicles return & park Drivers walk back to residence 1. Battle group pullout 2. Battle group return 3. People exiting woods near berm 4. People crossing the road Activities

33. 32 October 19-20, 1999 People Exiting Woods near Berm People Exit Tree Line 1. Battle group pullout 2. Battle group return 3. People exiting woods near berm 4. People crossing the road Activities

34. 33 October 19-20, 1999 People Crossing Road People Exit Tree Line and cross the road 1. Battle group pullout 2. Battle group return 3. People exiting woods near berm 4. People crossing the road Activities

35. 34 October 19-20, 1999 Preliminary Event Statistics Results from 2 flights with high contrast imagery

36. 35 October 19-20, 1999 Preliminary Whole Vignette Statistics Results from 2 flights with high contrast imagery

37. 36 October 19-20, 1999 Summary Accomplishments: AMIS – Activity Monitoring Integrated System Activity Templates – an initial representation for activities An initial technique for recognizing activities based on augmented finite state machines An extension to dynamic belief networks to activity recognition A technique for identifying moving objects due to motion parallax A technique for classifying moving objects as animate or inanimate A semi-automatic video registration technique A realtime moving object detection technique Increase the productivity of an image analyst by a factor of 10 to 15 by multiplexing a high- performance sensor and automatically identifying potentially significant activities. Goal:

38. 37 October 19-20, 1999 Evaluation of ‘99 Accomplishments Moving object classification -- Components only Sensor Control -- manual versus computer- controlled HCI -- primarily on PC, not integrated into CAGS-Ground

39. 38 October 19-20, 1999 Plans for ‘00 Represent & recognize more complex activities, such as checkpoint monitoring Call PVR for video registration Place sensor under computer-control (based on MTS results) Integrate moving object classification Integrate the HCI into CAGS-Ground