1. Lecture 2 Project Topic 1: Augmented NYC CSC 59866CD Fall 2004 Zhigang Zhu, NAC 8/203A http://www-cs.engr.ccny.cuny.edu/~zhu/ Capstone2004/Capstone_Sequence2004.html Capstone: Vision, Video and VR

2. 7/15/2015Copyright @Zhigang Zhu2 Augmented NYC: Interaction, Modeling and Visualization of a Human-In-the-Loop Traffic Sensor Network Zhigang Zhu Department of Computer Science The City College / CUNY Or CityEyes: City Traffic Sensor Network for Everyone and by Everyone for a Safer Life

3. 7/15/2015Copyright @Zhigang Zhu3 Outline A Bigger Picture The Real-World Problem Capstone Project Scale-down project Visualization and Vehicle Extraction parts

4. 7/15/2015Copyright @Zhigang Zhu4 A BIGGER Picture

5. 7/15/2015Copyright @Zhigang Zhu5 Related Research Efforts at CUNY Prof. Robert Haralick, PR Lab Prof. Brian Schwartz, New Media Lab Prof. Robert Paaswell, UTRC II Prof. George Wolberg, Vision & Graphics Prof. Zhu – Video Computing

6. 7/15/2015Copyright @Zhigang Zhu6 Background NYC DOT’s Traffic Management Center 86 closed circuit TV cameras on major arteries of NYC DOT’s Real-Time Traffic Cameras 44 video streams or updated still images on the web City Drive Live on Crosswalks Channel 74 22 live streams, weekday mornings from 6:30 to 9:00 am

7. 7/15/2015Copyright @Zhigang Zhu7 NYC DOT’s Traffic Management Center 86 cameras NOW more in the future 11,000 signals in NYC ITS

8. 7/15/2015Copyright @Zhigang Zhu8 Real Player plug-in At times when demand is high, video streaming may not be available to each visitor Live camera locations will be limited during the introductory phase of this new feature Still web cam photos are available at all locations.

9. 7/15/2015Copyright @Zhigang Zhu9 City Drive Live on Crosswalks Channel 74 Problems (Internet/TV): separated views cannot give global information raw video streams bandwidth camera failures unavoidable shaking NO (little) processing/ extraction Solutions New ways for Information extraction/compression enhancement integration and presentation Twenty-two of DOT's traffic cameras Channel 74 on weekday mornings from 6:30 to 9 AM Storm Feb 18 2003

10. 7/15/2015Copyright @Zhigang Zhu10 Sensor Network for NYC Traffic Sensor network in 4D (3D space + Time) Traffic Cameras (above ground) Inductive Loops (underground) Inspection Cameras (under vehicles) Aerial Cameras (high in air) Sensors (GPS, EZ-Pass) on Vehicles (mobile)

11. 7/15/2015Copyright @Zhigang Zhu11 Research Issues – in cope with NSF Sensors and Sensor Networks localization versus distributed versus centralized processing information extraction on-sensor self-calibration Adaptive management and control of sensor nodes human-in-the-loop or close-loop adaptive feedback

12. 7/15/2015Copyright @Zhigang Zhu12 Human In the Loop DARPA Total Information Awareness Humans in all the three stages – sensors, alarm and response Google Search Engine Relevance feedback Amozon.com recommends books based upon what other customers have selected Can we do it in ITS?

13. 7/15/2015Copyright @Zhigang Zhu13 Human Involvement in ITS How to get user feedbacks? Web browsers As of Feb 18, 2003, the hit of the San Diego Area Traffic Page of Caltrans since 1996 is 385,379,722, which means more than 100 visits every minute! Drivers Real traffic information from sites via wireless connection

14. 7/15/2015Copyright @Zhigang Zhu14 Information Sharing Homeland security Department “ would make this a top priority ” Lockheed Martin Transportation Systems one of the biggest challenges confronting ITS today is information sharing Data and control

15. 7/15/2015Copyright @Zhigang Zhu15 How to make these happen? Advanced HCI Interaction ( by everyone) Not only provide info to, but also collect info from people User Internet !! Visualization (for everyone) 3D immersive and augmented interface Instead of locator map + video streams

16. 7/15/2015Copyright @Zhigang Zhu16 Proposed Research Traffic Information Extraction (Front End) Vision-based vehicle detection and tracking Human “sensors” - feedbacks Traffic Information Analysis (in Background) For traffic management and advisory Distributed inference model (DBNs) Traffic Visualization (Human Interface) Augmented and immersive user interface 3D model + real video 2D Map + animation

17. 7/15/2015Copyright @Zhigang Zhu17 Outcome of the Research Traffic Control and Management (DOT) signal timing, city evacuation,... City and Traffic Show (Web, TV) realtime, interactive, 3D walkthrough Traffic Forecasting/Advisory (Web, TV) predicting the future traffic, personalized advisory Traffic Open Source for Education and Research (Internet) interdisciplinary – sensors, vision, graphics, mm, transportation, networking, virtual reality, decision making system... open for students and researchers!

18. 7/15/2015Copyright @Zhigang Zhu18 Camera views registered with map…and take a virtual walk..

19. 7/15/2015Copyright @Zhigang Zhu19 Grid Traffic Project Brian Schwartz and Lei Zhou Gridlock simulation Cellular automaton

20. 7/15/2015Copyright @Zhigang Zhu20 Virtual Fly-through + Real Video Zhu & Schwartz (CUNY) 2003

21. 7/15/2015Copyright @Zhigang Zhu21 Traffic Information Extraction Z. Zhu, et al, VISATRAM: A real-time vision system for automatic traffic monitoring, Image and Vision Computing Journal, 18(10), July 2000: pp 781-794

22. 7/15/2015Copyright @Zhigang Zhu22 Security – Under Vehicle Inspection Full FOV coverage when a car drives over an a array of cameras... One of the cameras P. Dickson, J. Li, Z. Zhu, A. Hanson, E. Riseman, H. Sabrin, H. Schultz and G. Whitten, Mosaic generation for under-vehicle inspection. IEEE Workshop on Applications of Computer Vision, Orlando, Florida, Dec 3-4, 2002, p 251- 256.

23. 7/15/2015Copyright @Zhigang Zhu23 Video Mosaics and Bird-Eye View Z. Zhu, E. M. Riseman, A. R. Hanson, Parallel-perspective stereo mosaics, the Eighth IEEE International Conference on Computer Vision, Vancouver, Canada, July 9-12, 2001, vol I, 345-352.

24. 7/15/2015Copyright @Zhigang Zhu24 Dynamic Mosaic from 2 Cams Wide FOV movie

25. 7/15/2015Copyright @Zhigang Zhu25 Sensors on Vehicles J. S. Jin, Z. Zhu, G. Xu, A stable vision system for moving vehicles, IEEE Transactions on Intelligent Transportation Systems, vol. 1, no. 1, March 2000, pp 32-39 Video Stabilization and Geo- Registration

26. 7/15/2015Copyright @Zhigang Zhu26 Vehicle Extraction [Ye Ming Ye, Robert M. Haralick and Linda G. Shapiro, " Estimating Optical flow Using a Global Matching Formulation and Graduated Optimization ", I6th International Conference on Image Processing, 2002

27. 7/15/2015Copyright @Zhigang Zhu27 Fig.6. System diagram of the human-in-loop distributed computing System Diagram

28. 7/15/2015Copyright @Zhigang Zhu28 Traffic Information Extraction from 4D distributed environment (PAST) spatially distributed sensors over time provide real-time information (NOW) predict the future traffic (FUTURE) Challenging Research Issues various road situations, realtime optical flow, traffic volume/velocity, 24/7, 4 season, rain &dry

29. 7/15/2015Copyright @Zhigang Zhu29 Traffic Information Analysis Problems Gridlock, traffic jams... Traffic Information NOT ONLY space-time data from sensors BUT events, people in NYC! Research Issues Mathematic model – differential equations or Simulation – cellular automata

30. 7/15/2015Copyright @Zhigang Zhu30 City and Traffic Show Three modes 2D map + cartoon-like animation (w/ info) 3D model + real video 3D model + (real video) + traffic info Research Issues camera calibration...static and dynamic real-virtual registration... geo-referenced fast rendering... realistic?? new media presentation – enhanced video??

31. 7/15/2015Copyright @Zhigang Zhu31 Traffic Forecasting (Advisory) give the best route show the animation Two modes On TV (like weather report) Internet (interactive) WHEN start... end WHERE start... end Research Issues prediction data from sensors data from models (events, facilities) presentation pseudo-colored real video on map interactive and personalized

32. 7/15/2015Copyright @Zhigang Zhu32 Traffic Education Open Source interdisciplinary ubiquitous sensors in NYC vision, graphics, mm,virtual reality networking, video transmission transportation, traffic, city, tourism control, decision making system... open for students and researchers! study various traffic problems

33. 7/15/2015Copyright @Zhigang Zhu33 Related Work Blending simulations with data from sensors Schreckenberg in Germany for up-to-the minute traffic show and 1-hour traffic forecast (http://traf2.uniduisburg.de/OLSIM) Texas A&M for computerized traffic control Source : 1999 Science News Online

34. 7/15/2015Copyright @Zhigang Zhu34 Come Down to the Capstone…

35. 7/15/2015Copyright @Zhigang Zhu35 What you need to do Augmented NYC demo code: starting point CCNY\Working\NYC_VC\AugmentedNYC\AugmentedNYC.exe C++ and OpenGL NeHe OpenGL Tutorials (with code samples) http://nehe.gamedev.net/ To-Do List – Real Work of the Senior Project 2D or 3D NYC map and virtual walk-through (basics) Multiple Real Video Streams in 3D (basics) Vehicle Extraction (advanced) Traffic Analysis and Prediction (optional) Reports and Papers

36. 7/15/2015Copyright @Zhigang Zhu36 Next… Project Topic 1 Update Everyone contributes to the update of the system design Project Topic 2 Virtualized Classroom