1. Augmenting Film and Video Footage with Sensor Data N. Su, H. Park, E. Bostrom, J. Burke, M. Srivastava, D. Estrin PerCom ’04: March 14-17, 2004

2. March 10, 2005Herman Li2 Augmented Recording System  Wireless sensor network application for filmmaking and media production Seamless integration Mobility Increased expressiveness  Synchronize sensor data with video footage  Sensor data allows post processing of video

3. March 10, 2005Herman Li3 Why not use image processing?  Image processing techniques can infer position, motion, light condition…  For example: Van Helsing Infrared LED Marker http://www.fxguide.com

4. March 10, 2005Herman Li4 Why not use image processing? http://www.fxguide.com

5. March 10, 2005Herman Li5 Why not use image processing?  Do not offer fine grain data  Cannot infer conditions outside of view, or quantities such as wind speed or temperature

6. March 10, 2005Herman Li6 Architecture  Sensor node neighbourhood  Serial port server  Timecode generator  Sylph server middleware  Jini client  SQL database

7. March 10, 2005Herman Li7 1.Sylph server lookup 2.Sylph translates query, forward to serial port server 3.Serial port server dispatches messages to base stations 4.Sensors begin collecting data 5.Base station forwards data to serial port server 6.Serial port server interpolates data 7.Sylph server announces new data 8.Jini client stores data in database

8. March 10, 2005Herman Li8 Sensors node neighbourhood  Uses CrossBow Mica 1 / Mica 2 motes Thermistor, light sensor, microphone, accelerometer.  Radio range of few hundred feet, ~10kbps http://computer.howstuffworks.com/mote.htm  Runs on PALOS (Power Aware Light-weight Operating System)

9. March 10, 2005Herman Li9 Sensors node neighbourhood  Clustering Each base station responsible for a few motes  Base station Potentiometer calibration Admit closest X sensors  Neighbours Internal frame counter from 2 to infinity Sends data every 13 frames Filters redundant values

10. March 10, 2005Herman Li10 Serial port server  Controls base stations via serial connections  Takes SMPTE timecode and synchronizes with sensor data  Combines all sensor data for a frame and sends as one packet  Interpolates missing data

11. March 10, 2005Herman Li11 Serial port server  Time synchronization At most 7  s per second drift One frame error per 4766.7 frames Time sync every 2.5 min  Latency test shows delay of at most 3 frames

12. March 10, 2005Herman Li12 Sylph server middleware  UCLA project, used in Smart Kindergarten  Allows queries on sensors  Defines JINI attributes such as light, period, command  JINI query: “READ LIGHT EVERY 30 SECONDS”  Translated: “SET PERIOD 30 SECONDS”, “SET COMMAND=STARTSENDING”

13. March 10, 2005Herman Li13 JINI client  Client retrieves sensor data per frame and stores in MS Access DB  Provides playback features and data collection controls

14. March 10, 2005Herman Li14 Evaluations  Deployed 2 base stations, 4 sensors each  Clustering algorithm took 2.59 min  Exp1: Graduate light intensity changes

15. March 10, 2005Herman Li15 Evaluations  Exp2: Delay measurement Some delay of ~10 frames Maximum of 20 frames

16. March 10, 2005Herman Li16 Future work  Dynamic control Real time on-the-fly adjustment to studio equipments  Semantic indexing of video streams Express interest and query high level events  Continuity Management Allows checks for continuity in different footage

17. March 10, 2005Herman Li17 References  Augmenting Film and Video Footage with Sensor Data, Norman Makoto Su, Heemin Park, Eric Bostrom, Jeff Burke, Mani B. Srivastava, Deborah Estrin  ARS: http://www.ee.ucla.edu/~hmpark/arshttp://www.ee.ucla.edu/~hmpark/ars  PALOS: http://deerhound.ats.ucla.edu:7777/pls/portal/docs/PAGE/CENS_REP OSITORIES/TECHNOLOGIES/PALOS-TUTORIAL.PDF  http://mmsl.cs.ucla.edu/sylph  http://www.fxguide.com  http://computer.howstuffworks.com/mote.htm