REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha
The Problem Video effects: Difficult Expensive Time consuming
The Solution Application: Automated Real time
Automation Depth sensor Capture colour with camera (RGBD) Capture depth with sensor (RGBD)
Research Question Can image based effects be applied to video footage in real-time?
Aim Develop a system to: Capture RGBD from a camera and depth sensor Render a variety of different video effects Test the system: Benchmark performance Evaluate the viability of real-time video effects
System Design
Mapping System Prepare the data Depth map Point cloud Normal map Refine the data Real time input Reflectance map
Video Capture
Normal Map Generation Depth Map to Normal Map
Normal Map Generation Depth Map to Normal Map
Reflectance Map Estimate initial lighting conditions Use normals Estimate reflectance at each point
Map Refinement
Effects System Four effects: Fog Synthesis Scene Relighting Depth of Field Cartoon Shading
Fog Synthesis
Scene Relighting
Depth of Field
Cartoon Shading
User Interface Interface with cameras View effects Edit parameters
Performance Utilise graphics processors Fragment shaders CUDA
Testing Benchmark o Different effects o Different parameters o Different resolutions Performance profiling
Success Criterion Performance Entire process at 25fps
Work Allocation
Thank You Questions?