1. Research & Innovation 1 An Industry Perspective on VVG Research Oliver Grau BBC Research & Innovation VVG SUMMER SCHOOL '07

2. Research & Innovation 2 Overview Introduction –(Computer) Vision, Video & Graphics in media production Integration of (real) Video & Graphics Production visualisation tools Free-viewpoint video for Sport visualisation Summary

3. Research & Innovation 3 Introduction BBC is a public funded broadcaster producing different kinds of media. Mainly: Radio, TV and Online Video: One of the ‘Main businesses’ Graphics – used to: –Add editorial value –Make programmes (visually) more exciting –Create new user’s experience Vision: Big toolbox to make things happen

4. Research & Innovation 4 Introduction Added editorial value

5. Research & Innovation 5 Introduction New user experience: 3D Virtual environment CBBC Adventure Rock: www.bbc.co.uk/cbbc/adventurerock/

6. Research & Innovation 6 Introduction

7. Research & Innovation 7 Integration of video & graphics ‘Classical’ application: Weather forecast Studio with chroma-key facility

8. Research & Innovation 8 What makes an integrated ‘virtual scene’ looking ‘real’ Real Scene Virtual Scene Depth perception Camera perspective Occlusions Shadows Reflections

9. Research & Innovation 9 Integration of video & graphics Virtual studio –inserting actor into virtual scene Virtual background Moving camera Camera Tracking

10. Research & Innovation 10 3D reconstruction of dynamic scenes in the studio Virtual studios have limited features regarding optical interaction of real and virtual objects More realistic integration requires full optical interaction in 3D –Shadows + reflections –Occlusions –Free choice of camera angle

11. Research & Innovation 11 3D reconstruction of dynamic scenes in the studio 12 fixed, calibrated cameras with chroma-keying facility Developed in the IST-ORIGAMI project

12. Research & Innovation 12 Overview Block diagram Volumetric visual hull computation.. Input images, Camera parameters Surface computation Marching cube algorithm Volumetric model Surface model Texture mapping

13. Research & Innovation 13 Visual hull computation Visual hull computation from image silhouettes Camera-1 Camera-2 Camera-3

14. Research & Innovation 14 Visual hull computation Visual hull computation from image silhouettes Camera-1 Camera-2 Camera-3

15. Research & Innovation 15 Artefacts in Visual hull-based 3D reconstruction Fundamental feature: –only parts of the volume that are visible as background in at least one of the silhouette images are taken out  no concavities can be modelled Occlusion errors  Phantom volumes  Reduced by increasing number of cameras Approximation errors Sampling errors

16. Research & Innovation 16 Visual hull-based 3D reconstruction Sampling errors –Binary volumetric scene representation  Quantisation noise 12 cameras 6 cameras12 cameras

17. Research & Innovation 17 Artefacts in dynamic visual hulls “Moving edges” as effect of discrete voxel grid C2 C1 object reconstruction (moving) edge

18. Research & Innovation 18 Improved visual hull computation: Super-sampling –Sub-divide voxels by L SS levels –Assign voxel a (pseudo-) continuous value Camera-1

19. Research & Innovation 19 Improved visual hull computation Gaussian smoothing (Optional) complexity reduction

20. Research & Innovation 20 Results 3D model using super-sampling (20000 triangles) After Gaussian smoothing Standard visual hull / marching cube

21. Research & Innovation 21 Results Video from ORIGAMI demo production

22. Research & Innovation 22 Results Non-photorealistic rendering

23. Research & Innovation 23 Re-lighting Studio illumination map 3D scene model irradiance Specular component Original image Light-neutral image

24. Research & Innovation 24 Re-lighting