1. M15672: View synthesis software and assessment of its performance
Mateusz Gotfryd Krzysztof Wegner Marek Domański
Chair of Multimedia Telecommunications and Microelectronics Poznań University of Technology, Poland
July, 21th 2008

2. Outline Algorithm description Test sequences Used depth maps
Experiment description
Objective video quality measure
Subjective video quality measure
Results
Summary

3. Algorithm description
Virtual camera can be synthesized from a real camera (the real reference view) and the respective depth map
Unfortunately such an approach suffers from occlusion
Virtual camera can be synthesized more correctly if two reference views are used

4. Block diagram
View synthesis algorithm

5. View synthesis algorithm
Proposed algorithm is composed of two identical, separate paths
Each virtual view is synthesized from one reference view
In the end both images of virtual view are merged into one
Futher we describe single path in detail (right one)

6. Virtual view depth map
Coordinates of each point in a reference view are transformed into coordinates of a point in the virtual view and virtual view depth map is created
In the case when two points from reference view are transformed into the virtual view and have the same coordinates, always the points closer to the camera location are chosen
Visibility problem
Proper depth values

7. Virtual view depth map
Resultant virtual view depth map has many small black holes on surfaces which have been rotated during the transformation from reference view into a virtual view
Median filter is used on virtual view depth map
Some regions in view have been uncovered
Black regions have unknown depth
Uncovered regions

8. Virtual view synthesis
Color information is copied from reference view based on inverse homography matrix
Calculated coordinates of corresponding point in reference view are not on pixel gird therefore interpolation is used

9. Merging of two virtual view images
Images from two paths are merged into one
Unknown regions from first path are filled with information from second path

10. Merging of two virtual view images
Images from two paths are merged into one
Unknown regions from first path are filled with information from second path

11. Filling holes
Missing areas are interpolated from neighboring pixels

12. Contour correction
Aliasing and blurring on the edges of the object are main reasons of „ghosting” effect
Unknown regions are outlined by 1 pixel-width

13. Test sequences Sequences provided by Fraunhofer HHI
Book Arrival,
Leaving Laptop,
Alt Moabit.
Only 3 views (2nd, 3rd and 4th) have been used in the experiments

14. Depth maps
Required depth maps were calculated with software provided by Nagoya University

15. Experiment description
Video objective quality measured by PSNR
Video subjective quality measured by Mean Opinion Score (MOS)
15 human subjects
Rating range from 1 to 10
Collected opinions have been averaged

16. Results

17. Single frame subjective quality

18. View synthesis quality

19. View synthesis quality

20. Average synthesis quality

21. Summary We have presented a new view synthesis software
Subjective experiments have been carried out with view-synthesis software provided to MPEG
Our proposal received good results
in criteria of objective quality measure (PSNR)
and in case of subjective quality measure (MOS)