Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR & MIT & SFU 5 November 2014 1 1

Motivations 3D content is popular especially in theaters  Controlled, homogenous environments 2 2

Internet 3D Video Streaming … … is a lot more challenging because of … complexity of perceiving and rendering depth on different displays & in dynamic network conditions 3 3

Sample 3D Displays 4

How do we see 3D? 5

Approximate Model for Depth [Holliman 04] Positive disparity Negative disparity z d d Left eye Left eye z e e p p Right eye Right eye 6 6

Depth & Comfort Zone [Shibata 11] Comfort zone depends on: Presented content Viewing condition Screen distance … This plot shows how the depth range that is comfortable for viewers depends on viewing distance. For large viewing distances we can reproduce depth up to infinity. This is why stereo works currently so well for cinema but not for desktop and mobile devices where the depth range is significantly reduced. “The zone of comfort: Predicting visual discomfort with stereo displays” by Shibata et al. 2011 7

Rendering 3D Content Many different display technologies From anaglyph to auto stereoscopic 8 8

3D Display: Active/Passive Glasses 9

3D Display: Glasses-Free Autostereoscopic Parallax Barriers: Blocks light in certain direction Lenticular Lens: Reflects light to certain direction 10

3D Display: Multi-views 11

Network Dynamics Dynamic conditions Bandwidth Loss rate Delay 12 12

In Summary … Depth perception heavily depends on Display size and technology User preferences Current 3D streaming systems are simple add-on to 2D streaming  Poor depth/visual perception and/or discomfort 13 13

3D Streaming Systems Need to .. Customize depth for different displays Big TV … to phone Serve different formats Stereo, V+D, MV + D, … Adapt to network dynamics Wired, wireless, …, varying bandwidth Manage many versions Compared to 1—3 versions for 2D videos Compared to 1-few versions for 2D videos 14

5 x 7 x 3 = 105 versions for each video Example Display Technology Network Bitrate Display Size Side By Side Top Bottom Frame Sequential Anaglyph Row-Interleaved Column-Interleaved Video Plus Depth High Quality TV Desktop Laptop Tablet Cell Phone Medium Quality Low Quality 5 x 7 x 3 = 105 versions for each video 15

Our Challenge Enable all devices to render best possible 3D videos From mobiles to big TVs 16

Our Solution … In ancient time, Anahita is the source of all clean water streams that flow through golden channels to all lands and oceans on Earth.  In Internet age, Anahita is the source of all high-quality 3D video streams that flow through network channels to all types of displays. 17

Our Contributions Show the need for depth optimization Subjective study on different displays Anahita: complete system for 3D streaming Supports most current displays/technologies Provides efficient management of 3D versions Enables personalized depth Is scalable and dynamic (using DASH) Method for depth expansion and compression Optimizes depth: from mobile phones to big TVs Preserves scene structure (important for sports) Does not introduce visual artifacts Is computationally inexpensive 18 18

The Need for Depth Customization Subjective study 10 subjects 2 displays (phone & 55” TV) Series of short 3D clips (from soccer) Show six different versions Original Depth compression Depth expansion (4 cases) Versions are shown in random to subjects Subjects are asked which version is preferred 19 19

Results Best version is NOT the original one Depth customization depends on display size 20 20

Depth Customization SPSS: Structure Preserving Scene Shifting Expands or compresses depth in stereo video Simple image processing operations Preserves lines and planes Suitable for field sports: soccer, football, tennis, … 21 21

SPSS: Basic Idea Control disparity of pixels Two operations  increase/decrease depth Two operations Slant (affects vertical component of disparity) Stretch (affects horizontal component of disparity) 22 22

Depth Gradient 2D+ depth - Depth map g = (gx, gy) 23

Depth Gradient 2D+ depth - Depth map g = (gx, gy) 24

SPSS: Slant Left eye image Right eye image We propose depth manipulation technique which uses simple operations which does not require dense depth information or image-based rendering. As a result the manipulartion are very efficient and do not produce noticable artifacts which are often for image-based rendering techniques. 25

SPSS: Slant gy is changed 26

SPSS: Stretch gx is changed 27

SPSS: Main Steps Calculate gradient g = (gx, gy) of scene disparity Compute slant and stretch factor Remap left/right views using: 28 28

SPSS: Sample Results Before e is the expansion value After 29

SPSS: Sample Results Before After e is the expansion value 30

SPSS: Coverage SPSS applies to scenes with planar depth  long shots in field sports Our analysis of multiple full games 60—70% of the game Close-ups and short shots do not benefit from SPSS Anahita: has automatic shot classifier 31 31

Anahita: System Architecture 32

Anahita: 3D Version Manager 33

Adaptive Streamer Based on DASH Versions are organized in segments Most suitable 3D version is chosen for each client Client can personalize: ask for more/less depth Served using DASH 34 34

Implementation: Whole System Server Deployed on Amazon Implements many 3D video processing operations, and our SPSS depth customization method Clients Mobile, tablet (mobile apps) Stereo TV (web) Desktop display (web) Auto stereoscopic 55” display (custom app) 35 35

Evaluation: Subjective Study 5 displays mobile, tablet, 15.6” laptop, 27” desktop, 55” TV 3 soccer 3D video clips from YouTube Man United vs. Wigan (60 sec) Chelsea vs. Wigan (24 sec) Chelsea vs. Plymouth (20 sec) 15 subjects viewed all clips (original & optimized) on all displays in random Ranked depth from 1-poor to 5-Excellent 36 36

Sample Results Significant improvements in all cases up to 35% 37 37

Conclusions 3D streaming systems are more complex than 2D Showed the need for depth adaptation Proposed method to customize depth Designed 3D streaming system Evaluation study shows large gains 38 38

Anahita: Meaning Ancient Persia: Anahita is source of all water, where warm and clear streams flow through golden channels to all lands and oceans on Earth  Internet age: Anahita is source of all 3D videos, where high-quality streams of 3D videos flow through network channels to all types of displays