1. Anahita: A System for 3D Video Streaming with Depth Customization
K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda
QCR & MIT & SFU
5 November 2014 1 1

2. Motivations 3D content is popular especially in theaters
 Controlled, homogenous environments 2 2

3. Internet 3D Video Streaming …
… is a lot more challenging because of …
complexity of perceiving and rendering depth on different displays
&
in dynamic network conditions 3 3

4. Sample 3D Displays 4

5. How do we see 3D? 5

6. Approximate Model for Depth [Holliman 04]
Positive disparity
Negative disparity z d d
Left eye
Left eye z e e p p
Right eye
Right eye 6 6

7. Depth & Comfort Zone [Shibata 11]
Comfort zone depends on:
Presented content
Viewing condition
Screen distance …
This plot shows how the depth range that is comfortable for viewers depends on viewing distance.
For large viewing distances we can reproduce depth up to infinity. This is why stereo works currently so well for cinema but not for desktop and mobile devices where the depth range is significantly reduced.
“The zone of comfort: Predicting visual discomfort with stereo displays” by Shibata et al. 2011 7

8. Rendering 3D Content Many different display technologies
From anaglyph to auto stereoscopic 8 8

9. 3D Display: Active/Passive Glasses9

10. 3D Display: Glasses-Free
Autostereoscopic
Parallax Barriers:
Blocks light in certain direction
Lenticular Lens:
Reflects light to certain direction 10

11. 3D Display: Multi-views11

12. Network Dynamics
Dynamic conditions
Bandwidth
Loss rate
Delay 12 12

13. In Summary … Depth perception heavily depends on
Display size and technology
User preferences
Current 3D streaming systems are simple add-on to 2D streaming 
Poor depth/visual perception and/or discomfort 13 13

14. 3D Streaming Systems Need to ..
Customize depth for different displays
Big TV … to phone
Serve different formats
Stereo, V+D, MV + D, …
Adapt to network dynamics
Wired, wireless, …, varying bandwidth
Manage many versions
Compared to 1—3 versions for 2D videos
Compared to 1-few versions for 2D videos 14

15. 5 x 7 x 3 = 105 versions for each video
Example
Display Technology
Network Bitrate
Display Size
Side By Side
Top Bottom
Frame Sequential
Anaglyph
Row-Interleaved
Column-Interleaved
Video Plus Depth
High Quality TV
Desktop
Laptop
Tablet
Cell Phone
Medium Quality
Low Quality
5 x 7 x 3 = 105 versions for each video 15

16. Our Challenge
Enable all devices to render best possible 3D videos From mobiles to big TVs 16

17. Our Solution …
In ancient time, Anahita is the source of all clean water streams that flow through golden channels to all lands and oceans on Earth. 
In Internet age, Anahita is the source of all high-quality 3D video streams that flow through network channels to all types of displays. 17

18. Our Contributions Show the need for depth optimization
Subjective study on different displays
Anahita: complete system for 3D streaming
Supports most current displays/technologies
Provides efficient management of 3D versions
Enables personalized depth
Is scalable and dynamic (using DASH)
Method for depth expansion and compression
Optimizes depth: from mobile phones to big TVs
Preserves scene structure (important for sports)
Does not introduce visual artifacts
Is computationally inexpensive 18 18

19. The Need for Depth Customization
Subjective study
10 subjects
2 displays (phone & 55” TV)
Series of short 3D clips (from soccer)
Show six different versions
Original
Depth compression
Depth expansion (4 cases)
Versions are shown in random to subjects
Subjects are asked which version is preferred 19 19

20. Results Best version is NOT the original one
Depth customization depends on display size 20 20

21. Depth Customization SPSS: Structure Preserving Scene Shifting
Expands or compresses depth in stereo video
Simple image processing operations
Preserves lines and planes
Suitable for field sports: soccer, football, tennis, … 21 21

22. SPSS: Basic Idea Control disparity of pixels Two operations
 increase/decrease depth
Two operations
Slant (affects vertical component of disparity)
Stretch (affects horizontal component of disparity) 22 22

23. Depth Gradient
2D+ depth Depth map
g = (gx, gy) 23

24. Depth Gradient
2D+ depth Depth map
g = (gx, gy) 24

25. SPSS: Slant Left eye image Right eye image
We propose depth manipulation technique which uses simple operations which does not require dense depth information or image-based rendering.
As a result the manipulartion are very efficient and do not produce noticable artifacts which are often for image-based rendering techniques. 25

26. SPSS: Slant
gy is changed 26

27. SPSS: Stretch
gx is changed 27

28. SPSS: Main Steps Calculate gradient g = (gx, gy) of scene disparity
Compute slant and stretch factor
Remap left/right views using: 28 28

29. SPSS: Sample Results
Before
e is the expansion value
After 29

30. SPSS: Sample Results
Before
After
e is the expansion value 30

31. SPSS: Coverage SPSS applies to scenes with planar depth
 long shots in field sports
Our analysis of multiple full games
60—70% of the game
Close-ups and short shots do not benefit from SPSS
Anahita: has automatic shot classifier 31 31

32. Anahita: System Architecture32

33. Anahita: 3D Version Manager33

34. Adaptive Streamer Based on DASH Versions are organized in segments
Most suitable 3D version is chosen for each client
Client can personalize: ask for more/less depth
Served using DASH 34 34

35. Implementation: Whole System
Server
Deployed on Amazon
Implements many 3D video processing operations, and our SPSS depth customization method
Clients
Mobile, tablet (mobile apps)
Stereo TV (web)
Desktop display (web)
Auto stereoscopic 55” display (custom app) 35 35

36. Evaluation: Subjective Study
5 displays
mobile, tablet, 15.6” laptop, 27” desktop, 55” TV
3 soccer 3D video clips from YouTube
Man United vs. Wigan (60 sec)
Chelsea vs. Wigan (24 sec)
Chelsea vs. Plymouth (20 sec)
15 subjects
viewed all clips (original & optimized) on all displays in random
Ranked depth from 1-poor to 5-Excellent 36 36

37. Sample Results
Significant improvements in all cases
up to 35% 37 37

38. Conclusions 3D streaming systems are more complex than 2D
Showed the need for depth adaptation
Proposed method to customize depth
Designed 3D streaming system
Evaluation study shows large gains 38 38

39. Anahita: Meaning
Ancient Persia: Anahita is source of all water, where warm and clear streams flow through golden channels to all lands and oceans on Earth 
Internet age: Anahita is source of all 3D videos, where high-quality streams of 3D videos flow through network channels to all types of displays