1. Current Work on Video Coding Raj Kumar

2. Overview Fine-Grained Scalability (FGS) What is it and why do we need it What are its drawbacks Our new scheme (FGS+) B-Frame only scheme All-Frame Scheme Future Directions

3. Fine Grained Scalability What is it What are the pros and cons IPBBB Base Layer Enhancement Layer

4. Traditional Vs Fine-grained Encoding IBPBB IPBBB Base Layer Enhancement Layer

5. Temporal FGS IP Base Layer Spatial Enhancement Layer Temporal Enhancement Layer

6. Problem with Temporal FGS Reference Frames are poor quality Doesn’t use motion-prediction well Compression ratio suffers compared to traditional encoders (approx 2-2.5 DB)

7. Insight FGS provides two degrees of freedom SNR Temporal Is there an optimal path that will maximize overall quality. How do we determine it How do we take advantage of it

8. Idea Can we use a part of the enhancement layer, in addition to the base layer, to predict motion? We can if we know how much of the base layer will be present at the decoder when the temporal layers are introduced So if we build in rules about when temporal layers are introduced relative to spatial quality, we can!

9. Implementation Increase SNR quality to a predetermined point Then improve temporal quality Further improve SNR quality at the new frame-rate And so on….

10. FGS+ Scheme (B-frame only) IP Base Layer Spatial Enhancement Layer Temporal Enhancement Layer

11. Two issues When (at what SNR quality) do we introduce new temporal-frames? How much extended-reference do we use

12. Introducing Temporal Frames We conducted a study where users chose preferred frame-rate at different bit-rates: As the bit-rate goes up, people prefer better frame- rates High-motion videos (Stefan, Coastguard) require a quicker transition

13. Determining the size of the reference Plot SNR performance as a function of enhancement at various bit-rates Choose enhancement corresponding to best quality

14. Performance Improvement over FGS varies from 0.19 dB to 1.28 dB At low bit-rates simple videos benefit (Coastguard) At high bit-rates complex videos benefit (Mobile)

15. FGS+ for All-Frames We can improve performance further by using enhancement for base-layer P-frames also But P-frames are present at all bit-rates. How much enhancement do we use?

16. FGS+ Scheme (All-frames) IP Base Layer Spatial Enhancement Layer Temporal Enhancement Layer

17. Improvement in Performance At 3 bit-planes Performance varies from –0.03 dB to 0.59 dB Performance will degrade at low bit- rates, due to incomplete references

18. Conclusion FGS+ uses a spatio-temporal notion of video-quality Improves performance of traditional FGS from 0.16 to 1.61 dB

19. Current Issues Spatio-Temporal preference curves vary for different videos How do we choose the correct curve automatically Use Motion-Vectors, Image-Complexity

20. Future Directions Factors other than video-characteristics do affect spatio-temporal preferences Environment (resolution, display-type, lighting) User preferences (moods, some prefer spatial- quality always) How do we account for these other dimensions ? Large dimensional space requires many (thousands) of sample points

21. Future Directions Build online experiment Interactive web-based tool that lets user watch many videos Allow user to change spatio-temporal settings Record data corresponding to changes (video-characteristics, display resolution and type, user information)