1 School of Computing Science Simon Fraser University, Canada Rate-Distortion Optimized Streaming of Fine-Grained Scalable Video Sequences Mohamed Hefeeda & ChengHsin Hsu 2 February 2007

2 Motivations  Multimedia streaming over the Internet is becoming very popular -More multimedia content is continually created -Users have higher network bandwidth and more powerful computers  Users request more multimedia content  And they look for the best quality that their resources can support

3 Motivations (cont’d)  Users have quite heterogeneous resources (bandwidth) -Dialup, DSL, cable, wireless, …, high-speed LANs  To accommodate heterogeneity  scalable video coding:  Layered coded stream -Few accumulative layers -Partial layers are not decodable  Fine-Grained Scalable (FGS) coded stream -Stream can be truncated at bit level

4 Motivations (cont’d)  Goal: Optimize quality for heterogeneous receivers  In general setting -FGS-coded streams -Multiple senders with heterogeneous bandwidth and store different portions of the stream  Why multiple senders? -Required in P2P streaming: Limited peer capacity and Peer unreliability -Desired in distributed streaming environment: Disjoint network path  Better streaming quality

5 Our Optimization Problem  Assign to each sender a rate and bit range to transmit such that the best quality is achieved at the receiver.  Consider a simple example to illustrate the importance of this problem

6 Example: Different Streaming Schemes Non-scalableLayered

7 Example: Different Streaming Schemes FGS ScalableOptimal FGS Scalable

8 Problem Formulation  First: single-frame case -Optimize quality for individual frames  Then: multiple-frame case -Optimize quality for a block of frames -More room for optimization

9 Input Parameters  T : fixed frame period  n : number of senders  b i : outgoing bandwidth of sender i  b I : incoming bandwidth of receiver  s i : length of (contiguous) bits held by sender i  Assume s 1 <= s 2 <= …… <= s n

10  Allocation: A = {(Δ i, r i ) | i=1, 2, ……, n} -Δ i : number of bits assigned to i -r i : streaming rate assigned to i  Specifies: -Sender 1 sends range [0, Δ 1 -1] at rate r 1 -Sender 2 sends range [Δ 1, Δ 1 +Δ 2 -1] at rate r 2 -… -Sender i sends range at rate r i Outputs

11 Integer Programming Problem  Minimize distortion  Subject to: -on-time delivery -assigned range is available -assigned rate is feasible -Aggregate rate not exceeds receiver’s incoming BW

12 How do we Compute Distortion?  Using Rate-Distortion (R-D) models -Map bit rates to perceived quality -Optimize quality rather than number of bits  Approaches to construct R-D models -Empirical Models: Many empirical samples  expensive -Analytic Models: Quality is a non-linear function of bit rate, e.g., log model [Dai 06] and GGF model [Sun 05] -Semi-analytic Models: A few carefully chosen samples, then interpolate, e.g., piecewise linear R-D model [Zhang 03]  Detailed analysis of R-D models in [Hsu 06]

13  Within each bitplane, approximate R-D function by a line segment  Line segments of different bitplanes have different slopes The Linear R-D Model

14 Visual Validation of Linear R-D Model Mother & Daughter, frame 110 Foreman, frame 100

15 Rigorous Validation of Linear R-D Model  Average error is less than 2% in most cases

16  Let y i be number of bits transmitted from bitplane i  Distortion is: -d : base layer only distortion -g i : slope of bitplane i -z : total number of bitplanes Using the Linear R-D Model

17 Integer Linear Programming (ILP) Problem  Linear objective function  Additional constraints -number of bits transmitted from bit plane h does not exceed its size l h -bits assigned to senders are divided among bitplanes

18 Solution of ILP is a Valid FGS Stream  Lemma 1: -An optimal solution for the integer linear program produces a contiguous FGS-encoded bit stream with no bit gaps  Proof sketch -minimizing -Since g 1 < g 2 < …… <g n <0 (line segment slopes), -the ILP will never assign bits to y i+1 if y i is not full

19  Solving ILP problem is expensive  Solution: Transform it to Linear Programming (LP) problem -Relax variables to take on real values  Objective function and constraints remain the same Linear Programming Relaxation

20  Solve LP  -Result is real values  Then, use the following rounding scheme for solution of the ILP Efficient Rounding Scheme

21 Correctness/Efficiency of Proposed Rounding  Lemma 2 (Correctness) -Rounding of the optimal solution of the relaxed problem produces a feasible solution for the original problem  Lemma 3 (Efficiency: Size of Rounding Gap) -The rounding gap is at most nT + n, where n is the number of senders and T is the frame period -Example: T=30 fps, n=30, the gap is 32 bits -Indeed negligible (frame sizes are in order of KBs)

22 FGSAssign: Optimal Allocation Algorithm  Solving LP (using Simplex method for example) may still be too much -Need to run in real-time on PCs (not servers)  Our solution: FGSAssign -Simple yet optimal allocation algorithm -Greedy: Iteratively allocate bits to sender with smallest s i (stored segment) first

23 Pseudo Code of FGSAssign  Sort senders based on s i, s 1 ≤ s 2 ≤ …… ≤ s n ;  x 0 = …… = x n = 0; Δ 1 = …… = Δ n = 0; r agg = 0;  for i = 1 to n do  x i = min(x i−1 + b i T, s i );  r i = (x i − x i−1 )/T ;  if (r agg + r i < b I ) then  r agg = r agg + r i ;  Δ i = x i − x i−1 ;  else  r i = b I − r agg ;  Δ i = T × r i ;  return  endfor

24 Optimality of FGSAssign  Theorem 1 -The FGSAssign algorithm produces an optimal solution in O(n log n) steps, where n is the number of senders.  Proof: see paper  Experimentally validated as well.

25 Multiple-Frame Optimization  Solve the allocation problem for blocks of m frames each  Objective: minimize total distortion in block  Why consider multiple-frame optimization? -More room for optimization -Less computation overhead: solve the problem less often

26 Multiple-Frame Optimization: Why?  More room for optimization: higher quality and less quality fluctuation

27 Multiple-Frame Optimization  Formulation (in the paper): -Straightforward extension to single-frame with lager number of variables and constraints -Computationally expensive to solve  Our Solution: mFGSAssign algorithm -Heuristic (close to optimal results) -Achieves two goals: Minimize total distortion in a block Reduce quality fluctuations among successive frames

28 mFGSAssign: High-Level Description 1.Estimate a target distortion D that is feasible and achieves the two goals (binary search) 2.Compute for each frame f in the block its bit budget B f 3.For each frame f, call FGSAssign to allocate B f among senders

29 Computing Target Distortion  Is bit budget enough to transmit all frames at distortion level D ? -D u : distortion upper bound -D l : distortion lower bound -D : distortion estimate

30 Efficiency of mFGSAssign  Lemma 5 -mFGSAssign terminates in O(m n log n) steps, where n is the number of senders and m is the number of frames in a block  Much more efficient than linear programming approach

31 Experimental Setup  Software used -MPEG-4 Reference Software ver 2.5 Augmented to extract R-D model parameters  Algorithms implemented (in Matlab) -LP solutions using Simplex for the single-frame and multiple-frame problems -FGSAssign algorithm -mFGSAssign algorithm -Nonscalable algorithm for baseline comparisons

32 Experimental Setup (cont’d)  Streaming scenarios -Four typical scenarios for Internet and corporate environments  Testing video sequences -Akiyo, Mother, Foreman, Mobile (CIF) -Sample results shown for Foreman and Mobile

33 Single Frame: Quality (PSNR) Foreman, Scenario I Mobile, Scenario III  Quality Improvement: 1--8 dB  FGSAssign is optimal

34 Multiple Frame: Quality (PSNR) Foreman, Scenario II Mobile, Scenario III  Scalable: higher improvement than single frame  mFGSAssign: almost optimal (< 1% gap)

35 Fluctuation Reduction Foreman, Scenario II Mobile, Scenario III  Small quality fluctuations in successive frames

36 Running Time Foreman, Scenario I Foreman, Scenario IV  Small and stable running times for mFGSAssign, unlike mOPT (Simplex)  mFGSAssign can be used in real time

37 Reconstructed Pictures Nonscalable mFGSAssign

38 Conclusions  Formulated and solved the bit allocation problem for single and multiple frame cases  Nonlinear problem  integer linear program -Using linear R-D model  Integer linear program  linear program -Using simple rounding scheme  Efficient Algorithms -FGSAssign: Optimal and efficient -mFGSAssign: close to optimal in terms of average distortion, reduces quality fluctuations, runs in real time  Significant quality improvements shown by our experiments

39 Thank You! Questions??  All programs/scripts/videos are available: