H.264/AVC for Wireless Applications Thomas Stockhammer, and Thomas Wiegand Institute for Communications Engineering, Munich University of Technology, Germany Image Processing Department, Fraunhofer Institute for Telecommunications Heinrich-Hertz-Institut MoMuC 2003 INTERNATIONAL WORKSHOP ON MOBILE MULTIMEDIA COMMUNICATIONS

Outline Introduction H.264/AVC  Compression Efficiency and Encoder Flexibility  Features for Multimedia Messaging and Wireless packet-based Streaming  Features for Wireless Conversational Services- Rate Control and Error Resilience  Rate-Distortion Optimized Mode Selection Selected Simulation Results Conclusion

Introduction The primary goals for H264/AVC are improved coding efficiency and improved network adaptation. H.264/AVC distinguishes between two different conceptual layers, the Video Coding Layer (VCL), and the Network Abstraction Layer (NAL).

Introduction Due to the likely business models in emerging wireless systems, in which the end-user’s costs are proportional to the transmitted data volume, and also due to limited resources bandwidth and transmission power, compression efficiency is the main target for wireless video and multimedia applications. This makes H.264/AVC coding an attractive candidate for all wireless applications including Multimedia Messaging Services (MMS), packet- switched streaming services (PSS) and conversational applications.

Introduction For efficient transmission in different environments not only coding efficiency is relevant, but also the seamless and easy integration of the coded video into all current and possible future protocol and multiplex architectures. The VCL specifies an efficient representation for the coded video signal. The NAL defines the interface between the video codec itself and the outside world.

Structure of H.264/AVC video encoder Control Data Video Coding Layer Data Partitioning Network Abstraction Layer H.320MP4FFH.323/IPMPEG-2Etc. Coded Macroblock Coded Slice/Partition

H.264/AVC Compression Efficiency and Encoder Flexibility Figure 1 H.264/AVC Encoder realization with coding options.

H.264/AVC Features for Multimedia Messaging and Wireless packet-based Streaming MMS (Multimedia Messaging Service)  Compression efficiency : due to the strict separation of encoding, transmission and decoding  IDR (Instantaneous Decoder Refresh): for random access and fast forward.  Rate control: for constant video quality  Reliable transmission: for wireless link layer

H.264/AVC Features for Multimedia Messaging and Wireless packet-based Streaming Streaming: online transmission and decoding Short-term variances in the bit-rate  With an appropriate setting of the initial delay and receiver buffer a certain quality of service can be guaranteed.  Wireless channels commonly provide a constant bit-rate and reliable transmission by using an acknowledged mode within a window of a few seconds.

H.264/AVC Features for Multimedia Messaging and Wireless packet-based Streaming  Long-term variances in the bit-rate: due to distance, shadowing, re-newed resource allocation Channel adaptive streaming technologies  Adaptive media playout  Rate-Distortion optimized packet scheduling  Frame dropping if the channel rate fluctuates In small range: non-reference frames => temporal scalability In large scale: I frames, SP (Switching Predictive) frames

H.264/AVC Features for Wireless Conversational Services-Rate Control and Error Resilience The low delay constraint has two main impacts on the video transmitted over wireless bearer services with constant bit-rate.  Fast quantization parameter adaptation  Temporally backward references in MC An error-resilient video coding standard suitable for conversational wireless services has to provide to combat two problems:  it is necessary to minimize the visual effect of errors within one frame, and  as errors cannot be avoided, the well-known problem of spatio-temporal error propagation in hybrid video coding has to be limited.

H.264/AVC Features for Wireless Conversational Services-Rate Control and Error Resilience Error-resilience features included in the H.264/AVC standard Slice-Structured Coding  A slice is a sequence of MBs and provides spatially distinct resynchronization points within the video data for a single frame.  Advantages: Packet loss probability can be reduced if slices and transmission packets are relatively small. re-synchronization possibility within one frame  Disadvantages: Increase packet overhead Loss of intra-frame prediction  Group-of-Block (GOB) and Slice Interleaving  Reduce no coding overhead in the VCL, but the costly RTP overhead of up to 40 bytes per packet.

H.264/AVC Features for Wireless Conversational Services-Rate Control and Error Resilience Flexible MB Ordering (FMO)  FMO permits the specification of different patterns for the mapping of MBs. Data partitioning  reduce visual artifacts resulting from packet losses, especially if prioritization or unequal error protection is provided by the network. Encoding of single MBs for regions Multiple reference frames limit error propagation

H.264/AVC Rate-Distortion Optimized Mode Selection The concept of selecting appropriate coding options in optimized encoder designs for many video coding standards is based on rate-distortion optimization algorithms The Lagrange parameter for appropriate weighting of rate and distortion has to be selected appropriately. In the H.264/AVC test model, the Lagrangian mode selection is used for motion vector search as well as MB mode and reference frame selection.

Selected Simulation Results Compression Efficiency Figure 2 Coding performance of H.264/AVC codec compared to state-of-the-art video coding standards for QCIF test sequence foreman at frame rate 10 Hz.

Selected Simulation Results Slices and Channel-Adaptive Intra Updates Figure 3 Cumulative distribution of decoded PSNR for different NAL unit erasure rates for the estimation of the expected distortion in the encoder. 8% 30% Channel statistics are taken into account into the selection of the coding option in the encoder. Packet loss rate=4% Pure R-D

Selected Simulation Results Slices and Channel-Adaptive Intra Updates Figure 4 Cumulative distribution of decoded PSNR for different error-resilience strategies: channel-optimized intra updates with and without slice structuring for different assumed loss probabilities p. Slice structuring + Channel adaptive Slice structuring Channel adaptive

Selected Simulation Results Exploiting Feedback in Video Encoding Figure 5 Cumulative distribution of decoded PSNR for different error-resilience strategies: RD-optimized intra updates with slice structuring, and feedback mode with and without slice structuring for delay d=2 and d=4. RD-optimized + slice structuring Feedback (2) + slice structuring Feedback (2) Feedback (4) + slice structuring Feedback (4)

Conclusion In addition to excellent coding efficiency, the design of H.264/AVC also takes into account network adaptation providing large flexibility for its use in wireless applications. In experimental results based on common test conditions it has been shown that in case without any feedback, several slices in combination with channel-adaptive rate-distortion optimized mode selection is a promising approach. In case of available feedback, the application of multiple reference frames to exclude error propagation without slice structuring provides excellent results.