Limin Liu, Member, IEEE Zhen Li, Member, IEEE Edward J. Delp, Fellow, IEEE CSVT 2009

Outline  Introduction  Low-complexity Video Encoding Approaches for SVC  SVC Using Wyner-Ziv Video Coding  SVC Using Backward-Channel Aware Wyner- Ziv Video Coding  Experimental Results  Conclusion

Basic Diagram of Video Surveillance Systems

Introduction  Video surveillance system has been widely used to enhance public safety and privacy protection.  Needs efficient transmission and storage.  H.264/AVC is high computational complexity at the encoder. This can increase the cost of a surveillance system.

Video Compression Design Considerations for Surveillance Video  General case Movie studio only needs to encode once to put on DVD but decoded by the consumers many times. Heavy encoder, light decoder.  Surveillance systems Encoder is implemented in a simple and low-cost video surveillance camera. The video is decoded and analyzed at central server. Light encoder, heavy decoder.  Other requirements High coding efficiency Error resilience requirements

Surveillance Video Compression (SVC)  Low-complexity encoder based on Wyner-Ziv coding principles to address the tradeoff between computational complexity and coding efficiency.  Backward-Channel Aware Wyner-Ziv (BCAWZ) video coding to improve coding efficiency while maintaining low-complexity at the encoder.  Error resilience scheme for BCAWZ to keep reliable transmission in the backward-channel.

Low-complexity Video Encoding Approaches for SVC  Slepian-Wolf Theorem Construct correlation between X and Y R X ≥ H(X|Y ) R Y ≥ H(Y |X) R X + R Y ≥ H(X, Y ) Regardless of its access to side information Y, encoder A can encode X with high fidelity as long as the decoder A has access to Y. Lower bound

Low-complexity Video Encoding Approaches for SVC  Wyner-Ziv Video Coding (WZVC ) Temporal correlation among frames is exploited at the decoder instead of encoder. Each frame is independently encoded at the encoder. Computational intensive job of motion estimation is shifted to decoder.  WZVC is an ideal choice for SVC due to the light encoder characteristics.

SVC Using Wyner-Ziv Video Coding video Initial estimate Turbo code ->parity bits LDPC code ->syndrome bits

SVC Using Wyner-Ziv Video Coding  Video sequence is divided into two groups. Key frames ○ Encoded by H.264 INTRA encoder. ○ Serve to side information at decoder Wyner-Ziv frames ○ Encoded by channel coding method. ○ Turbo code  parity bits ○ Low-density-parity-check(LDPC) code  syndrome bits

SVC Using Wyner-Ziv Video Coding  Key frames Decoded by the H.264 INTRA decoder. Supply side information to other frames by find initial estimate. Initial estimate is derived from previously reconstructed key frames. ○ Use co-located pixel value in (n-1)th frame. ○ Take average of the co-located pixel value at (n-1)th and (n+1)th frames.  Wyner-Ziv frames Quality is low, temporal correlation is not fully exploited. To obtain higher quality, motion estimation can be done at decoder.

SVC Using Wyner-Ziv Video Coding  Higher quality initial estimate method1 ○ Search for the motion vector MVn-1 of the co-located block in (n-1)th frame as the predictor of current block. ○ The reference block in (n-1)th frame is the initial estimate. method2 ○ Obtain MV F and MV B ○ Use to find P F1 Use to find P F2 Use to find P B1 Use to find P B2 ○ Initial estimate

SVC Using Wyner-Ziv Video Coding  The channel decoder uses initial estimate and the incoming parity or syndrome bits to decode the frame.  Assume decoder can communicate with the encoder to request more bits until correctly decoded. Wyner-Ziv video codec formulates the video decoding problem as an error correction problem. If the parity or syndrome bits are lost or corrupted during transmission, it will add complexity. Standard video codec may not be correctly decoded when bits are lost and error can propagate to the following frame.

SVC Using Backward-Channel Aware Wyner-Ziv Video Coding  WZVC with INTRA key frames coding efficiency is higher than INTRA but much lower than INTER.  The distance between two neighboring key frames too far or too close will result in lower coding efficiency and degrading the quality of side information.  Use Backward-Channel Aware Motion estimation (BCAME).  The basic idea BCAME is to perform motion estimation at decoder and send the motion information back to encoder through backward-channel.  May increase the latency and cause problem for some SVC.

SVC Using Backward-Channel Aware Wyner-Ziv Video Coding video Even frames Odd frames(excluding 1, 3) 1, 3 frames  H.264 INTRA encoder  H.264 INTRA decoder Mode selection

SVC Using Backward-Channel Aware Wyner-Ziv Video Coding  Encode first and third frame as INTRA frames.  All the other odd frames are encoded with BCAME called BP frames.  All the even frames are encoded as a Wyner-Ziv frame.

SVC Using Backward-Channel Aware Wyner-Ziv Video Coding  BP frames Mode1Mode2 Motion vectors are sent back to encoder. Encoder do mode selection by MSE or SAD. Encoder use received motion vector with previous reconstructed BP frames to generate MC reference for current BP frame. Residual between current BP and reference is than transformed and entropy-coded.

Error Resilience In Backward-Channel Aware Wyner-Ziv Video Coding  Problem If there is error or delay at backward- channel. Motion vector is not updated and encoder continues to use it. Encoder and decoder use different motion vectors in motion compensation for the same frame.

Error Resilience In Backward-Channel Aware Wyner-Ziv Video Coding  Two-stage error-resilient procedure Insert the index of the frame as header info before sending Report error when receive index not match the frame number at the encoder Encodes the key frames as INTRA and send frame type to decoder Coding key frames as BP frames decoderencoder De-synchronization OK Synchronization channel

Experimental Results  H.264/AVC JM8.0  WZVC with INTRA, INTER or BP key frames Odd-frames -> key frame Even-frame ->Wyner-Ziv frame  Block-size -> 8*8  WZVC with INTER key frames First-frame -> INTRA Other key frames -> P frame  H.264/AVC INTER coding First-frame -> INTRA Other frames -> IBPBP GOP structure Quarter-pel motion search Search range -> 32 Number of reference frames -> 3

Experimental Results  Comparison WZVC with INTRA key frames ○ dB gain for low motion sequences ○ dB gain for high motion sequences H.264 INTRA coding  Discussions Low motion sequence is more continuous. Hence motion vectors derived from previous reconstructed frame is good. Many surveillance video are low motion. WZVC encoder complexity is comparable to H.264 INTRA coding. WZVC can provide significant rate reduction compared to H.264 INTRA coding.

Experimental Results  Comparison WZVC with INTRA key frames ○ 7-8 dB loss ○ dB loss H.264 INTER coding  Discussions INTER coding is much more efficient than INTRA coding especially in low motion sequences.

Experimental Results  Comparison BCAWZ ○ 5-7 dB gain WZVC with INTRA key frames  Comparison BCAWZ ○ dB loss H.264 INTER coding  Discussions This gain is achieved with marginal complexity increase at the encoder. BCAWZ ○ Two motion vector send from decoder H.264 INTER coding ○ 4*2M*2M candidate motion vector for a search range of M. (in this paper is 32)

Experimental Results  Comparison BCAWZ ○ 2-3 dB loss WZVC with INTER key frames  Discussions Use INTER key frames significantly increase the encoder complexity.

Experimental Results  Comparison WZVC with INTRA key frames BCAWZ (BP key frame)  Discussions The visual quality of BP frame is significantly higher than WZVC with INTRA key frames.

Experimental Results  Backward-channel /forward- channel bandwidth Lower rate : 10-15% Higher rate : 5% This usage can be satisfied in SVC.  Error resilience performance for BCAWZ Without error resilience ○ Sharply drops With error resilience ○ Effectively recover from backward- channel delay or erasure.

Conclusion  Present a low-complexity video encoding framework for surveillance video compression and transmission to address the tradeoff between computational complexity and coding efficiency.  BCAWZ can achieve significantly higher coding efficiency than H.264/AVC INTRA coding as well as existing Wyner–Ziv video coding methods and is close to H.264/AVC INTER coding, while maintaining similar coding complexity with INTRA coding.  Error resilience scheme for BCAWZ to address the concern of reliable transmission in the backward-channel, which is essential to the quality of video data for real-time and reliable activity and event analysis.