1. Challenges in Wireless Multimedia CSE Department Seminar Series September 26, 2003 Borko Furht

2. OUTLINE Scalable Video Source Coding Channel Coding and Error Control Power-Aware Coding and Transmission Techniques Networking Issues  Rate control Multimedia Security Application: Virtual Workplace

3. Mobile Internet Access Source: Ericsson Internet Subscribers (millions) Year

4. Wireless Multimedia Architecture

5. Bandwidth Problem “ Bandwidth is like money and sex - only too much seems to be enough.” Arnold Penzias, former chief scientist of Bell Labs

6. Generations of WAN Air Interfaces Based on Access Technologies 1G: FDMA (Frequency Division Multiple Access)  1980s - each caller has a dedicated frequency channel (3 callers use 3 channels) 2G: TDMA (Time Division Multiple Access) and GSM (Groupe Speciale Mobile)  1990s - callers timeshare a frequency channel (9 callers use 3 channels) 3G: CDMA (Code Division Multiple Access) and WCDMA (Wide Code Division Multiple Access)  1990s - callers use a shorter bandwidth  2000s - “spread spectrum”. Each code is spread, randomly broken down and mixed (14 callers use the full bandwidth of 1 channel)

7. Data Services 2,000 0 32 64 9.6 128 144 384 1G2G 3G Voice Text Messaging Video Streaming Still Imaging Audio Streaming Data Transmission Speed - k bps Electronic Newspaper Remote Medical Service (Medical image) Video Conference (High quality) Telephone (Voice) Voice Mail E-Mail Fax Electronic Publishing Karaoke Video Conference (Lower quality) JPEG Still Photos Mobile Radio Video Surveillance, Video Mail, Travel Image Audio Voice-driven Web Pages Streaming Audio Data Weather, Traffic, News, Sports, Stock updates Mobile TV E-Commerce Video on Demand: Sports, News Weather

8. M-Commerce Applications Transaction Management Digital Content Delivery Telemetry Services Searching for Killer Applications!

9. Transaction Management On-line shopping tailored to mobile phones and PDAs  on-line catalogs  shopping carts  back office functions Initiate and pay for purchases and services Micro-transactions - subway fees, digital cash

10. Digital Content Delivery Information browsing  weather  transit schedules  sport scores  ticket availability  market prices Downloading entertainment products Transferring software, high-resolution images, and full-motion video Innovative video applications

11. Telemetry Services Wide range of new applications Transmission of receipt of status, sensing, and measurement information Communication with various devices from homes, offices, or in the field Activation of remote recording devices or service systems

12. AT&T Wireless Welcome to mlife Find breaking news, flight information, entertainment.. Get the latest weather forecasts Get the business and investments news Get the latest weather forecasts Get the business and investments news

13. Future of Wireless Technology Mobile networks have already begun the migration to IP-based networks  IP as the routing protocol 4G, New spectrum, and Emerging wireless air interfaces (very high bandwidth 10 Mbps+)  It may entirely be IP-based and packet-switched Increasing usage of wireless spectrum  On average, the number of channels has doubled every 30 months since 1985 (Cooper’s law)

14. Wireless Multimedia Challenges Adaptive Decoding - Optimizing rich digital media for mobile information devices with limited processing power, limited battery life and varying display sizes Error Resilience - Delivering rich digital media over wireless networks that have high error rates and low and varying transmission speeds Network Access - Delivering rich digital media without adversely affecting the delivery of voice and data services Negotiable QoS for IP multimedia sessions as well as for individual media components

15. Components of a Wireless Video System Video Encoder Video Decoder Depacketizer Packetizer Demodulator Modulator Channel Decoder Channel Encoder Wireless Channel Input Video Output Video Transport + Network Layer Tradeoff: Throughput, Reliability, Delay

16. Source and Channel Coding Trade-off Classic goal of source coding  Achieve the lowest possible distortion for a given target bit rate Classic goal of channel coding  Deliver reliable information at a rate that is as close as possible to the channel capacity Shannon’s separation principle:  It is possible to independently consider source and channel coding without loss in performance  The separation principle applies only to point-to- point communications and it is not valid for multiuser or broadcast scenarios

17. Pragmatic Approach Keep the source coder and channel coder separate, but optimize their parameters jointly  Key problem in this optimization is the bit allocation between the source and channel coder Joint source-channel coding schemes  In the infancy today  Exploit the redundancy in the source signal for channel decoding (Source-controlled channel decoding)  Designing the source codec for a given channel characteristic (Channel-optimized source coding)

18. Characteristics of a Wireless Video System The capacity of wireless channel is limited by the available bandwidth of the radio spectrum and various types of noise and interference The wireless channel is the weakest link of multimedia networks – mobility causes fading and error bursts Resulting transmission errors require error control techniques (such as FEC - forward error control and ARQ – automatic repeat request)

19. The Case for Scalable Video Coding In emerging wireless applications, multimedia data will be streamed:  over various access networks (GPRS, UMTS, WLANs, etc.)  to a variety of devices (PCs, TVs, PDAs, cellular phones, etc.) The transmission of multimedia data need to cope with unpredictable bandwidth variations:  due to heterogeneous access technologies of receivers (3G, 802.11a, etc.) or  due to dynamic changes of network conditions (interference, etc.)

20. Scalable Video Coding Techniques Scalable video coding methods can adapt in real time to the bandwidth variations over heterogeneous networks and to the terminal capabilities while using the same pre-encoded system. Scalable video coding uses multiple bit streams – layered video coding For example, in a two-layer coding, the codec generates two bit streams:  Base layer – the most vital video information  Enhancement layer – the residual information to enhance the quality of the base layer image  This form of two-layer coding is known as SNR scalability

21. Scalability Techniques Data partitioning SNR scalability Spatial scalability Temporal scalability Hybrid scalability

22. Data Partitioning Data partitioning is used when two channels are available for transmission (it is not true scalable coding) Divides the bitstream of a single layer into two parts, or layers. Single layer encoder Data Partitioner Multiplexer Video in Base-layer bitstream Enhancement- layer bitstream Output bitstream

23. Block Diagram Two-Layer SNR Scalable Coder Base layer Encoder (MPEG 1) Multiplexer Base layer Decoder (MPEG 1) Enhancement layer Encoder (MPEG 2) Video in + - Base layer bitstream Enhancement layer bitstream Output bitstream

24. Adaptive Video Coder Based on 3D-DCT Original video cube 8x8x8 3D Discrete Cosine Transform

25. Motion Analysis for Various Blocks Partition image into NxN inspection areas Examine each area for motion content based on Normalized Pixel Difference (NPD) between frames 1 and 8 Three motion types defined:  No Motion  Low Motion, and  High Motion 3D-DCT block size adapts based on determined motion content

26. Example of a Video Hallway Clip 8 Frames of luminance (Y) component Inspection area size => 16x16 Inspection areas used to determine NPD thresholds

27. Video Example, 4:2:0 Original Cr=120 Cr=190 Cr=408

28. Architecture of 3D-DCT Adaptive Encoder

29. Example of a Scalable Coding Adaptive 3D-DCT Coder Original

30. Adaptive 3D-DCT Coder Layer 1: Cr=164 (in vehicles, 144 Kbps)

31. Adaptive 3D-DCT Coder Adding Enhancement Layer 2: Cr=96 (For pedestrians, 384 Kbps)

32. Adaptive 3D-DCT Coder Adding Enhancement Layer 3: Cr=54 (for indoor use, 2 Mbps)

33. Channel Coding and Error Control Effects of Transmission Errors Error-free frame Example 2: Corrupted group number causing a GOB misplacement Example 1: The extra insertion bit causing the loss of the first GOB Example 3: Corruption of the group quantizer parameter that resulted in employing the wrong quantizer in decoder

34. Channel Coding and Error Control Trade-off between throughput, reliability, and delay Forward Error Correction (FEC) Automatic Repeat Request (ARQ) Error Resilience Techniques for Low Bit Rate Video  Techniques that reduce the amount of introduced errors for a given error event (Resynchronization)  Techniques that limit interframe error propagation

35. Recovery From Packet Loss FEC scheme “Piggyback lower quality stream” Send lower resolution audio stream as the redundant information For example, nominal stream PCM at 64 kbps and redundant stream GSM at 13 kbps. Sender creates packet by taking the nth chunk from nominal stream and appending to it the (n-1)st chunk from redundant stream. Whenever there is non-consecutive loss, the receiver can conceal the loss. Only two packets need to be received before playback Can also append (n-1)st and (n-2)nd low-bit rate chunk

36. Joint Source Coding and Transmission Power Management Goal: to limit the amount of distortion in the received video sequence, while minimizing transmission energy Combines:  Error resilience and concealment techniques at the source coding level, and  Transmission power management at the physical layer Optimization problem: Minimizing the energy required to transmit video under distortion and delay constraints

37. Joint Source Coding and Transmission Power Management Video Encoder Wireless Channel Controller Demodulator Channel Decoder Video Decoder Channel Encoder Modulator Decoder Concealment Strategy Channel State Information Video in Video out Goal: to limit the amount of distortion in the received video sequence, while minimizing transmission energy Control powerControl coding parameters

38. Transmission Energy Total energy to transmit all the packets in a frame: The algorithm calculates the power needed to achieve the desired probability of loss

39. Controlling the Bit Rate Most video codecs use variable-length coding techniques Most existing mobile radio systems transmit at a fixed bit rate Goal: Constant signaling rate leading to a different constant bit rate for each modulation scheme Rate Control Techniques - determine the sending rate of video traffic based on the estimated bandwidth in the network  Source-Based Rate Control  Receiver-Based Rate Control  Hybrid-Based Rate Control

40. Rate Shaping Techniques Techniques that adapt the rate of pre- compressed video stream to a target rate constraint Rate shaper is an interface (or filter) between the compression layer and the network transport layer Compression Layer Rate Shaper Network Transport Layer Video in Variable rate Constant bit rate

41. Rate Shapers Codec filters Frame-dropping filters (dropping B,P, or I frames) Layer-dropping filters (in scalable video coding schemes) Frequency filters (discard DCT coefficients of the highest frequency) Requantization filters (reqauntizes the DCT coefficients with a larger quantizers, resulting in rate reduction)

42. Multimedia Content Security Access control in applications such as video-on- demand and videoconferencing, so only selected users can access the data Established encryption algorithms (DES or AES) are very complicated and involve large number of computations. Software implementations of these schemes are not fast enough to process the large amount of multimedia data Hardware implementations require additional costs to both data generation and receivers

43. General Architecture Selective Encryption System

44. Example of Video Encryption MPEG Encoder Secret Key Selective encryption algorithm That operates on sign bits of DC coefficients Secret Key Randomly change the sign bits of motion vectors Secret Key Permutation of the Huffman codeword list

45. Example: Encrypting Frames of a MPEG-4 Video Sequence Original frameEncrypted VLC only Encrypted FLC only Encrypted VLC and FLC

46. Virtues of the Virtual Workplace Universal access to information, applications, services, processes, and people, from any device, over any network connection - wired, wireless, or Web

47. Virtual Workplace Video Clip Wireless Internet and Web Wireless appliances Security Redundant systems Wireless applications: videoconferencing

49. The Portable Office Take the office with you, wherever you go

50. Secure Authentication High Security Authentication, including Bio-Authentication

51. Integrated Messaging and Communication Integrated messaging (eg. voice, chat), voice to text, with intelligent alerting

52. Information Portability Access information over any connection – wired or wireless, regardless of form factor

53. Business Collaboration Collaborative capabilities allow on-line information sharing and communication

54. Business Continuity Resilient to network interruptions

55. Further Readings Hanzo, Cherriman, and Streit, “Wireless Video Communications,” IEEE Press, 2001. IEEE Trans. On Circuits and Systems for Video Technology, Special Issue on Wireless Video, June 2002. Sun and Reibman, “Compressed Video over Networks,” Marcel Dekker, 2001 Wang, Ostermann, and Zhang, “Video Processing and Communications, Prentice Hall, 2002. Furht and Ilyas, “Wireless Internet Handbook,” CRC Press, 2003.