1. Multimedia Processing Lab NH 140
Advisor : Dr. K.R. Rao Phone : (817)
Website:

2. Multimedia Network
Now a typical multimedia network looks like this. There are three main domains : 1. Authoring, 2. Distribution and 3. playback. We as end users are no longer restricted to the “Playback” domain of such a typical multimedia network. Each of these domains have promising research potential that is explored in our Multimedia Processing Lab under the guidance of Dr K. R. Rao.

3. Home Media Ecosystem A case for importance of research in multimedia
The Home Media Ecosystem shown here is an example of how multimedia and its related technology has pervaded all aspects of our lives. The MPL provides a good platform to start your career in such a promising field.
A case for importance of research in multimedia

4. Video Redundancy – An Example

5. The need for video compression
Video signal : Sequence of frames (images) related among temporal dimension
TV video quality: 704x576 pixels per frame, 12 bpp, 25 frames per second - > 121 Mbps
Too much data for video transmission or storage
Increasing importance of multimedia communication t
NEED FOR VIDEO COMPRESION

6. Research Focus Areas MVC SVC mobile Year H.265(?) 2010
Coding Efficiency
Network awareness
+ implementation?
2005
2010
1999
1994
MPEG4
H.264
1992
MPEG1
Video Conferencing
H.263
2003
Mobile Phone
Hand PC
Mobile TV
SVC
HDTV
Year
MPEG2
H.265(?)
mobile
2009
MVC
Blue ray DVD
With each new compression technique that are standardized, we at the lab obtain the codec software through the networking influence of the MPL advisor Dr K. R Rao and work on optimizing the codec, either by reducing the complexity, encoding time, improving the quality, or improving the robustness of the standard using algorithms for error concealment and error correction. Research by a former student (Rahul Panchal) was accepted by the Joint Video Team to be incorporated into the h.264 standard and he is also working on H.265 which maybe a standard released in the near future.

7. Research : Image, Video, Audio
JPEG, JPEG-LS, LOCO, CALIC
MPEG 1,2,3,4,7
Dolby True HD
JPEG 2000
H.264, H.265(?)
HD-AAC
JPEG XR–AIC
VP6, VP7, VP8
MP3, MP3 Pro
JBIG1,2
VC–1 (WMV–9)
AAC–SBR
PNG
Wyner Ziv
HE–AC3
GIF
AVS China p2
AVS China p3
Dirac,Dirac Pro(BBC)
ATSC
Real Networks-RV10
WMA
Most of these software codecs are proprietary material. But Dr Rao’s influential networking has enabled him to get most of these codecs strictly for academic research in his lab. So one will have hands on experience in the latest technology by the time one leaves the university. For example work is being done h.265 in the industry but it is yet to be released as a standard; Dr Rao has his former students working on this and they collaborate with MPL to do some research on this. 7

8. Video Compression Standards
Main Applications
Year
JPEG, JPEG2000
Image
, 2000
JBIG, JBIG2
Fax
H.261
Video Conferencing
1990
H.262, H.262+
DTV, SDTV, HDTV
1995, 2000
H.263, H.263++
Videophone
1998, 2000
MPEG-1
Video CD
1992
MPEG-2
DTV, SDTV, HDTV, DVD
1995
MPEG-4 Part 2
Interactive video
2000
MPEG-7
Multimedia Content description
2001
MPEG-21
Multimedia Framework
2002
H.264/MPEG-4 part 10
Advanced Video Coding
2003

9. Under standardization
Latest Video Codecs
Standard
Main Applications
Year
Dirac (B.B.C.)
Internet streaming to Ultra-high definition TV
2008
Dirac pro/VC-2
Studio and professional use
Under standardization
VC-1 (SMPTE/Microsoft)
Internet streaming to High definition TV
2006
VC-3
Compositing, mastering, and multi-generational use
VP6 (On2 technologies)
Broadcasting
2003
VP7
2005
VP8
RV10 (Real Networks)
Internet streaming
AVS China
IP TV , Terrestrial digital TV, Satellite broadcast, Video surveillance
H.264 Fidelity Range Extensions
Studio editing, Post processing, Digital cinema
2004
H.264 SVC, MVC
Scalable video coding, panaromic video
These latest codecs are proprietary softwares acquired by Dr Rao for academic purposes to be used in MPL

10. Comparison of various video compression standards
Algorithmic Element
MPEG-2 Video
(H.262)
MPEG-4 AVC
(H.264)
SMPTE VC-1
(Windows Media Video 9)
Dirac
DiracPRO
AVS
Part 2
Part 7
Intra Prediction
None: MB encoded DC predictors
4x4 spatial
16x16 spatial
I-PCM
Frequency domain coefficient
4x4
Spatial
(forward, backward)
8×8 block based Intra Prediction
Intra_4x4 (4x4 spatial).
Direct Intra Prediction
Picture coding type
Frame
Field
Picture AFF
MB AFF
Intra – Frame,
Field (Interlace, Progressive)
Motion compensation block size
16×16, 16×8, 8×16
16×16, 16×8, 8×16, 8×8, 8×4, 4×8, 4×4
16×16, 8×8
4×4
N/A
16×16, 16×8, 8×16, 8×8
16×16, 16×8, 8×16, 8×8, 8×4, 4×8
Motion vector Precision
Full pel
Half pel
Quarter pel
1/8 pel
1/4 pel

11. Comparison of various video compression standards
Algorithmic Element
MPEG-2 Video
(H.262)
MPEG-4 AVC
(H.264)
SMPTE VC-1
(Windows Media Video 9)
Dirac
DiracPRO
AVS
Part 2
Part 7
P frame type
Single reference
Multiple reference
Single reference,
Intensity compensation
No P frames
Single and multiple reference (maximum of 2 reference frames)
B frame type
One reference each way
One reference each way,
Multiple reference,
Direct & spatial direct weighted prediction.
No B frames
One reference each way, Multiple reference.
Direct and symmetrical mode.
No B frames.
In loop filters
None
De-blocking
Overlap transform
De-blocking filter.

12. Comparison of various video compression standards
Algorithmic Element
MPEG-2 Video
(H.262)
MPEG-4 AVC
(H.264)
SMPTE VC-1
(Windows Media Video 9)
Dirac
DiracPRO
AVS
Part 2
Part 7
Entropy coding
VLC
CAVLC,CABAC
Adaptive VLC
Arithmetic coding
Context based adaptive binary arithmetic coding,
Exp-Golomb coding.
2D variable length coding.
Context based adaptive 2D variable length coding.
Transform
8×8 DCT
4×4 integer DCT
8×8 integer DCT
8×4 & 4×8 integer DCT
4×4 wavelet transform
4×4
wavelet transform
4×4 DCT
Other
Quantization scaling matrices.
Range reduction.
Instream-post processing control

13. Audio Compression Standards
Main Applications
Year
Dolby True HD
Lossless audio, Blu-ray Disc players, A/V receivers, and home-theater
2006
HD-AAC
Soundtrack applications
1997
MP3
Handheld devices
1991
MP3 Pro
2001
AAC–SBR
DAB – High quality audio
2003
HE–AC3
Satellite or terrestrial audio broadcasting
2005
AVS China part 3
Handheld and broadcasting
2004
AC3 Pro

14. Current Research Activities of MPL
Mobile Applications
Development of virtual lab platform for mobile software application
Developing a low complexity video codec for mobile application
Complexity reduction
Complexity reduction in existing video codecs
Complexity reduction in existing audio codecs
Quality Improvement
Optimizing existing video codecs using perceptual coding techniques
Improve Robustness
Error Resilience of video streams in a Lossy Wireless Environment
Error concealment techniques for wireless video transmission
Transcoders
Video transcoders : VP6 to H.264, H.264 to VC-1, Wyner Ziv to H.264, H.264-to-AVS China transcoders
Video/Audio Integration
AVS China – Audio/Video codec – Multiplex/demultiplex and lip sync
DIRAC video codec and AAC - Multiplex/demultiplex and lip sync 14

15. Virtual lab. Platforms for Mobile SW Applications

16. Low complexity Codec Applications
SensorCamPillCamWearableCamDisposable cam.ScanCam

17. Transcoding Applications
Low complexity Decoder
Low complexity Encoder
Some codecs have a complex encoder but a simple decoder, whereas others have a simple encoder and a complex decoder. Combining the two by will enable us to employ these codecs for low complexity applications such as mobile platforms. The high complexity decoding of bitstream from one codec to re-encoding it to the bitstream of another codec is what is called transcoding.
Also for cross platform applications : transcoding enables us to use products from any company on any device. For example if a erricsson device only supports MPEG2 and we have h.264 viddeo, then using a transcoder we can convert it to MPEG2 btstream to play it on our device.
The transcoding platforms handle the high complexity decoding on one side and high complexity encoding on the other (right) side

18. An application scenario for transcoding

19. Error Concealment in Lossy Wireless Environment
Reconstruct lost information
Source
Destination
When video is transmitted over a wireless network, it may experience loss of some information due to the lossy nature of the wireless environment. For a lot of video applications, especially real time video applications, re-transmission is not possible. So we embed some error concealment algorithm at the decoder side to conceal the errors so that we can have uninterrupted transmission and veiwing.
Original Information
Information lost due to lossy wireless network
Typical situation of 3G/4G cellular telephony 19

20. Multiplexing of Audio/Video
Most compression work is done on video and there are a lot of codecs available for the same. But video is useless without audio.. Hence this is a viable research area where we can multiplex the best available audio codec to the latest video codecs that are available.
AVS – Audio Video Standard of China

21. A quick view on H.264

22. Profiles in H.264

23. Intra Prediction 4x4

24. Intra Prediction 8x8

25. Intra Prediction 16x16

26. Motion Estimation/Compensation Sizes

27. Sub pixel accuracy

28. Scanning
Zig-zag scan
Alternate scan

29. SVC Extensions

30. (HDR storage/display) (conventional display)
Future Standards Activities – Bit depth Scalability
LCD dynamic range – 500:1
HDR displays: Sharp “Mega-contrast”, LG.Philips - 1,000,000:1, Dolby – 250,000:1
Bit Depth
Scalable Coder
HDR video input
10, 12, 14 bits/pixel
HDR video output
(HDR storage/display)
LDR video output
(conventional display) + =
Tone
Mapping
HDR
range
8-bit
HDR = High Dynamic Range
LDR = Low Dynamic Range

31. Future Standards Activities – 3D Video
Consumer Electronics auto-stereoscopic display, 10+ views required
Digital Cinema
polarized glasses, 2 views sufficient
3D Video (3DV)/Free View-Point Video (FVV) effort initiated in MPEG. Similar concept to MPEG-C. Any number of views can be recreated using depth map in the decoder.
2D video data + depth
HDR = High Dynamic Range
LDR = Low Dynamic Range

32. Future Standards Activities – 3D Video
Paramount Pictures' Beowulf is benefiting from theaters utilizing next-generation 3D technology (grossed approximately $23.4 million of a total domestic gross over 79.4 million.”
“U2 3D, the first live-action movie to be shot, produced, and screened exclusively with digital 3-D technology
DreamWorks Animation is joining the digital 3-D wave
Studio plans to release all its pics in 3-D starting in 2009.”
HDR = High Dynamic Range
LDR = Low Dynamic Range

33. Original and compressed Lena image with different methods
Original Lena
(51251224)
(b) AIC: 0.22bpp,
PSNR=28.84dB
(c) JPEG2000: 0.22bpp,
PSNR=29.57dB

34. Compressed Lena image with different methods(contd.)
(d) M-AIC: 0.22bpp, PSNR=29.02dB
(e) JPEG: 0.22bpp, PSNR=24.29dB

35. Current Interns & Alumni Network
Jennie Abraham
Muniyappa
Radhika Veerla (Aug 08) - Nokia Seimens  
Theju Jacob (Aug 08) - Fast VDO
Pooja Agawane (Aug 08) - Intel
Leena Agarwal (Dec 07) - Intel
Rahul Panchal (May 07) - Qualcomm
Harishankar Murugan (May 07)- NVidia
Sreejana Sharma (May 07)- Intel
Hitesh Yadav (August 06)- Intel
Basavaraj S. M. (May 06)- Fast VDO
Rochelle Pereira (Dec 05)- NVidia
Sandya Sheshadri (Dec 05) - Microsoft
Tarun Bhatia (Dec 05)- FastVDO
Current & Recent Grads:
Vineeth Shetty Kolkeri (M.S) Qualcomm, Apple
Subrahmanya MV (M.S.) – Qualcomm, Qualcomm
Jay R Padia (M.S) - Intel,
Vidhya Vijayakumar (M.S) - Intern at Adobe,
Att Kruafak (Ph.D) – Engineer CAT, Thailand
Sangseok Park (Ph.D) – DiaLogic
Swaroop
Sadaf
MPL has an impressively strong network of employed alumni giving us the opportunity to explore the latest things in the video coding field. MPL has had a record of 100% placement for all thesis and phD students so far. Companies like Intel and fastVDP have in the past specifically asked for graduating students of MPL

36. Explore the field of multimedia processing in MPL @
- Dr. K.R. Rao
(817)
NH 140