Automatic Video Shot Detection from MPEG Bit Stream Jianping Fan Department of Computer Science University of North Carolina at Charlotte Charlotte, NC.

Slides:

Advertisements

Similar presentations

Learning Techniques for Video Shot Detection Under the guidance of Prof. Sharat Chandran by M. Nithya.

Advertisements

Automated Shot Boundary Detection in VIRS DJ Park Computer Science Department The University of Iowa.

Taxonomic classification for web- based videos Author: Yang Song et al. (Google) Presenters: Phuc Bui & Rahul Dhamecha.

-1/20- MPEG 4, H.264 Compression Standards Presented by Dukhyun Chang

B. Prabhakaran1 Multimedia Metadata Multimedia information needs to be “interpreted” Popular example: “A picture is worth thousand words” Who will “write”

MPEG-4 Objective Standardize algorithms for audiovisual coding in multimedia applications allowing for Interactivity High compression Scalability of audio.

Content-based Video Indexing and Retrieval

A presentation by Modupe Omueti For CMPT 820:Multimedia Systems

Discussion on Video Analysis and Extraction, MPEG-4 and MPEG-7 Encoding and Decoding in Java, Java 3D, or OpenGL Presented by: Emmanuel Velasco City College.

Content-based Video Indexing, Classification & Retrieval Presented by HOI, Chu Hong Nov. 27, 2002.

1 Adaptive slice-level parallelism for H.264/AVC encoding using pre macroblock mode selection Bongsoo Jung, Byeungwoo Jeon Journal of Visual Communication.

CMPT-884 Jan 18, 2010 Error Concealment Presented by: Cameron Harvey CMPT 820 October

Chapter 11 Beyond Bag of Words. Question Answering n Providing answers instead of ranked lists of documents n Older QA systems generated answers n Current.

Content-based Image Retrieval CE 264 Xiaoguang Feng March 14, 2002 Based on: J. Huang. Color-Spatial Image Indexing and Applications. Ph.D thesis, Cornell.

Image Search Presented by: Samantha Mahindrakar Diti Gandhi.

Issues for Multimedia Privacy & Security ---- Video Content Privacy Protection, Copyright Protection & Database Access Control Jianping Fan Dept of Computer.

1 Motion in 2D image sequences Definitely used in human vision Object detection and tracking Navigation and obstacle avoidance Analysis of actions or.

Supervised by Prof. LYU, Rung Tsong Michael Department of Computer Science & Engineering The Chinese University of Hong Kong Prepared by: Chan Pik Wah,

Multimedia Search and Retrieval Presented by: Reza Aghaee For Multimedia Course(CMPT820) Simon Fraser University March.2005 Shih-Fu Chang, Qian Huang,

T.Sharon 1 Internet Resources Discovery (IRD) Video IR.

Efficient MPEG Compressed Video Analysis Using Macroblock Type Information Soo-Chang Pei, Yu-Zuong Chou IEEE TRANSACTIONS ON MULTIMEDIA, DECEMBER,1999.

Traditional Database Indexing Techniques for Video Database Indexing Jianping Fan Department of Computer Science University of North Carolina at Charlotte.

LYU 0102 : XML for Interoperable Digital Video Library Recent years, rapid increase in the usage of multimedia information, Recent years, rapid increase.

Automatic Key Video Object Plane Selection Using the Shape Information in the MPEG-4 Compressed Domain Berna Erol and Faouzi Kossentini, Senior Member,

Decision Trees for Error Concealment in Video Decoding Song Cen and Pamela C. Cosman, Senior Member, IEEE IEEE TRANSACTION ON MULTIMEDIA, VOL. 5, NO. 1,

Visual Information Retrieval Chapter 1 Introduction Alberto Del Bimbo Dipartimento di Sistemi e Informatica Universita di Firenze Firenze, Italy.

Presented by Zeehasham Rasheed

Stockman MSU Fall Computing Motion from Images Chapter 9 of S&S plus otherwork.

E.G.M. PetrakisVideo Processing1  Video is a rich information source  frames (individual images)  links between frames (cuts, fades, dissolves)  changes.

1 Motion in 2D image sequences Definitely used in human vision Object detection and tracking Navigation and obstacle avoidance Analysis of actions or.

A fuzzy video content representation for video summarization and content-based retrieval Anastasios D. Doulamis, Nikolaos D. Doulamis, Stefanos D. Kollias.

Object Tracking for Retrieval Application in MPEG-2 Lorenzo Favalli, Alessandro Mecocci, Fulvio Moschetti IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR.

On Error Preserving Encryption Algorithms for Wireless Video Transmission Ali Saman Tosun and Wu-Chi Feng The Ohio State University Department of Computer.

CSE679: MPEG r MPEG-1 r MPEG-2. MPEG r MPEG: Motion Pictures Experts Group r Standard for encoding videos/movies/motion pictures r Evolving set of standards.

Video Streaming via Transcoding Jianping Fan Department of Computer Science University of North Carolina at Charlotte Charlotte, NC 28223

Multimedia Databases (MMDB)

Multimedia Information Retrieval

Adaptive Multi-path Prediction for Error Resilient H.264 Coding Xiaosong Zhou, C.-C. Jay Kuo University of Southern California Multimedia Signal Processing.

June, 1999 An Introduction to MPEG School of Computer Science, University of Central Florida, VLSI and M-5 Research Group Tao.

IEEE Int'l Symposium on Signal Processing and its Applications 1 An Unsupervised Learning Approach to Content-Based Image Retrieval Yixin Chen & James.

IBM QBIC: Query by Image and Video Content Jianping Fan Department of Computer Science University of North Carolina at Charlotte Charlotte, NC 28223

Performance Characterization of Video-Shot-Change Detection Methods U. Gargi, R. Kasturi, S. Strayer Presented by: Isaac Gerg.

Compression video overview 演講者：林崇元. Outline Introduction Fundamentals of video compression Picture type Signal quality measure Video encoder and decoder.

Advances in digital image compression techniques Guojun Lu, Computer Communications, Vol. 16, No. 4, Apr, 1993, pp

Implementation, Comparison and Literature Review of Spatio-temporal and Compressed domains Object detection. By Gokul Krishna Srinivasan Submitted to Dr.

Bachelor of Engineering In Image Processing Techniques For Video Content Extraction Submitted to the faculty of Engineering North Maharashtra University,

Content-Based Image Retrieval QBIC Homepage The State Hermitage Museum db2www/qbicSearch.mac/qbic?selLang=English.

Digital Video Library Network Supervisor: Prof. Michael Lyu Student: Ma Chak Kei, Jacky.

Soon Joo Hyun Database Systems Research and Development Lab. US-KOREA Joint Workshop on Digital Library t Introduction ICU Information and Communication.

Page 11/28/2016 CSE 40373/60373: Multimedia Systems Quantization  F(u, v) represents a DCT coefficient, Q(u, v) is a “quantization matrix” entry, and.

Query by Image and Video Content: The QBIC System M. Flickner et al. IEEE Computer Special Issue on Content-Based Retrieval Vol. 28, No. 9, September 1995.

Attila Kiss, Tamás Németh, Szabolcs Sergyán, Zoltán Vámossy, László Csink Budapest Tech Recognition of a Moving Object in a Stereo Environment Using a.

Instructor : Dr. K. R. Rao Presented by : Vigneshwaran Sivaravindiran

Introduction to MPEG  Moving Pictures Experts Group,  Geneva based working group under the ISO/IEC standards.  In charge of developing standards for.

Ontology-based Automatic Video Annotation Technique in Smart TV Environment Jin-Woo Jeong, Hyun-Ki Hong, and Dong-Ho Lee IEEE Transactions on Consumer.

Course Project Lists for ITCS6157 Jianping Fan. Project Implementation Lists Automatic Image Clustering You can download 1,000,000 images from You can.

Visual Information Retrieval

Automatic Video Shot Detection from MPEG Bit Stream

CS644 Advanced Topics in Networking

Presenter: Ibrahim A. Zedan

JPEG Image Coding Standard

MPEG-1 Video Coding Standard

Security Issues for Visual Data: Copyright and Access Control

Project Implementation for ITCS4122

Research Topic Error Concealment Techniques in H.264/AVC for Wireless Video Transmission Vineeth Shetty Kolkeri EE Graduate,UTA.

Image Segmentation Techniques

DC Image Extraction and Shot Segmentation

Multimedia Information Retrieval

A Block Based MAP Segmentation for Image Compression

Presentation transcript:

Automatic Video Shot Detection from MPEG Bit Stream Jianping Fan Department of Computer Science University of North Carolina at Charlotte Charlotte, NC

Why we need video shots? a. Text Retrieval: Keyword Extraction Indexing Document Storage Reverse File Indexing

Why we need video shots? b. Database Query: Entity Extraction Indexing Database Storage B-Tree Indexing

Why we need video shots? c. Image Retrieval: Object Indexing Indexing Database Storage Ontology Indexing

Why we need video shots? MPEG Video Sequence Video shot frames

Why we need video shots? Story Unit in News Video Each story unit may consist of multiple connected video shots!

Why we need video shots? Indexing Video Shots in Storage Shot Indexing ???

Storage Database Query Processing Server Networks ??? Video shot =?= keyword in video? Shot is used as basic unit for video indexing!

1. Why we need shot detection? Potential Search on Video Shots: a.Give me some videos which consist of similar video shots! b. Give me all the videos which are related to this video shot!

2. Content Structure of Video Sequence Video sequence Video shot frames Shot-Based Video Content Interpretation!

3. Coding Structure in MPEG Video Video sequence GOP MPEG encoding shot ?

Interesting Questions If you are leader of MPEG around 1994, how can you design new MPEG standards which can be used for video indexing?

New MPEG Standard with Video Shot Video sequence GOP Video shot frames GOP is Video Shot!

4. What’s mean of video shot? (a) Scene Cuts; (b) Fade in & Fade out; © Dissolves & Wipes Why We Have Shots: (a) Human Editing; (b) Content Change & Camera Motion Shot Types:

4. What’s mean of video shot? time video Shot 1 a.Scene Cuts: Sudden change of video content or focus

b. Fade: gradual transition between a screen and a 4. What’s mean of video shot? constant image (fade out).

c. Dissolve: gradual transition from one screen to 4. What’s mean of video shot? another, the first screen fade & the second one fade out.

4. What’s mean of video shot? Dissolve

d. Wipe: a line moves across the screen & new appears behind the line 4. What’s mean of video shot?

Wipe

Video Sequence Video shot frames Conclusion: Why we have video shots? Big change between two continuous video frames!

How can we detect video shots? Why we can define them as cuts, wipe, ….? Because they are changed between neighboring video frames!!!! How can we measure the changes or similarities between the neighboring video frames? How can make decision on whether they are change or not?

5. How to detect video shot? Major Components for Shot Detection: a.Visual Representation of Video Frames: Color, Texture… b.Difference Calculation for Neighboring Video Frames c.Threshold for Decision Making: How Large is Enough?

5. How to detect video shot? How to measure statistical property of video frames? Color Histogram

5. How to detect video shot? How to measure statistical property of video frames? Color Histogram

5. How to detect video shot? How to measure statistical property of video frames? Texture Histogram

5. How to detect video shot?

How we can do this more efficient on MPEG videos? MPEG videos MPEG Decoder Color Histogram Difference Scene Cut Frame Output Automatic Threshold Determination a. Easy but not smart

Scene Cut

Effects on color histogram difference by scene cuts: 5. How to detect video shot?

How to measure the change of video content? Color Histogram Difference

5. How to detect video shot? How we can select the threshold automatically? a.J. Fan, D.K.Y. Yau, W.G. Aref, A. Rezgui, ``Adaptive motion- compensated video coding scheme towards content-based bit rate allocation”, Journal of Electronic Imaging, vol.9, no.4, pp , This algorithm can adapt the thresholds to different videos. b. J. Fan, et al., ``ClassView: Hierarchical video shot classification and retrieval”, IEEE Trans. on Multimedia, This algorithm can adapt thresholds to different video shots.

5. How to detect video shot? Relationships among continuous frames can defined as: scene cuts versus non-scene cuts probability T non-scene cut scene cut

5. How to detect video shot? How to obtain T automatically? Entropy for non-scene cut frames:

Interesting Questions What will happen on MPEG bit stream if the current video frame is a shot boundary? I B P How can we use such properties for shot detection?

5. How to detect video shot? b. Complex but smart approach GOP MPEG encoding If scene cuts happen on I frames in MPEG Video? How to calculate color histograms of I frames efficiently? Only decode DC coefficient!!!

5. How to detect video shot? b. Complex but smart approach What will happen on MPEG bit stream if there is a shot boundary? 1.5Mb/s 4Mb/s Picture Quality high low

Reference frame( I or P)Current P frame Most macroblocks can not find their correspondences!! If content change happens on P frames in MPEG video? Only decode the block coding types in P frames!!!

Previous ReferenceCurrent B frame Future Reference Most macroblocks in B current frame can only be predicted by backward way!! If content change happens on B frames in MPEG video? Only decode the coding types in B frames!!!!

6. Shot Detection via Edge Extraction Edges indicate the content structure of video frames!

6. Shot Detection via Edge Extraction MPEG videos MPEG Decoder Color Edge Detection Scene Cut Frame Output Automatic Threshold Determination a. Easy but not smart

6. Shot Detection via Edge Extraction a.J. Fan, W.G. Aref, M.-S. Hacid and A.K. Elmagarmid, ``An automatic isotropic color edge detection technique”, Pattern Recognition Letters, vol.22, pp ,

6. Shot Detection via Edge Extraction b. complex but smart Only decode only part of MPEG videos DCT What you find from this figure?

6. Shot Detection via Edge Extraction 1. Calculate the directional edge histogram Horizontal edges Vertical edges Northwest diagonal edges Northeast diagonal edges 2. Calculate the differences of the directional edge histogram 3. Threshold to obtain the scene cuts

Effects on color histogram different by Fades & Dissolves 7. Complex Shot Detection

Wipe Modeling Dissolve Modeling 7. Complex Shot Detection

8. Camera Motion Detection a.Zoom in

8. Camera Motion Detection b. Zoom out

8. Camera Motion Detection c. Left Pan

8. Camera Motion Detection d. Right Pan

Why We Need Keyframe Extraction?

Keyframe Extraction What are keyframes? Different from others! How to measure the significance of video frames? How to make the decision?

9. Background Modeling for Shot Detection

10. Shot-Based Video Content Representation Statistical Information Keyframe-Based Information (a)Color histogram & variance; (b) edge histogram & variance © motion histogram & variance; (a)Color histogram; (b) edge histogram © motion histogram

11. Shot-Oriented Video Semantics Interpretation

Who cares?

12. Who works on this topic? 1. IBM research center 2. Intel 3. Microsoft Redmond & Beijing 4. Kodak at Rochester 5. HP at Polo Alto 6. Google 7. Yahoo!

12. Requirement If you are asked to work on video shot detection from compressed video stream, what you should know (basic requirement)? a.MPEG video coding standard: how to decode and read MPEG file b. Knowledge on video shot analysis: Color, motion, edge, fade, dissolve, … c. Skills on C++

Project Introduction Design a system for automatic shot detection from MPEG video streams Interface design: read in MPEG files and display shot boundaries, or even decision process!