1. EE569 Digital Video Processing Copyright Xin Li 2007 1 EE569: Digital Video Processing Instructor: Xin Li (office: ESB939) –Email: xinl@csee.wvu.edu xinl@csee.wvu.edu or Xin.Li@mail.wvu.edu (preferred) Xin.Li@mail.wvu.edu Lecture time: MWF 10:00-10:50AM Office hour: M 2:00-3:00PM or by appointment

2. EE569 Digital Video Processing Copyright Xin Li 2007 2 Prerequisites EE465 (Introduction to Digital Image Processing) or equivalent course or equivalent working experience Mathematical background: elementary geometry, probability and random process, differential equations Signal processing background: signal and system (EE327/329), coding (quantization and entropy coding), processing (interpolation, filtering and restoration)

3. EE569 Digital Video Processing Copyright Xin Li 2007 3 Course Homepage I will put lots of additional teaching materials (papers, demos, data) in addition to all PPT slides It will also contain course-related announcements, computer assignments and their solutions http://www.csee.wvu.edu/~xinl/courses/ee569/

4. EE569 Digital Video Processing Copyright Xin Li 2007 4 Textbooks A.M. Tekalp, Prentice-Hall, 1995 Yao Wang, Jörn Ostermann, Ya-Qin Zhang Prentice Hall, 2002

5. EE569 Digital Video Processing Copyright Xin Li 2007 5 Course Objectives Knowledge learning –A comprehensive coverage of video processing –Width is preferred over depth Research skill training –What kind of problems should you work on? –How to solve problems? (Polya has a famous book about this) –From an idea to a paper

6. EE569 Digital Video Processing Copyright Xin Li 2007 6 Research-oriented I will downplay the role of textbooks and emphasize hand-on experiences In-class discussion is strongly encouraged especially for hot topics and current trends After each chapter, I will spend some time on brainstorming – identify good topics for your research Although the lecturing style of this course is deductive, computer assignments and course projects involve inductive teaching

7. EE569 Digital Video Processing Copyright Xin Li 2007 7 Performance Evaluation Basic skills –Computer assignments 10%x5=50% –Extra bonus points available Advanced skills –Individual midterm project 25% (5% will be about writing skills) –Group-based final Project 25% (5% will be about oral presentation)

8. EE569 Digital Video Processing Copyright Xin Li 2007 8 Computer Assignments Purpose: collect basic experience of processing video signals under MATLAB Example: –Video I/O (SIF/CIF/QCIF format) –Block matching algorithm (central component in all current video coding standards) –Optical flow estimation –Object tracking –Scene change detection

9. EE569 Digital Video Processing Copyright Xin Li 2007 9 Mid-Term Project Individual-based You will be given several papers and asked to choose one to implement The objective is to reproduce the reported experimental results (assuming the paper is flawless) You also need to write an abstract of the paper using your own words and propose new ideas (which you will have the opportunity to implement during the final project)

10. EE569 Digital Video Processing Copyright Xin Li 2007 10 Term Project It is recommended to form a two-person group The hardest thing in research is to find a good problem to start with and have a reasonable attack (your midterm project might be a good starting point) The objective is to help you finish a full cycle of doing research

11. EE569 Digital Video Processing Copyright Xin Li 2007 11 Digital Video Processing Digital –Use a digital computer as the platform Video –a sequence of images along the temporal axis –We will talk this more next Processing –A running software program or computing operation –Driven by the real-world applications (e.g., compression, filtering, retrieval)

12. EE569 Digital Video Processing Copyright Xin Li 2007 12 A Brief History Born of Television (1920s) Cable TV system (1968) Video games (1970s) All-digital HDTV (1990s) Video streaming (2000s) Everyday video transmission through internet and wireless networks (20??)

13. EE569 Digital Video Processing Copyright Xin Li 2007 13 Why Video? The magic of Tele-Vision –Our vision capability is extended in space The unfulfilled 3D dream (3D TV) You don’t need to travel to north pole to watch polar bears

14. EE569 Digital Video Processing Copyright Xin Li 2007 14 Why Video? (Cont’d) Our vision capability is extended in time –If time can be reversed, I will not need a Gigabyte hard-drive to store the moments of how my daughter has grown The fundamental interplay between time and motion –We measure time by the motion of material things –Motion offers a new horizon for us to understand the world

15. EE569 Digital Video Processing Copyright Xin Li 2007 15 Importance of Motion Our HVS routinely perceives and interprets motion (neurobiology) Functional MRI (fMRI) –By measuring the increase in blood flow to the local vasculature that accompanies neural activity in the brain, fMRI studies brain function instead of anatomy Gait-based biometrics –The characteristics of an individual’s walk

16. EE569 Digital Video Processing Copyright Xin Li 2007 16 Diversity of Motion

17. EE569 Digital Video Processing Copyright Xin Li 2007 17 Motion Perception is Tricky

18. EE569 Digital Video Processing Copyright Xin Li 2007 18 Motion Perception is Tricky

19. EE569 Digital Video Processing Copyright Xin Li 2007 19 Motion Perception in HVS

20. EE569 Digital Video Processing Copyright Xin Li 2007 20 Motion in Video It is not an arbitrary concatenation of images, but a sequence of images carrying a coherent interpretation of natural scene –Ordering is important –Sampling rate is important –The role of a single frame is less important due to the masking effect of HVS (e.g., frames 71-73 of coastguard sequence)

21. EE569 Digital Video Processing Copyright Xin Li 2007 21 How to Understand Video? Understand the source –How to model the motion of a camera? (relatively easy) –How to model the motion in the real world? (notoriously difficult) Understand the mechanism of time- varying image formation model –Two sides: geometric and photometric

22. EE569 Digital Video Processing Copyright Xin Li 2007 22 Camera Motion Rent a movie or watch TV for two hours, do the following simple experiments: –How many scene changes? –Within each scene, what kind of camera motion do you see? camera panning zoom in/out combination

23. EE569 Digital Video Processing Copyright Xin Li 2007 23 Real-world Motion Bring a notebook with you and record every motion you observe for a day –Can you classify them into a few simple classes? Rigid motion vs. deformable motion –IF you observe multiple motions at the same time, how about the spatial relationship among different moving objects? Overlapping vs. non-overlapping

24. EE569 Digital Video Processing Copyright Xin Li 2007 24 Geometric Image Formation Models

25. EE569 Digital Video Processing Copyright Xin Li 2007 25 Photometric Image Formation Models Modeling surface reflectance function –Lambertian vs. specular Modeling illumination condition –Light source location and intensity Modeling the photometric impact of 3D motion

26. EE569 Digital Video Processing Copyright Xin Li 2007 26 Why Video is Hard? The daunting modeling complexity –Scene geometry, lighting condition, object/camera motion, sensor characteristics We have to rely on digital computers to process video –Limited memory and computation resource –Fundamental question about computing

27. EE569 Digital Video Processing Copyright Xin Li 2007 27 Toy Example: 2D Motion Estimation 1 st frame 2 nd frame

28. EE569 Digital Video Processing Copyright Xin Li 2007 28 Toy Example: Stereo Matching Left-eye imageRight-eye image

29. EE569 Digital Video Processing Copyright Xin Li 2007 29 Fundamental Assumption the n-1-th frame the n-th frame (v x,v y ) I n (x,y)I n-1 (x,y) Image intensity field is smooth along the motion trajectory

30. EE569 Digital Video Processing Copyright Xin Li 2007 30 Almost There Ground truthState-of-the-art

31. EE569 Digital Video Processing Copyright Xin Li 2007 31 Tantalizing Questions Why constrained to two frames for ME? –A simplified solution: N-frame ME problem boils down to N-1 two-frame ME ones Is HVS really establishing point-to-point correspondence during motion perception? –Psychological studies do not support this model New direction in joint spatio-temporal analysis –What do we gain by handling N frames together? –We will discuss this issue later in detail