1. FrameFree Video Compression Critical Points versus Block Matching FrameFree Technologies http://framefree.com Igor Borovikov

2. Overview Motion compensation for video compression, Critical Points Filter: brief introduction, FrameFree Video: a bird eye view and side by side comparison of schematics with MPEG, Occlusions and CPF, Side by side comparison CPF3 and H.264, FFV features overview, Future work.

3. Motion Compensation Block Matching Algorithm – dominant motion predictor. Advantages: –Conceptually simple yet effective algorithm, –Low CPU requirement on decoder side, –Motion vectors data is well suited for entropy compression.

4. BMC Downsides Inherently discrete nature of blocks leads to: Resolution dependence (partially addressed by adaptive block size), Block artifacts (addressed by deblocking filters), Optimal motion vectors (exhaustive search) are usually prohibitively expensive, No simple way to reconstruct good quality non- integer frames, BMC has no direct relation to Human Visual System.

5. Alternatives Optical flow and related methods. A large group of pixel matching algorithms, yet not directly suitable for motion compensation in video compression. Critical Points Filter. The exceptional robustness of CPF pixel matching and adaptive mesh brings CPF to practical applications.

6. Critical Points Filter (CPF) Y. Shinagawa, and T. L. Kunii, Unconstrained Automatic Image Matching Using Multi-resolutional Critical-Point Filters, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.20, no.9, pp994-1001, Sep 1998 K. Habuka and Y. Shinagawa, Image Interpolation Using Enhanced Multi-resolutional Critical-Point Filters, International Journal of Computer Vision, vol.58, no.1, pp19–35, 2004

7. CPF mimics HVS HVS can reconstruct transition between these two images So can CPF.

8. Motion Compensation with CPF I Frame B-Frame P-Frame Original frame compressed with, say, JPEG. CPF- Interpolated between I and P frames: information from both sides of interval is used. Forward motion compensated and corrected frame.

9. Motion Compensation with CPF

10. BMC vs. CPF Motion compensation with CPF (uncorrected) Destination image Block motion compensation, Search radius 16 (uncorrected) Source image

11. CPF vs. BMC BMC Metaphor: - moving around tiles; CPF Metaphor: - deforming flexible film fixed on a rectangular frame, no tearing or folding allowed.

12. Bijectivity – no holes left after compensation, Adaptive mesh - resolution independence, Continuous time - no discretization whatsoever, Key features – critical points – are matched with high precision, thus: Remaining artifacts are not too conspicuous for HVS, Residual error of motion compensation is relatively well suited for combination of custom JPEG-like and entropy coding. Motion Compensation with CPF

13. MPEG vs. FFV encoding comparison

15. Practical Scheme TRANS: Image Translator : Differential Encoder & Noise Filter INTERP: Interpolation Unit : Differential Decoder

16. Occlusions and CPF Boy and Toys (NHK test sequence) with ball bouncing off the floor: CPF3 Original CPF CPF3 is the latest modification of FFV Practical Scheme that addresses occlusion problem.

17. Compression ratio and PSNR Sequence FFV Practical Scheme, 12.2005 FFV CPF3 12.2006 H.264 MedH.264 Low Yachting313Kb, 27.93 328Kb, 30.12 301Kb, 34.82 94Kb, 30.71 Intersection257Kb, 30.48 206Kb, 31.47 253Kb, 33.91 63Kb, 29.96 Horse race653Kb, 28.35 807Kb, 31.13 582Kb, 35.25 215Kb, 32.07 Boy and Toys 265Kb, 31.14 162Kb, 32.80 209Kb, 33.94 54Kb, 30.44 FFV is approaching H.264: many sequences are in between Med and Low settings in terms of PSNR and compression ratio. The table represents work in progress.

18. Side by Side Comparison Original Original CPF* CPF3 H.264Low H.264Med *) The worst frames (occlusion artifacts are apparent) were picked for original CPF to show CPF3 improvements.

19. FFV/CPF3 Features Overview 1.Continuous time – arbitrary playback speed, 2.Mesh based rendering – eliminates resolution dependence. Combination of 1 and 2 liberates video from frames, hence FrameFree video ; 3.Playback fits well modern graphics hardware – it is nothing else but rendering of textured meshes with simple pixel shader enhancements (ps_1_1), 4.CPF3 allows fallback Practical Scheme playback in case of dropped packets or CPU- bounded client.

20. Future work Simplify encoding and decoding by better occlusion handling, Coming soon: –DirectShow filters and QT components, –CODEC, –Integration with popular players on both Mac and PC, –Parallel and GPU-based encoding, –Hardware support, –Mobile platforms support, –Lightweight decoder/player. Some applications: –Sports video: arbitrary speed playback and web streaming, –Security video archiving, –Integration with FrameFree Studio, –Much more.

21. Thank you! Q&A Business contact: Tom Randolph tom@framefree.ustom@framefree.us Engineering contact: Igor Borovikov igor@framefree.usigor@framefree.us General information: www.framefree.comwww.framefree.com