1. M4 – Video Processing, Brno University of Technology1 M4 – Video Processing Igor Potůček, Michal Španěl, Ibrahim Abu Kteish, Olivier Lai Kan Thon, Pavel Zemčík Faculty of Information Technolgoy, Brno University of Technology, Czech Republic M4 Meeting, September 2003, Delft, The Netherlands

2. M4 – Video Processing, Brno University of Technology2 Outline: Video processing block diagram Mouth parametrization Face detection and feature extraction Demo Other video work in Brno Conclusions

3. Gesture interpretation Face (rough) positioning, recognition M4 – Video Processing, Brno University of Technology3 Video Processing Block Diagram Finished In progress Planned Raw (hyperbolic mirror image) „Corrected“ geometry Pre-identified body parts Image acquisition Colour-based methods Identification of face, eyes, mouth (hands) Gabor wavelet networks Detected mouth in large video (reference) Geometry/graphics Mouth parametrization Mouth area (motion and shape) in low res. Statistical methods Features for audio Pattern matching Annotation/control functions Annotation functions

4. M4 – Video Processing, Brno University of Technology4 Mouth Parametrization Mouth parametrization may be useful for speech recognition anhancement Low-resolution videos do not allow „proper“ lips tracking (in the case of meting rooms the heads are always small) Statistical mouth area parametrization needed; however, we also need a reference method with „visibly correct“ mouth parametrization to be able to compare the results 2 methods - „full size“ lips tracking and mouth parametrization and „statistical“ approach for small-size face images while the „full size“ version serves just as a reference (currently no intention for further research in it)

5. M4 – Video Processing, Brno University of Technology5 Mouth Parametrization Three main parameters: width w, height h, and curvature c A hope exists that these parameters are enough as they can be quite well extracted from the low resolution video If the parametrization model is not sufficient, it can be further extended (probably through subcontracting)

6. M4 – Video Processing, Brno University of Technology6 Face Detection and Feature Extraction The process of face detection if based on pre-processing of the colours in the video It is necessary to positively identify the faces in the video Gabor wavelet networks seem to be suitable instrument for that purpose (fixed-size networks with 56 or 112 wavelets, see paper of M. Španěl) Training for general face detection and positioning (spatial orientation, 5 pan, 5 tilt positions, 15° step) Can be re-applied for face features, such as eyes or mouth to fine-tune the results and specifically to adjust the scale

7. M4 – Video Processing, Brno University of Technology7 Face Detection and Feature Extraction The Gabor wavelet networks can olso be used for (limited capability) face recognition Well suitable for occlusion problems solution – does not loose track of people Can be used to distinguish „registered“ people and „newcomers“ Requires totally different training – not „zero price“ but still cheap

8. M4 – Video Processing, Brno University of Technology8 Demo Face tracking with feature detection and identification

9. M4 – Video Processing, Brno University of Technology9 Demo Face tracking with feature detection and identification

10. M4 – Video Processing, Brno University of Technology10 Demo Face tracking with feature detection and identification

11. M4 – Video Processing, Brno University of Technology11 Demo Face tracking with feature detection and identification

12. M4 – Video Processing, Brno University of Technology12 Demo Face tracking with feature detection and identification

13. M4 – Video Processing, Brno University of Technology13 Demo Mouth parametrization

14. M4 – Video Processing, Brno University of Technology14 Other Video Work in Brno Automatic video editing – PROLOG-based, prolog clauses generated from scenarios (general intentions and rules) and processed audiovisual sequences – generates „cutting“ instructions that are used by an off-line video editing software Video annotation tool – shown before, recently extended to multi-channel capabilitiy and extended XML usage The more detailed description of the tools/approaches can be seen at: http://www.fit.vutbr.cz/research/projects/m4

15. M4 – Video Processing, Brno University of Technology15 Conclusions Audio/video multimodal processing not yet very successfull Video features detection fairly successfull Face detection very reliable In the future work (possibly in AMI), we would like to perform some fo the tasks in real-time and in embedded video camera systems Thanks for the attention