Presentation is loading. Please wait.

Presentation is loading. Please wait.

Österreichisches Forschnungsinstitut für Artificial Intelligence Featuring the GEMEP Corpus Experiences and Future Plans Hannes Pirker OFAI, Vienna.

Published bySeth Slates Modified over 10 years ago

Similar presentations

Presentation on theme: "Österreichisches Forschnungsinstitut für Artificial Intelligence Featuring the GEMEP Corpus Experiences and Future Plans Hannes Pirker OFAI, Vienna."— Presentation transcript:

1 Österreichisches Forschnungsinstitut für Artificial Intelligence Featuring the GEMEP Corpus Experiences and Future Plans Hannes Pirker OFAI, Vienna

2 Österreichisches Forschnungsinstitut für Artificial Intelligence Overview Features from Audio Channel Segments and pitch contours Features from Video Channel Faces, silhouettes and hands Discussion What we do have & what we need

3 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien3 Features from Audio Channel Phonetic segmentation into Phonemes Syllables Pitch Extraction

4 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien4 Speech Analysis Phonetic Segmentation Phonetic segmentation works quite good (forced alignment with HTK) Bootstrapping circle with Manual labelling of training data Training of HMMs Automatic alignment Manual correction of alignment results Re-training of HMMs Processed data: Typ 1 sentences (Ne kal ibam sud molen)

5 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien5 Speech Analysis Pitch Extraction Testing different pitch extraction methods from SFS (Mark Huckvale, UCL) Promising results with fine-tuning of parameters for contour- smoothing & high-pitch correction

6 Österreichisches Forschnungsinstitut für Artificial Intelligence Sample 06joi112

7 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien7 Features from Video Channel Face detection Silhouettes & Bounding Boxes Hand tracking

8 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien8 FaceDetection Using openCV Feature based face detection with pre-trained cascaded classifier (almost) out of the box Very good results under close-to- optimal conditions

9 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien9 FaceDetection Using openCV Feature based face detection with pre-trained cascaded classifier Clip:06joi112

10 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien10 Silhouettes & Bounding Boxes Combing silhouettes & bounding boxes in frontal and sideview as simple & robust estimator of Dynamics of movements Amount of expansion

11 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien11 Silhouettes & Bounding Boxes Combing silhouettes & bounding boxes in frontal and sideview as simple & robust estimator

12 Österreichisches Forschnungsinstitut für Artificial Intelligence 1 st Results: 3D Bounding Box Volume per Emotion – Actor 01

13 Österreichisches Forschnungsinstitut für Artificial Intelligence 1 st Results: 3D Bounding Box Volume per Emotion – Actor 07

14 Österreichisches Forschnungsinstitut für Artificial Intelligence 3D Bounding Box Volume: JOI vs. TRI / Actor 01 vs. Actor 06 ACTOR 01ACTOR 06

15 Österreichisches Forschnungsinstitut für Artificial Intelligence Temporal Dynamics: Plotting BoundingBox & Speech-Timing Clip:06joi112

16 Österreichisches Forschnungsinstitut für Artificial Intelligence Hand Tracking Find skin areas in 1 st frame

17 Österreichisches Forschnungsinstitut für Artificial Intelligence Hand Tracking Find skin areas in 1 st frame

18 Österreichisches Forschnungsinstitut für Artificial Intelligence Hand Tracking Find skin areas in 1 st frame Find hand & use center of area as hand position

19 Österreichisches Forschnungsinstitut für Artificial Intelligence Hand Tracking Find skin areas in 1 st frame Find hand & use center of area as hand position Interactively accept or correct

20 Österreichisches Forschnungsinstitut für Artificial Intelligence Hand Tracking Find skin areas in 1 st frame Find hand & use center of area as hand position Interactively accept or correct Perform automatic tracking (using mean shift algorithm)

21 Österreichisches Forschnungsinstitut für Artificial Intelligence Hand Tracking Perform automatic tracking (using mean shift algorithm)

22 Österreichisches Forschnungsinstitut für Artificial Intelligence Hand Tracking Perform automatic tracking (using mean shift algorithm) Interactively classify quality of tracking The GOOD (70%) The BAD (15%) The UGLY (15%)

23 Österreichisches Forschnungsinstitut für Artificial Intelligence Different Actors – Different Results

24 Österreichisches Forschnungsinstitut für Artificial Intelligence Different Actors – Different Results Difficult Easier

25 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien25 Discussion What have we gained? What do we need?

26 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien26 Investigations 1: Interaction Influence of Affect on Gesture AND speech Activation dimension should be reflected in speech and body movement in a parallel way. i.e. look at speed, effort etc.

27 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien27 Investigations 1: Interaction Influence of Affect on Gesture AND speech Activation dimension should be reflected in speech and body movement in a parallel way. i.e. look at speed, effort etc. Data already usable

28 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien28 Investigations 2: Timing Temporal aspects Look at the relative timing of speech and non-verbal signals, e.g. The location of strokes in relation to accented syllables

29 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien29 Timing: Traditional Anchor points in Speech Borders of Syllables Words (Prosodic and Syntactic) Phrases Utterances Turns Location of Pauses (Pitch) Accents

30 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien30 Timing: Traditional Anchor points in Speech Borders of Syllables +++ Words +++ (Prosodic & Syntactic) Phrases + - Utterances +? Turns -- Location of Pauses + (Pitch) Accents + --

31 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien31 Temporal alignment Traditional Anchor points in Gestures Prepare Stroke Hold Retract (Graphics by A. Marshall http://twiki.isi.edu/Public/BML_Specification)

32 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien32 Temporal alignment Traditional Anchor points in Gestures Phases difficult to obtain, but general information on dynamics available (Graphics by A. Marshall http://twiki.isi.edu/Public/BML_Specification)

33 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien33 Investigations 3: Bodily Expression of Emotion Relevant Features (cf Wallbott 1998): Upper Body: away from camera, collapsed Shoulders: up, backward, forward Head: down, back, turned or bent Arms: lateral, stretched out frontal/sideways, crossed

34 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien34 Investigations 3: Bodily Expression of Emotion Relevant Features (cf Wallbott 1998): Hand-form: fist(s), opening/closing Movement Qualities: Activity,Expansiveness, Dynamics/Energy/Power

35 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien35 Investigations 3: Bodily Expression of Emotion Relevant Features (cf Wallbott 1998): Data currently focussing on hand location. Upper-body and posture not directly assessed. Movement qualities partially accessible

36 Österreichisches Forschnungsinstitut für Artificial Intelligence What we DO have by Now

37 Österreichisches Forschnungsinstitut für Artificial Intelligence What we might WANT to have?

38 Österreichisches Forschnungsinstitut für Artificial Intelligence Possible Representations H-ANIM/MPEG-4-style joint angles

39 Österreichisches Forschnungsinstitut für Artificial Intelligence Possible Representations H-ANIM/MPEG-4-style joint angles Gesticon/MURML/HamNoSys- style wrist position in relation to body encoding of the stroke-phase etc.

40 Österreichisches Forschnungsinstitut für Artificial Intelligence Possible Representations Less ambitious Symmetric/Assymetric Extended/Collapsed Static/Dynamic … In any case: Need to relate pixel- numbers to anthropomorphic measures!

41 Österreichisches Forschnungsinstitut für Artificial Intelligence Possible Representations Also consider (manually supported) classification into prototypical classes

42 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien42 Summary Numerous possibilities for improvements, e.g., in hand tracking (but is it necessary?) Rather concentrating on representations in order to ensure data really is useful and re-usable (e.g. ECAs)

43 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien43 Summary Time to bundle expertises and distribute efforts within

44 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien44 Thanks for your attention

45 Österreichisches Forschnungsinstitut für Artificial Intelligence

46 Österreichisches Forschnungsinstitut für Artificial Intelligence

47 Österreichisches Forschnungsinstitut für Artificial Intelligence Sample 06col112

48 Österreichisches Forschnungsinstitut für Artificial Intelligence Sample: 06peu111

49 Österreichisches Forschnungsinstitut für Artificial Intelligence

50 Österreichisches Forschnungsinstitut für Artificial Intelligence

51 Österreichisches Forschnungsinstitut für Artificial Intelligence

52 Österreichisches Forschnungsinstitut für Artificial Intelligence

53 Österreichisches Forschnungsinstitut für Artificial Intelligence

Download ppt "Österreichisches Forschnungsinstitut für Artificial Intelligence Featuring the GEMEP Corpus Experiences and Future Plans Hannes Pirker OFAI, Vienna."

Similar presentations

Gesture recognition using salience detection and concatenated HMMs Ying Yin Randall Davis Massachusetts Institute.

Gesture recognition using salience detection and concatenated HMMs Ying Yin Randall Davis Massachusetts Institute.
Descriptive schemes for facial expression introduction.

Descriptive schemes for facial expression introduction.
Pose Estimation and Segmentation of People in 3D Movies Karteek Alahari, Guillaume Seguin, Josef Sivic, Ivan Laptev Inria, Ecole Normale Superieure ICCV.

Pose Estimation and Segmentation of People in 3D Movies Karteek Alahari, Guillaume Seguin, Josef Sivic, Ivan Laptev Inria, Ecole Normale Superieure ICCV.
4 Mei 2009Universitaet WIEN1 Stability of Equilibrium Points Roger J-B Wets, Univ. California, Davis with Alejandro Jofr é, Universidad de Chile.

4 Mei 2009Universitaet WIEN1 Stability of Equilibrium Points Roger J-B Wets, Univ. California, Davis with Alejandro Jofr é, Universidad de Chile.
DDDAS: Stochastic Multicue Tracking of Objects with Many Degrees of Freedom PIs: D. Metaxas, A. Elgammal and V. Pavlovic Dept of CS, Rutgers University.

DDDAS: Stochastic Multicue Tracking of Objects with Many Degrees of Freedom PIs: D. Metaxas, A. Elgammal and V. Pavlovic Dept of CS, Rutgers University.
Advances in Phonetics-based Sub-Unit Modeling for Transcription, Alignment and Sign Language Recognition. Vassilis Pitsikalis 1, Stavros Theodorakis 1,

Advances in Phonetics-based Sub-Unit Modeling for Transcription, Alignment and Sign Language Recognition. Vassilis Pitsikalis 1, Stavros Theodorakis 1,
BODY LANGUAGE. Body Language Charles Darwin 1873 “ The expression of the emotions in man and animals” Desmond Morris 1967 “ The naked ape” Lie detectors.

BODY LANGUAGE. Body Language Charles Darwin 1873 “ The expression of the emotions in man and animals” Desmond Morris 1967 “ The naked ape” Lie detectors.
Designing Facial Animation For Speaking Persian Language Hadi Rahimzadeh 81271003 June 2005.

Designing Facial Animation For Speaking Persian Language Hadi Rahimzadeh June 2005.
M. Emre Sargın, Ferda Ofli, Yelena Yasinnik, Oya Aran, Alexey Karpov, Stephen Wilson,Engin Erzin, Yücel Yemez, A. Murat Tekalp Combined Gesture- Speech.

M. Emre Sargın, Ferda Ofli, Yelena Yasinnik, Oya Aran, Alexey Karpov, Stephen Wilson,Engin Erzin, Yücel Yemez, A. Murat Tekalp Combined Gesture- Speech.
CENTER FOR SPOKEN LANGUAGE UNDERSTANDING 1 PREDICTION AND SYNTHESIS OF PROSODIC EFFECTS ON SPECTRAL BALANCE OF VOWELS Jan P.H. van Santen and Xiaochuan.

CENTER FOR SPOKEN LANGUAGE UNDERSTANDING 1 PREDICTION AND SYNTHESIS OF PROSODIC EFFECTS ON SPECTRAL BALANCE OF VOWELS Jan P.H. van Santen and Xiaochuan.
International Conference on Automatic Face and Gesture Recognition, 2006 A Layered Deformable Model for Gait Analysis Haiping Lu, K.N. Plataniotis and.

International Conference on Automatic Face and Gesture Recognition, 2006 A Layered Deformable Model for Gait Analysis Haiping Lu, K.N. Plataniotis and.
 INTRODUCTION  STEPS OF GESTURE RECOGNITION  TRACKING TECHNOLOGIES  SPEECH WITH GESTURE  APPLICATIONS.

 INTRODUCTION  STEPS OF GESTURE RECOGNITION  TRACKING TECHNOLOGIES  SPEECH WITH GESTURE  APPLICATIONS.
Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004.

Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004.
Creating Dances - 3 Motif & Development. Objectives Recognise motifs Know how to create & develop motifs.

Creating Dances - 3 Motif & Development. Objectives Recognise motifs Know how to create & develop motifs.
LYU0103 Speech Recognition Techniques for Digital Video Library Supervisor : Prof Michael R. Lyu Students: Gao Zheng Hong Lei Mo.

LYU0103 Speech Recognition Techniques for Digital Video Library Supervisor : Prof Michael R. Lyu Students: Gao Zheng Hong Lei Mo.
CS335 Principles of Multimedia Systems Multimedia and Human Computer Interfaces Hao Jiang Computer Science Department Boston College Nov. 20, 2007.

CS335 Principles of Multimedia Systems Multimedia and Human Computer Interfaces Hao Jiang Computer Science Department Boston College Nov. 20, 2007.
Learn how to make your drawings come alive…  Lecture 2: SKETCH RECOGNITION Analysis, implementation, and comparison of sketch recognition algorithms,

Learn how to make your drawings come alive…  Lecture 2: SKETCH RECOGNITION Analysis, implementation, and comparison of sketch recognition algorithms,
LYU0103 Speech Recognition Techniques for Digital Video Library Supervisor : Prof Michael R. Lyu Students: Gao Zheng Hong Lei Mo.

LYU0103 Speech Recognition Techniques for Digital Video Library Supervisor : Prof Michael R. Lyu Students: Gao Zheng Hong Lei Mo.
Text-To-Speech Synthesis An Overview. What is a TTS System  Goal A system that can read any text Automatic production of new sentences Not just audio.

Text-To-Speech Synthesis An Overview. What is a TTS System  Goal A system that can read any text Automatic production of new sentences Not just audio.
KINS 151 Website www.hhs.csus.edu/HomePages/KHS/imamura/KINS151.

KINS 151 Website

Similar presentations

Ads by Google