Presentation is loading. Please wait.

Presentation is loading. Please wait.

Österreichisches Forschnungsinstitut für Artificial Intelligence Featuring the GEMEP Corpus Experiences and Future Plans Hannes Pirker OFAI, Vienna.

Similar presentations


Presentation on theme: "Österreichisches Forschnungsinstitut für Artificial Intelligence Featuring the GEMEP Corpus Experiences and Future Plans Hannes Pirker OFAI, Vienna."— Presentation transcript:

1 Österreichisches Forschnungsinstitut für Artificial Intelligence Featuring the GEMEP Corpus Experiences and Future Plans Hannes Pirker OFAI, Vienna

2 Österreichisches Forschnungsinstitut für Artificial Intelligence Overview Features from Audio Channel Segments and pitch contours Features from Video Channel Faces, silhouettes and hands Discussion What we do have & what we need

3 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien3 Features from Audio Channel Phonetic segmentation into Phonemes Syllables Pitch Extraction

4 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien4 Speech Analysis Phonetic Segmentation Phonetic segmentation works quite good (forced alignment with HTK) Bootstrapping circle with Manual labelling of training data Training of HMMs Automatic alignment Manual correction of alignment results Re-training of HMMs Processed data: Typ 1 sentences (Ne kal ibam sud molen)

5 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien5 Speech Analysis Pitch Extraction Testing different pitch extraction methods from SFS (Mark Huckvale, UCL) Promising results with fine-tuning of parameters for contour- smoothing & high-pitch correction

6 Österreichisches Forschnungsinstitut für Artificial Intelligence Sample 06joi112

7 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien7 Features from Video Channel Face detection Silhouettes & Bounding Boxes Hand tracking

8 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien8 FaceDetection Using openCV Feature based face detection with pre-trained cascaded classifier (almost) out of the box Very good results under close-to- optimal conditions

9 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien9 FaceDetection Using openCV Feature based face detection with pre-trained cascaded classifier Clip:06joi112

10 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien10 Silhouettes & Bounding Boxes Combing silhouettes & bounding boxes in frontal and sideview as simple & robust estimator of Dynamics of movements Amount of expansion

11 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien11 Silhouettes & Bounding Boxes Combing silhouettes & bounding boxes in frontal and sideview as simple & robust estimator

12 Österreichisches Forschnungsinstitut für Artificial Intelligence 1 st Results: 3D Bounding Box Volume per Emotion – Actor 01

13 Österreichisches Forschnungsinstitut für Artificial Intelligence 1 st Results: 3D Bounding Box Volume per Emotion – Actor 07

14 Österreichisches Forschnungsinstitut für Artificial Intelligence 3D Bounding Box Volume: JOI vs. TRI / Actor 01 vs. Actor 06 ACTOR 01ACTOR 06

15 Österreichisches Forschnungsinstitut für Artificial Intelligence Temporal Dynamics: Plotting BoundingBox & Speech-Timing Clip:06joi112

16 Österreichisches Forschnungsinstitut für Artificial Intelligence Hand Tracking Find skin areas in 1 st frame

17 Österreichisches Forschnungsinstitut für Artificial Intelligence Hand Tracking Find skin areas in 1 st frame

18 Österreichisches Forschnungsinstitut für Artificial Intelligence Hand Tracking Find skin areas in 1 st frame Find hand & use center of area as hand position

19 Österreichisches Forschnungsinstitut für Artificial Intelligence Hand Tracking Find skin areas in 1 st frame Find hand & use center of area as hand position Interactively accept or correct

20 Österreichisches Forschnungsinstitut für Artificial Intelligence Hand Tracking Find skin areas in 1 st frame Find hand & use center of area as hand position Interactively accept or correct Perform automatic tracking (using mean shift algorithm)

21 Österreichisches Forschnungsinstitut für Artificial Intelligence Hand Tracking Perform automatic tracking (using mean shift algorithm)

22 Österreichisches Forschnungsinstitut für Artificial Intelligence Hand Tracking Perform automatic tracking (using mean shift algorithm) Interactively classify quality of tracking The GOOD (70%) The BAD (15%) The UGLY (15%)

23 Österreichisches Forschnungsinstitut für Artificial Intelligence Different Actors – Different Results

24 Österreichisches Forschnungsinstitut für Artificial Intelligence Different Actors – Different Results Difficult Easier

25 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien25 Discussion What have we gained? What do we need?

26 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien26 Investigations 1: Interaction Influence of Affect on Gesture AND speech Activation dimension should be reflected in speech and body movement in a parallel way. i.e. look at speed, effort etc.

27 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien27 Investigations 1: Interaction Influence of Affect on Gesture AND speech Activation dimension should be reflected in speech and body movement in a parallel way. i.e. look at speed, effort etc. Data already usable

28 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien28 Investigations 2: Timing Temporal aspects Look at the relative timing of speech and non-verbal signals, e.g. The location of strokes in relation to accented syllables

29 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien29 Timing: Traditional Anchor points in Speech Borders of Syllables Words (Prosodic and Syntactic) Phrases Utterances Turns Location of Pauses (Pitch) Accents

30 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien30 Timing: Traditional Anchor points in Speech Borders of Syllables +++ Words +++ (Prosodic & Syntactic) Phrases + - Utterances +? Turns -- Location of Pauses + (Pitch) Accents + --

31 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien31 Temporal alignment Traditional Anchor points in Gestures Prepare Stroke Hold Retract (Graphics by A. Marshall

32 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien32 Temporal alignment Traditional Anchor points in Gestures Phases difficult to obtain, but general information on dynamics available (Graphics by A. Marshall

33 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien33 Investigations 3: Bodily Expression of Emotion Relevant Features (cf Wallbott 1998): Upper Body: away from camera, collapsed Shoulders: up, backward, forward Head: down, back, turned or bent Arms: lateral, stretched out frontal/sideways, crossed

34 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien34 Investigations 3: Bodily Expression of Emotion Relevant Features (cf Wallbott 1998): Hand-form: fist(s), opening/closing Movement Qualities: Activity,Expansiveness, Dynamics/Energy/Power

35 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien35 Investigations 3: Bodily Expression of Emotion Relevant Features (cf Wallbott 1998): Data currently focussing on hand location. Upper-body and posture not directly assessed. Movement qualities partially accessible

36 Österreichisches Forschnungsinstitut für Artificial Intelligence What we DO have by Now

37 Österreichisches Forschnungsinstitut für Artificial Intelligence What we might WANT to have?

38 Österreichisches Forschnungsinstitut für Artificial Intelligence Possible Representations H-ANIM/MPEG-4-style joint angles

39 Österreichisches Forschnungsinstitut für Artificial Intelligence Possible Representations H-ANIM/MPEG-4-style joint angles Gesticon/MURML/HamNoSys- style wrist position in relation to body encoding of the stroke-phase etc.

40 Österreichisches Forschnungsinstitut für Artificial Intelligence Possible Representations Less ambitious Symmetric/Assymetric Extended/Collapsed Static/Dynamic … In any case: Need to relate pixel- numbers to anthropomorphic measures!

41 Österreichisches Forschnungsinstitut für Artificial Intelligence Possible Representations Also consider (manually supported) classification into prototypical classes

42 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien42 Summary Numerous possibilities for improvements, e.g., in hand tracking (but is it necessary?) Rather concentrating on representations in order to ensure data really is useful and re-usable (e.g. ECAs)

43 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien43 Summary Time to bundle expertises and distribute efforts within

44 Österreichisches Forschnungsinstitut für Artificial Intelligence © ÖFAI, Wien44 Thanks for your attention

45 Österreichisches Forschnungsinstitut für Artificial Intelligence

46 Österreichisches Forschnungsinstitut für Artificial Intelligence

47 Österreichisches Forschnungsinstitut für Artificial Intelligence Sample 06col112

48 Österreichisches Forschnungsinstitut für Artificial Intelligence Sample: 06peu111

49 Österreichisches Forschnungsinstitut für Artificial Intelligence

50 Österreichisches Forschnungsinstitut für Artificial Intelligence

51 Österreichisches Forschnungsinstitut für Artificial Intelligence

52 Österreichisches Forschnungsinstitut für Artificial Intelligence

53 Österreichisches Forschnungsinstitut für Artificial Intelligence


Download ppt "Österreichisches Forschnungsinstitut für Artificial Intelligence Featuring the GEMEP Corpus Experiences and Future Plans Hannes Pirker OFAI, Vienna."

Similar presentations


Ads by Google