1. Speech Prosody Conversion using Sequence Generative Adversarial Nets
with Continuous Wavelet Transform F0 features
Zhaojie Luo, Tetsuya Takiguchi, and Yasuo Ariki (Kobe University)
Canonical Correlation Analysis
Overview
Background
Problems
Goal
1. The representation of fundamental frequency (F0) is too simple for prosody conversion.
2. GANs can only give the score/loss for an entire sequence when it has been generated.
1. Applying the continuous wavelet transform (CWT) to systematically capture the F0 features of different temporal scales.
2. Using sequence-GAN to address the continuity of each matrix in a completed sentence, which can train both long-term and short-term dependencies.
Sentence
Sentence
Keep linguistic information unchanged　
Neutral
Sad
Happy
Angry
Speech prosody conversion
Emotional robot
Framework
Input voice
Features extraction
Features processing
Matrics processing
Training
Conversion
seq-GANs
seq-GANs
Dataset
Feature extraction
Dataset
Training model
Mexican hat
Interpolated log-normalized F0 and wavelet transform features (i=30, i=24, i=18, i=12, i=6)
Illustration of training the sentence features in the Seq-GANs
Dataset
Results
(a) recorded voice
(b) DBNs+LG
(c) DBNs+NNs
(d) GAN
(e) Sequence-GAN
Tar./Percept
Ang.
Sad
Hap.
Neu.
Angry 90 2 8 27 5 68 59 13 3 25 33 21 44 71 4 1 24 97 62 53 43 39 45 15 12 17
Happy 98 18 74 38 47 48 80
Neutral 95 14 10 7 69 56 22 23 37 11 60