1. Regression Approaches to Voice Quality Control Based on One-to-Many Eigenvoice Conversion Kumi Ohta, Yamato Ohtani, Tomoki Toda, Hiroshi Saruwatari, and Kiyohiro Shikano Nara Institute of Science and Technology (NAIST), Japan August 23rd, 2007

2. 1 –Amusement device –Speech enhancement device for a speaking aid system recovering a disabled person’s voice for a hearing aid system to make speech sounds more intelligible Voice Quality Control Technique for converting user’s voice quality into another one Applications Development of voice quality control with high quality and high controllability is desired! Controller Hello.

3. 2 Contents 1. Conventional voice quality control methods 2. Proposed voice quality control methods 3. Experimental verification 4. Conclusions 1. Conventional voice quality control methods 2. Proposed voice quality control methods 3. Experimental verification 4. Conclusions

4. 3 Arbitrary speakers Multiple pre-stored target speakers Conversion Training Source speaker Hello. Thank you. Hello. Thank you. Hello. Thank you. Hello. Thank you. Let’s convert. Eigenvoice GMM (EV-GMM) Manually setting Parallel data One-to-Many Eigenvoice Conversion (EVC) [Toda et al., 2006] A source speaker’s voice is statistically converted into an arbitrary speaker’s one.

5. 4 Converted voice quality is controlled by weights for eigenvectors. Eigenvoice GMM (EV-GMM) Weight Mean vector Covariance matrix Eigenvectors (for eigenvoices) Bias vector (for average voice) Parameters of the i th mixture Source mean vector Target mean vector = + Weights for eigenvoices (free parameters) Problem: eigenvoices do NOT represent a specific physical meaning (such as a masculine voice or a clear voice). Intuitive control of the converted voice quality is difficult! : Speaker independent parameters : Free parameters

6. 5 Contents 1. Conventional voice quality control methods 2. Proposed voice quality control methods 3. Experimental verification 4. Conclusions

7. 6 Proposed Framework We would like to intuitively control the converted voice quality! We propose multiple regression approaches to one-to-many EVC. Converted voice quality is controlled with the voice quality control vector. * Similar approaches have been proposed in HMM-based speech synthesis [Tachibana et al., 2006].

8. 7 Process of Proposed Framework 1. Preparing multiple parallel data sets 2. Setting the voice quality control vector for every pre-stored target speaker 3. Modeling the target mean vectors with voice quality control vector 1. Preparing multiple parallel data sets 2. Setting the voice quality control vector for every pre-stored target speaker 3. Modeling the target mean vectors with voice quality control vector

9. 8 Setting Voice Quality Control Vector We manually assign scores for expression word pairs to each pre-stored target speaker. Assigned scores are used as components of the voice quality control vector. Tense Hoarse Masculine Elderly Thin Feminine Clear Youthful Deep Lax -3-20123 Very Quite Some- what No preference 2 1 1 -2 Voice quality control vector for the speaker A Assigned scores for the speaker A

10. 9 Process of Proposed Framework 1. Preparing multiple parallel data sets 2. Setting the voice quality control vector for every pre-stored target speaker 3. Modeling the target mean vectors with voice quality control vector We propose 3 regression methods.

11. 10 Proposed Method A Regression parameters Principal components for the s th target speaker  Modeling principal components is modeled by  Minimizing the following error function: Error of principal components for the s th pre-stored target speaker Total error over all pre- stored target speakers Least-squares (LS) estimation of regression parameters converting the voice quality control vector into principal components Voice quality control vector for the s th target speaker

12. 11 Resulting EV-GMM in Method A = Weight Mean vector Covariance matrix Eigenvectors Bias vector Parameters of the i th mixture Target mean vector Regression parameters Voice quality control vector + + Problem: the desired voice characteristics might not be represented as a linear combination of eigenvectors. Changing the eigenvectors themselves is necessary! : Training parameters : Speaker independent EV-GMM parameters

13. 12 Proposed Method B  Minimizing the following error function:  Target mean vector is modeled by Error of target mean vectors for the s th pre-stored target speaker Total error over all pre- stored target speakers LS estimation of a regression parameters converting the voice quality control vector into the target mean vectors = + Regression parameters Target mean vector for the s th target speaker Voice quality control vector for the s th target speaker

14. 13 Resulting EV-GMM in Method B = Weight Mean vector + Covariance matrix Regression parameters Parameters of the i th mixture Target mean vector Voice quality control vector Problem: the desired voice quality might not be obtained because the converted voice quality is affected by all EV- GMM parameters. : Training parameters : Speaker independent EV-GMM parameters

15. 14 Proposed Method C  Maximizing the following likelihood function: * This process is considered as speaker adaptive training (SAT) of EV-GMM [Ohtani et al., Interspeech 2007]. Likelihood of the adapted EV-GMM for each pre-stored target speaker Maximum Likelihood (ML) estimation of all EV-GMM parameters while fixing the voice quality control vector Total likelihood over all pre- stored target speakers  Target mean vector is modeled by = + Regression parameters Target mean vector for the s th target speaker Voice quality control vector for the s th target speaker

16. 15 Resulting EV-GMM in Method C = Weight Mean vector + Covariance matrix Parameters of the i th mixture Target mean vector Voice quality control vector Regression parameters : Training parameters

17. 16 Comparison of Proposed Methods Dependent variables Tied parameters of EV-GMM Training criterion Method A Principal components Speaker independent LS Method B Target mean vectors Speaker independent LS Method C Target mean vectorsOptimizedML

18. 17 Contents 1. Conventional voice quality control methods 2. Proposed voice quality control methods 3. Experimental verification 4. Conclusions

19. 18 Verification of Proposed Methods Objective verification Subjective verification Source speakerOne female Pre-stored target speakers15 males and 15 females SentencesPhonetically balanced 50 sentences per a speaker Expression word pairsmasculine / feminine, hoarse / clear, elderly / youthful, thin / deep, lax / tense Number of mixtures128 Number of Eigenvectors29 (no loss of information) Experimental conditions

20. 19 Objective Verification Is a correspondence of the voice quality control vector into the converted voice quality appropriately modeled? For each pre-stored target speaker in the training data, the following two voice quality control vectors were compared. 1. Manually assigned one 2. Adjusted one on the trained EV-GMM so that the converted voice quality becomes similar to the target * approximately determined by maximum likelihood eigen- decomposition for EV-GMM [ Toda et al., 2006 ] using two sentences Euclidean distance and correlation coefficient between those two vectors were calculated as objective measures.

21. 20 Results of Objective Verification * Reassigned: assigned scores by the same listener a second time on a different day Worse Better Worse Better! 1. The method A does not work at all. 2. The method B works but not so good. 1. The method A does not work at all. 2. The method B works but not so good. 3. The method C works reasonably well. Too consistent compared with human judgment? Better!

22. 21 Subjective Verification Preference test on the converted speech quality was conducted. –Comparison of average voices* by the trained EV-GMMs * converted voices when setting every component of the voice quality control vector to zero Test sentences50 sentences not included in training data Number of subjects5 Experimental conditions Having very similar speaker individuality in both method B and C Which is better, the method B or the method C?

23. 22 Result of Subjective Verification The method B outperforms the method C. Possibility to be thought –The EV-GMM parameters trained in EM algorithm converged to local optima due to using inappropriate initial model (i.e., the target independent GMM).

24. 23 Contents 1. Conventional voice quality control methods 2. Proposed voice quality control methods 3. Experimental verification 4. Conclusions

25. 24 Conclusions Proposal of regression approaches to the voice quality control based on one-to-many eigenvoice conversion (EVC) –Based on a statistical conversion framework –Allowing intuitive control of converted voice quality with voice quality control vector Experimental verification –Showing the possibility that voice quality control with high quality and high controllability is realized.