1. Functional Data Analysis for Speech Research Michele Gubian Radboud University Nijmegen The Netherlands London, March 24 th 2010 Cambridge, March 26 th 2010

2. Content What and why Functional Data Analysis (FDA)  Motivation  Case study 1  Case study 2 – pitch re-synthesis How to use FDA  Using the R package ‘fda’

3. Motivation

4. Analyzing curves PCA ANOVA Linear models x x x x ? durext 58 48 98 … 2.8 3.8 2.9 … dur ext

5. Problems x x x x ? dur ext Decide what are the important features of a curve using  models  intuition / trial and error However Those features may not capture all the relevant dynamic aspects  e.g. concavity/convexity  long range correlatioins

6. Problems (2) x x x x ? dur ext Identify those feature points  manually  (semi)automatically However The identification may be hard, even ill-posed  time consuming  risk of subjective judgment

7. Analyzing curves with FDA x x x x ? dur ext Functional Data Analysis

8. Analyzing curves with FDA All the information contained in the curve (dynamics) is used  No need to reduce a curve to a set of significant features  No need to introduce assumptions on what is relevant in a curve shape and what is not FDA provides both VISUAL and QUANTITATIVE results  input is curves, output is also curves  plus classic statistical output like p-values, confidence intervals …

9. Functional Data Analysis: an extension of (some) statistical techniques to the domain of functions Example Ask people:  How old are you?  How much do you earn? Each data point is a point in 2D CLASSICFDA age salary x x x x x x x x Record people salary through the years Each “data point” is a whole CURVE age salary

10. Case study

11. Diphthong vs. hiatus in Spanish /ja/ vs. /i.a/ contrast is unstable in European Spanish  Diachronically, in Romance languages /i.a/ becomes /ja/  Diatopically, in Latin American Spanish the contrast seems to be lost  It is not present in orthography (“ia” in either case)  No strict minimal pairs Investigate  Consistent realization of the contrast  Inter-speaker variation  Cues used in the realization

12. Cues DIPHTHONG /ja/ HIATUS /i.a/ Duration Formants Pitch shortlong f1 f2 f1 f2 f0

13. Example diphthong

14. Example hiatus

15. Dataset Read speech  Diphthong ‘ Emiliana no, …’ /e.mi.lja.na#no#.../ (‘Not Emiliana, …’)  Hiatus ‘Mi liana no, … ‘ /mi#li.a.na#no#.../ (‘Not my liana, …’) 9 speakers (gender balanced) 20 repetitions per speaker per type In total 365 utterances

16. Duration

17. Pitch Pitch was extracted from the beginning of /l/ to the end of the rising gesture  In Spanish the pitch rising peak falls beyond the accented syllable ljalia

18. The raw data speaker /ja/ vs /i.a/

19. FDA data preparation Each sampled curve has to be turned into a function Decide how much detail to retain (smoothing)

20. FDA data preparation (2) All functions will be obtained by a combination of so-called basis functions, usually B-splines All functions will be linearly stretched in time to become of equal duration Functional representation B-spline

21. Classic Principal Component Analysis (PCA) age2565 salary x x x x x x x x x x x x x x x x x xx x x x x x x x x x x x x x x x x x PC1 PC2

22. Functional PCA on pitch contours

23. PCA does not know about labels !!

24. Functional PCA on pitch contours PC1

25. Functional PCA on pitch contours PC1

26. Functional PCA on pitch contours PC2

27. Functional PCA on pitch contours PC2

28. Functional PCA on formants PC2 PC1 f1 f2

29. Functional PCA on formants PC1

30. Cues coordination Duration vs formantsDuration vs pitch

31. Summary FDA provides tools to extract relevant dynamic characteristics of a set of curves Traditional tools like PCA (and linear regression) are extended to curves Functional PCA revealed the main dynamic cues used in the realization of a (weak) contrast in Spanish  Without using the labels information  Without extracting features from the curves (e.g. peaks)  Combining multi-dimensional curves (formants) without effort

32. References Functional Data Analysis website: www.functionaldata.org Books: Software: a bilingual (R and MATLAB) tool is freely available online

33. Appendix

34. Functional linear models y(t) = a(t) + b(t) x diphthong, x = 0 hiatus, x = 1 Confidence intervals for a(t) and b(t) R 2 (t) = percentage of explained variance