2. The Reliability of Formant Measurements in High Quality Audio Data: The Effect of Agreeing Measurement Procedures Martin Duckworth, Kirsty McDougall, Gea de Jong, Linda Shockey

3. Introduction Formant measurement implicitly required legally in the UK in speaker comparison cases Measurements on analogue spectrograms had to be by hand and eye Measurements on digital spectrograms can be assisted by formant trackers, LPC is common

4. Introduction How replicable are measurements by eye on digital spectrograms?

5. Introduction How replicable are measurement by eye on digital spectrograms? If LPC tracking is used what can lead to variability?

6. Introduction How replicable are measurement by eye on digital spectrograms? If LPC tracking is used what can lead to variability? −Software settings

7. Introduction How replicable are measurement by eye on digital spectrograms? If LPC tracking is used what can lead to variability? −Software settings −Point at which data is extracted

8. Study Aims What is required in order to make measurements more replicable?

9. Study Aims What is required in order to make measurements more replicable? If software (but not method) is held constant and data is high quality, can different laboratories make the same F1-3 measurements?

10. Study Aims What is required in order to make measurements more replicable? If software (but not method) is held constant and data is high quality, can different laboratories make the same F1-3 measurements? If method of analysis is the same does this lead to statistically improved reliability between laboratories?

11. Aims continued We are aiming to find a reliable means of obtaining formant values We are examining reliability, not validity

12. Data read speech from Cambridge DyViS database male Standard Southern British English aged 18-25 40 speakers:Set 1 (20 speakers) Set 2 (20 speakers)

13. Data 6 monophthongs: / i ː, æ, ɑː, ɔː, ʊ, u ː / 6 repetitions per vowel per speaker elicited in hVd contexts in sentences: It’s a warning we’d better HEED today. It’s only one loaf, but it’s all Peter HAD today. We worked rather HARD today. We built up quite a HOARD today. He insisted on wearing a HOOD today. He hates contracting words, but he said a WHO’D today.

14. Measurements Analysts from 3 labs – Cambridge, Plymouth, Reading Task: to measure F1, F2, F3 for each vowel token using Praat Set 1 – using individual – but constrained- methods Set 2 – after a meeting at which a single method is agreed

15. Set 1 Methods Measure the formants at a relatively early point in the vowel

16. Set 1 Methods Measure the formants at a relatively early point in the vowel Measure formants over no more than 5 glottal pulses

17. Set 1 Methods Measure the formants at a relatively early point in the vowel Measure formants over no more than 5 glottal pulses Use either: −LPC tracking checked against the spectrogram or

18. Set 1 Methods Measure the formants at a relatively early point in the vowel Measure formants over no more than 5 glottal pulses Use either: −LPC tracking checked against the spectrogram or −hand/eye measures

19. Set 2 Method Measure towards the start of the vowel

20. Set 2 Method Measure towards the start of the vowel Measure in a relatively steady early part of the vowel

21. Set 2 Method Measure towards the start of the vowel Measure in a relatively steady early part of the vowel Measure around the vowel's maximum intensity

22. Set 2 Method Measure towards the start of the vowel Measure in a relatively steady early part of the vowel Measure around the vowel's maximum intensity Use a single time slice

23. Set 2 Method (continued) Use the LPC formant tracker adjusted for best visual fit

24. Set 2 Method (continued) Use the LPC formant tracker adjusted for best visual fit When values generated by Praat are judged by visual inspection to be incorrect, replace them by correct values from a time-slice immediately preceding or following the slice being measured.

25. Results: HAD, F1 Lab1 Lab2 Lab3 Set 1

26. Results: HAD, F1 Lab1 Lab2 Lab3 Set 1

27. Results: HAD, F1 Lab1 Lab2 Lab3 Set 1 Set 2

28. Results: HAD, F1 Lab1 Lab2 Lab3 Set 1 Set 2

29. Statistical Analysis 3 formants  6 vowels  2 datasets = 36 tests Two-way ANOVA - repeated measures on the factor Lab (3) - between-groups factor Speaker (20) If Lab signficant at p < 0.05: Pairwise comparisons with Sidak correction

30. Results: HAD, F1 Lab1 Lab2 Lab3 Set 1 Set 2

31. Results: HAD, F1 Lab1 Lab2 Lab3 Lab: significant Set 1 Set 2

32. Results: HAD, F1 Lab1 Lab2 Lab3 Lab: significant 0.001 0.000 Set 1 Set 2

33. Results: HAD, F1 Lab1 Lab2 Lab3 0.001 0.000 Set 1 Set 2 Lab: significant Lab: significant but pairwise comparisons NS

34. Results: HAD, F1 Lab1 Lab2 Lab3 Lab: significant 0.001 0.000 Set 1 Set 2 NS Lab: significant but pairwise comparisons NS

35. Results: HAD, F2

36. Lab1 Lab2 Lab3 Set 1 Set 2 NS Lab: not significant NS

37. Results: HAD, F3

38. Lab1 Lab2 Lab3 Set 1 Set 2 Lab: significant Lab: not significant NS 0.000 NS

39. Summary - HAD F1F2F3F1F2F3 LabsigNSsig NS 1 vs 2sigNS 1 vs 3sigNSsigNS 2 vs 3sigNSsigNS Set 1 Set 2

40. Summary - HAD F1F2F3F1F2F3 LabsigNSsig NS 1 vs 2sigNS 1 vs 3sigNSsigNS 2 vs 3sigNSsigNS Set 1 Set 2 main effect

41. Summary - HAD F1F2F3F1F2F3 LabsigNSsig NS 1 vs 2sigNS 1 vs 3sigNSsigNS 2 vs 3sigNSsigNS Set 1 Set 2 pairwise comparisons

42. Summary - HAD F1F2F3F1F2F3 LabsigNSsig NS 1 vs 2sigNS 1 vs 3sigNSsigNS 2 vs 3sigNSsigNS Set 1 Set 2

43. Summary - HAD F1F2F3F1F2F3 LabsigNSsig NS 1 vs 2sigNS 1 vs 3sigNSsigNS 2 vs 3sigNSsigNS Set 1 Set 2 improvement

44. Summary - HAD F1F2F3F1F2F3 LabsigNSsig NS 1 vs 2sigNS 1 vs 3sigNSsigNS 2 vs 3sigNSsigNS Set 1 Set 2

45. Summary - HAD F1F2F3F1F2F3 LabsigNSsig NS 1 vs 2sigNS 1 vs 3sigNSsigNS 2 vs 3sigNSsigNS Set 1 Set 2

46. Summary - HAD F1F2F3F1F2F3 LabsigNSsig NS 1 vs 2sigNS 1 vs 3sigNSsigNS 2 vs 3sigNSsigNS Set 1 Set 2 improvement

47. Summary - HAD F1F2F3F1F2F3 LabsigNSsig NS 1 vs 2sigNS 1 vs 3sigNSsigNS 2 vs 3sigNSsigNS Set 1 Set 2

48. Summary - HAD F1F2F3F1F2F3 LabsigNSsig NS 1 vs 2sigNS 1 vs 3sigNSsigNS 2 vs 3sigNSsigNS Set 1 Set 2 Set 2: good news

49. Effect of Lab - 6 vowels Set 1 F1F2F3 heedsigNSsig hadsigNSsig hardsig hoardsig who’dsig NS hoodsig

50. Effect of Lab - 6 vowels Set 1 Set 2 F1F2F3F1F2F3 heedsigNSsig NSsig hadsigNSsig NS hardsig NS sig hoardsig NS who’dsig NSsig hoodsig NS sig NS

51. Influence of Speaker Interaction Lab x Speaker significant (p < 0.05) for F1-F3 of all 6 vowels for both Set 1 and Set 2  certain speakers lead to measurement differences among labs for example…

52. F3 of HARD (Set 2) means by speaker

53. Agreement across labs in most cases, but certain individuals lead to measurement differences among labs

54. F3 of HARD (Set 2) means by speaker Agreement across labs in most cases, but certain individuals lead to measurement differences among labs

55. Subject 42 HARD6 F3 = 3325 Hz Subject 42 HARD4 F3 = 2219Hz Subject 42 HARD2 F3 = 2579Hz Difficult cases: subject 42 F3

56. Difficult cases: subject 43 F3 Subject 43 HARD2 F3? Subject 43 HARD1 F3? Visual inspection Visual inspection vs formant tracker Visual inspection

57. Subject 43 HARD2 F3? Subject 43 HARD1 F3? Visual inspection Tracker

58. The effect of intraspeaker variability, possibly voice quality This can affect: −The visibility of formants −The functioning of the LPC tracker for example…

59. The effect of intraspeaker variability Subject 37: HAD1 F1=??Subject 37: HAD6 F1..had today.

60. Discussion: Laboratory Effects Do different laboratories produce different formant values?

61. Discussion: Laboratory Effects Do different laboratories produce different formant values? YES

62. Discussion: Laboratory Effects Do different laboratories produce different values formant values? YES Does replicating the measurement method reduce these differences?

63. Discussion: Laboratory Effects Do different laboratories produce different formant values? YES Does replicating the measurement method reduce these differences? YES

64. Discussion: Laboratory Effects Do different laboratories produce different formant values? YES Does replicating the measurement method reduce these differences? YES Could these be reduced further?

65. Discussion: Laboratory Effects Do different laboratories produce different formant values? YES Does replicating the measurement method reduce these differences? YES Could these be reduced further? YES

66. Other sources of variability Settings (e.g. No. of poles; No of Formants in Praat)

67. Other sources of variability Settings The exact point in the vowel at which the measure is taken

68. Other sources of variability Settings The exact point in the vowel at which the measure is taken The ‘readability’ of the spectrogram which can be affected by speaker characteristics

69. Conclusion Developing standard ways of collecting formant values could assist comparisons between experts in case work If records are kept relating to time points, software and settings then the measurement process can be replicated

70. Acknowledgements IAFPA Research Grant for travel expenses Economic and Social Research Council UK for funding the DyViS Project ‘Dynamic Variability in Speech: A Forensic Phonetic Study of British English’ [RES-000-23-1248] Other members of the DyViS project – Francis Nolan and Toby Hudson