Connecting Acoustics to Linguistics in Chinese Intonation Greg Kochanski (Oxford Phonetics) Chilin Shih (University of Illinois) Tan Lee (CUHK) with Hongyan Jing (IBM) Jiahong Yuan (Cornell)

Questions Can we usefully include biomechanics into a phonetics model? Can we objectively assign an importance to a syllable? Can we write a unified description of F 0 for both tone and accent languages? Goal Build a mathematical model that takes a sequence of discrete symbols as input and produces a quantitative prediction for f0.

The Challenge

Existing work Rising?

Basic assumptions used in modeling People plan their utterances several syllables in advance. People produce speech optimized to communicate with minimal effort. A realistic model for the muscles that control f 0

Realistic model of muscle control for F 0 We’d like a model of prosody that can apply beyond F 0.

People talk nearly as fast as possible.

Speech could be optimal Most of what we say is made from bits and pieces we’ve said before. There are only 4 (Mandarin) or 6 (Cantonese) tones to combine. A speaker has the chance to practice and optimize all the common 3- and 4- tone sequences.

Optimize what? People want to minimize effort and/or talk faster –Chairs, Cars People want to minimize the chance that they will be misunderstood. –Risk = P(misinterpreted) * cost(misinterpreted) Minimize: Effort + cost*Error –We allow each syllable to have a different weight, so error is a sum over syllables or words. –Perhaps cost matches importance.

Modeling math is the muscle tension (~frequency) at time t. “Effort” Each target encodes some linguistic information, r i is the error of the i th target, and s i is its importance. y is the i th pitch target and a bar denotes an average over a target. “Error”

Effort and Error How does Effort depend on the form of the pitch curve? Error = mean-squared deviation between the f0 and the templates.

Model behavior For cost>>1, Error dominates, and pitch matches target. For cost<<1, Effort dominates, both speaker and listener accept large deviations, and pitch smoothly interpolates. For cost~1, everything compromises. Cost plays the role of a prosodic strength.

Another Challenge Time (10 ms intervals) F 0 (Hz) Tone shapes

The rest of the model. A model is a sequence of targets (used to compute the Error terms). Each target has a strength (i.e. the cost of misinterpretation). One target per tone. Targets are stretched to fit syllable duration. Only one phonological rule: 33  23

Model fits for Mandarin Chinese Tone class (input) Strength (result) Inside a word, strength is distributed by the metrical pattern

What’s the procedure? Compute the pitch curve as a function of phonological inputs and prosodic strength. Sequence of tones (phonology) Prosodic strengths Predicted F 0 Data Nonlinear least-squares fitting algorithm

Model fits to Mandarin Chinese 0.61 free parameters per syllable, 13 Hz RMS error.

Strengths are stable under small changes in the model. The two models have words defined by different labelers This model allows extra freedom: different tones are allowed to define their targets differently This model allows less freedom: all tones have the same type of target.

Model parameters Mandarin Cantonese Phrasing is marked in speech. Cantonese data courtesy of Prof. Tan Lee

Model parameters Cantonese Mandarin Nouns are relatively important.

Model parameters Cantonese Mandarin Longer words tend to be spoken more carefully.

Metrical patterns inside words Mandarin “Normal” segmentation of characters into words. Random segmentation of characters into words. Lexical acquisition

Other nice properties Strengths are correlated with duration: (duration is a proxy for prominence) r = 0.40 (sentence final) r = 0.27 (non-final) >95% confidence Strength is correlated with mutual information of neighboring syllables: r = >95% confidence Sloppy when generating unsurprising syllables, and precise for surprising syllables.

Local Conclusion Intonation can be represented as: – a small set of discrete symbols, in sequence, with –a per-person or per-style shape for each symbol; –modulated by a variable prosodic strength. One symbol per syllable seems enough The strength parameter seems real –Similar across languages –Matches language structure

Q: But does it work for English? A: Yes, under circumstances where the intonational phonology is simple enough to be obvious.

Reminder: Limitations of f 0 and complexity of prosody. To show the range of information that can be carried by prosody, observe an elegant experiment by Stan Freberg (1950): The text has virtually no lexical information, but it still tells a story. Even so, it is very hard to label individual words.

English Sentences in the form “ ?” Speaker is trying to confirm a single digit. Models have just 1.1 parameter per sentence.

The model for English There are identical boundary tones on every utterance. All target shapes are identical, except the focus. %X B B B | B A B | B B B B Y% %X B B B | A B B | B B B B Y% %X B A B | B B B | B B B B Y% Rather simple phonology. Accent prominence depends on position in phrase and in utterance.

Model details Strength time 910 – Decline over utterance Decline over phrase Local effect around accent Compress range after accent

The rest of the model. Where do you put the targets? What are the targets? –Pitch values? –Slopes? Do the targets change in f 0 range with changes in strength?

Model fits well over a range of speeds. Suppressed phrasing Low speed High speed Merger of accent with boundary tone

Model reproduces nontrivial features of the data and fits well over a range of speeds. Suppressed phrasing Low speed High speed Merger of accent with boundary tone

Conclusion Physiologically-based models can capture important aspects of speech. A very compact representation of behavior. It can be applied broadly: Two dialects of Chinese Some aspects of English It raises questions about where the phonetics/phonology boundary actually sits. Introduces an objective acoustic measure of prosodic prominence. Suggests that the speaker may help the listener segment the speech stream.