AES 120 th Convention Paris, France, 2006 Adaptive Time-Frequency Resolution for Analysis and Processing of Audio Alexey Lukin AES Student Member Moscow State University, Moscow, Russia Jeremy Todd AES Member iZotope, Inc., Cambridge, MA

A. Lukin, J. Todd “Adaptive Time-Frequency Resolution” 2/15 Short-Time Fourier Transform Most commonly used transform for audio: Most commonly used transform for audio: ► Spectral analysis ► Noise reduction (spectral subtraction algorithms) ► Time-variable filters and other effects Very fast implementation for large number of bands via FFT Very fast implementation for large number of bands via FFT Good energy compaction for many musical signals Good energy compaction for many musical signals Many oscillations in basis functions → ringing (Gibbs phenomenon) Many oscillations in basis functions → ringing (Gibbs phenomenon) Uniform frequency resolution → inadequate resolution at lows Uniform frequency resolution → inadequate resolution at lows + –

A. Lukin, J. Todd “Adaptive Time-Frequency Resolution” 3/15 Filter banks Idea: Idea: Decompositions of time-frequency plane Decompositions of time-frequency plane Decomposition Processing of subband signals Synthesis x[n]y[n] …… f t STFT f t DWT Uncertainty principle

A. Lukin, J. Todd “Adaptive Time-Frequency Resolution” 4/15 Suggested approach Transforms must vary their time-frequency resolution in a perceptually motivated way ► Imitation of time-frequency resolution of human hearing ► Adaptation of resolution to local signal features

A. Lukin, J. Todd “Adaptive Time-Frequency Resolution” 5/15 Spectrograms Problems: Problems: ► Most perceptually meaningful energy is concentrated in the narrow band below 4 kHz → can’t see useful details ► Time/frequency resolution trade-off Conventional STFT spectrogram (linear frequency scale)

A. Lukin, J. Todd “Adaptive Time-Frequency Resolution” 6/15 Spectrograms Problems: Problems: ► Poor frequency resolution at low frequencies → can’t separate bass harmonics from bass drum ► Time/frequency resolution trade-off Mel-scale STFT spectrogram (window size = 12 ms)

A. Lukin, J. Todd “Adaptive Time-Frequency Resolution” 7/15 Spectrograms Problems: Problems: ► Poor time resolution at transients → time-smearing of drums Mel-scale STFT spectrogram (window size = 93 ms)

A. Lukin, J. Todd “Adaptive Time-Frequency Resolution” 8/15 Spectrograms Simple solution: combine spectrograms with different resolutions Simple solution: combine spectrograms with different resolutions ► Take bass from spectrogram with good freq. resolution ► Take treble from spectrogram with good time resolution

A. Lukin, J. Todd “Adaptive Time-Frequency Resolution” 9/15 Spectrograms Simple solution: Simple solution: ► Combine spectrograms with different resolutions: take bass from spectrogram with good frequency resolution, take treble from spectrogram with good time resolution Combined resolution spectrogram (window sizes from 12 to 93 ms)

A. Lukin, J. Todd “Adaptive Time-Frequency Resolution” 10/15 Spectrograms Better approach: select best resolution for each time-frequency neighborhood Better approach: select best resolution for each time-frequency neighborhood Criteria? Criteria? ► Better frequency resolution at bass (reflects a-priori psychoacoustical knowledge) ► Maximal energy compaction (to minimize spectral smearing in both time and frequency) 6 ms12 ms24 ms48 ms96 ms best STFT window size

A. Lukin, J. Todd “Adaptive Time-Frequency Resolution” 11/15 Spectrograms Calculation of energy compaction Calculation of energy compaction (energy smearing in the given block for all given resolutions) 6 ms12 ms24 ms48 ms96 ms best STFT window size a i,r Here a i,r are descendingly sorted STFT magnitudes in the block, r S r is the energy smearing for the given resolution r, r 0 r 0 is the resolution with best energy compaction.

A. Lukin, J. Todd “Adaptive Time-Frequency Resolution” 12/15 Spectrograms Benefits: Benefits: ► Sharper bass drum hits and other transients, even in mid- frequency range ► Sharper guitar harmonics in high frequencies Adaptive resolution spectrogram (window sizes from 12 to 93 ms)

A. Lukin, J. Todd “Adaptive Time-Frequency Resolution” 13/15 Spectrograms Tone onset waveform More examples Conventional STFT spectrogram

A. Lukin, J. Todd “Adaptive Time-Frequency Resolution” 14/15 Spectrograms Combined resolution spectrogram More examples Adaptive resolution spectrogram

A. Lukin, J. Todd “Adaptive Time-Frequency Resolution” 15/15 Processing framework General framework for General framework for multi-resolution processing ► Perform processing with several different resolutions ► Adaptively combine (mix) results in time-frequency space ► Mixing is controlled by a-priori knowledge of psychoacoustics and analysis of local signal features (e.g. transience)

A. Lukin, J. Todd “Adaptive Time-Frequency Resolution” 16/15 Noise reduction Spectral subtraction (short windows) Mixer of coefficients y[t] x 3 [t] Spectral subtraction (long windows) STFT Synthesis x 1 [t] x 2 [t] Transience analysis control Spectral subtraction algorithm modifications Spectral subtraction algorithm modifications ► Better frequency resolution at low frequencies (according to the human hearing resolution) ► Better temporal resolution near signal transients (for reduction of Gibbs phenomenon)

A. Lukin, J. Todd “Adaptive Time-Frequency Resolution” 17/15 Noise reduction Results of single-resolution and multi-resolution algorithms Results of single-resolution and multi-resolution algorithms Noisy recording (guitar + castanets)

A. Lukin, J. Todd “Adaptive Time-Frequency Resolution” 18/15 Noise reduction Results of single-resolution and multi-resolution algorithms Results of single-resolution and multi-resolution algorithms Single resolution Multi-resolution

A. Lukin, J. Todd “Adaptive Time-Frequency Resolution” 19/15 Your questions Demo web page: Poster session P17: Monday, 9:00 – 10:30 a.m. ?