HIWIRE meeting ITC-irst Activity report Marco Matassoni, Piergiorgio Svaizer March Torino

Outline Beamforming and Adaptive Noise Cancellation Environmental Acoustics Estimation Audio-Video data collection Multi-channel pitch estimation Fixed-platform prototype acquisition module

Beamforming: D&S Availability of multi-channel signals allows to selectively capture the desired source: Issues: estimation of reliable TDOAs; Method: CSP analysis over multiple frames Advantages: robustness reduced computational power

D&S with MarkIII Test set: set N1_SNR0 of MC-TIDIGITS (cockpit noise), MarkIII channels clean models, trained on original TIDIGITS Results (WRR [%]): C_138.5 C_ DS_C879.9 DS_C DS_C DS_C6485.4

Adaptive Noise Cancellation A remote microphone can be used as reference for noise estimation: equivalent noise path filter (cockpit) noise (beamformed) speech Adaptive filter noisy speech filtered noise denoised speech

NMLS The tested algorithm is the Normalized Mean Least Squares: iterativelly estimate a FIR filter that minimizes the difference between the primary channel and the reference We implemented two algorithms: time domain frequency domain (subband)

D&S + ANC Test set: set N1_SNR0 of MC-TIDIGITS (cockpit noise), MarkIII channels clean models, trained on original TIDIGITS Results (WRR): C_32 (T)64.7 C_32 (F)72.4 DS_C64 (T)81.8 DS_C64 (F)88.4

Acoustics estimation Idea: Simulate in a realistic way an environment (and the noise) Method: Measure several impulse responses in an environment with a multi-channel equipment (through reproduction of chirp signals) preserving relative amplitudes and mutual delays; Generate appropriate noisy signals starting from clean data; The derived acoustics models perform better in the given environment (also) using real data.

Audio–Video Data Collection Idea: In a noisy environment exploit additional features from video data (collaboration with NTUA and TUC) Design of AV corpus: Task: English connected digits, HIWIRE commands/keywords Channels: 4 audio, 3 video Environment: acoustically-treated room + noise diffusion

Audio–Video Setup ) ) ) ) ) ) Cockpit noise ) ) ) cm

Audio–Video Setup Audio 4 omnidirectional PZM Shure microphones, 16 kHz/16 bits background noise diffused by 2 loudspeakers Video Webcam: 640x480, 30 fps – color, Unix timestamps Stereoscopic camera pair: 640x480, 30 fps - bw or 15 fps – color, perfectly synchronous Current data sets 8 speakers / connected digits 2 speakers / HIWIRE keyword lists

Fixed prototype acquisition device Hardware platform: 8 Shure microphones + RME Hammerfall Software environment: Linux, ALSA driver Acquisition module: acquires synchronously multiple channels (8); writes (to its standard output/file) the enhanced signal + additional information/features (start/end speech hyphoteses, voiced/unvoiced, pitch, …)

Multi-channel pitch analysis The basic principle is that we can exploit many observations of the same speech process Once located the speaker, we can take into account the different propagation time at the microphones and perform a time-alignment Pitch analysis can be performed using: adjacent time intervals extracted from different microphone signals Basic correlation techniques: AMDF, AUTOC, WAUTOC

Single Channel Method: Weighted Autocorrelation For every frame of length N: (see Shimamura-Kobayashi, Trans. on SAP, 2001) Hz Samples

WAUTOC is computed for each channel, and summed over the M channels. For a given frame: Issues: Weights w i may represent the channel reliability ; Use of possible intraframe smoothing of the resulting fundamental frequency contour, which could improve the overall accuracy A Multichannel WAUTOC Method

Video example: distant-talking speech recognition

Video example: multi-channel pitch estimation

Forthcoming activities more effective combination of beamforming and ANC; test also ANC before D&S beamforming; test post-filtering after D&S; audio-video collection: an improved audio/video synchronization would be advisable; audio-video collection: select best balance beetween quality and frame rate acoustically characterize the target environment (prototype); integrate the selected features in the multi-channel front-end;