Presentation is loading. Please wait.

Presentation is loading. Please wait.

NATO UNCLASSIFIED 1 The NATO Post-2000 Narrow Band Voice Coder: Test and Selection of STANAG 4591 CIS Division, NATO C3 Agency Technical.

Similar presentations


Presentation on theme: "NATO UNCLASSIFIED 1 The NATO Post-2000 Narrow Band Voice Coder: Test and Selection of STANAG 4591 CIS Division, NATO C3 Agency Technical."— Presentation transcript:

1 NATO UNCLASSIFIED 1 The NATO Post-2000 Narrow Band Voice Coder: Test and Selection of STANAG 4591 CIS Division, NATO C3 Agency Technical Presentation-001

2 NATO UNCLASSIFIED 2 Abstract and Conditions of Release Abstract The work described in this presentation was carried out under customer funded projects and N , conducted by NC3A on behalf of AC322(SC/6-AHWG/3). This presentation gives a general introduction to the work, which is documented in NC3A Technical Note-881 and NC3A Technical Memorandum-946. This presentation is a working paper that may not be cited as representing formally approved NC3A opinions, conclusions or recommendations.Abstract The work described in this presentation was carried out under customer funded projects and N , conducted by NC3A on behalf of AC322(SC/6-AHWG/3). This presentation gives a general introduction to the work, which is documented in NC3A Technical Note-881 and NC3A Technical Memorandum-946. This presentation is a working paper that may not be cited as representing formally approved NC3A opinions, conclusions or recommendations.

3 NATO UNCLASSIFIED 3 NC3A-NL, The HagueNC3A-BE, Brussels Scientific staffAcquisition staff Set up voice coding testbed Equipment Acquisition Process input data Contractual issues Blind and deblind data Support to AHWG NBVC, test labs and coder developers NATO Infrastructure Committee NATO Narrow Band Voice Coder Ad-Hoc Working Group NBVC and NC3A Customers Customer funded Voice coder developers Host Nation

4 NATO UNCLASSIFIED 4 Introduction to STANAG 4591

5 NATO UNCLASSIFIED 5 BackgroundBackground Voice Coding technology is constantly improvingVoice Coding technology is constantly improving driven by mobile telephonydriven by mobile telephony –narrow band –wireless channels new coders outperform existing NATO voice codersnew coders outperform existing NATO voice coders Voice Coding technology is constantly improvingVoice Coding technology is constantly improving driven by mobile telephonydriven by mobile telephony –narrow band –wireless channels new coders outperform existing NATO voice codersnew coders outperform existing NATO voice coders STANAG CVSD + good resilience to noise - poor speech quality in no noise - high rate (16 k) STANAG LPC10e + low rate (2.4k) - low speech quality - low resilience to noise AHWG NBVC tasked by NATO to select a future Narrow Band Voice Coder for NATO use at 1.2kbps and 2.4kbpsAHWG NBVC tasked by NATO to select a future Narrow Band Voice Coder for NATO use at 1.2kbps and 2.4kbps

6 NATO UNCLASSIFIED 6 Voice Coders Tested NATO requested candidates to be submitted by member nationsNATO requested candidates to be submitted by member nations Three candidates submittedThree candidates submitted NATO requested candidates to be submitted by member nationsNATO requested candidates to be submitted by member nations Three candidates submittedThree candidates submitted FranceFrance HSX (Harmonic Stochastic eXcitation)HSX (Harmonic Stochastic eXcitation) FranceFrance HSX (Harmonic Stochastic eXcitation)HSX (Harmonic Stochastic eXcitation) TurkeyTurkey SB-PLC (Split-Band Linear Predictive Coding)SB-PLC (Split-Band Linear Predictive Coding) TurkeyTurkey SB-PLC (Split-Band Linear Predictive Coding)SB-PLC (Split-Band Linear Predictive Coding) USAUSA MELP (Mixed Excitation Linear Prediction)MELP (Mixed Excitation Linear Prediction) USAUSA MELP (Mixed Excitation Linear Prediction)MELP (Mixed Excitation Linear Prediction) (each candidate operates at both 1.2k & 2.4k) (each candidate operates at both 1.2k & 2.4k) plus LPC-10e (2.4k) CELP (4.8k) CVSD (16k) as known reference codersplus LPC-10e (2.4k) CELP (4.8k) CVSD (16k) as known reference coders (each candidate operates at both 1.2k & 2.4k) (each candidate operates at both 1.2k & 2.4k) plus LPC-10e (2.4k) CELP (4.8k) CVSD (16k) as known reference codersplus LPC-10e (2.4k) CELP (4.8k) CVSD (16k) as known reference coders

7 NATO UNCLASSIFIED 7 Test Resources and Responsibilities NC3A ran raw audio data through coders and ‘blinded’ all outputNC3A ran raw audio data through coders and ‘blinded’ all output National test labs analysed raw audio from NC3A. Test labs were:National test labs analysed raw audio from NC3A. Test labs were: TNO, NLTNO, NL CELAR, FRCELAR, FR Arcon, USArcon, US NC3A impartially collated resultsNC3A impartially collated results NC3A ran raw audio data through coders and ‘blinded’ all outputNC3A ran raw audio data through coders and ‘blinded’ all output National test labs analysed raw audio from NC3A. Test labs were:National test labs analysed raw audio from NC3A. Test labs were: TNO, NLTNO, NL CELAR, FRCELAR, FR Arcon, USArcon, US NC3A impartially collated resultsNC3A impartially collated results Project was ‘customer funded’ by NATO Infrastructure Committee and nations submitting codersProject was ‘customer funded’ by NATO Infrastructure Committee and nations submitting coders NC3A host nation, but worked with specialist speech processing labsNC3A host nation, but worked with specialist speech processing labs Project was ‘customer funded’ by NATO Infrastructure Committee and nations submitting codersProject was ‘customer funded’ by NATO Infrastructure Committee and nations submitting coders NC3A host nation, but worked with specialist speech processing labsNC3A host nation, but worked with specialist speech processing labs The TNO test laboratory at Soesterberg, NL NATO data being analysed at TNO

8 NATO UNCLASSIFIED 8 NATO NBVC tests Phase 1 Floating Point vocoder implementationsFloating Point vocoder implementations PerformancePerformance IntelligibilityIntelligibility QualityQuality Noise ConditionsNoise Conditions QuietQuiet Modern officeModern office Acoustic noise, (6 dB, 12 dB)Acoustic noise, (6 dB, 12 dB) 5488 Mb of processed audio in 5848 files5488 Mb of processed audio in 5848 files Floating Point vocoder implementationsFloating Point vocoder implementations PerformancePerformance IntelligibilityIntelligibility QualityQuality Noise ConditionsNoise Conditions QuietQuiet Modern officeModern office Acoustic noise, (6 dB, 12 dB)Acoustic noise, (6 dB, 12 dB) 5488 Mb of processed audio in 5848 files5488 Mb of processed audio in 5848 files A typical test booth where subjects listen to speech for analysis

9 NATO UNCLASSIFIED 9 Processing by NC3A Nine raw audio output files Sent to test labs for analysis EncodeDecode CELP FR1200 FR2400 CVSD LPC10e TU1200 TU2400 US1200 US2400 CELP FR1200 FR2400 CVSD LPC10e TU1200 TU2400 US1200 US2400 BITSTREAMBITSTREAM CELP FR1200 FR2400 CVSD LPC10e TU1200 TU2400 US1200 US2400 CELP FR1200 FR2400 CVSD LPC10e TU1200 TU2400 US1200 US Raw audio file 8kHz sample rate, 16 bit samples

10 NATO UNCLASSIFIED 10 Double blinding process CELP FR1200 FR2400 CVSD LPC10e TU1200 TU2400 US1200 US2400 To test lab Nine audio output files Vocoder3 Vocoder4 Vocoder5 Vocoder2 Vocoder1 Vocoder6 Vocoder7 Vocoder8 Vocoder9 Decoded output files Double blinded files BLINDBLIND Coder3 Coder4 Coder5 Coder2 Coder1 Coder6 Coder7 Coder8 Coder9 Single blinded files BLINDBLIND Blinded by NC3A Blinded by DSTL LPC10e Coder7 Vocoder9

11 NATO UNCLASSIFIED Modulated Noise Reference Unit CELP FR1200 FR2400 CVSD LPC10e TU1200 TU2400 US1200 US2400 One raw audio file Nine raw audio output files CELP FR1200 FR2400 CVSD LPC10e TU1200 TU2400 US1200 US2400 BITSTREAMBITSTREAM MNRU 15dB MNRU 20dB MNRU 25dB MNRU 10dB MNRU 5db MNRU 30dB MNRU 35dB MNRU 40dB MNRU is a standard method to apply known levels of noise. It provides known references against which listeners can compare vocoder outputs MNRU 15dB MNRU 20dB MNRU 25dB MNRU 10dB MNRU 5db MNRU 30dB MNRU 35dB MNRU 40dB raw audio output files. MNRU files to test labs as references for analysing speech quality

12 NATO UNCLASSIFIED 12 NATO NBVC tests Phase II Fixed point implementationFixed point implementation C plus ETSI librariesC plus ETSI libraries Performance MeasurementsPerformance Measurements Intelligibility, QualityIntelligibility, Quality Speaker recognitionSpeaker recognition Language dependencyLanguage dependency –English, French, German, Dutch, Polish, Turkish Fixed point implementationFixed point implementation C plus ETSI librariesC plus ETSI libraries Performance MeasurementsPerformance Measurements Intelligibility, QualityIntelligibility, Quality Speaker recognitionSpeaker recognition Language dependencyLanguage dependency –English, French, German, Dutch, Polish, Turkish 10 acoustic noise environments10 acoustic noise environments Transmission channelTransmission channel 1% BER1% BER TandemTandem 16 kbps CVSD - vocoder16 kbps CVSD - vocoder Whispered speechWhispered speech

13 NATO UNCLASSIFIED 13 Phase 2 additional test conditions Coder n Audio input file Audio output file Decoder n Bitstream 1% random bit errors Audio input file CVSD Coder Audio output file Decoder nCoder n CVSD Decoder Test configuration: 1% Bit error rate Test configuration: Voice coder tandem B itsB its B itsB its Audio Audio

14 NATO UNCLASSIFIED 14 NATO NBVC tests - Phase 2 NATO NBVC tests - Phase 2 Noise Conditions Phase 1 plus …….. HMMWV Bradley Fighting Vehicle Le Clerc Tank Volvo (staff car) MCE field shelter Blackhawk helicopter Mirage 2000 F-15

15 NATO UNCLASSIFIED 15 NATO NBVC Phase 2 Over 36,000 filesOver 36,000 files Over 30 GB of processed speech dataOver 30 GB of processed speech data   500 hours of speech   500 hours of speech –Some voice coders ran approx 10 times real time Over 36,000 filesOver 36,000 files Over 30 GB of processed speech dataOver 30 GB of processed speech data   500 hours of speech   500 hours of speech –Some voice coders ran approx 10 times real time 3 test labs x   5 tests x  12 noise conditions x 9 coders (+ 8 MNRU levels) x   88 files per test

16 NATO UNCLASSIFIED 16 Need for Precision Weighted Ranking Graphs show variation between intelligibility tests performed by the 3 test labsGraphs show variation between intelligibility tests performed by the 3 test labs General trends are the sameGeneral trends are the same Absolute scores varyAbsolute scores vary Need to combine all results accurately and fairlyNeed to combine all results accurately and fairly Simple scaling is not sufficientSimple scaling is not sufficient Graphs show variation between intelligibility tests performed by the 3 test labsGraphs show variation between intelligibility tests performed by the 3 test labs General trends are the sameGeneral trends are the same Absolute scores varyAbsolute scores vary Need to combine all results accurately and fairlyNeed to combine all results accurately and fairly Simple scaling is not sufficientSimple scaling is not sufficient US24 CELP FR24 CVSD TU24 US12 LPC TU12 FR12

17 NATO UNCLASSIFIED 17 Range of test results divided into segments or binsRange of test results divided into segments or bins Precision Weighted Ranking Coders in subsequent intervals score bin numberCoders in subsequent intervals score bin number Bin 1Bin 3Bin 4 Bin 5Bin 7 Coder scores are determined by which bin their test result falls intoCoder scores are determined by which bin their test result falls into Confidence interval of test Confidence interval of test The resolution (or 95% confidence interval) of the test determines bin sizeThe resolution (or 95% confidence interval) of the test determines bin size Worst coder always scores 1. In this test Vocoder 7 came lastWorst coder always scores 1. In this test Vocoder 7 came last Scores for vocoders 6, 8 and 9 were confidence intervals above that of V7. They all score 5Scores for vocoders 6, 8 and 9 were confidence intervals above that of V7. They all score 5 Score = 1 Score = 7

18 NATO UNCLASSIFIED 18 Combined Performance Index

19 NATO UNCLASSIFIED 19 Phase 2 Combined Performance Index Selection made on combined scores at 2400 and 1200 bps 60% bps score 40% bps score

20 NATO UNCLASSIFIED 20 Phase 2 Combined Performance Index

21 NATO UNCLASSIFIED 21 Specific Results - Intelligibility Results of all coders in all noise conditions (CVC test)Results of all coders in all noise conditions (CVC test) US24 CELP FR24 CVSD TU24 US12 LPC TU12 FR12 Intelligibility score (%) Intelligibility score (%)

22 NATO UNCLASSIFIED 22 Specific Results - Speech Quality Range of Mean Opinion Score testRange of Mean Opinion Score test 1 (Bad)1 (Bad) 2 (Poor)2 (Poor) 3 (Fair)3 (Fair) 4 (Good)4 (Good) 5 (Excellent)5 (Excellent) Results of all coders in all noise conditions (MOS test)Results of all coders in all noise conditions (MOS test) Range of Mean Opinion Score testRange of Mean Opinion Score test 1 (Bad)1 (Bad) 2 (Poor)2 (Poor) 3 (Fair)3 (Fair) 4 (Good)4 (Good) 5 (Excellent)5 (Excellent) Results of all coders in all noise conditions (MOS test)Results of all coders in all noise conditions (MOS test) US24 CELP FR24 CVSD TU24 US12 LPC TU12 FR12 Mean Opinion Score

23 NATO UNCLASSIFIED 23 Specific Results - Language Dependency Language dependency of all tested codersLanguage dependency of all tested coders The closer a point lies to the x=y diagonal, the less language dependant the voice coderThe closer a point lies to the x=y diagonal, the less language dependant the voice coder Language dependency of all tested codersLanguage dependency of all tested coders The closer a point lies to the x=y diagonal, the less language dependant the voice coderThe closer a point lies to the x=y diagonal, the less language dependant the voice coder

24 NATO UNCLASSIFIED 24 Current position Phase 1Phase 1 CompletedCompleted Results available in NC3A Technical Note-881Results available in NC3A Technical Note-881 Phase 2Phase 2 All material processed and analysedAll material processed and analysed Results collatedResults collated Results analysed and blind removedResults analysed and blind removed Coder selected on 24 October 2001Coder selected on 24 October 2001 Stanag 4591 knownStanag 4591 known MELPeMELPe Phase 1Phase 1 CompletedCompleted Results available in NC3A Technical Note-881Results available in NC3A Technical Note-881 Phase 2Phase 2 All material processed and analysedAll material processed and analysed Results collatedResults collated Results analysed and blind removedResults analysed and blind removed Coder selected on 24 October 2001Coder selected on 24 October 2001 Stanag 4591 knownStanag 4591 known MELPeMELPe

25 NATO UNCLASSIFIED 25 NC3A - Current activity Test Process Phase 3Test Process Phase 3 Real-time Implementation of Phase 2 winnerReal-time Implementation of Phase 2 winner Communicability testsCommunicability tests –real-life communication problem –end-to-end delay effects Assist in drafting STANAG 4591Assist in drafting STANAG 4591 Advise on the use and implementation of STANAG 4591Advise on the use and implementation of STANAG 4591 Test Process Phase 3Test Process Phase 3 Real-time Implementation of Phase 2 winnerReal-time Implementation of Phase 2 winner Communicability testsCommunicability tests –real-life communication problem –end-to-end delay effects Assist in drafting STANAG 4591Assist in drafting STANAG 4591 Advise on the use and implementation of STANAG 4591Advise on the use and implementation of STANAG 4591

26 NATO UNCLASSIFIED 26 Stanag 4591 vs COTS voice coders Male speaker Female speaker COTS X = 6 kbps COTS Y = 4.56 kbps COTS X = 4.56 kbps MELPe = 2.4 kbps

27 NATO UNCLASSIFIED 27 ConclusionConclusion STANAG 4591 providesSTANAG 4591 provides substantially improved performancesubstantially improved performance – speech quality – intelligibility – noise immunity reduced throughput requirementsreduced throughput requirements interoperabilityinteroperability STANAG 4591 providesSTANAG 4591 provides substantially improved performancesubstantially improved performance – speech quality – intelligibility – noise immunity reduced throughput requirementsreduced throughput requirements interoperabilityinteroperability

28 NATO UNCLASSIFIED 28 Further information Stanag 4591 test and selection process Street MD, “Future NATO narrow band voice coder selection: Stanag 4591”, NC3A Technical Note 881, The Hague, December Street MD and Collura JS, “Interoperable Voice Communications: test and selection of STANAG 4591”, RTA IST Symposium - NATO Research and Technology Agency (Information Systems and Technology panel) Tactical Military Communications symposium, Warsaw, October Street MD and Collura JS, “The test and selection of the future NATO narrow band voice coder”, RCMCIS - NATO Regional Conference on Military CIS, Warsaw, Zegrze, October MELPe: the selected voice coder Collura JS and Rahikka DJ, “Interoperable secure voice communications in tactical systems, IEE coll. on Speech coding algorithms for radio channels, London, February An overview of the MELP voice coder and its use in military environments Collura JS, Rahikka DJ, Fuja TE, Sridhara D and Fazel T, “Error coding strategies for MELP vocoder in wireless and ATM environments”, IEE coll. on Speech coding algorithms for radio channels, London, February Performance of MELP with a variety of different error correction mechanisms

29 NATO UNCLASSIFIED 29 Information and Source Code available from: Applied Communication Technologies Branch CIS Division NATO C3 Agency PO Box CD, The Hague The Netherlands Tel: Fax:


Download ppt "NATO UNCLASSIFIED 1 The NATO Post-2000 Narrow Band Voice Coder: Test and Selection of STANAG 4591 CIS Division, NATO C3 Agency Technical."

Similar presentations


Ads by Google