Presentation is loading. Please wait.

Presentation is loading. Please wait.

NATO UNCLASSIFIED CIS Division, NATO C3 Agency

Similar presentations


Presentation on theme: "NATO UNCLASSIFIED CIS Division, NATO C3 Agency"— Presentation transcript:

1 NATO UNCLASSIFIED CIS Division, NATO C3 Agency
The NATO Post-2000 Narrow Band Voice Coder: Test and Selection of STANAG 4591 Technical Presentation-001 CIS Division, NATO C3 Agency NATO UNCLASSIFIED NATO UNCLASSIFIED 6 April 2017

2 Abstract and Conditions of Release
NATO UNCLASSIFIED Abstract and Conditions of Release Abstract The work described in this presentation was carried out under customer funded projects and N , conducted by NC3A on behalf of AC322(SC/6-AHWG/3). This presentation gives a general introduction to the work, which is documented in NC3A Technical Note-881 and NC3A Technical Memorandum-946. This presentation is a working paper that may not be cited as representing formally approved NC3A opinions, conclusions or recommendations. Conditions of Release With reference to NATO Documents C-M(55)15(Final) and AC/322-D/1, this document is released to a NATO Government at the direction of the NATO Consultation, Command and Control (C3) Agency subject to the following conditions 1. The recipient NATO Government agrees to use its best endeavours to ensure that the information herein disclosed, whether or not it bears a security classification, is not dealt with in any manner (a) contrary to the intent of the provisions of the Charter of the NATO C3 Organization, or (b) prejudicial to the rights of the owner thereof to obtain patent, copyright or other likely statutory protection therefor. 2. If the technical information was originally released to the Agency by a NATO Government subject to restrictions clearly marked on this document the recipient NATO Government agrees to use its best endeavours to abide by the terms of the restrictions so imposed by the releasing Government. NATO UNCLASSIFIED 6 April 2017

3 Customers Host Nation Customer funded NBVC and NC3A
NATO UNCLASSIFIED Customers NATO Infrastructure Committee Voice coder developers NATO Narrow Band Voice Coder Ad-Hoc Working Group Host Nation Customer funded NC3A-NL, The Hague NC3A-BE, Brussels Scientific staff Acquisition staff Set up voice coding testbed Equipment Acquisition Process input data Contractual issues Blind and deblind data Support to AHWG NBVC, test labs and coder developers The NATO C3 Agency operates under a single management structure but is located in two locations, Brussels, Belgium and The Hague, Netherlands. The General Manager, Executive, Planning and Acquisition Staffs are based in Brussels whereas the major part of the Agency, which includes the Scientific Divisions and the scientific laboratories are in the Hague. The Agency has a wide ranging responsibility including C3 Planning, Operational Research, Air Command and Control, Communications and Information Systems, Acquisition and Procurement. It operates under the authority of the NATO C3 Board and under a regime of customer funding. This project was jointly “customer funded” by the NATO Infrastructure Committee and the three nations which submitted candidate voice coders - France, Turkey and the USA. NC3A Acquisition staff in Brussels supported contractual aspects of the project and acquisition of equipment. NC3A Scientific staff in The Hague established the voice processing testbed on which the speech processing was performed, processed the data and liased with test labs, coder developers and the AHWG NBVC. NC3A-NL were appointed technical project manager by AHWG NBVC. NATO UNCLASSIFIED 6 April 2017

4 Introduction to STANAG 4591
NATO UNCLASSIFIED Introduction to STANAG 4591 The voice coder selected by AHWG NBVC is being standardised within NATO as STANAG 4591. STANAG 4591 will provide high quality speech in harsh acoustic environments and can be used on the constrained channels often found in the military environment. This voice coder will enhance the quality of voice communications to NATO commanders in the field. Voice coders were submitted to AHWG NBVC by NATO member nations. They were then tested in a wide range of noise environments and conditions which were representative of military scenarios. The voice coder which provided the best combined performance was adopted as STANAG The selection process finished in October 2001. NATO UNCLASSIFIED 6 April 2017

5 Voice Coding technology is constantly improving
Background NATO UNCLASSIFIED Voice Coding technology is constantly improving driven by mobile telephony narrow band wireless channels new coders outperform existing NATO voice coders STANAG LPC10e + low rate (2.4k) - low speech quality - low resilience to noise STANAG CVSD + good resilience to noise - poor speech quality in no noise - high rate (16 k) AHWG NBVC tasked by NATO to select a future Narrow Band Voice Coder for NATO use at kbps and 2.4kbps The performance of voice coders currently in-service within NATO on narrow band channels no longer represents the state-of-the-art in voice coding. Voice coders currently in use such as LPC-10e and CVSD do not provide high quality speech, and their performance can be degraded substantially when operating the harsh acoustic environments in which NATO Commanders operate e.g. helicopters, tracked vehicles. Technological advances in algorithms now allow voice coders with higher speech quality and better performance in noisy conditions, improved speaker recognition, and superior non-native speaker intelligibility to be developed. Advances in signal processing hardware allows practical implementations of these algorithms in user terminals. The standardisation of such new voice coders is essential if the element of interoperability is to be added to the technical benefits available. AHWG NBVC (NATO AC322/SC-6/AHWG-3) was tasked to provide a voice coder capable of high quality speech in harsh acoustic environments at rates of 2.4kbps and 1.2 kbps. These data rates ensure it can operate on constrained channels e.g. HF radio, UHF satcom. NATO UNCLASSIFIED 6 April 2017

6 Voice Coders Tested NATO UNCLASSIFIED NATO requested candidates to be submitted by member nations Three candidates submitted France HSX (Harmonic Stochastic eXcitation) Turkey SB-PLC (Split-Band Linear Predictive Coding) USA MELP (Mixed Excitation Linear Prediction) (each candidate operates at both 1.2k & 2.4k) plus LPC-10e (2.4k) CELP (4.8k) CVSD (16k) as known reference coders NATO member nations were invited to submit candidate voice coders to the AHWG NBVC for test and evaluation. The test procedure and selection criteria were defined by the AHWG NBVC. The selection process tested voice coders in a wide range of representative noise environments and conditions. The voice coder which offered the best combined performance was adopted as STANAG 4591. Three candidate voice coders were submitted by NATO member nations. Each candidate operated at both kbps and 2.4 kbps. For comparison, the well-known coders CVSD (16 kbps), CELP (4.8 kbps) and LPC-10e (2.4 kbps) were also tested to provide a range of known reference points. NATO UNCLASSIFIED 6 April 2017

7 Test Resources and Responsibilities
NATO UNCLASSIFIED Project was ‘customer funded’ by NATO Infrastructure Committee and nations submitting coders NC3A host nation, but worked with specialist speech processing labs The TNO test laboratory at Soesterberg, NL NC3A ran raw audio data through coders and ‘blinded’ all output National test labs analysed raw audio from NC3A. Test labs were: TNO, NL CELAR, FR Arcon, US NC3A impartially collated results NATO data being analysed at TNO The selection process for STANAG 4591 took place in two phases. Phase one of the selection process assessed the intelligibility and the voice quality of the voice coders in a small number of noise environments. In addition to processing and ‘blinding’ the data, NC3A-NL also performed pre-processing and post- processing of data where required. NC3A-NL were appointed technical project manager by the customers. NATO UNCLASSIFIED 6 April 2017

8 NATO NBVC tests Phase 1 Floating Point vocoder implementations
NATO UNCLASSIFIED NATO NBVC tests Phase 1 Floating Point vocoder implementations Performance Intelligibility Quality Noise Conditions Quiet Modern office Acoustic noise, (6 dB, 12 dB) 5488 Mb of processed audio in 5848 files A typical test booth where subjects listen to speech for analysis Phase one used floating point implementations of all voice coders. Different tests - and different types of input data - are required for testing the two distinct criteria of speech quality and speech intelligibility. Intelligibility tests were performed by all three test labs; using Inteltrans (CELAR), Consonant-Vowel-Consonant (TNO) and Diagnostic Rhyme Test (Arcon) tests. Speech quality tests were performed by only two labs (both using the Mean Opinion Score test). All tests were performed in four noise environments. The Signal to Noise Ratio (SNR) indicates the relative levels of speech signal to background noise. The noise environments were: Quiet (no background noise) Modern office (SNR 20 dB) Acoustic (speech-shaped) noise (SNR 12 dB) Acoustic (speech-shaped) noise (SNR 6 dB) NATO UNCLASSIFIED 6 April 2017

9 Processing by NC3A 3 2 9 4 5 8 7 6 1 Encode Decode BITSTREAM Nine raw
NATO UNCLASSIFIED Encode Decode LPC10e CELP FR1200 FR2400 CVSD LPC10e TU1200 TU2400 US1200 US2400 BITSTREAM LPC10e CVSD CVSD 3 2 9 4 5 8 7 6 1 Nine raw audio output files Sent to test labs for analysis CELP CELP FR1200 FR1200 Raw audio file 8kHz sample rate, 16 bit samples FR2400 FR2400 TU1200 TU1200 TU2400 TU2400 US1200 US1200 Input speech material was provided to NC3A by the test labs. The material was processed through all coders, blinded and returned to the test labs for analysis. The different types of speech test employed required the use of different types of input material e.g. complete sentences, isolated words. Each input speech file provided by a test lab generated at least nine output speech files, one for each voice coder. All files used UNIX raw audio format (8 kHz sample rate, 16 bits per sample). A strict blinding process was used to mask the identities of all voice coders and guarantee impartiality during the analysis of individual voice coders, in the combination of scores, and in the final selection of the voice coder. US2400 US2400 NATO UNCLASSIFIED 6 April 2017

10 Double blinding process
NATO UNCLASSIFIED Decoded output files Single blinded files Double blinded files LPC10e LPC10e Coder1 Vocoder1 CVSD Coder2 Vocoder2 CELP Coder3 Vocoder3 B L I N D B L I N D Nine audio output files FR1200 Coder4 Vocoder4 To test lab FR2400 Coder5 Vocoder5 TU1200 Coder6 Vocoder6 TU2400 Coder7 Coder7 Vocoder7 US1200 Coder8 Vocoder8 The blinding process masked the identity of all voice coder outputs. This ensured impartiality during the analysis and scoring of the voice coder outputs by the test labs. Performing a double blind ensured that no one person (or organisation) knew the identity of output files sent for analysis. Removing the blind required both parties to co-operate. The blind was not removed until all results had been returned by all test labs, checked, and combined, ready for the final selection of the voice coder for STANAG 4591. US2400 Coder9 Vocoder9 Vocoder9 Blinded by NC3A Blinded by DSTL NATO UNCLASSIFIED 6 April 2017

11 Modulated Noise Reference Unit
NATO UNCLASSIFIED MNRU is a standard method to apply known levels of noise. It provides known references against which listeners can compare vocoder outputs BITSTREAM LPC10e LPC10e Nine raw audio output files CVSD CVSD CELP CELP FR1200 FR1200 FR2400 FR2400 TU1200 TU1200 TU2400 TU2400 US1200 US1200 US2400 US2400 MNRU 5db MNRU 15dB MNRU 20dB MNRU 25dB MNRU 10dB MNRU 5db MNRU 30dB MNRU 35dB MNRU 40dB 12 13 16 15 11 17 raw audio output files. MNRU files to test labs as references for analysing speech quality 10 17 14 MNRU 10dB One raw audio file MNRU 15dB MNRU 20dB MNRU 25dB For speech quality tests the audio input files were also processed through MNRU software. Industry standard software from ITU-T (International Telecommunications Union-Telecom Standardisation). This produces an additional eight output files from each input file. Listeners can then compare the quality of speech processed through the voice coders with the quality of the same speech to which known levels of noise have been added. The MNRU output files are not blinded. MNRU 30dB MNRU 35dB MNRU 40dB NATO UNCLASSIFIED 6 April 2017

12 NATO NBVC tests Phase II
NATO UNCLASSIFIED NATO NBVC tests Phase II Fixed point implementation C plus ETSI libraries Performance Measurements Intelligibility, Quality Speaker recognition Language dependency English, French, German, Dutch, Polish, Turkish 10 acoustic noise environments Transmission channel 1% BER Tandem 16 kbps CVSD - vocoder Whispered speech Phase two of the selection process extended the range of noise environments and the characteristics tested. Phase two also employed fixed-point implementations of the candidate and reference voice coders. Fixed-point software is much closer to that used for the final implementation of a voice coder. The libraries and source code used to create the floating-point implementations was rigorously controlled. This was to prevent coders using techniques which would be difficult to implement in practical (mass-produced) versions of a voice coder. Phase two tests the intelligibility and speech quality as in phase one. It also tests language dependency (Dutch, English, French, German), speaker recognition and delay of the voice coders. Three new conditions were tested in phase two which mirror the practical environments under which military voice coders operate. Tandem. Speech passes through two voice coders. Whispered speech. Errored channel. A random 1% bit error rate is applied to the encoded voice bitstream before it is decoded NATO UNCLASSIFIED 6 April 2017

13 Phase 2 additional test conditions
NATO UNCLASSIFIED 1% random bit errors Audio output file Coder n Bitstream Decoder n Audio input file Test configuration: 1% Bit error rate Audio B i t s B i t s Audio output file CVSD Coder CVSD Decoder Coder n Decoder n Audio input file To test the voice coders with a 1% bit error rate, the encoded bitstream was XORed with a static file containing randomly generated bit errors. The rate and distribution of errors in this file was analysed extensively by the AHWG NBVC prior to use. This errored file was then decoded. Because the different coder rates result in different lengths of bitstream file for a given duration of audio input, the error file generated was longer than the maximum bitstream file (generated by the CVSD vocoder). However, the overall error rate applied to the coders was always the same. The tandem condition emulates a common scenario in voice networks, where the speech must be converted from one voice coding standard to another. Such conversion inevitably leads to a degradation in voice quality. In the test, speech was coded and decoded with CVSD, then coded and decoded with the nine voice coders under test (including CVSD). Data for the whispered speech tests was processed in the same way as speech intelligibility tests, although clearly the input material differed. Test configuration: Voice coder tandem NATO UNCLASSIFIED 6 April 2017

14 NATO NBVC tests - Phase 2 Noise Conditions Phase 1 plus ……..
NATO UNCLASSIFIED NATO NBVC tests - Phase 2 Noise Conditions Phase 1 plus …….. Blackhawk helicopter HMMWV MCE field shelter Mirage 2000 Bradley Fighting Vehicle Le Clerc Tank Volvo (staff car) F-15 Additional noise environments used in phase two included the MCE field shelter, staff car, wheeled military vehicle (HMMMV and P4), helicopter (UH60 Black Hawk), tracked vehicle (M2A2 Bradley and LeClerc) and aircraft (F-15 and Mirage 2000). The P4, LeClerc and Mirage noise environments were used by the French test lab. The HMMWV, Bradley and F-15 by the US and Dutch test labs. NATO UNCLASSIFIED 6 April 2017

15 Over 30 GB of processed speech data @ 500 hours of speech
NATO NBVC Phase 2 NATO UNCLASSIFIED 3 test labs x coders (+ 8 MNRU levels) x £ 5 tests x £ 12 noise conditions x £ 88 files per test Over 36,000 files Over 30 GB of processed speech data @ 500 hours of speech Some voice coders ran approx 10 times real time Not all types of test were conducted in all noise environments. Even so, the combination of 3 test labs, performing up to 5 tests, in up to 12 noise environments, with 9 voice coders (plus eight MNRU levels in some cases) led to a large quantity of processed data to be analysed. The time taken to process this data was considerable, with some voice coders operating considerably slower than ‘real time’. Phase two represented an extensive and substantial test of voice coder performance. The reliability and dependability of these results benefits greatly from the diversity of test methods, test laboratories and languages used. In order to combine the results from diverse test labs, diverse tests and multiple languages, a precision weighted ranking was developed. This allowed all results to be combined accurately and fairly. Different tests produce scores in different ranges e.g. MOS results range from 0 - 5, CVC results are a percentage. The precision weighted ranking scaled all results to the same level. The accuracy of the results also plays a part in this scaling e.g. test results with a higher precision are weighted higher. NATO UNCLASSIFIED 6 April 2017

16 Need for Precision Weighted Ranking
NATO UNCLASSIFIED Graphs show variation between intelligibility tests performed by the 3 test labs General trends are the same Absolute scores vary Need to combine all results accurately and fairly Simple scaling is not sufficient US24 CELP FR24 CVSD TU24 US12 LPC TU12 FR12 Many speech tests rely on human listeners making subjective assessments of intelligibility and quality. Such subjectivity can be negated by using the same group of listeners for all tests, and measurements will be valid when comparing one voice coder with others within the same test group. However, comparisons between different groups of tests cannot be made with any accuracy. The graphs above show the variation in scores between the different test laboratories. While the relative performance of all voice coders remains broadly similar between all test labs and in all noise conditions, the absolute scores vary widely between test labs and between noise conditions. It should be noted that all test labs used different tests for measuring speech intelligibility. By combining results from a number of well-established intelligibility tests, the possible effect of any spurious behaviour by a given coder in a given test is reduced. N.B. Speech testing methods which rely solely on automated techniques, such as Perceptual Speech Quality Measurement (PSQM) and Perceptual Evaluation of Speech Quality (PESQ), do not involve humans in the test mechanism. Such tests are not subjective, but they are not able to accurately reflect the complete response of a human listener. This is particularly true for low bit rate coders where speech quality is lower, and the cognitive power of a human listener makes a considerable impact on the perception of the speech. US24 CELP FR24 CVSD TU24 US12 LPC TU12 FR12 NATO UNCLASSIFIED 6 April 2017

17 Precision Weighted Ranking
NATO UNCLASSIFIED Range of test results divided into segments or bins Confidence interval of test The resolution (or 95% confidence interval) of the test determines bin size Bin 1 Bin 3 Bin 4 Bin 5 Bin 7 Score = 7 Coder scores are determined by which bin their test result falls into Score = 1 Worst coder always scores 1. In this test Vocoder 7 came last The precision weighted ranking allocates scores depending on the performance of each voice coder compared to the worst coder in the test. The scores are also a function of the accuracy of the test method, given by the 95% confidence interval. A number of bins are created which span the range of results for that particular test. The bins start at the lowest result. The width of the bins is given by the confidence interval of the test and the best straight line fit between all points. In this case the best least squares straight line fit is given by y = x Coders receive a score for each test according to which bin their result falls into. In this test voice coder 7 came last, its result lies in ‘bin 1’ and therefore it scores 1 in this test. (LPC-10e was the worst coder in all but one test). Results for voice coders 6, 8 and 9 were substantially above that of voice coder 7. Their results were all between 4 and 5 confidence intervals higher than vocoder 7. Therefore their results fall in ‘bin 5’ and they score 5. Scores for vocoders 1 and 3 were confidence intervals above that of vocoder 7. They lie in ‘bin 6’ and score 6. Scores for vocoders 2, 4 and 5 were confidence intervals above that of vocoder 7. They all score 7, the highest of this test. With precision weighted ranking, coders with similar performance receive the same mark, while large differences between coders are also reflected in the marks. Coders in subsequent intervals score bin number Scores for vocoders 6, 8 and 9 were confidence intervals above that of V7. They all score 5 NATO UNCLASSIFIED 6 April 2017

18 Combined Performance Index
NATO UNCLASSIFIED The scores generated by the precision weighted ranking were then fed into the matrix above in order to generate a combined performance index for each coder. The combined performance index was made up of all the test results, each weighted according to its relative importance. The speech intelligibility results contributed 42% of the combined performance index. Within this, the quiet (baseline) noise condition accounted for 27% of the intelligibility score, with 11% contributed by each of the acoustic noise environments e.g. office, F-15 etc. Speech quality accounted for 34% of the combined performance index. The speech quality performance in a quiet environment made the biggest contribution to this, with the performance in different acoustic noise environments contributing varying amounts, depending on the importance (as determined by AHWG NBVC). Speech quality tests with an error channel accounted for 1.8%; and whispered speech 2.2%. Both the precision weighted ranking and the combined performance index mechanisms had been tested with the results of phase one. NATO UNCLASSIFIED 6 April 2017

19 Phase 2 Combined Performance Index
NATO UNCLASSIFIED US FR TU The generation of precision weighted rankings and combined performance indices for each coder was conducted using double blind results. Once the performance index score for each of the nine coders was generated, one blind was removed (by DSTL). NC3A then identified the reference coders and the pairs of candidate coders. Scores for each pair were then combined according to the previously agreed weights. The selection criteria combined the scores for each candidate at 1.2 kbps and 2.4 kbps in the ratio 40:60. This gave a single metric for each candidate coder in the wide variety of tests and noise environments. Only when the best performing candidate coder pair (at 1.2 kbps and 2.4 kbps) was agreed was the final blind removed to reveal the winner. The winning coder was the US candidate, MELPe (Mixed Excitation Linear Prediction-enhanced). Selection made on combined scores at 2400 and 1200 bps 60% bps score 40% bps score NATO UNCLASSIFIED 6 April 2017

20 Phase 2 Combined Performance Index
NATO UNCLASSIFIED US2400 CELP FR2400 CVSD TU2400 US1200 FR1200 TU1200 LPC10 This graph shows the combined performance index for all coders tested. The minimum possible score using the precision weighted ranking and combined performance index is 1. It should be noted that the performance index is non-linear. The inclusion of CELP, CVSD and LPC-10e provide reference points when evaluating the scores. All candidates (even at 1.2 kbps) offer large improvements over LPC-10e, the previous 2.4 kbps NATO voice coder. All candidates (at 2.4 kbps) offer performance comparable, or superior, to CVSD, the 16 kbps NATO voice coder. The selected voice coder not only outperforms LPC-10e and CVSD, but also CELP, a modern 4.8 kbps voice coder. NATO UNCLASSIFIED 6 April 2017

21 Specific Results - Intelligibility
NATO UNCLASSIFIED Results of all coders in all noise conditions (CVC test) US24 CELP FR24 CVSD TU24 US12 LPC TU12 FR12 Intelligibility score (%) Intelligibility score (%) The graphs above show results of a specific test in more detail. These are the results from the TNO Intelligibility (CVC) tests. As the results are from a single test lab they can be compared (and are shown) without application of the Precision Weighted Ranking or Combined Performance Index. General trends in comparative performance of the voice coders can be easily seen, across all coders and across all noise conditions. The results have been plotted on two separate graphs for clarity. Points on the coloured lines show the performance of the coders in a given noise condition. The coloured lines connecting the points serve no purpose other than to group points in the same noise condition. E.g. LPC-10 is consistently the worst performer of all voice coders in these tests. The effect of different noise environments on a given coder can be seen by comparing different points within a given column. As expected, there is a clear degradation in intelligibility as the acoustic noise environment becomes harsher. The black range bars extending above and below each plotted point show the 95% confidence interval for that particular test result. US24 CELP FR24 CVSD TU24 US12 LPC TU12 FR12 NATO UNCLASSIFIED 6 April 2017

22 Specific Results - Speech Quality
NATO UNCLASSIFIED Range of Mean Opinion Score test 1 (Bad) 2 (Poor) 3 (Fair) 4 (Good) 5 (Excellent) Results of all coders in all noise conditions (MOS test) Mean Opinion Score US24 CELP FR24 CVSD TU24 US12 LPC TU12 FR12 Mean Opinion Score The graphs above show the results of the Arcon Speech Quality (MOS) tests. Again, these results are from a single test lab without application of the Precision Weighted Ranking or Combined Performance Index. General trends in comparative performance of the voice coders can be easily seen, across all coders and across all noise conditions. Again, the results have been plotted on two separate graphs for clarity. The results are displayed in the same format as those in the previous graphs, with coloured lines linking the points showing all coders in a given noise condition. Bars showing the 95% confidence interval of each test result are also shown. As noted in an earlier slide, these results show that while the speech quality of CVSD is not particularly high in quiet conditions, in many acoustic noise environments CVSD is affected very little. US24 CELP FR24 CVSD TU24 US12 LPC TU12 FR12 NATO UNCLASSIFIED 6 April 2017

23 Specific Results - Language Dependency
NATO UNCLASSIFIED Language dependency of all tested coders The closer a point lies to the x=y diagonal, the less language dependant the voice coder The selected coder exhibited a strong language independence (for those languages tested). The level of language dependence of a particular voice coder will vary from language to language. In addition to testing in the official languages of NATO (English and French) Dutch and German were also included in the test to give a broader base for the results. Language dependence tests of the coders have also been conducted for Polish and Turkish, although these results were used in the selection of Stanag 4591. SRT - Speech Reception Threshold. This is a measure of speech quality. NATO UNCLASSIFIED 6 April 2017

24 Current position Stanag 4591 known MELPe Phase 1 Phase 2 Completed
NATO UNCLASSIFIED Phase 1 Completed Results available in NC3A Technical Note-881 Phase 2 All material processed and analysed Results collated Results analysed and blind removed Coder selected on 24 October 2001 Stanag 4591 known MELPe The selection process for STANAG 4591 has been completed. However, the test process has not finished. The test procedure always included a third phase of work which would follow once the voice coder was selected at the end of phase 2. NATO UNCLASSIFIED 6 April 2017

25 NC3A - Current activity Test Process Phase 3
NATO UNCLASSIFIED Test Process Phase 3 Real-time Implementation of Phase 2 winner Communicability tests real-life communication problem end-to-end delay effects Assist in drafting STANAG 4591 Advise on the use and implementation of STANAG 4591 Phase 3 involves communicability tests of the selected coder. This requires the development of real time implementations of the selected coder (at 1.2 kbps and 2.4 kbps) which will be used for real time tests. In communicability tests, speech quality and intelligibility play a part, but the result is also affected by end-to-end delays and other aspects relating to the practical implementation of the coder. NATO UNCLASSIFIED 6 April 2017

26 Stanag 4591 vs COTS voice coders
NATO UNCLASSIFIED COTS X = kbps COTS Y = 4.56 kbps COTS X = 4.56 kbps MELPe = 2.4 kbps The performance of 2.4 kbps MELPe is comparable to that of voice coders in current Commercial Off The Shelf (COTS) mobile radio systems. The graph above shows MELPe compared to three COTS systems. All the COTS voice coders use a higher bit rate, but performance is comparable or worse than 2.4 kbps MELPe. NC3A is working with civil communications standards bodies e.g. ETSI, to influence civil standards such that future COTS equipment will better meet military requirements. STANAG 4591 is an area where military communications standards may offer something to civil communications standardisation. Particularly as new civil standards aim to use “new voice coders utilising the latest low bit-rate technology” [Terms of reference for TETRA release 2, ETSI]. NC3A participates in civil standards working groups in order to influence future civil standards. e.g. consideration of Stanag 4591 for inclusion in future civil standards; modification of TETRA Security and Fraud Prevention Group’s Recommendation 02 on End to end encryption to allow the encryption framework to support the Stanag 4591 voice coder. Male speaker Female speaker NATO UNCLASSIFIED 6 April 2017

27 Conclusion STANAG 4591 provides substantially improved performance
NATO UNCLASSIFIED STANAG 4591 provides substantially improved performance speech quality intelligibility noise immunity reduced throughput requirements interoperability The introduction of a new NATO narrow band voice coder (STANAG 4591) will improve communications within NATO and support end-to-end security and interoperability. Modern technology allows such voice coders to provide improved speech performance in a variety of harsh acoustic noise environments. The test and selection process ensured that the selected voice coder provides the above performance improvements. The rigorous and comprehensive nature of the testing plan ensures that these benefits will be available in the difficult military environments in which NATO forces must communicate. The NATO standardisation of the state-of-the-art voice coder STANAG 4591 ensures that NATO and Partnership for Peace member nations will benefit from improved voice communications, with enhanced performance and increased interoperability. NATO UNCLASSIFIED 6 April 2017

28 Further information Stanag 4591 test and selection process
NATO UNCLASSIFIED Stanag 4591 test and selection process Street MD, “Future NATO narrow band voice coder selection: Stanag 4591”, NC3A Technical Note 881, The Hague, December 2001 Street MD and Collura JS, “Interoperable Voice Communications: test and selection of STANAG 4591”, RTA IST Symposium - NATO Research and Technology Agency (Information Systems and Technology panel) Tactical Military Communications symposium, Warsaw, October 2001 Street MD and Collura JS, “The test and selection of the future NATO narrow band voice coder”, RCMCIS - NATO Regional Conference on Military CIS, Warsaw, Zegrze, October 2001. MELPe: the selected voice coder Collura JS and Rahikka DJ, “Interoperable secure voice communications in tactical systems, IEE coll. on Speech coding algorithms for radio channels, London, February 2000. An overview of the MELP voice coder and its use in military environments Collura JS, Rahikka DJ, Fuja TE, Sridhara D and Fazel T, “Error coding strategies for MELP vocoder in wireless and ATM environments”, IEE coll. on Speech coding algorithms for radio channels, London, February 2000. Performance of MELP with a variety of different error correction mechanisms NATO UNCLASSIFIED 6 April 2017

29 Information and Source Code available from:
NATO UNCLASSIFIED Information and Source Code available from: Applied Communication Technologies Branch CIS Division NATO C3 Agency PO Box CD , The Hague The Netherlands Tel: Fax: Further work to support this new standard is continuing, with NC3A assisting potential users within NATO and NATO member nations, manufacturers and others. Stanag 4591 was released for NATO under terms which make it available for government use by NATO and PfP nations free of Intellectual Property Rights. NATO UNCLASSIFIED 6 April 2017


Download ppt "NATO UNCLASSIFIED CIS Division, NATO C3 Agency"

Similar presentations


Ads by Google