Presentation is loading. Please wait.

Presentation is loading. Please wait.

Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez.

Similar presentations


Presentation on theme: "Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez."— Presentation transcript:

1 Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez (Respeaking consultant) ITU-T Workshop onTelecommunications relay services for persons with disabilities (Geneva, 25 November 2011)

2 Accuracy in Respeaking Quality in respeaking Delay Accuracy

3 Accuracy in Respeaking 97-98% accuracy

4 Basic requirements for a model 1) Functional and easy to apply 2) Include the basic principles of WER calculations in SR 3) Different programmes, different editing 4) Possibility of edited and yet accurate respeaking 5) Compare subtitles with original spoken text 6) Include other relevant info (delay, position, speed) 7) Provide both percentage and food for thought in training

5 Traditional WER methods US National Institute of Standards and Technology N - Errors Accuracy Rate ------------------------ × 100 = % N But... Well, you know, you have to try and put out a good performance, I mean, yeah, its kind of a stepping stone, isnt it, really? You have to try to put out a good performance. Its a stepping stone.

6 Traditional WER methods US National Institute of Standards and Technology N - Errors Accuracy Rate ------------------------ × 100 = 16% N But... Well, you know, you have to try and put out a good performance, I mean, yeah, its kind of a stepping stone, isnt it, really? You have to try to put out a good performance. Its a stepping stone.

7 Spain = SDH guidelines Different European countries UN Accessibility Focus Group N – E – R Accuracy ------------------------ × 100 = % N Correct editions: Serious errors: Assessment:

8 NER Model 205 – 3 – 2 Accuracy ------------------------ × 100 = 98.6% 205

9 NER Model 226 – 13 – 1 Accuracy ------------------------ × 100 = 93.8% 226 Assessment: poor editing (not quantity, but quality)

10 NER Model 257 – 1 – 13 Accuracy ------------------------ × 100 = 94.3% 257 Assessment: poor recognition (including serious mistakes)

11 WGBH: There is a wide range of error types in real time captioning and they are not all equal in their impact to caption viewers. Treating all errors the same does not provide a true picture of caption accuracy.

12 Types of errors (feedback from DTV4ALL project) 1) There are errors, yes, but you can easily figure out what the correct form was meant to be. Now Im bilingual –I can speak English and teletext 2)Live subtitles? - Sound like gobbledygook to me 3)As far as Im concerned they are not errors, but lies

13 Types of errors (feedback from DTV4ALL project) 1) Minor edition or recognition errors (0.25) 2)Normal edition or recognition errors (0.5) 3)Serious errors(1)

14 Minor Errors What a great goal by a Ryan Giggs! Simon brown has been appointed new chairman of Rolls Royce. For people are still missing following Sundays tornado.

15 Standard Errors Hes a buy you a bull asset. Is it really attend Tatian?

16 Serious errors Public funding for universities has been cut by 15% this year. He never talks dirty.

17 Serious errors Public funding for universities has been cut by 15% this year. He never talks dirty. He never talks to Rudy.

18

19 NER MODEL N – E – R Accuracy ------------------------ × 100 = % N Correct editions: Comments: Target = 98%

20 NER Orange with apples

21 Pablo Romero-Fresco (p.romero-fresco@roehampton.ac.uk) Graciñas

22 Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) (p.romero-fresco@roehampton.ac.uk) Juan Martínez (Respeaking consultant) ITU-T Workshop onTelecommunications relay services for persons with disabilities (Geneva, 25 November 2011)


Download ppt "Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez."

Similar presentations


Ads by Google