Presentation is loading. Please wait.

Presentation is loading. Please wait.

Dipartimento di Meccanica e Tecnologie Industriali Università degli Studi di Firenze On TFSR (semi)automatic systems supportability: novel instruments.

Similar presentations


Presentation on theme: "Dipartimento di Meccanica e Tecnologie Industriali Università degli Studi di Firenze On TFSR (semi)automatic systems supportability: novel instruments."— Presentation transcript:

1 Dipartimento di Meccanica e Tecnologie Industriali Università degli Studi di Firenze On TFSR (semi)automatic systems supportability: novel instruments for analysis and compensation Francesco Borchi, Monica Carfagni, Matteo Nunziati

2 2 Dipartimento di Meccanica e Tecnologie Industriali Outline Main goal TFSR Systems LogR estimation Common test procedures for TFSR systems System behaviour classification Supportability evaluation tools Score compensation tools Quality assessment logics Conclusion

3 3 Dipartimento di Meccanica e Tecnologie Industriali Main goal Our goal is to propose a general purpose set of tools for system compensation and quality assessment Specific goals: 1. to build a generic framework for system analysis 2. to develop a novel generic tool for system compensation 3. to assess system quality level on the basis of the amount of compensation required by the system itself

4 4 Dipartimento di Meccanica e Tecnologie Industriali TFSR Systems Voice sample 1 Voice sample 2 TFSR system LogR We define a TFSR system as a black box which receives two or more recordings as inputs and produces one or more scores (LogR) as outputs

5 5 Dipartimento di Meccanica e Tecnologie Industriali LogR estimation 1/2 LogR = log 10 [P(E | H 0 ) / P( E | H 1 )] Log-likelihood ratio defines the most supportable hypotesis Hypotesis 0: the two samples belong to the same speaker Hypotesis 1: the two samples belong to different speakers If LogR>0 support goes to the H 0 hypotesis If LogR<0 support goes to the H 1 hypotesis If LogR=0 no support is provided

6 6 Dipartimento di Meccanica e Tecnologie Industriali LogR estimation 2/2 The system goodness depends on a number of factors: The way we have used to retrieve voice samples The kind of parameters employed in the recognition The algorithms used for parameter extraction The mathematic model used to estimate LogR Experimentation is the best way to assess system behaviour The real LogR value is unknown. We can estimate it using some approximations. Our systems are error-prone.

7 7 Dipartimento di Meccanica e Tecnologie Industriali Common test procedures for TFSR systems 1/2 The system is tested against a set of recordings having known origin: … Speaker1…SpeakerN… 2 or more recordings

8 8 Dipartimento di Meccanica e Tecnologie Industriali Common test procedures for TFSR systems 2/2 Recordings are mixed up and grouped in pairs: Same speaker pairs (SS) Different speaker pairs (DS) SS: test system behaviour when H 0 is true. Is LogR>0? DS: test system behaviour when H 1 is true. Is LogR<0?

9 9 Dipartimento di Meccanica e Tecnologie Industriali System behaviour classification 1/3 Tippett Plot: a common method to show system behaviour % SS % DS False negatives False positives H1H1 H0H0

10 10 Dipartimento di Meccanica e Tecnologie Industriali System behaviour classification 2/3 Only false scores Wrong support Provide a solution to eliminate false score only areas (red boxes)

11 11 Dipartimento di Meccanica e Tecnologie Industriali System behaviour classification 3/3 ipoperforming isoperforming Provide a solution to reduce the amount of false scores

12 12 Dipartimento di Meccanica e Tecnologie Industriali Supportability evaluation tools 1/3 A quantitative evaluation of false scores has been proposed by P. Rose et Al. (2003): LR test =P(LogR>0 | H 0 ) / P(LogR>0 | H 1 ) Percentage of true positivesPercentage of false positives Interpretable via Evett Table No information is provided about false negatives No information about the distribution of false scores Do they affect a narrow range of scores? Do they widely perturb the system response?

13 13 Dipartimento di Meccanica e Tecnologie Industriali Supportability evaluation tools 2/3 We propose to generalize the LR test index using a new tool: the Supportability of System function (SoS): Interpretable via Evett Table Defined for both false positives and negatives Univocally detects the amount of false scores for each LogR Provides the accuracy of each score SoS(x) = P(LogR>x | H0) / P(LogR>x | H1) if x>0 SoS(x) = [1- P(LogR>x | H1)] / [1-P(LogR>x | H0)] if x<0 We know how much we can rely on our system, time by time!

14 14 Dipartimento di Meccanica e Tecnologie Industriali Supportability evaluation tools 3/3 90% true 20% false SoS=90/20=4.5 LogR = -13

15 15 Dipartimento di Meccanica e Tecnologie Industriali Score compensation tools 1/3 original translated 0X X Preliminary operation: Eliminate false score only areas encreasing or reducing all scores

16 16 Dipartimento di Meccanica e Tecnologie Industriali Score compensation tools 2/3 LogR=1 LogR=2 LogR=3 LogR=4 New LogR = LogR*tanh( Log 10 (SoS) )

17 17 Dipartimento di Meccanica e Tecnologie Industriali Score compensation tools 3/3 Compress all scores by a value defined by the SoS function Reduced amount of false scores compressed original Decreased values for true scores Reduce the amount of false scores at the cost of a lower discriminative power

18 18 Dipartimento di Meccanica e Tecnologie Industriali Quality assessment logics 1/3 Compensation increases for decreasing values of SoS Compensation is intrinsic to the system A good system must have a strong SoS for each LogR value – Score compensation reduces systems discriminative power + Score compensation is required to prevent unbalanced responses

19 19 Dipartimento di Meccanica e Tecnologie Industriali Quality assessment logics 2/3 DMTI procedure Step 1: test the system against a dataset (LogR) Step 2: calculate supportability (SoS) Step 3: calculate compensated scores (New LogR) Step 4: calculate the percentage P of new LogR which has a strong SoS score (fixed by our standards) Step 5: evaluate the Degree of Supportability (DoS): DoS = atanh (2P-1)

20 20 Dipartimento di Meccanica e Tecnologie Industriali Quality assessment logics 3/3 Regardless of the specific procedure, our DoS score is equivalent to a LogR score!

21 21 Dipartimento di Meccanica e Tecnologie Industriali Conclusion A general purpose tool has been developed to score system supportability An additional mathematic tool has been developed to compensate unbalanced systems The tools are system independent and theoretically motivated rather than empirically built The tools are useful to reduce both false positives and false negatives False score reduction produces a decrement in discriminative power Such decrement is intrinsic to the system response and is univocally usable for system quality assessment The proposed procedure for system quality assessment (degree of supportability) uses the well known Evett scale to score the system supportability

22 22 Dipartimento di Meccanica e Tecnologie Industriali Thank You for your attention… Questions?


Download ppt "Dipartimento di Meccanica e Tecnologie Industriali Università degli Studi di Firenze On TFSR (semi)automatic systems supportability: novel instruments."

Similar presentations


Ads by Google