1 Using Bayesian Network for combining classifiers Leonardo Nogueira Matos Departamento de Computação Universidade Federal de Sergipe

2 Agenda Why combining classifiers? Bayesian network principles Bayesian network as an ensemble of classifiers Experimental results Future works and conclusions

3 Why combining classifiers? Classifiers can colabore with each other Minimizes computational effort for training Maximizes global recognition rate

4 Why not to do so? Because combining individual preditions can be so difficult as divising a robust single classifier

5 Why not to do so? Because combining individual preditions can be so difficult as divising a robust single classifier Decision Classifiers Combiner

6 Approaches for combining classifiers L1. Data LevelL3. Decision Level L2. Feature Level Fixed rules Trainable rules

7 Approaches for combining classifiers L1. Data LevelL3. Decision Level L2. Feature Level Fixed rules Trainable rules

8 Why not to do so? Because combining individual preditions can be so difficult as divising a robust single classifier Decision Classifiers Combiner p(w|x)

9 Approaches for combining classifiers L1. Data LevelL3. Decision Level L2. Feature Level Fixed rules Trainable rules

10 Existent scenarios Pattern space Pattern 2 1 classifiers

11 Our scenery Pattern space classifiers

12 A closed look

13 A closed look – discriminant function

14 A closed look – using multiple classifiers

15 A closed look – using multiple classifiers The challegers: How can we combine classifier's output? How can we identify regions in pattern space?

16 Agenda Why combining classifiers? Bayesian network principles Bayesian network as an ensemble of classifiers Experimental results Future works and conclusions

17 Bayesian network principles A B C Those circles represent binary random variables

18 Bayesian network principles A B C Those circles represent binary random variables

19 Bayesian network principles A B C Those circles represent binary random variables dataset

20 Bayesian network principles A B C Those circles represent binary random variables instance

21 Bayesian network principles A B C Jointly probability inference is a combinatorial problem 2 possibilities 4 possibilities

22 Bayesian network principles A B C Jointly probability inference is a combinatorial problem Independence makes computation a little more simple

23 Bayesian network principles A B C Arest – indicates statistical dependence between variables

24 Bayesian network principles A B C Arc – represents causality

25 Bayesian network principles A B C A Bayesian network is a DAG (Direct Aciclic Graph) where nodes represent random variables and arcs represent causality relatioship

26 Bayesian network principles A B C There are polinomial time algorithms to compute inference in BN

27 Bayesian network principles A B C There are polinomial time algorithms to compute inference in BN Evidence

28 Bayesian network principles A B C There are polinomial time algorithms to compute inference in BN Evidence messages

29 Bayesian network principles A B C There are polinomial time algorithms to compute inference in BN Evidence

30 Agenda Why combining classifiers? Bayesian network principles Bayesian network as an ensemble of classifiers Experimental results Future works and conclusions

31 A Fundamental Goal

32 Another insight From a statistical point-of-view a Bayesian network is also a graphical model to represents a complex and factored probability distribution function

33 Another insight From a statistical point-of-view a Bayesian network is also a graphical model to represents a complex and factored probability distribution function

34 Another insight From a statistical point-of-view a Bayesian network is also a graphical model to represents a complex and factored probability distribution function The challegers: How can we combine classifier's output? How can we identify regions in pattern space?

35 How can we combine classifier's output? We use a BN as a graphical model of the pdf P(w|x) We assume that classifier participate in computing that function Each classifier must be a statistical classifier

36 How can we identify regions in pattern space?

37 Splitting pattern space

38 Defining a region

39 Patterns in a region

40 Algorithm

41 Bayesian Network Structure

42 Bayesian networks for combining classifiers

43 Agenda Why combining classifiers? Bayesian network principles Bayesian network as an ensemble of classifiers Experimental results Future works and conclusions

44 Results with UCI databases

45 Results with NIST database

46 System I classifiers

47 Preliminaries

48 Results with the complete dataset

49 Agenda Why combining classifiers? Bayesian network principles Bayesian network as an ensemble of classifiers Experimental results Future works and conclusions

50 Future works

51 Future works

52 Future works

53 Future works

54 Future works Pattern space Pattern 2 1 classifiers

55 Conclusions We have developed a method for combining classifiers using a Bayesian network A BN act as trainable ensemble of statistical classifiers The method is not suitable for small size dataset Experimental results reveal a good performance with a large dataset As a future work we intend to use a similar approach for splitting the feature vector and combine classifiers specialized on each piece of it.

56 Thank you!