1. A Similarity Evaluation Technique for Data Mining with Ensemble of Classifiers Seppo Puuronen, Vagan Terziyan International Workshop on Similarity Search 1-2 September, 1999 Florence (Italy)

2. Authors Department of Computer Science and Information Systems University of Jyvaskyla FINLAND Seppo Puuronen Vagan Terziyan Department of Artificial Intelligence Kharkov State Technical University of Radioelectronics, UKRAINE vagan@jytko.jyu.fi sepi@jytko.jyu.fi

3. Contents 4 The Research Problem and Goal 4 Basic Concepts 4 External Similarity Evaluation 4 Evaluation of Classifiers Competence 4 An Example 4 Internal Similarity Evaluation 4 Conclusions

4. The Research Problem During the past several years, in a variety of application domains, researchers in machine learning, computational learning theory, pattern recognition and statistics have tried to combine efforts to learn how to create and combine an ensemble of classifiers. The primary goal of combining several classifiers is to obtain a more accurate prediction than can be obtained from any single classifier alone.

5. Goal 4 The goal of this research is to develop simple similarity evaluation technique to be used for classification problem based on an ensemble of classifiers 4 Classification here is finding of an appropriate class among available ones for certain instance based on classifications produced by an ensemble of classifiers

6. Basic Concepts: Training Set (TS)  TS of an ensemble of classifiers is a quadruple: D is the set of instances D 1, D 2,..., D n to be classified; C is the set of classes C 1, C 2,..., C m, that are used to classify the instances; S is the set of classifiers S 1, S 2,..., S r, which select classes to classify the instances; P is the set of semantic predicates that define relationships between D, C, S

7. Basic Concepts: Semantic Predicate P

8. Problem 1: Deriving External Similarity Values Instances Classes Classifiers

9. External Similarity Values External Similarity Values (ESV): binary relations DC, SC, and SD between the elements of (sub)sets of D and C; S and C; and S and D. ESV are based on total support among all the classifiers for voting for the appropriate classification (or refusal to vote)

10. Problem 2: Deriving Internal Similarity Values Instances Classes Classifiers

11. Internal Similarity Values Internal Similarity Values (ISV): binary relations between two subsets of D, two subsets of C and two subsets of S. ISV are based on total support among all the classifiers for voting for the appropriate connection (or refusal to vote)

12. Why we Need Similarity Values (or Distance Measure) ? 4 Distance between instances is used by agents to recognize nearest neighbors for any classified instance 4 distance between classes is necessary to define the misclassification error during the learning phase 4 distance between classifiers is useful to evaluate weights of all classifiers to be able to integrate them by weighted voting

13. Deriving External Relation DC: How well class fits the instance Classifiers Instances Classes

14. Deriving External Relation SC: Measures Classifiers Competence in the Area of Classes 4 The value of the relation (S k,C j ) in a way represents the total support that the classifier S k obtains selecting (refusing to select) the class C j to classify all the instances.

15. Example of SC Relation Classifiers Instances Classes

16. Deriving External Relation SD: Measures “Competence” of Classifiers in the Area of Instances 4 The value of the relation (S k,D i ) represents the total support that the classifier S k receives selecting (or refusing to select) all the classes to classify the instance D i.

17. Example of SD Relation Instances Classes Classifiers

18. Standardizing External Relations to the Interval [0,1] n is the number of instances m is the number of classes r is the number of classifiers

19. Competence of a Classifier DiDi Conceptual pattern of features Conceptual pattern of class definition Instances Classes CjCj Classifier Competence in the Instance Area Competence in the Area of Classes

20. Classifier’s Evaluation: Competence Quality in an Instance Area - measure of the “classification abilities” of a classifier relatively to instances from the support point of view

21. Agent’s Evaluation: Competence Quality in the Area of Classes - measure of the “classification abilities” of a classifier in the correct use of classes from the support point of view

22. Quality Balance Theorem The evaluation of a classifier’s competence (ranking, weighting, quality evaluation) does not depend on the competence area “real world of instances” or “conceptual world of classes” because both competence values are always equal

23. Proof...

24. An Example 4 Let us suppose that four classifiers have to classify three papers submitted to a conference with five conference topics 4 The classifiers should define their selection of appropriate conference topic for every paper 4 The final goal is to obtain a cooperative result of all the classifiers concerning the “paper - topic” relation

25. C (classes) Set in the Example Classes - Conference Papers Notation AI and Intelligent SystemsC 1 Analytical TechniqueC 2 Real-Time SystemsC 3 Virtual RealityC 4 Formal MethodsC 5

26. S ( classifiers ) Set in the Example Classifiers - “Referees” Notation A.B.S 1 H.R.S 2 M.L.S 3 R.S.S 4

27. D (instances) Set in the Example

28. Selections Made for the Instance “Paper 1” D 1 P(D,C,S) C 1 C 2 C 3 C 4 C 5 S 1 1-1-10-1 S 2 0 + -1 ** 0 ++ 1 * -1 *** S 3 00-110 S 4 1-1001 Classifier H.R. considers “Paper 1” to fit to topic Virtual Reality* and refuses to include it to Analytical Technique** or Formal Methods***. H.R. does not choose or refuse to choose the AI and Intelligent Systems + or Real-Time Systems ++ topics to classify “Paper 1”.

29. Selections Made for the Instance “Paper 2” D 2 PC 1 C 2 C 3 C 4 C 5 S 1 -10-101 S 2 1-1-100 S 3 1-1011 S 4 -10010

30. Selections Made for the Instance “Paper 3” D 3 PC 1 C 2 C 3 C 4 C 5 S 1 101-10 S 2 010-11 S 3 -1-11-11 S 4 -1-11-11

31. Result of Cooperative Paper Classification Based on DC Relation

32. Results of Classifiers’ Competence Evaluation ( based on SC and SD sets) … Proposals obtained from the classifier A.B. should be accepted if they concern topics Real-Time Systems and Virtual Reality or instances “Paper 1” and “Paper 3”, and these proposals should be rejected if they concern AI and Intelligent Systems or “Paper 2”. In some cases it seems to be possible to accept classification proposals from the classifier A.B. if they concern Analytical Technique and Formal Methods. All four classifiers are expected to give an acceptable proposals concerning “Paper 3” and only suggestion of the classifier M.L. can be accepted if it concerns “Paper 2”...

33. Deriving Internal Similarity Values Via one intermediate setVia two intermediate sets

34. Internal Similarity for Classifiers: Instance-Based Similarity Instances Classifiers

35. Internal Similarity for Classifiers: Class-Based Similarity Classes Classifiers

36. Internal Similarity for Classifiers : Class-Instance-Based Similarity Classifiers Classes Instances

37. Conclusion 4 Discussion was given to methods of deriving the total support of each binary similarity relation. This can be used, for example, to derive the most supported classification result and to evaluate the classifiers according to their competence 4 We also discussed relations between elements taken from the same set: instances, classes, or classifiers. This can be used, for example, to divide classifiers into groups of similar competence relatively to the instance-class environment