Mining Several Databases with an Ensemble of Classifiers Seppo Puuronen Vagan Terziyan Alexander Logvinovsky 10th International Conference and Workshop on Database and Expert Systems Applications August 30 - September 3, 1999 Florence, Italy DEXA-99

Authors Department of Computer Science and Information Systems University of Jyvaskyla FINLAND Seppo PuuronenVagan Terziyan Department of Artificial Intelligence Kharkov State Technical University of Radioelectronics, UKRAINE Alexander Logvinovsky Department of Artificial Intelligence Kharkov State Technical University of Radioelectronics, UKRAINE

Contents The problem of “multiclassifiers” - “multidatabase” mining; Case “One Database - Many Classifiers”; Dynamic integration of classifiers; Case “One Classifier - Many Databases”; Weighting databases; Case “Many Databases - Many Classifiers”; Context-based trend within the classifiers predictions and decontextualization; Conclusion

Introduction

Introduction

Problem

Problem

Case ONE:ONE

Case ONE:MANY

Dynamic Integration of Classifiers Final classification is made by weighted voting of classifiers from the ensemble; Weights of classifiers are recalculated for every new instance; Weighting is based on predicted errors of the classifiers in the neighborhood area of the instance

Sliding Exam of a Classifier (Predictor, Interpolator) Remove an instance y(x i ) from training set; Use a classifier to derive prediction result y’(x i ); Evaluate difference  as distance between real and predicted values Continue for every instance

Brief Review of Distance Functions According to D. Wilson and T. Martinez (1997)

PEBLS Distance Evaluation for Nominal Values (According to ) PEBLS Distance Evaluation for Nominal Values (According to Cost S. and Salzberg S., 1993 ) The distance d i between two values v 1 and v 2 for certain instance is: where C 1 and C 2 are the numbers of instances in the training set with selected values v 1 and v 2, C 1i and C 2i are the numbers of instances from the i-th class, where the values v 1 and v 2 were selected, and k is the number of classes of instances

Interpolation of Error Function Based on Hypothesis of Compactness | x - x i | <  (   0)  |  (x) -  (x i ) |  0

Competence map absolute difference  weight function 

Solution for ONE:MANY

Case MANY:ONE

Integration of Databases Final classification of an instance is obtained by weighted voting of predictions made by the classifier for every database separately; Weighting is based on taking the integral of the error function of the classifier across every database

Integral Weight of Classifier Integral Weight of Classifier

Solution for MANY:ONE

Case MANY:MANY

Weighting Classifiers and Databases Prediction and weight of a databasePrediction and weight of a classifier

Solutions for MANY:MANY

Decontextualization of Predictions Sometimes actual value cannot be predicted as weighted mean of individual predictions of classifiers from the ensemble; It means that the actual value is outside the area of predictions; It happens if classifiers are effected by the same type of a context with different power; It results to a trend among predictions from the less powerful context to the most powerful one; In this case actual value can be obtained as the result of “decontextualization” of the individual predictions

Neighbor Context Trend x prediction in (1,2) neighbor context: “worse context” prediction in (1,2,3) neighbor context: “better context” actual value: “ideal context” y xixi y(x i ) y + (x i ) y - (x i )

Main Decontextalization Formula y Y y - - prediction in worse context y + - prediction in better context y ’ - decontextualized prediction y - actual value y’ y+y+ y-y- ’’ ++ --  ’ = - ·+- ·+- ·+- ·+  - +  +  ’ <  - ;  ’ <  +  + <  -

Decontextualization One level decontextualization All subcontexts decontextualization Decontextualized difference  New sample classification

Physical Interpretation of Decontextualization R1R1 R2R2 R res actual value decontextualized value predicted values Uncertainty is like a “resistance” for precise prediction actual value y+y+ y-y- y’y’ y y y i- - prediction in worse context y + - prediction in better context y ’ - decontextualized prediction y - actual value

Conclusion Dynamic integration of classifiers based on locally adaptive weights of classifiers allows to handle the case «One Dataset - Many Classifiers»; Integration of databases based on their integral weights relatively to the classification accuracy allows to handle the case «One Classifier - Many Datasets»; Successive or parallel application of the two abowe algorithms allows a variety of solutions for the case «Many Classifiers - Many Datasets»; Decontextualization as the opposite to weighted voting way of integration of classifiers allows to handle context of classification in the case of a trend