Presentation on theme: "Oblique Decision Trees Using Householder Reflection Chitraka Wickramarachchi Dr. Blair Robertson Dr. Marco Reale Dr. Chris Price Prof. Jennifer Brown."— Presentation transcript:
Oblique Decision Trees Using Householder Reflection Chitraka Wickramarachchi Dr. Blair Robertson Dr. Marco Reale Dr. Chris Price Prof. Jennifer Brown
Introduction Literature Review Methodology Results and Discussion Outline of the Presentation
Example: A bank wants to predict the potential status (Default or not) of a new credit card customer For the existing customers the bank has following data Introduction Salary No of Credit cards Total Credit Amount No of Loans Total Loan Installment Value of Other earnings GenderAge Possible approach - Generalized Linear models with binomial errors Model become complex if the structure of the data is complex.
Decision tree is a tree structured classifier. Decision Tree (DT) Salary <= s TCA < tc TLA < tl DNDD Root Node Non-Terminal Node Terminal Node Test based on features
Recursively partition the feature space into disjoint sub- regions until each sub-region becomes homogeneous with respect to a particular class Partitions X1 X2
Choosing the best split X2 <= X1<= X1<=
Types of DTs Decision Trees Univariate DT Multivariate DT Linear DT Non-Linear DT Axis parallel splits Oblique splits
Axis parallel splits Easy to implement Computer complexity is low Easy to interpret Advantages Disadvantage When the true boundaries are not axis parallel it produces complicated boundary structure
Disadvantages Implementation is challenging Computer complexity is high Therefore computationally less expensive oblique tree induction method would be desirable X1 X2 Oblique splits
Literature Review Oblique splits search for splits in the form of CART – LC Starts with the best axis parallel split Perturb each coefficient until find the best split Breiman et al. (1984) Can get trapped in local mimina Limitations No upper bound on the time spent at any node
Literature Review Heath et al. (1993) Simulated annealing Decision Trees (SADT) First places a hyperplane in a canonical location Perturb each coefficient randomly By randomization - try to escape from the local mimima Algorithm runs much slower than CART- LC Limitations
Literature Review Murthy et al. (1994) Oblique Classifier 1 (OC1) Start with the best axis parallel split Perturb each coefficient At a local mimima, perturb the hyperlane randomly Since 1994, there are many ODT induction methods have been developed based on EA algorithms and neural network concept
Proposed Methodology Our approach is to Transform the data set parallel to one of the feature axes Implement axis parallel splits Back-transform them in to the original space Transformation is done using Householder reflection.
Householder Reflection Let X and Y are vectors with the same norm there exists orthogonal symmetric matrix P such that where
Orientation of a cluster can be represented by the dominant Eigen vector of its variance covariance matrix. X1 X2 Householder Reflection
X1 X2 Householder Reflection
To avoid over-fitting Number of Terminal Nodes Accuracy Cost-complexity pruning
Results and Discussion Data sets - UCI Machine Learning Repository Data set Number of examples Number of features Number of Classes Iris Data15043 Boston Housing Data Estimate of the accuracy was obtained by ten 5-fold cross validation experiments.
Results and Discussion ClassiferIris DataHousing Data Householder Method CART-LC OC1 C4.5 Results High accuracy Computationally inexpensive