BOAI: Fast Alternating Decision Tree Induction based on Bottom-up Evaluation Bishan Yang, Tengjiao Wang, Dongqing Yang, and Lei Chang School of EECS, Peking University

Outline Motivation Related work Preliminaries Our Approach Experimental Results Summary

Motivation Alternating Decision Tree (ADTree) is an effective decision tree algorithm based on AdaBoost. High accurate classifier Small size of tree and easy to interpret Provide measures of prediction confidence Wide range of applications Customer churn prediction Fraud detection Disease trait modeling ……

Limitation of existing work Very expensive to apply to large sets of training data Takes hours to train on large number of examples and attributes The training time will grow up exponentially with the increasing size of data

Related work Several techniques have been developed to tackle the efficiency problem for traditional decision trees (ID3,C4.5) SLIQ (EDBT — Mehta et al.) Introduce data structures called attribute list and class list SPRINT (VLDB — J. Shafer et al.) Only construct attribute list PUBLIC (VLDB — Rastogi & Shim) Integrates MDL “prunning” into tree “building” process Rainforest (VLDB — Gehrke, Ramakrishnan & Ganti) Introduce AVC-groups which are sufficient for split evaluation BOAT (PODS — Gehrke, Ganti, Ramakrishnan & Loh) build tree based on a subset of data by using bootstrapping Can’t not directly apply to ADTree evaluate splits based on information of the current node

Related work Several optimizing methods for ADTree PAKDD – Pfahringer et al. Zpure cut off Merging Three heuristic mechanisms ILP – Vanassche et al. Caching optimization Cons: Little improvement until reaching large number of iterations Cons: can’t guarantee the model quality Cons :the additional memory consumption grows fast with increasing number of boosting rounds

Preliminaries Alternating Decision Tree (ADTree) (ICML) Classification: the sign of the sum of the prediction values along the paths defined by the instance Eg. Instance (age, income) = (35, 1300), sign(f(x)) = sign(0.5-0.5+0.4+0.3) = sign(0.7) = +1 +0.5 Age < 40 Income > 1000 -0.5 +0.2 +0.3 -0.6 Income > 1200 Age > 30 : prediction node +0.4 -0.2 -0.1 +0.1 : decision node

Complexity mainly lies in this part! Preliminaries Algorithm ADTree Induction Input: For t=1 to T do /* evaluation phase*/ for all such that for all such that calculate Select , which minimize /*Partition phase*/ Set update weights: , is the prediction value Pt is set of preconditions, C is set of base conditions, (c=(A≤(vi+vi+1)/2) or c=(A=vi)) Complexity mainly lies in this part! W+(c) (resp. W-(c)) is the total weight of the positive (resp. negative) instances that satisfying condition c Weights are increased for misclassified instances and decreased for correctly classified instances

Our approach – BOAI Evaluation phase in top-down evaluation instances need to be sorted on numeric attributes at each prediction node the weight distribution for all possible splits need to be calculated by scanning instances at each prediction node great deal of sorting and computing overlap BOAI (Bottom-up Evaluation for ADTree Induction) Presorting technique reduce the sorting cost Bottom-up evaluation evaluate splits from the leaf nodes to the root node avoid much redundant computing and sorting cost obtain the exactly same evaluation results with the top-down evaluation approach

Pre-sorting technique Preprocessing step Sort the values of the numeric attribute, and map the sorting space of values x0,x1,…,xm-1 to 0,1,…,m-1, and then replace the attribute values as the sorted indexes. use the sorted indexes to speed up the sorting time the split evaluation phase. Sorting space: 1500, 1600, 1800 Sorted index: 0,1,2 sorting Replace the original values in the data with their sorted indexes replacing

VW-set (Attribute-Value, Class-Weight) Only need weight distribution ( , ) on distinct attribute values for split evaluation. Just keep the necessary information! VW-set of attribute A at node p stores the weight distribution of each class for each distinct value of A in F(p). (F(p) denotes the instances projected onto p) If A is a numeric attribute, the distinct values in VW-set must be sorted. VW-group of node p the set of all VW-sets at node p Each prediction node can be evaluated based on its VW-group Our idea is to just keep the necessary weight distribution information for evaluation.

VW-set (Attribute-Value, Class-Weight) The size of the VW-set is determined by the distinct attribute values appeared in F(p) and is not proportional to the size of F(p) W+(Dept.=2) W-(Dept.=2)

Bottom-up evaluation The main idea: Evaluate based on VW-group evaluate splits from the leaf nodes to the root node use already computed statistics to evaluate parent nodes much computing and sorting redundancy can be avoided Evaluate based on VW-group

Bottom-up evaluation (Cont.) For leaf nodes directly construct VW-group by scanning instances at the node VW-set on categorical attribute: Use hash table to index different values and collect their weights VW-set on numeric attribute: Map weights to the corresponding index in the value space and compress them to VW-set Sort can take linear time! O(n+m)

Bottom-up evaluation (Cont.) For internal nodes construct VW-group by merging the VW-groups of its two children nodes (prediction nodes) Sort cost: O(V1+V2)

Evaluation Algorithm directly construction merge construction categorical split numeric split evaluate other children

Computation analysis Prediction node p, |F(p)|=n Top-down evaluation: sorting cost for each numeric attribute : O(nlogn) Z-value calculation cost for each attribute : O(n) Bottom-up evaluation: sorting cost for each numeric attribute : Leaf node: in most case O(n+m) Internal node: sort through merging, O(V1+V2), where V1,V2 are the numbers of distinct values in the two merged VW-groups. They always much smaller then n Z-value calculation cost for each attribute: O(V), V is the number of distinct values in the VW-group, always much smaller than n

Experiments Data sets Environment Synthetic data sets: Real data sets: IBM Quest data mining group, up to 500,000 instances Real data sets: China Mobile Communication Company, 290,000 subscribers covering 92 variables Environment AMD 3200+ CPU running windows XP with 768MB main memory

Experimental results (Synthetic data)

Experimental results (real data)

Experimental results (real data) Apply to churn prediction Calibration set: 20,083, validation set: 5,062 Imbalance problem: about 2.1% churn rate Re-balancing strategy: Multiply the weight of each instance in the minority class by Wmaj/Wmin (Wmaj(resp. Wmin) is the total weight of the majority (resp. minority) class instances) Little information loss and does not introduce more computing power on average Models F-measure G-mean W-accuracy Modeling Time (sec) ADT(w/o re-balancing) 56.04 65.65 44.53 75.56 Random Forests 19.21 84.04 84.71 960.00 TreeNet 72.81 79.61 64.40 30.00 BOAI 50.62 90.81 85.84 7.625

Summary We developed a novel approach for ADTree induction to speed up training time on large data sets Key insight: eliminate the great redundancy of sorting and computation in the tree induction by using a bottom-up evaluation approach based on VW-group Experiments on both synthetic and real datasets show that BOAI offers significant performance improvement while constructing exactly the same model. Its an attractive algorithm for modeling on large data sets!

Thanks!