Information of a split Pick a node, with a set S of size N Compute the impurity of the set Q(S) Pick a criteria A split the set S into M subsets The average impurity of these sets is Reduction of impurity (or increase of purity)
Algorithm Pick the test A which maximizes Q: how many values to consider? Lemma: ( see code below )
Algorithm Initialize: single leaf (what label?) Iterate: – Go over all leafs – Go over all features d – Go over all splitting values N – Pick (leaf, feature, splitting value) that reduces most impurity – Replace leaf with: new node two new leafs (their label?)
Issues (8.4) number of splits Missing features Prevent over-fitting – Early stopping – pruning Optimality vs greediness (Rivest et al, 76)
Example: xor Function: Tree with single node? Tree with two nodes labelinput 1(1,1) 1(-1,-1) (-1,1) (1,-1) X 1 >0 +1+1 X 2 >0 -1 -11-11 +1+1 yes No no
Regression (8.5) Value of leaf – Replace a single label with majority of outputs Impurity of a leaf – Replace discrete functions above with variance
Your consent to our cookies if you continue to use this website.