1. DM Performance: Decile (Lift) Analysis

2. Decile Maximization(DMAX)
Objective
Find model f(x) (predictor variables x)
such that performance in upper deciles (specified depth-of-file) is maximized
Explicitly manages resource constraint
mailings to particular depths-of file
Performance at different mailing depths
models optimized for different mailing depths

3. DMAX: Illustrative Example

4. Case II: 2% Response Rate Cum Response Lift Comparison

5. Response Model: Experimental Study
Two aspects to fitness
decile performance, overall fit to data
Modeling Response
Model =1/(1+exp(-wx))
Fitness f=w1D + w2C
Decile performance (responders in top d deciles)
Fit-to-data (Hosmer-Lemeshow goodness-of-fit)
Bhattacharyya, S., “Direct Marketing Response Models using Genetic Algorithms”,
KDD-98 Proceedings.

6. Top decile
(DMAX 10%)
2nd decile
(DMAX 20%)
3rd decile
(DMAX 30%)
7th decile
(DMAX 70%)

7. Learning with Resampling
Fitness as average of performance over multiple sub-samples
cross-validation
high variance in performance
sampling with replacement
member-wise, generation-wise, run-wise
DMAX
Logit
Improvement
10.1 %
10.9 %
13.0 %
-0.2 %

8. Modeling on Multiple Objectives
Model [y1,..,yk] = f (x)
simultaneously optimize on multiple objectives
Some common DM modeling desirables
response and high purchase revenues
likely churners with high usage of services
high tenure and usage
purchase and non-return
cross-selling, etc.
[or CPR (Combined Profit and Response) Models]

9. Multiple objectives Traditional approaches conflicting objectives
multiple single-objective models, and combine
weighted average of objectives
conflicting objectives
different levels of tradeoffs
frontier of non-dominated solutions
choice of final model based on diverse decision-maker objectives, can also be subjective

10. Pareto Frontier Non-dominated solutions Single GA run obtains
multiple objectives i, f a(x) better than f b(x) if
Single GA run obtains
tradeoff frontier of
non-dominated solutions f k(x) 2
non-dominated models
dominated models 1

11. Experimental Study: Data
Cellular-phone provider seeking to identify potential high-value churners
two dependent variables
binary Churn variable
continuous variable measuring revenue ($)
predictors: minutes-of-use (peak and off-peak), average charges, and payment information, etc.
obtained after EDA, normalized to 0 mean 1 s.d
50,000 sample: 25,000 for training, 25,000 for test set

12. Multiple Objectives: Performance
Churn lift
model capturing more churners in top deciles is better
$-Lift
model giving high revenue customers in upper deciles is better
overall modeling objective
maximize expected revenue saved through identification of high-value churners
Churn-Lift * $-Lift

13. Experimental Study Non-dominated models: Decile 1 (Training)
Decile 1 (trg)
400
350 GP
300 GA
250
$-Lift
Logistic
200
150
OLS
100 50
100
200
300
400
500
600
Churn-Lift
5 independent GA runs, aggregate the sets of non-dominated solutions

14. Experimental Study Non-dominated models: Decile 1 (Test)
400
350
300 GP
250 GA
$-Lift
200
Logistic
150
OLS
100 50
100
200
300
400
500
Churn-Lift

15. Experimental Study Non-dominated models: Decile 2 (Test)
300
250 GP
200 GA
$-Lift
Logistic
150
OLS
100 50 50
100
150
200
250
300
350
400
450
Churn-Lift

16. Experimental Study Non-dominated models: Decile 3 (Test)
250 GP
200 GA
150
Logistic
$-Lift
OLS
100 50 50
100
150
200
250
300
350
Churn-Lift

17. Experimental Study Non-dominated models: Decile 7 (Test)GP GA
Logistic
OLS

18. Experimental Study Performance Summary

19. Multi-Objective Models: Elitism and Population Size
preserves non-dominated solutions in next generation
Elitism particularly helpful when using smaller
populations

20. General Optimization of Lifts
Fitness function
Seeks a general maximization of lifts at all deciles
n = # of observations, nR = # of “responders”
= dependent-B value of ith obs.
= dependent-C value of ith obs. (obs. sorted on model scores)
(binary dependent var.)
(continuous dependent var.)

21. Specific vs. General Lift Opt
Table: Best Prod-Lifts in Deciles

22. Specific vs. General Lift Opt.
Table: Best $-Lift and Churn-Lifts in Deciles