1. to win Kaggle Data Mining Competitions
Using R
to win Kaggle Data Mining Competitions
Chris Raimondi
November 1, 2012

2. Overview of talk What I hope you get out of this talk Life before R
Simple model example
R programming language
Background/Stats/Info
How to get started
Kaggle
Antonio Possolo, Division Chief of Statistical Engineering at the National Institute of Science and Technology (NIST), was charged with making sense of these varied estimates to help the government coordinate the national response to the spill. As described in this video testimonial (starting at 2:20), Possolo was sitting in the company of the Secretaries of Energy and the Interior, when he broke out R on his laptop to run uncertainty analysis and harmonize the estimates from the various sources.

3. Overview of talk Individual Kaggle competitions HIV Progression Chess
Mapping Dark Matter
Dunnhumby’s Shoppers Challenge
Online Product Sales
Antonio Possolo, Division Chief of Statistical Engineering at the National Institute of Science and Technology (NIST), was charged with making sense of these varied estimates to help the government coordinate the national response to the spill. As described in this video testimonial (starting at 2:20), Possolo was sitting in the company of the Secretaries of Energy and the Interior, when he broke out R on his laptop to run uncertainty analysis and harmonize the estimates from the various sources.

4. What I want you to leave with
Belief that you don’t need to be a statistician to use R - NOR do you need to fully understand Machine Learning in order to use it
Motivation to use Kaggle competitions to learn R
Knowledge on how to start
Antonio Possolo, Division Chief of Statistical Engineering at the National Institute of Science and Technology (NIST), was charged with making sense of these varied estimates to help the government coordinate the national response to the spill. As described in this video testimonial (starting at 2:20), Possolo was sitting in the company of the Secretaries of Energy and the Interior, when he broke out R on his laptop to run uncertainty analysis and harmonize the estimates from the various sources.

5. My life before R Lots of Excel
Had tried programming in the past – got frustrated
Read NY Times article in January 2009 about R & Google
Installed R, but gave up after a couple minutes
Months later…
Antonio Possolo, Division Chief of Statistical Engineering at the National Institute of Science and Technology (NIST), was charged with making sense of these varied estimates to help the government coordinate the national response to the spill. As described in this video testimonial (starting at 2:20), Possolo was sitting in the company of the Secretaries of Energy and the Interior, when he broke out R on his laptop to run uncertainty analysis and harmonize the estimates from the various sources.

6. My life before R
Using Excel to run PageRank calculations that took hours and was very messy
Was experimenting with Pajek – a windows based Network/Link analysis program
Was looking for a similar program that did PageRank calculations
Revisited R as a possibility
Antonio Possolo, Division Chief of Statistical Engineering at the National Institute of Science and Technology (NIST), was charged with making sense of these varied estimates to help the government coordinate the national response to the spill. As described in this video testimonial (starting at 2:20), Possolo was sitting in the company of the Secretaries of Energy and the Interior, when he broke out R on his laptop to run uncertainty analysis and harmonize the estimates from the various sources.

7. My life before R Came across “R Graph Gallery” Saw this graph…
Antonio Possolo, Division Chief of Statistical Engineering at the National Institute of Science and Technology (NIST), was charged with making sense of these varied estimates to help the government coordinate the national response to the spill. As described in this video testimonial (starting at 2:20), Possolo was sitting in the company of the Secretaries of Energy and the Interior, when he broke out R on his laptop to run uncertainty analysis and harmonize the estimates from the various sources.

9. Addicted to R in one line of code
pairs(iris[1:4], main="Edgar Anderson's Iris Data", pch=21,
bg=c("red", "green3", "blue")[unclass(iris$Species)])
“pairs” = function
“iris” = dataframe

10. What do we want to do with R?
Machine learning
a.k.a. – or more specifically
Making models
We want to TRAIN a set of data with KNOWN answers/outcomes
In order to PREDICT the answer/outcome to similar data where the answer is not known
Antonio Possolo, Division Chief of Statistical Engineering at the National Institute of Science and Technology (NIST), was charged with making sense of these varied estimates to help the government coordinate the national response to the spill. As described in this video testimonial (starting at 2:20), Possolo was sitting in the company of the Secretaries of Energy and the Interior, when he broke out R on his laptop to run uncertainty analysis and harmonize the estimates from the various sources.

12. How to train a model
R allows for the training of models using probably over 100 different machine learning methods
To train a model you need to provide
Name of the function – which machine learning method
Name of Dataset
What is your response variable and what features are you going to use

13. Example machine learning methods available in R
Bagging
Partial Least Squares
Boosted Trees
Principal Component Regression
Elastic Net
Projection Pursuit Regression
Gaussian Processes
Quadratic Discriminant Analysis
Generalized additive model
Random Forests
Generalized linear model
Recursive Partitioning
K Nearest Neighbor
Rule-Based Models
Linear Regression
Self-Organizing Maps
Nearest Shrunken Centroids
Sparse Linear Discriminant Analysis
Neural Networks
Support Vector Machines

14. Code used to train decision tree
library(party) irisct <- ctree(Species ~ Sepal.Length + Sepal.Width + Petal.Length + Petal.Width, data = iris) Or use “.” to mean everything else - as in… irisct <- ctree(Species ~ ., data = iris)

15. That’s it
You’ve trained your model – to make predictions with it – use the “predict” function – like so: my.prediction <- predict(irisct, iris2) To see a graphic representation of it – use “plot”. plot(irisct) plot(irisct, tp_args = list(fill = c("red", "green3", "blue")))

18. R background Statistical Programming Language Since 1996
Powerful – used by companies like Google, Allstate, and Pfizer.
Over 4,000 packages available on CRAN
Free
Available for Linux, Mac, and Windows
Antonio Possolo, Division Chief of Statistical Engineering at the National Institute of Science and Technology (NIST), was charged with making sense of these varied estimates to help the government coordinate the national response to the spill. As described in this video testimonial (starting at 2:20), Possolo was sitting in the company of the Secretaries of Energy and the Interior, when he broke out R on his laptop to run uncertainty analysis and harmonize the estimates from the various sources.

19. Learn R – Starting Tonight
Buy “R in a Nutshell”
Download and Install R
Download and Install Rstudio
Watch 2.5 minute video on front page of rstudio.com
Use read.csv to read a Kaggle data set into R
Antonio Possolo, Division Chief of Statistical Engineering at the National Institute of Science and Technology (NIST), was charged with making sense of these varied estimates to help the government coordinate the national response to the spill. As described in this video testimonial (starting at 2:20), Possolo was sitting in the company of the Secretaries of Energy and the Interior, when he broke out R on his laptop to run uncertainty analysis and harmonize the estimates from the various sources.

20. Learn R – Continue Tomorrow
Train a model using Kaggle data
Make a prediction using that model
Submit the prediction to Kaggle
Antonio Possolo, Division Chief of Statistical Engineering at the National Institute of Science and Technology (NIST), was charged with making sense of these varied estimates to help the government coordinate the national response to the spill. As described in this video testimonial (starting at 2:20), Possolo was sitting in the company of the Secretaries of Energy and the Interior, when he broke out R on his laptop to run uncertainty analysis and harmonize the estimates from the various sources.

21. Learn R – This Weekend Install the Caret package
Start reading the four Caret vignettes
Use the “train” function in Caret to train a model, select a parameter, and make a prediction with this model
Antonio Possolo, Division Chief of Statistical Engineering at the National Institute of Science and Technology (NIST), was charged with making sense of these varied estimates to help the government coordinate the national response to the spill. As described in this video testimonial (starting at 2:20), Possolo was sitting in the company of the Secretaries of Energy and the Interior, when he broke out R on his laptop to run uncertainty analysis and harmonize the estimates from the various sources.

22. Buy This Book: R in a Nutshell
Excellent Reference
2nd Edition released just two weeks ago
In stock at Amazon for $37.05
Extensive chapter on machine learning

23. R Studio

25. Read the vignettes – some of them are golden.
R Tip
Read the vignettes – some of them are golden.
There is a correlation between the quality of an R package and its associated vignette.

27. What is kaggle? Platform/website for predictive modeling competitions
Think middleman – they provide the tools for anyone to host a data mining competition
Makes it easy for competitors as well – they know where to go to find the data/competitions
Community/forum to find teammates

28. Kaggle Stats Competitions started over 2 years ago
55+ different competitions
Over 60,000 Competitors
165,000+ Entries
Over $500,000 in prizes awarded

29. Why Use Kaggle? Rich Diverse Set of Competitions Real World Data
Competition = Motivation
Fame
Fortune

30. Who has Hosted on Kaggle?

31. Methods used by competitors
source:kaggle.com

32. Predict HIV Progression
Prizes:
1st
$500.00
Objective:
Predict (yes/no) if there will be an improvement in a patient's HIV viral load.
Training Data:
1,000 Patients
Testing Data:
692 Patients

33. Training Test Training Set Public Leaderboard Private Leaderboard
Answer
Various Features
Response
PR Seq
RT Seq
VL-t0
CD4-t0 1
CCTCAGATCA
TACCTTAAAT
4.7
473
CACTCTAAAT
CTTAAATTTY
5.0 7
AAGAAATCTG
3.2
349
CTCTTTGGCA
5.1 51
GAGAGATCTG
3.7 77
5.7
206
TCTAAATTTC
3.9
144
CACTTTAAAT
TCTAAACTTT
4.4
496
3.4
252
TGGAAGAAAT
5.5
TTCGTCACAA
4.3
109
AAGAGATCTG 70
ACTAAATTTT
570
CCTCAAATCA
4.0
217
2.8
730
ATTAAATTTT
4.5 56
TACTTTAAAT 21
249
CTTAAATTTT
269
AAGGAATCTG
4.6
165 91
Training
Training Set
Test
N/A
Public Leaderboard
Private Leaderboard

34. Predict HIV Progression

35. Predict HIV Progression
Features Provided:
PR: 297 letters long – or N/A
RT: 193 – 494 letters long
CD4: Numeric
VLt0: Numeric
Features Used:
PR1-PR97: Factor
RT1-RT435: Factor
CD4: Numeric
VLt0: Numeric

36. Predict HIV Progression
Concepts / Packages:
Caret
train
rfe
randomForest

37. Random Forest
Tree 1:
Take a random ~ 63.2% sample of rows from the data set
For each node – take mtry random features – in this case 2 would be the default
Tree 2:
Take a different random ~ 63.2% sample of rows from the data set
And so on…..
Sepal.Length
Sepal.Width
Petal.Length
Petal.Width
5.1
3.5
1.4
0.2
4.9 3
4.7
3.2
1.3
4.6
3.1
1.5 5
3.6
5.4
3.9
1.7
0.4
3.4
0.3
4.4
2.9
0.1
3.7
4.8
1.6
4.3
1.1
5.8 4
1.2
5.7
3.8

38. Caret – train TrainData <- iris[,1:4] TrainClasses <- iris[,5]
knnFit1 <- train(TrainData, TrainClasses,
method = "knn",
preProcess = c("center", "scale"),
tuneLength = 3,
trControl = trainControl(method = "cv",
number=10))

39. Caret – train > knnFit1 150 samples 4 predictors
3 classes: 'setosa', 'versicolor', 'virginica'
Pre-processing: centered, scaled
Resampling: Cross-Validation (10 fold)
Summary of sample sizes: 135, 135, 135, 135, 135, 135, ...
Resampling results across tuning parameters:

40. Caret – train k Accuracy Kappa Accuracy SD Kappa SD
Accuracy was used to select the optimal model using the largest value.
The final value used for the model was k = 17.

42. Benefits of winning Cold hard cash Several newspaper articles
Quoted in Science magazine
Prestige
Easier to find people willing to team up
Asked to speak at STScI
Perverse pleasure in telling people the team that came in second worked at….

43. IBM Thomas J. Watson Research Center

44. Training Data Provided:
Chess Ratings Comp
Prizes:
1st
$10,000.00
Objective:
Given 100 months of data predict game outcomes for months 101 – 105.
Training Data Provided:
Month
White Player #
Black Player #
White Outcome – Win/Draw/Lose (1/0.5/0)

45. How do I convert the data into a flat 2D representation?
Think:
What are you trying to predict?
What Features will you use?

46. Percentage of Games Won Number of Games won as White
Outcome
White Feature 1
White Feature 2
White Feature 3
White Feature 4
Black Feature 1
Black Feature 2
Black Feature 3
Black Feature 4
White/Black 1
White/Black 2
White/Black 3
White/Black 4
Game Feature 1
Game Feature 2 1
0.5
Percentage of Games Won
Number of Games won as White
Number of Games Played
Percentage of Games Won
Number of Games won as White
Number of Games Played
White Games Played/Black Games Played
Type of Game Played

47. Packages/Concepts Used:
igraph
1st real function

49. Mapping Dark Matter Mapping Dark Matter
Prizes:
1st
~$3,000.00
The prize will be an expenses paid trip to the Jet Propulsion Laboratory (JPL) in Pasadena, California to attend the GREAT10 challenge workshop "Image Analysis for Cosmology".
Objective:
“Participants are provided with 100,000 galaxy and star pairs. A participant should provide an estimate for the ellipticity for each galaxy.”

50. dunnhumby's Shopper Challenge
Prizes:
1st
$6,000.00
2nd
$3,000.00
3rd
$1,000.00
Objective:
Predict the next date that the customer will make a purchase
AND
Predict the amount of the purchase to within £10.00

51. Data Provided For 100,000 customers: April 1, 2010 – June 19, 2011
customer_id
visit_date
visit_spend
For 10,000 customers: April 1, 2010 – March 31, 2011

52. Really two different challenges:
1) Predict next purchase date
Max of ~42.73% obtained
2) Predict purchase amount to within £10.00
Max of ~38.99% obtained
If independent 42.73% * 38.99% = 16.66%
In reality – max obtained was 18.83%

53. dunnhumby's Shopper Challenge
Packages Used & Concepts Explored:
1st competition with real dates
zoo
arima
forecast
SVD
svd
irlba

54. SVD Singular value decomposition

55. X X = U D V Original Matrix 807 x 1209 T 1st Most Important
2nd Most Important
3rd Most Important
4th Most Important
. . .
Nth Most Important 1 2 3 4 … N
N x N
1st
2nd
3rd
4th
. . .
Nth
N x N X X =
Row Features
Column Features
Col 1
Col 2
Col 3
Col 4 …
Col N
Row 1
Row 2
Row 3
Row 4
Row N

56. X X ~ U D V Original Matrix 807 x 1209
1st Most Important 1
1st X X ~
x <- read.jpeg("test.image.2.jpg")
im <- imagematrix(x, type = "grey")
im.svd <- svd(im)
u <- im.svd$u
d <- diag(im.svd$d)
v <- im.svd$v

57. X X ~ U D V Original Matrix 807 x 1209 new.u <- as.matrix(u[, 1:1])
1st Most Important 1
1st X X
new.u <- as.matrix(u[, 1:1])
new.d <- as.matrix(d[1:1, 1:1])
new.v <- as.matrix(v[, 1:1])
new.mat <- new.u %*% new.d %*% t(new.v)
new.im <- imagematrix(new.mat, type = "grey")
plot(new.im, useRaster = TRUE) ~

58. U D V T
Original Matrix
807 x 1209
1st Most Important 1
1st X X ~

59. X X ~ U D V Original Matrix 807 x 1209 T 1 1st 2 2nd
1st Most Important
2nd Most Important 1 2
1st
2nd X X ~

60. X X ~ U D V Original Matrix 807 x 1209 T 1 1st 2 2nd 3 3rd
1st Most Important
2nd Most Important
3rd Most Important 1 2 3
1st
2nd
3rd X X ~

61. X X ~ U D V Original Matrix 807 x 1209 T 1st Most Important
2nd Most Important
3rd Most Important
4th Most Important 1 2 3 4
1st
2nd
3rd
4th X X ~

62. X X ~ U D V Original Matrix 807 x 1209 T 1st Most Important
2nd Most Important
3rd Most Important
4th Most Important
5th Most Important 1 2 3 4 5
1st
2nd
3rd
4th
5th X X ~

63. X X ~ U D V Original Matrix 807 x 1209 T 1st Most Important
2nd Most Important
3rd Most Important
4th Most Important
5th Most Important
6th Most Important 1 2 3 4 5 6
1st
2nd
3rd
4th
5th
6th X X ~

64. X X ~ U D V Original Matrix 807 x 1209 T 1st Most Important
2nd Most Important
3rd Most Important
4th Most Important …
807th Most Important 1 2 3 4 .
807
1st
2nd
3rd
4th …
807th X X ~

65. X X = U D V Original Matrix 100,000 x 365 100,000 x 365 T
1st Most Important
2nd Most Important
3rd Most Important
4th Most Important
. . .
Nth Most Important 1 2 3 4 … N
365x365
1st
2nd
3rd
4th
. . .
Nth
365 x
365 X X =
Customer Features
Day Features
Day 1
Day 2
Day 3
Day 4 …
Day N
Cust 1
Cust 2
Cust 3
Cust 4
Cust 5
100,000 x
365

66. D
Original Matrix
100,000 x 365 1 2 3 4 … N
365x365

67. = U[,1] = 100,000 x 1 V = 365 x 1 = Original Matrix 100,000 x 365
1st Most Important V T
= 365 x 1 = =
1st

68. V T
= 365 x 1 [first 28 shown]=
1st

69. V T
= 365 x 1 [first 28 shown]=
2nd

70. V T
= 365 x 1 [first 28 shown]=
3rd

71. V T
= 365 x 1 [first 28 shown]=
4th

72. V T
= 365 x 1 [first 28 shown]=
5th

73. V T
= 365 x 1 [first 28 shown]=
6th

74. V T
= 365 x 1 [first 28 shown]=
7th

75. V T
= 365 x 1 [all 365 shown]=
8th

76. Online Product Sales Prizes: 1st $15,000.00 2nd $ 5,000.00 3rd
$ 2,500.00
Objective:
“[P]redict monthly online sales of a product. Imagine the products are online  self-help programs following an initial advertising campaign.”

77. Packages/Concepts Explored:
Online Product Sales
Packages/Concepts Explored:
Data analysis – looking at data closely
gbm
Teams

78. Looking at data closely
Online Product Sales
Looking at data closely
Cat_1=0
Cat_1=1
6274 1
6532
6661
7696
7701
8229
8412
8895
9596
9772

79. On the public leaderboard:
Online Product Sales
On the public leaderboard:

80. On the private leaderboard:
Online Product Sales
On the private leaderboard:

81. Thank You!
Questions?

82. Extra Slides

83. R Code for Dunnhumby Time Series

87. X X = U D V > my.svd <- svd(iris[,1:4]) > objects(my.svd)
[1] "d" "u" "v"
> my.svd$d
[1]
> dim(my.svd$u)
[1] 150 4
> dim(my.svd$v)
[1] 4 4