1. Data Mining (and machine learning)
DM Lecture 6: Confusion, Correlation and Regression
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

2. Today A note on ‘accuracy’: Confusion matrices
A bit more basic statistics: Correlation
Undestanding whether two fields of the data are related
Can you predict one from the other?
Or is there some underlying cause that affects both?
Basic Regression:
Very, very often used
Given that there is a correlation between A and B (e.g. hours of study and performance in exams; height and weight ; radon levels and cancer, etc … this is used to predict B from A.
Linear Regression (predict value of B from value of A)
But be careful …
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

3. 80% ‘accuracy’ ...
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

4. 80% ‘accuracy’ ...
… means our classifier predicts the correct class 80% of the time, right?
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

5. 80% ‘accuracy’ ... Could be with this confusion matrix: True \ Pred A10 5 15
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

6. 80% ‘accuracy’ ... Could be with this confusion matrix: Or this one
True \ Pred A B C 80 10 5 15
True \ Pred A B C
100 60 40
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

7. 80% ‘accuracy’ ... Could be with this confusion matrix: Or this one
True \ Pred A B C 80 10 5 15
80%
True \ Pred A B C
100 60 40
100%
60%
Individual class accuracies
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

8. 80% ‘accuracy’ ... Could be with this confusion matrix: Or this one
True \ Pred A B C 80 10 5 15
80%
True \ Pred A B C
100 60 40
100%
60%
86.7%
Individual class accuracies
Mean class accuracy
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

9. Another example / unbalanced classes
True \ Pred A B C
100
400
300
500
1500
8000
100%
30%
80%
Overall accuracy: ( ) /  75.7%
Individual class accuracies
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

10. Another example / unbalanced classes
True \ Pred A B C
100
400
300
500
1500
8000
100%
30%
80%
70%
Overall accuracy: ( ) /  75.7%
Individual class accuracies
Mean class accuracy
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

11. Yet another example / unbalanced classes
True \ Pred A B C
100
4000
2000
100%
20%
73.3%
Overall accuracy: ( ) /  21.6%
Individual class accuracies
Mean class accuracy
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

12. Another example / unbalanced classes
True \ Pred A B C
100
400
300
500
1500
8000
100%
30%
80%
70%
Overall accuracy: ( ) /  75.7%
Individual class accuracies
Mean class accuracy
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

13. Another example / unbalanced classes
True \ Pred A B C
100
400
300
500
1500
8000
100%
30%
80%
70%
Overall accuracy: ( ) /  75.7%
Individual class accuracies
Mean class accuracy
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

14. Correlation Are these two things correlated?
Phone use (hrs)
Life expectancy 1 84 2 78 3 91 4 79 5 69 6 80 7 76 8 9 75 10 70
Are these two things
correlated?
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

15. Correlation
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

16. What about these (web credit)
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

17. What about these (web credit)
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

18. Correlation Measures
It is easy to calculate a number that tells you how well two
things are correlated. The most common is “Pearson’s r”
The r measure is:
r = 1 for perfectly positively correlated data
(as A increases, B increases, and the line
exactly fits the points)
r = -1 for perfectly negative correlation
(as A increases, B decreases, and the line
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

19. Correlation Measures
r = No correlation – there seems to me not the slightest
hint of any relationship between A and B
More general and usual values of r:
if r >= 0.9 (r <= -0.9) a `strong’ correlation
else if r >= 0.65 (r <= -0.65) -- a moderate correlation
else if r >= 0.2 (r <= -0.2) -- a weak correlation,
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

20. Calculating r You will remember the Sample standard deviation,
when you have a sample of n different values
whose mean is
Sample Std is square root of
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

21. Calculating r If we have pairs of (x,y) values, Pearson’s r is:
Interpretation of this should be obvious (?)
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

22. Correlation (Pearson’s R) and covariance
As we just saw, this is Pearson’s R
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

23. Correlation (Pearson’s R) and covariance
And this is the covariance between x and y
often called cov(x,y)
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

24. R = −0.556 cov(x,y) = − 4.35 r = 0.772 cov(x,y) = 6.15 R = −0.556
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

25. So, we calculate r, we think there is a correlation, what next?
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

26. We find the `best’ line through the data
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

27. Linear Regression AKA simple regression, basic regression, etc …
Linear regression simply means: find the `best’ line through the data.
What could `best line’ mean?
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

28. Best line means …
We want the “closest” line, which minimizes the error. I.e. it minimizes the degree to which the actual points stray from the line . For any given point, its distance from the line is called a residual.
We assume that there really is a linear relationship ( e.g. for a top long distance runner, time = miles x 5 minutes), but we expect that there is also random `noise’ which moves points away from the line (e.g. different weather conditions, different physical form, when collecting the different data points). We want this noise (deviations from line) to be as small as possible.
We also want the noise to be unrelated to A (as in predicting B from A) – in other words, we want the correlation between A and the noise to be 0.
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

29. Noise/Residuals: the regression line
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

30. Noise/Residuals: the residuals
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

31. Same again, with key bits:
We want the line, which minimizes the sum of the (squared) residuals.
And we want the residuals themselves to have zero correlation with the variable on the x axis
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

32. Calculating the regression line
We get both of those things like this:
First, recall that any line through (x,y) points can be represented by:
where m is the gradient, and c is where it crosses the y-axis at x=0
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

33. Calculating the regression line
To get m, we can work out Pearson’s r, and then we calculate:
to get c, we just do this:
Job done
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

34. Now we can: Try to gain some insight from m – e.g.
“since m = −4, this suggests that every extra hour of watching TV leads to a reduction in IQ of 4 points”
“since m = 0.05, it seems that an extra hour per day of mobile phone use leads to a 5% increase in likelihood to develop brain tumours”.
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

35. BUT
Be careful --
It is easy to calculate the regression line, but always remember what the r value actually is, since this gives an idea of how accurate is the assumption that the relationship is really linear.
So, the regression line might suggest:
“… extra hour per day of mobile phone use leads to a 5% increase in likelihood to develop brain tumours”
but if r ~ 0.3 (say) we can’t say we are very confident about this. Either not enough data, or the relationship is different from linear
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

36. Prediction
Commonly, we can of course use the regression line for prediction:
So, predicted life expectancy for 3 hrs per day is: fine
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

37. BUT II We cannot extrapolate beyond the data we have!!!!
So, predicted life expectancy for 11.5 hrs per day is: 68 >>>
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

38. BUT II We cannot extrapolate beyond the data we have!!!!
Extrapolation outside the range is invalid
Not allowed, not statistically sound.
Why?
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

39. What would we predict for 8 hrs if we did regression based only on
the first cluster?
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources:

40. So, always be careful
When we look at data like that, we can do piecewise linear
regression instead. What do you think that is?
Often we may think some kind of curve fits best. So there are
things called polynomial regression, logistic regression etc,
which deal with that.
David Corne, and Nick Taylor, Heriot-Watt University -
These slides and related resources: