1. Cluster analysis and segmentation of customers

2. What is Clustering analysis?
Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense or another) to each other than to those in other groups (clusters).
It is a collection of algorithms
Knowledge Discovery in Databases (KDD) - It is a main task of exploratory data mining, and a common technique for statistical data analysis, used in many fields.
The difference with classification and other techniques is that we do not know a priori the identity of the clusters

3. Why Clustering? A very wide range of marketing research application:
Customer segmentation, profiling etc
Market segmentation based on demographics, consumer behavior or other dimensions
Very useful for the design of recommending systems and search engines

4. Examples
A chain of radio-stores uses cluster analysis for identifying three different customer types with varying needs.
An insurance company is using cluster analysis for classifying customers into segments like the “self confident customer”, “the price conscious customer” etc.
A producer of copying machines succeeds in classifying industrial customers into “satisfied” and “non-satisfied or quarrelling” customers.

5. Dependence and Independence methods
Dependence Methods: We assume that a variable (i.e. Y) depends on (are caused or determined by) other variables (X1, X2 etc.)
Examples: Regression, ANOVA, Discriminant Analysis
Independence Methods: We do not assume that any variable(s) is (are) caused by or determined by others. Basically, we only have X1, X2 ….Xn (but no Y)
Examples: Cluster Analysis, Factor Analysis etc.
When using independence methods we let the data speak for themselves!

6. Input-data 1 Output-data 2 3 4 Cluster: X1 X2 … Xn Obs. 1 Obs. 2
Obs. i
Obs, m
Cluster 1
Cluster 2
Classify
rows
Factor 1
Factor 2
Factor:
X1 X2 X3… Xj…Xn
Obs. 1
Obs. 2 …
Obs. m 3 4
Classify
columns
Relatedness of multivariate methods: cluster analysis and factor analysis

7. Multiple Regression The primary focus is on the variables! Y (Sales)
X1 (Price)
X2 (Price Competitor)
X3 (Adverting)
Obs1
Obs2
Obs3
Obs4
Obs5
Obs6
Obs7
Obs8
Obs9
Obs10 . 95 90 80 85 ..
100 75
The primary focus is on the variables!

8. The primary focus is on the observations!
Cluster analysis X1 X2 X3
Obs1
Obs2
Obs3
Obs4
Obs5
Obs6
Obs7
Obs8
Obs9
Obs10 5 3 2 . 4 1
Cluster 1
Cluster 2
Cluster 3
The primary focus is on the observations!

9. Cluster analysis output: A new cluster-variable with a cluster-number on each respondentX1 X2 X3
Cluster
Obs1
Obs2
Obs3
Obs4
Obs5
Obs6
Obs7
Obs8
Obs9
Obs10 5 3 2 . 4 .. 1

10. Cluster analysis: A cross-tab between the cluster- variable and background + opinions is established
Age
%-Females
Household size
Opinion 1
Opinion 2
Opinion 3 32 31
1.4
3.2
2.1
2.2 44 54
2.9
4.0
3.4
3.3 56 46
2.6
3.0
Core-families with
Traditional values
“Younger male nerds”
“Senior-relaxers”

11. Cluster profiling: (hypothetical)
“Ecological shopper”
Cluster 2:
“Traditional shopper”
Buy ecological food
Advertisements funny
Low price important
1 = Totally Agree

12. Governing principle
Maximization of homogeneity within clusters and simultaneously Maximization of heterogeneity across clusters

13. Clustering Steps
1. Data cleaning and processing 2. Select similarity measurement 3. Select and apply clustering method 4. Interpret the result

14. Data Cleaning and Processing
The data can be numerical, binary, categorical, text, text and numbers, images etc
A typical marketing data set may contain:
How recently the customer has purchased from our company (POS data)
The value of the items purchased (POS data)
The customer’s zip code (derived from loyalty cards, credit cards, manual input at check-outs)
The median house value for that zip code (secondary data found online)
Promo variables (e.g. if the customer redeemed a coupon, rebate or an offer)
Other demographic and geographic data (age, gender, income, country, etc)
Other supplemental survey data yield the customer’s favorite TV shows, Internet sites etc.
Do we need all that?
Can we calculate a new figure and add this to our data set?
GIGO – Garbage In – Garbage Out

15. Similarity
The success of clustering depends on the measurement of similarity (or the opposite)
We need to establish a distance metric.
This depends on the type of the data we have available

16. Euclidean distance (Default in SPSS):Y
(x1, y1)
(x2, y2)
y2-y1
x2-x1 B * A * X
d = (x2-x1)2 + (y2-y1)2
Other distances available in SPSS: City-Block uses  of absolute differences instead of squared differences of coordinates. Moreover: Minkowski distance, Cosine distance, Chebychev distance, Pearson Correlation.

17. Euclidean distance Y B (3, 5) * 5-2 A * (1, 2) 3-1 X

18. Which two pairs of points are to be clustered first?G * A B * * F C * * D * E H * *

19. Maybe A/B and D/E (depending on algorithm!)* A B * * F C * * D * E H * *

20. Quo vadis, C? G * A B * * C * D * E H * *

21. Quo vadis, C? (Continued)G * A B * * C * D * E H * *

22. How does one decide which cluster a “newcoming” point is to join?
Measuring distances from point to clusters or points:
“Farthest neighbour” (complete linkage)
“Nearest neighbour” (single linkage)
“Neighbourhood centre” (average linkage)

23. Quo vadis, C? (Continued)G * A B * *
10,5
8,5
7,0
11,0 C *
8,5 D *
9,0
12,0
9,5 E H * *

24. Complete linkage G * A B * * 10,5 C * D * 9,5 E H * *
Minimize longest distance from cluster to point G * A B * *
10,5 C * D *
9,5 E H * *

25. Average linkage G * A B * * 8,5 C * D * 9,0 E H * *
Minimize average distance from cluster to point G * A B * *
8,5 C * D *
9,0 E H * *

26. Single linkage Minimize shortest distance from cluster to point G * AB * *
7,0 C *
8,5 D * E H * *

27. Single linkage: Pitfall*
A and C merge into the same cluster omitting B! *
Chaining or Snake-like clusters *
Cluster formation begins A C *
All the time the closest observation is put into the existing cluster(s) * B * * * * *

28. Single linkage: Advantage* * * * ** * *
Outliers * * * *
Entropy group * * * *
Good outlier detection and removal procedure in cases with “noisy” data sets

29. Similarity in textual or categorical data
Edit distance
Assume that we want to find the distance between the words Garamond and Gautami. I need to perform 5 changes and 1 deletion. Distance 6. G A R M O N D U T I

30. Similarity in textual or categorical data
Hamming distance
Let’s assume that we want to compare ‘‘GOR234’’ with ‘‘GHT335’’. The distance is 4. G O R 2 3 4 H T 5

31. Single Linkage:
Minimum distance *
Complete Linkage:
Maximum distance *
Average Linkage:
Average distance *
Centroid method:
Distance between centres *
Wards method:
Minimization of
within-cluster variance *

32. Which similarity metric to select?
Average Linkage
Tends to combine clusters with small within-cluster variance. Tends to be biased towards establishing clusters with about the same variance
Single linkage
Often produces big chain like clusters. It does not require metric data
Complete linkage
Highly sensitive to outliers. It does not require metric data
Centroid
Sometimes produces muddled results, but it is less affected from outliers
Median
Appropriate for non-metric data
Ward
Produces small clusters with the same size

33. 2 Non overlapping Overlapping Hierarchical Non-hierarchical 1a 1b 1c
Agglomerative
Divisive

34. Overview of clustering methods (in SPSS)
Non-overlapping
(Exclusive) Methods
Overlapping Methods
Non-hierarchical
Hierarchical
Non-hierarchical/
Partitioning/k-means
- Overlapping k-centroids
Overlapping k-means
Latent class techniques
- Fuzzy clustering
- Q-type Factor analysis (9)
Agglomerative
Divisive
- Sequential threshold
- Parallel threshold
- Neural Networks
- Optimized partitioning (8)
Linkage
Methods
Centroid
Variance
Name in SPSS 1 2 3 4 5 6 7 8 9
Between-groups linkage
Within-groups linkage
Nearest neighbour
Furthest neighbour
Centroid clustering
Median clustering
Ward’s method
K-means cluster
(Factor)
- Centroid (5)
- Median (6)
- Average
- Between (1)
- Within (2)
- Weighted
- Single
- Ordinary (3)
- Density
- Two stage Density
- Complete (4)
- Ward (7)
Overview of clustering methods (in SPSS)

35. Cluster analysis More potential pitfalls & problems:
Do our data at all permit the use of means?
Some methods (i.e. Wards) are biased toward production of clusters with approximately the same number of observations.
Other methods (i. e. Centroid) require data as input that are metric scaled. So, strictly speaking it is not allowable to use this algorithm, when clustering data containing interval scales (Likert- or semantic differential scales).

36. Hierarchical Clustering Example I

37. Steps Let’s assume that each point is a cluster.
We find the closest distance between clusters, and we merge them.
We recalculate the distances (many alternatives here!).
We continue until all points belong to one cluster
We draw the dendrogram and we select were to cut.

38. Dendrogram

39. Clustering cities in Italy

40. Clustering cities in Italy

41. Clustering cities in Italy

42. Clustering cities in Italy

43. Clustering cities in Italy

44. Clustering cities in Italy

45. Hierarchical Clustering Example II

46. Cluster analysis: Small artificial example1
0,68 3
0,42 2 6
0,92 4 5
0,58

47. Cluster analysis: Small artificial example1
0,68 3
0,42 2 6
0,92 4 5
0,58

48. Dendrogram * * * * * * 0,2 0,4 0,6 0,8 1,0 OBS 1 OBS 2 Step 0: OBS 3
Each observation
is treated as a
separate cluster
OBS 3 *
OBS 4 *
OBS 5 *
OBS 6 *
Distance Measure
0, , , , ,0

49. Dendrogram (Continued)
OBS 1 *
Cluster 1
OBS 2 *
Step 1:
Two observations
with smallest
pairwise distances
are clustered
OBS 3 *
OBS 4 *
OBS 5 *
OBS 6 *
0, , , , ,0

50. Dendrogram (Continued)
OBS 1 *
Cluster 1
OBS 2 *
Step 2:
Two other observations
with smallest distances
amongst remaining points/clusters
are clustered
OBS 3 *
OBS 4 *
OBS 5 *
Cluster 2
OBS 6 *
0, , , , ,0

51. Dendrogram (Continued)
OBS 1 *
Cluster 1
OBS 2 *
OBS 3 *
Step 3:
Observation 3 joins
with cluster 1
OBS 4 *
OBS 5 *
Cluster 2
OBS 6 *
0, , , , ,0

52. Dendrogram (Continued)
OBS 1 *
OBS 2 *
“Supercluster”
OBS 3 *
OBS 4 *
Step 4:
Cluster 1 and 2 - from
Step 3 joint into a
“Supercluster”
OBS 5 *
OBS 6 *
0, , , , ,0
A single observation
remains unclustered (Outlier)

53. Hierarchical Clustering Example III

54. Market research for exporting activities
Manager
Risk aversion
Propensity to internationalize Α 2 6 Β 3 9 C 7 1 D 8 E

55. Running hierarchical cluster analysis in SPSS

56. SPSS hierarchical cluster analysis dialogue boxes
Note the above deviations from default settings. For instance, if Squared Euclidean distance is used the clustering will look different compared to the clustering displayed on next slide.
SPSS hierarchical cluster analysis dialogue boxes

57. Output from SPSS between-groups (average) linkage

58. Single linkage B A D E C Cluster 2 Cluster 1 3,162 7,071 4,000 6,083
High B
3,162
Cluster 2 A
7,071
Cluster 1
Propensity to Internationalize (X2)
4,000
6,083 D
2,236 E
5,099 C
Low
Low
Risk aversion (X1)
High

59. Complete linkage B A D E C Cluster 2 Cluster 1 7,071 6,083 High
Propensity to Internationalize (X2)
Cluster 1 D
6,083 E C
Low
Low
Risk aversion (X1)
High

60. Average linkage (between)
High B
Cluster 2 A
7,071
Propensity to Internationalize (X2)
4,000
Cluster 1
6,083 D E
5,099 C
Low
Low
Risk aversion (X1)
High

61. Average linkage (within)
High B
3,162
Cluster 2 A
7,071
Cluster 1
Propensity to Internationalize (X2)
4,000
6,083 D
2,236 E
5,099 C
Low
Low
Risk aversion (X1)
High

62. Wards method B A D E C Cluster 2 Cluster 1 3,162 7,071 4,000 6,083
High B
3,162
Cluster 2 A
7,071
Cluster 1
Propensity to Internationalize (X2)
4,000
6,083 D
2,236 E
5,099 C
Low
Low
Risk aversion (X1)
High

63. Centroid method B A D E C Cluster 2 Cluster 1 5,522 5,500 High
Propensity to Internationalize (X2)
Cluster 1 D
5,500 E C
Low
Low
High

64. “outlier” CL2 CL1 Cluster Method: Hierarchical cluster
Algorithm: Between Groups Linkage
Measure: Squared Euclidean Distance
Plots: Dendrogram flagged

65. Non-hierarchical Clustering
K-means clustering algorithm

66. Concept
At first we select the number of groups and we randomly assign each point to each team.
Next, we revisit the assignments.
The above process is repeated until a criterion is met.
We need a priori to determine the number of clusters
Popular algorithm in this category (non-hierarchical and non- overlapping) is the k-means. Synonyms partitioning and nearest centroid method

67. K-means Algorithm
We select k cluster centers from the points of the data set .
We assign each point to the closest center.
We recalculate the centers.
If the centers remain the same, we stop; Otherwise, we continue with step 2.

68. Example
A study has been made among export managers within twenty industries (pharmaceuticals, heavy machinery, banking etc).
Within each industry, interviews were carried out with 10 to 20 managers. Each manager has asked to comment on 2 statements.

69. Numeric example of k-means cluster analysis1
1.19
2.33 2
1.25
4.50 3
1.17
5.06 4
4.72
5.35 5
1.47
0.95 6
0.82
3.24 7
1.50
2.64 8
1.60
4.86 9
0.51
2.20 10
3.89
2.89 11
2.07
4.53 12
3.06
5.30 13
5.07
4.67 14
1.54
4.35 15
1.71
1.56 16
0.83
5.12 17
2.78
5.27 18
2.73
4.03 19
1.04
4.02 20
5.45
5.40
Note that observations on X1 and X2 are averages on an industrial level. Within each
industry, between 10 and 20 interviews were carried out with managers
Table: Managers’ responses to two statements
Expected market development (0 = Very negative, 6 = Very positive)
Willingness to export (0 = None, 6 = Very high)
Industry X1 X2
(Observation 1)

70. Very
high 4 12 20 17 16 3 8 11 2 19 13 14 18 6
(1,19, 2,33) 10
Willingness to export (X2) 7 1 9 15 5
None
Very
negative
Very
positive

71. Introduction of cluster Seeds (Start of iteration 1)
Very
high 4 12 20 17 16 3 8 11 2 19 13 14 18 6 10 7 1 9 15
Cluster
centres
(seeds) 5
None
Very
negative
Very
positive

72. Measurement of distances
between observations and seeds (average linkage)
Very
high 4 17 12 20 16 3 8
CL2 11 2
5,25 19 13 14 18 6
2,96 7 10 1 9 15
1,41
3,10 5
CL1
None
Very
negative
Very
positive

73. Allocation of observations to closest cluster seed
Very
high 4 17 12 20 16 3 8
CL2 11 2 19 13 14 18 6 7 10 1 9 15 5
CL1
None
Very
negative
Very
positive

74. * * Recalculation of seeds (cluster centroids) for Iteration 2 4 12 3
Very
high 4 12 3 17 16 8 20 11 * 2
(3,72; 4,68) 19 13 14
(1,22; 3,40) 18 6 * 10 7
CL2 1 9 15
Obs. x1 x2 4
4.72
5.35 10
3.89
2.89 11
2.07
4.53 12
3.06
5.30 13
5.07
4.67 17
2.78
5.27 18
2.73
4.03 20
5.45
5.40
Sum
29.77
37.44
Centroid
3.72
4.68 5
CL1
None
Very
negative
Very
positive
CL2
29.77/8 = 3.72

75. Change of seeds between iterations may cause observations to change from one cluster to another
Very
high
(3,49) 20 11 *
1,66
1,50 *
CL2
(3,63) 5
CL1
None
Very
negative
Very
positive

76. * * 20 11 CL2 5 CL1 Very high None Very Very negative positive
Change in centroid coordinates
CL1
CL2
Iteration X1 X2 1
1.47
0.95
5.45
5.4 2
1.22
3.3
3.72
4.68 3
1.28
3.49
3.96
4.7 4

77. Running k-means cluster analysis in SPSS

78. SPSS k-means cluster analysis dialogue boxes

79. Determining the best number of clusters: ESS (Error Sum of Squares)
ESS and number of clusters
Sum = = ESS for 2 clusters
26.42
Reduction in ESS is big between 2 and 3 clusters. So probably 3 clusters comprises the best solution

80. Two clusters
Very
high 4 12 3 17 16 8 20 11 2 19 13 14 18 6 10 7
CL2 1 9 15 5
CL1
None
Very
negative
Very
positive

81. Three clusters
Very
high 17 4 16 3 8 20 12 11 2 19 18 13 14
CL2 6 10
Willingness to export (X2) 7
CL3 1 9 15
CL1 5
None
Very
negative
Very
positive

82. Four clusters
Very
high 3 17 4 8
CL3 20 12 16 11 2 13 19 14
CL4 18 6 10
Willingness to export (X2) 7
CL1 1 9 15 5
CL2
None
Very
negative
Very
positive

83. The principle of replicated clustering
Clustering of numerical example using SPSS default seed and 1 iteration
New clustering of data this time using observations 10 and 16 as initial seeds (1 iteration)
See Table 12.10
Column V
Listing of cluster assignment of observations in both cluster runs
Panel 4: Crosstab of clusterings
Replication 1 versus 2
Replication 1 versus 3
Notice the consistency between the first and the third clustering. Cluster membership number is reversed but the cluster number variable is nominal scaled and therefore the cluster number is arbitrary across runs.
The principle of replicated clustering

84. Non-hierarchical method: Recap
It is an iterative procedure, so – unlike when using the hierarchical methods – there is no guarantee that the optimal solution is found (however, it usually comes quite close)
The analyst must subjectively select the “best” number of clusters (some clues may help, though)
The procedure is fast
Applying (running) is straightforward
It is rather easy to understand (interpret)
However, in applied settings, differences between clusters may be small, thereby disappointing the analyst.

85. Non-hierarchical cluster analysis
How does one determine the optimal number of clusters?
Unfortunately, no formal help (i.e. a built-in option) is presently available in SPSS for determining the best number of clusters.
So: Try 2, 3, and 4 clusters and choose the one that “feels” best. It is recommended to use the ESS-procedure covered above in combination with sound reasoning.

86. Textbooks in Cluster Analysis
Brian S. Everitt
Cluster Analysis for Social Scientists, 1983
Maurice Lorr
Cluster Analysis for Researchers, 1984
Charles Romesburg
Cluster Analysis, 1984
Aldenderfer and Blashfield