1. Mohammad Hosein Mazrouee
CLARA & CLARENS
Faramarz Naderi
Mohammad Hosein Mazrouee
Saeid Sharifzade
Kashan University, Dec. 2017

2. K-Medoids Method C Minimize the sensitivity of k-means to outliers
C Pick actual objects to represent clusters instead of mean values
C Each remaining object is clustered with the representative object (Medoid) to which is the most similar
C The algorithm minimizes the sum of the dissimilarities between each object and its corresponding reference point
‹ E: the sum of absolute error for all objects in the data set
‹ P: the data point in the space representing an object
‹ Oi: is the representative object of cluster Ci

3. K-Medoids Method: The Idea
C Initial representatives are chosen randomly
C The iterative process of replacing representative objects by no representative objects continues as long as the quality of the clustering is improved
C For each representative Object O
‹ For each non-representative object R, swap O and R
C Choose the configuration with the lowest cost
C Cost function is the difference in absolute error-value if a current representative object is replaced by a non-representative object

4. K-Medoids Method: Example9 8 7 6 5 4 3
Data Objects 3 A1 A2 O1 2 6 O2 3 4 O 8 O4 7 O5 O6 O7 O8 O9 5 4 10 1 9 2 6 8 3 7 5 2 3 4 5 6 7
8 9
Goal: create two clusters
Choose randmly two medoids
7 6
O10
O2 = (3,4)
O8 = (7,4)

5. K-Medoids Method: Example9 8 7 6 5 4 3
Data Objects
A1 A2
cluster1 3
cluster2 4 O1 10 2 6 O2 3 4 O 8 O4 7 1 9 2 6 8 3 7 5
O5 O6 O7 O8 O9 O
6 2
6 4
7 3
7 4
8 5
7 6 2 3 4 5 6 7
8 9
‹Assign each object to the closest representative object
‹Using L1 Metric (Manhattan), we form the following clusters
Cluster1 = {O1, O2, O3, O4} 10
Cluster2 = {O5, O6, O7, O8, O9, O10}

6. K-Medoids Method: Example9 8 7 6 5 4 3
Data Objects
A1 A2
cluster1 3
cluster2 4 O1 10 2 6 O2 3 4 O 8 O4 7 O5 O6 O7 O8 O9 5
O10 1 9 2 6 8 3 7 5 2 3 4 5 6 7
8 9
‹Compute the absolute error criterion [for the set of Medoids (O2,O8)]
E | poi | | o1 o2 | | o3 o2 | | o4 o2 |
i1 pCi k
| o5 o8 | | o6 o8 | | o7 o8 | | o9 o8 | | o10 o8 |

7. K-Medoids Method: Example9 8 7 6 5 4 3
Data Objects
A1 A2
cluster1 3
cluster2 4 O1 10 2 6 O2 3 4 O 8 O4 7 O5 O6 O7 O8 O9 5 1 9 2 6 8 3 7 5 2 3 4 5 6 7
8 9
‹The absolute error criterion [for the set of Medoids (O2,O8)]
7 6
O10
E  (3  4  4)  (3 11 2  2)  20

8. K-Medoids Method: Example9 8 7 6 5 4 3
Data Objects
A1 A2
cluster1 3
cluster2 4 O1 10 2 6 O2 3 4 O 8 O4 7 O5 O6 O7 5 1 9 2 6 8 3 7 5 2 3 4 5 6 7
8 9
‹Choose a random object O
‹Swap O8 and O7 7 8 9
O10
7 6
‹Compute the absolute error criterion [for the set of Medoids (O2,O7)]
E  (3  4  4)  (2  2 1 3  3)  22

9. K-Medoids Method: Example9 8 7 6 5 4 3
Data Objects
A1 A2
cluster1 3
cluster2 4 O1 10 2 6 O2 3 4 O 8 O4 7 O5 O6 O7 O8 5 1 9 2 6 8 3 7 5 2
3 4 5 6 7
8 9
‹Compute the cost function
Absolute error [for O2,O7] – Absolute error [O2,O8]
S  22  20 9
O10
7 6
S> 0  it is a bad idea to replace O8 by O7

10. K-Medoids Method: Example9 8 7 6 5 4 3
Data Objects
A1 A2
cluster1 3
cluster2 4 O1 10 2 6 O2 3 4 O 8 O4 7 O5 O6 O7 O8 5 1 9 2 6 8 3 7 5
5 6 7 2
3 4
8 9
C In this example, changing the medoid of cluster 2 did not change the assignments of objects to clusters. 9
O10
7 6
C What are the possible cases when we replace a medoid by another object?

11. K-Medoids Algorithm(PAM)
PAM : Partitioning Around Medoids
C Input
‹ K: the number of clusters
‹ D: a data set containing n objects
C Output: A set of k clusters
C Method:
Arbitrary choose k objects from D as representative objects (seeds)
Repeat
Assign each remaining object to the cluster with the nearest representative object
For each representative object Oj
Randomly select a non representative object Orandom
Compute the total cost S of swapping representative object Oj with Orandom
if S<0 then replace Oj with Orandom
Until no change

12. K-Medoids Properties(k-medoids vs.K-means)
C The complexity of each iteration is O(k(n-k)2)
C For large values of n and k, such computation becomes very costly
C Advantages
‹ K-Medoids method is more robust than k-Means in the presence of noise and outliers
C Disadvantages
‹ K-Medoids is more costly that the k-Means method
‹ Like k-means, k-medoids requires the user to specify k
‹ It does not scale well for large data sets

13. CLARA should closely represent the original data
C CLARA (Clustering Large Applications) uses a sampling-based method to deal with large data sets
PAM
C A random sample
should closely represent the original data
sample
C The chosen medoids will likely be similar
to what would have been chosen from the whole data set

14. CLARA PAM PAM PAM … sample1 sample2 samplem
C Draw multiple samples of the data set
C Apply PAM to each sample
Choose the best clustering
Clusters Clusters
PAM PAM
Clusters
C Return the best clustering
PAM …
sample1
sample2
samplem

15. CLARA - Algorithm
C Set mincost to MAXIMUM;
C Repeat q times // draws q samples
Create S by drawing s objects randomly from D;
Generate the set of medoids M from S by applying the PAM algorithm;
Compute cost(M,D)
If cost(M, D)<mincost
Mincost = cost(M, D);
Bestset = M;
Endif;
C Endrepeat;
C Return Bestset;

16. CLARA Properties finds the best k medoids among the selected samples
C Complexity of each Iteration is: O(ks2 + k(n-k))
‹ s: the size of the sample
‹ k: number of clusters
‹ n: number of objects
C PAM finds the best k medoids among a given data, and CLARA
finds the best k medoids among the selected samples
C Problems
‹ The best k medoids may not be selected during the sampling process, in this case, CLARA will never find the best clustering
‹ If the sampling is biased we cannot have a good clustering

17. CLARANS
C CLARANS (Clustering Large Applications based upon RANdomized Search ) was proposed to improve the quality and the scalability of CLARA
C It combines sampling techniques with PAM
C It does not confine itself to any sample at a given time
C It draws a sample with some randomness in each step of the search

18. CLARANS: The idea Clustering view Current medoids medoids …
Cost=10
Cost=5
Cost=20
Cost=1
Cost=3
Cost=5
Cost=2 …
Keep the current medoids

19. CLARANS: The idea Sample CLARA medoids Current medoids …
C Draws a sample of nodes at the beginning of the search
C Neighbors are from the chosen sample
C Restricts the search to a specific area of the original data
medoids
Current medoids
First step of the search
Neighbors are from the chosen sample
second step of the search
Neighbors are from the chosen sample …

20. CLARANS: The idea Original data CLARANS Current medoids medoids …
C Does not confine the search to a localized area
C Stops the search when a local minimum is found
C Finds several local optimums and output the clustering with the best local optimum
Current
medoids
First step of the search
Draw a random sample of neighbors
medoids
second step of the search
Draw a random sample of neighbors …
The number of neighbors sampled from the original data is specified by the user

21. CLARANS - Algorithm J = 1; // counter of neighbors Repeat
C Set mincost to MAXIMUM;
C For i=1 to h do // find h local optimum
Randomly select a node as the current node C in the graph;
J = 1; // counter of neighbors
Repeat
Randomly select a neighbor N of C;
If Cost(N,D)<Cost(C,D)
Assign N as the current node C;
J = 1;
Else J++;
Endif;
Until J > m
Update mincost with Cost(C,D) if applicableEnd for;
C End For
C Return bestnode;

22. CLARANS Properties C Advantages C Disadvantages
‹ Experiments show that CLARANS is more effective than both PAM and CLARA
‹ Handles outliers
C Disadvantages
‹ The computational complexity of CLARANS is O(n2), where n is the number of objects
‹ The clustering quality depends on the sampling method

23. Comparison with PAM
‹ For large and medium data sets, it is obvious that CLARANS is
much more efficient than PAM
‹ For small data sets, CLARANS outperforms PAM significantly

24. When n=80, CLARANS is 5 times faster than PAM, while the cluster quality is the same.

25. Comparision with CLARA
‹ CLARANS is always able to find clusterings of better quality than
those found by CLARA; CLARANS may use much more time
than CLARA
‹ When the time used is the same, CLARANS is still better than CLARA