1. Clustering Techniques
What goes together?

2. DRAFT: Copyright G A Tagliarini, PhD
The Problem
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

3. DRAFT: Copyright G A Tagliarini, PhD
Central Issues
What does “similar” mean? Least squared difference
Maximum pair-wise distance
How many classes “should” there be?
Sometimes the problem will dictate; e.g., classifying letters or numerals
Sometimes there is no clear a priori knowledge; e.g., the operational states of a satellite, airplane
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

4. DRAFT: Copyright G A Tagliarini, PhD
Approaches
K-means
Fuzzy k-means
Nearest neighbor
Kohonen networks
Adaptive resonance theory (ART) networks
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

5. A Representative K-Means Clustering Algorithm
Choose:
A number of classes k
e > 0
Initial class “mean” vectors m1, m2, …, mk Do
Classify entities by “nearest” mean
Update the means based upon the classification
While (Dmi > e)
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

6. K-Means Algorithm: Notes
Initial means
May be chosen randomly
May be selected from the data vectors xi
Cluster discriminant function d
If d(xi, mm) ≤ d(xi, mj) for all j  m, then xi is in cluster m
New means are the averages of the vectors assigned to each cluster
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

7. K-Means Algorithm: A Cluster Assignment Matrix
A matrix U
A row r = 1,…,k for each of the k clusters
A column c = 1,…, n for each of the n data vectors
The entries Urcare each binary valued
Urc= 1, if xc is assigned to cluster r
Urc= 0, if xc is not assigned to cluster r
Each column must sum to 1
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

8. K-Means Algorithm: A Cluster Assignment Matrix Example
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

9. Fuzzy K-Means Algorithm:
A.k.a., fuzzy C-means clustering
Similar to k-means clustering
Different because fuzzy membership grades are used in the cluster assignment matrix
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

10. Fuzzy Set Theory Basics
Conventional set theory
Derives from symbolic, two-valued (T/F) logic
Depends upon binary decisions to determine set membership
Fuzzy set theory
Permits continuous valued grading of set membership
Allows for reasoning subject to imprecision
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

11. Fuzzy Set Theory Basics
Assume:
A domain of discourse X with entities generically denoted by x
A fuzzy set A is a set of ordered pairs
A = { (x, mA(x)) | x e X }
Where mA(x) is the fuzzy membership function (MF) for A and 0 ≤ mA(x) ≤ 1
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

12. Fuzzy Set Theory Basics
A⊆B (A is a subset of B)
If and only if mA(x) ≤ mB(x) ∀ x
C = A⋂B (C is the intersection of A with B)
Where mC(x) = min( mA(x) , mB(x))
C = A⋃B (C is the union of A with B)
Where mC(x) = max( mA(x) , mB(x))
Ac (Ac is the complement of A)
Where mAC (x) = 1 - mA(x)
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

13. Fuzzy Membership Functions and Set Operations Illustrated
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

14. Fuzzy K-Means (FKM) Algorithm
Clusters n data vectors xi
The degree of membership in a cluster is specified by a fuzzy membership grade
Clustering is into fuzzy sets
Cluster centers arise by minimizing a cost function of dissimilarity
The corresponding cluster assignment matrix U may have non binary entries 0≤Urc≤1
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

15. Normalizing The Cluster Assignment Matrix U
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

16. DRAFT: Copyright G A Tagliarini, PhD
A Cost Function
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

17. DRAFT: Copyright G A Tagliarini, PhD
Update Equations
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

18. The Fuzzy K-Means Algorithm
Initialize the cluster assignment matrix subject to FKM Eq. 1 Do
Calculate k fuzzy cluster centers using FKM Eq. 3
Compute the cost function using FKM Eq. 2
Compute a new cluster assignment matrix using FKM Eq. 4
While (cost is decreasing “significantly”)
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

19. Nearest Neighbor Algorithm
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

20. DRAFT: Copyright G A Tagliarini, PhD
Kohonen Networks
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

21. Adaptive Resonance Theory (ART)
Gail Carpenter and Stephen Grossberg
Center for Adaptive Systems, Boston University
ART 2: Self-organization of stable category recognition codes for analog input patterns
Applied Optics, Vol. 26, No. 23, pp , 1 December 1987.
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

22. Design Goals and Trade-Offs
Stability-Plasticity trade-off
Search-Direct Access trade-off
Match-Reset trade-off
STM invariance under readout of matched LTM
LTM readout and STM normalization coexist
No LTM recoding by superset inputs
Stable choice until reset
Contrast enhancement, noise suppression, and mismatch attenuation
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

23. DRAFT: Copyright G A Tagliarini, PhD
System Architecture
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

24. System Equations: F1-Layer
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

25. System Equations: F2-Layer
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

26. System Equations: Vigilance
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD

27. System Equations: Long Term Memory
1/13/2019
DRAFT: Copyright G A Tagliarini, PhD