Classification Problem Class F1 F2 F3 F The objects to be classified are flowers The two classes are: 1) Was pollinated by a honey-bee 2) Was not pollinated by a honey-bee The biologist measured four features: F1: The longitude of the plant F2: The latitude of the plant F3: The height of the flower from ground F4: The diameter of the flower.

This plot is F Class F1 F2 F3 F

This plot is F1 vs F Class F1 F2 F3 F

This plot is F3 vs F4 Class F1 F2 F3 F

This plot is F1 vs F2 Class F1 F2 F3 F

F1 F

Which algorithms would work well on this dataset.

F2 < -1.1? Blue class (1)? YN

F2 < -1.1? Blue class (1)F1 < 0.98 YN

Class F1 F2 F3 F Problem: You are given a problem with four features, but you are not told which features (if any) are useful for classification. How can you figure out which are useful? You could try plotting all pairs, and visual inspection.. For four features that is only 6 combinations For forty features it is 780 combinations However it might not be a pair of features that is best, it could be a subset of one, two, three… For forty features there are (over a trillion) combinations.

1234 3,42,41,42,31,31,2 2,3,41,3,41,2,41,2,3 1,2,3,4 50%

1234 3,42,41,42,31,31,2 2,3,41,3,41,2,41,2,3 1,2,3,4 62%61%51%52%

1234 3,42,41,42,31,31,2 2,3,41,3,41,2,41,2,3 1,2,3,4 60% 58% 97%

1234 3,42,41,42,31,31,2 2,3,41,3,41,2,41,2,3 1,2,3,4 87% 88%

1234 3,42,41,42,31,31,2 2,3,41,3,41,2,41,2,3 1,2,3,4 72%

,42,4 1,4 2,31,31,2 2,3,4 1,3,41,2,4 1,2,3 1,2,3, Class F1 F2 F3 F F1 and F2

1234 3,42,41,42,31,31,2 2,3,41,3,41,2,41,2,3 1,2,3,4 One nearest neighbor cross validation Cross-validation accuracy Data Binary vector indicating features to use [ ] How I would solve this problem… Ignore the search part for now Just get the cross validation working

Extra Credit Possibility Also do search on a UCI dataset, and explain why the result is intuitive. Don’t do this, unless the rest of your project is perfect!

School Employees Simpson's Family What is a natural grouping among these objects?

We can look at the dendrogram to determine the “correct” number of clusters. In this case, the two highly separated subtrees are highly suggestive of two clusters. (Things are rarely this clear cut, unfortunately)

Outlier One potential use of a dendrogram is to detect outliers The single isolated branch is suggestive of a data point that is very different to all others

Chimpanzee Pygmy Chimp Human Gorilla Orangutan Sumatran Orangutan Gibbon Hellenic Armenian Persian Hellenic Armenian “Armenian borrowed so many words from Iranian languages that it was at first considered a branch of the Indo- Iranian languages, and was not recognized as an independent branch of the Indo-European languages for many decades” Do Trees Make Sense for non-Biological Objects? The answer is “Yes”. There are increasing theoretical and empirical results to suggest that phylogenetic methods work for cultural artifacts. Does horizontal transmission invalidate cultural phylogenies? Greenhill, Currie & Gray. Branching, blending, and the evolution of cultural similarities and differences among human populations. Collard, Shennan, & Tehrani...results show that trees constructed with Bayesian phylogenetic methods are robust to realistic levels of borrowing

Because trees are powerful in biology They make predictions – Pacific Yew produces taxol which treats some cancers, but it is expensive. Its nearest relative, the European Yew was also found to produce taxol. They tell us the order of events – Which came first, classic geometric spider webs, or messy cobwebs? They tell us about.. – “Homelands”, where did it come from. – “Dates” when did it happen. – Rates of change – Ancestral states Why would we want to use trees for non biological things?

Markus Pudenz Mostly Belgium beers Irish beers Mostly Californian beers Clustered based on crowd sourced user subjective ranking

Piotr Pyotr Petros Pietro Pedro Pierre Piero Peter Peder Peka Peadar Pedro (Portuguese/Spanish) Petros (Greek), Peter (English), Piotr (Polish), Peadar (Irish), Pierre (French), Peder (Danish), Peka (Hawaiian), Pietro (Italian), Piero (Italian Alternative), Petr (Czech), Pyotr (Russian) Where is Pet'Ka from? Where is Bedros from?

How do we know the dates? If we can get dates, even upper/lower bounds, some events, we can interpolate to the rest of the tree. “ Irish/Welsh Split: Must be before 300AD. Archaic Irish inscriptions date back to the 5 th century AD – divergence must have occurred well before this time.” Gray, R.D. and Atkinson, Q. D., Language tree divergence times support the Anatolian theory of Indo-European origin

Family Tree of Languages Has Roots in Anatolia, Biologists Say By NICHOLAS WADE Published: August 23, 2012 NEW York TimesNICHOLAS WADE Biologists using tools developed for drawing evolutionary family trees say that they have solved a longstanding problem in archaeology: the origin of the Indo-European family of languages. The family includes English and most other European languages, as well as Persian, Hindi and many others. Despite the importance of the languages, specialists have long disagreed about their origin..

Partitional Clustering Nonhierarchical, each instance is placed in exactly one of K nonoverlapping clusters. Since only one set of clusters is output, the user normally has to input the desired number of clusters K.

Squared Error Objective Function

Algorithm k-means 1. Decide on a value for k. 2. Initialize the k cluster centers (randomly, if necessary). 3. Decide the class memberships of the N objects by assigning them to the nearest cluster center. 4. Re-estimate the k cluster centers, by assuming the memberships found above are correct. 5. If none of the N objects changed membership in the last iteration, exit. Otherwise goto 3.

K-means Clustering: Step 1 Algorithm: k-means, Distance Metric: Euclidean Distance k1k1 k2k2 k3k3

K-means Clustering: Step 2 Algorithm: k-means, Distance Metric: Euclidean Distance k1k1 k2k2 k3k3

K-means Clustering: Step 3 Algorithm: k-means, Distance Metric: Euclidean Distance k1k1 k2k2 k3k3

K-means Clustering: Step 4 Algorithm: k-means, Distance Metric: Euclidean Distance k1k1 k2k2 k3k3

K-means Clustering: Step 5 Algorithm: k-means, Distance Metric: Euclidean Distance k1k1 k2k2 k3k3

Comments on the K-Means Method Strength – Relatively efficient: O(tkn), where n is # objects, k is # clusters, and t is # iterations. Normally, k, t << n. – Often terminates at a local optimum. The global optimum may be found using techniques such as: deterministic annealing and genetic algorithms Weakness – Applicable only when mean is defined, then what about categorical data? – Need to specify k, the number of clusters, in advance – Unable to handle noisy data and outliers – Not suitable to discover clusters with non-convex shapes

The K-Medoids Clustering Method Find representative objects, called medoids, in clusters PAM (Partitioning Around Medoids, 1987) – starts from an initial set of medoids and iteratively replaces one of the medoids by one of the non-medoids if it improves the total distance of the resulting clustering – PAM works effectively for small data sets, but does not scale well for large data sets

EM Algorithm Initialize K cluster centers Iterate between two steps –Expectation step: assign points to clusters –Maximation step: estimate model parameters

Iteration 1 The cluster means are randomly assigned

Iteration 2

Iteration 5

Iteration 25

Nearest Neighbor Clustering Not to be confused with Nearest Neighbor Classification Items are iteratively merged into the existing clusters that are closest. Incremental Threshold, t, used to determine if items are added to existing clusters or a new cluster is created. What happens if the data is streaming…

10 Threshold t t 1 2

10 New data point arrives… It is within the threshold for cluster 1, so add it to the cluster, and update cluster center

10 New data point arrives… It is not within the threshold for cluster 1, so create a new cluster, and so on Algorithm is highly order dependent… It is difficult to determine t in advance…

How can we tell the right number of clusters? In general, this is a unsolved problem. However there are many approximate methods. In the next few slides we will see an example. For our example, we will use the familiar katydid/grasshopper dataset. However, in this case we are imagining that we do NOT know the class labels. We are only clustering on the X and Y axis values.

When k = 1, the objective function is 873.0

When k = 2, the objective function is 173.1

When k = 3, the objective function is 133.6

0.00E E E E E E E E E E E We can plot the objective function values for k equals 1 to 6… The abrupt change at k = 2, is highly suggestive of two clusters in the data. This technique for determining the number of clusters is known as “knee finding” or “elbow finding”. Note that the results are not always as clear cut as in this toy example k Objective Function