1.  2003, G.Tecuci, Learning Agents Laboratory 1 Learning Agents Laboratory Computer Science Department George Mason University Prof. Gheorghe Tecuci 9 Instance-Based Learning

2.  2003, G.Tecuci, Learning Agents Laboratory 2 Overview Exemplar-based representation of concepts The k-nearest neighbor algorithm Discussion Recommended reading Lazy Learning versus Eager Learning

3.  2003, G.Tecuci, Learning Agents Laboratory 3 Concepts representation Let us consider a set of concepts C = {c 1, c 2,..., c n }, covering a universe of instances I. Each concept c i represents a subset of I. How is a concept usually represented? How does one test whether an object ‘a’ is an instance of a concept “c 1 ”?

4.  2003, G.Tecuci, Learning Agents Laboratory 4 Intentional representation of concepts How could we represent a concept extensionally? How could we represent a concept extensionally, without specifying all its instances? Usually, a concept is represented intentionally by a description covering the positive examples of the concept and not covering the negative examples. How is a concept usually represented? How does one test whether an object ‘a’ is an instance of a concept “c i ”? The set of instances represented by a concept c i is the set of instances of the description of ci. Therefore, testing if an object a is an instance of a concept c i reduces to testing if the description of c i is more general than the description of a.

5.  2003, G.Tecuci, Learning Agents Laboratory 5 Exemplar based representation of concepts A concept c i may be represented extensionally by: - a collection of examples c i = {e i1, e i2,...}, - a similarity estimation function f, and - a threshold value . How could a concept c i be generalized in this representation? An instance ‘a’ belongs to the concept c i if ‘a’ is similar to an element e ij of c i, and this similarity is greater than , that is, f(e ij, c i ) > .

6.  2003, G.Tecuci, Learning Agents Laboratory 6 Generalization in exemplar based representations Generalizing the concept c i may be achieved by: - adding a new exemplar; - decreasing . How could a concept c i be generalized in this representation? Why are these generalization operations? Is there an alternative to considering the threshold value  for classification of an instance?

7.  2003, G.Tecuci, Learning Agents Laboratory 7 Prediction with exemplar based representations Let us consider a set of concepts C = {c 1, c 2,..., c n }, covering a universe of instances I. Each concept c i is represented extensionally as a collection of examples c i = {e i1, e i2,...}. Let ‘a’ be an instance to classify. How to decide to which concept does ‘a’ belong? Different answers to this question lead to different learning methods.

8.  2003, G.Tecuci, Learning Agents Laboratory 8 Let ‘a’ be an instance to classify in one of the classes {c 1, c 2,..., c n }. How to decide to which concept does it belong? Method 1 ‘a’ belongs to the concept c i if ‘a’ is similar to an element e ij of c i, and this similarity is greater than the similarity between ‘a’ and any other concept exemplar (1-nearest neighbor). Prediction (cont) What is a potential problem with 1-nearest neighbor? Hint: Think of an exemplar which is not typical.

9.  2003, G.Tecuci, Learning Agents Laboratory 9 Method 2 Consider the k most similar exemplars. ‘a’ belongs to the concept c i that contains most of the k exemplars (k-nearest neighbor). Prediction (cont) How could the problem with method 1 be alleviated? Use more than one example. What is a potential problem with k-nearest neighbor? Hint: Think of the intuition behind instance-based learning.

10.  2003, G.Tecuci, Learning Agents Laboratory 10 Answer 3 Consider the k most similar exemplars, but weight their contribution to the class of ‘a’ by their distance to ‘a’, giving greater weight to the closest neighbors (distance-weighted nearest neighbor). Prediction (cont) How could the problem with method 2 be alleviated? Weight the exemplars.

11.  2003, G.Tecuci, Learning Agents Laboratory 11 Overview Exemplar-based representation of concepts The k-nearest neighbor algorithms Discussion Recommended reading Lazy Learning versus Eager Learning

12.  2003, G.Tecuci, Learning Agents Laboratory 12 The k-nearest neighbor algorithm Training algorithm Each example is represented as a feature-value vector. For each training example (eik Ci) add eik to the exemplars of Ci. Classification algorithm Let ‘a’ be an instance to classify. Find the k most similar exemplars. Assign ‘a’ to the concept that contains the most of the k exemplars. Each example is represented using the feature-vector representation: ei = (a1=vi1, a2=vi2, …, an=vin) The distance between two examples ei and ej is the Euclidean distance: d(ei, ej) = √ Σ (vik - vjk) 2

13.  2003, G.Tecuci, Learning Agents Laboratory 13 Nearest neighbors algorithms: illustration + + + + - - - - - - e1 1-nearest neighbor: the concept represented by e1 5-nearest neighbors: q1 is classified as negative q1

14.  2003, G.Tecuci, Learning Agents Laboratory 14 Overview Exemplar based representation of concepts The k-nearest neighbor algorithms Discussion Recommended reading Lazy Learning versus Eager Learning

15.  2003, G.Tecuci, Learning Agents Laboratory 15 What is the inductive bias of the k-nearest neighbor algorithm? The assumption that the classification of an instance ‘a’ will be most similar to the classification of other instances that are nearby in the Euclidian space. Nearest neighbors algorithms: inductive bias

16.  2003, G.Tecuci, Learning Agents Laboratory 16 Application issues Which are some practical issues in applying the k-nearest neighbor algorithms? Because the algorithm delays all processing until a new classification/prediction is required, significant processing is needed to make the prediction. Because the distance between instances is based on all the attributes, less relevant attributes and even the irrelevant ones are used in the classification of a new instance. How to alleviate these problems? Because the algorithm is based on a distance function, the attribute values should be such that a distance could be computed.

17.  2003, G.Tecuci, Learning Agents Laboratory 17 Application issue: the use of the attributes Weight the contribution of each attribute, based on its relevance. The classification of an example is based on all the attributes, independent of their relevance. Even the irrelevant attributes are used. How to alleviate this problem? How to determine the relevance of an attribute? Use an approach similar to cross-validation. How?

18.  2003, G.Tecuci, Learning Agents Laboratory 18 Tress where the leaves are exemplars, nearby exemplars are stored at nearby nodes, and internal nodes sort the query to the relevant leaf by testing selected attributes. Application issue: processing for classification Because the algorithm delays all processing until a new classification/prediction is required, significant processing is needed to make the prediction. How to alleviate this problem? Use complex indexing techniques to facilitate the identification of the nearest neighbors at some additional cost in memory. How?

19.  2003, G.Tecuci, Learning Agents Laboratory 19 Instance-based learning: discussion Which are the advantages of the instance-based learning algorithms? Which are the disadvantages of the instance-based learning algorithms?

20.  2003, G.Tecuci, Learning Agents Laboratory 20 Instance-based learning: advantages Model complex concept descriptions using simpler example descriptions. Information present in the training examples is never lost, because the examples themselves are stored explicitly.

21.  2003, G.Tecuci, Learning Agents Laboratory 21 Instance-based learning: disadvantages Efficiency of labeling new instances is low, because all processing is done at prediction time. It is difficult to determine an appropriate distance function, especially when examples are represented as complex symbolic expressions. Irrelevant features have a negative impact of on the distance metric.

22.  2003, G.Tecuci, Learning Agents Laboratory 22 Lazy Learning versus Eager Learning Lazy learning Defer the decision of how to generalize beyond the training data until each new query instance is encountered. Eager learning Generalizes beyond the training data before observing the new query, committing at the training time to the learned concept. Lazy learners require less computation time for training and more for prediction. How do the two types of learning compare in terms of computation time?

23.  2003, G.Tecuci, Learning Agents Laboratory 23 Exercise Suggest a lazy version of the eager decision tree learning algorithm ID3. What are the advantages and disadvantages of your lazy algorithm compared to the original eager algorithm?

24.  2003, G.Tecuci, Learning Agents Laboratory 24 Recommended reading Mitchell T.M., Machine Learning, Chapter 8: Instance-based learning, pp. 230 - 248, McGraw Hill, 1997. Kibler D, Aha D., Learning Representative Exemplars of Concepts: An Initial Case Study, in J.W.Shavlik, T.G.Dietterich (eds), Readings in Machine Learning, Morgan Kaufmann, 1990.