Instance Based Learning Bob Durrant School of Computer Science University of Birmingham (Slides: Dr Ata Kabán) 1

Outline Today we learn: K-Nearest Neighbours Case-based reasoning Lazy and eager learning 2

Instance-based learning One way of solving tasks of approximating discrete or real valued target functions Have training examples: (x n, f(x n )), n=1..N. Key idea: – just store the training examples – when a test example is given then find the closest matches 3

“Nearest Neighbours” 1-Nearest neighbour: – given a query instance x q – locate the nearest training example x n – then f(x q ):= f(x n ) K-Nearest neighbour: – given a query instance x q – locate the k nearest training examples – if discrete values target function then take vote among its k nearest neighbours – if real valued target function then take the mean of the f values of the k nearest neighbours: 4

The distance between examples We need a measure of distance in order to know who are the neighbours Assume that we have T attributes for the learning problem. Then one example point x has elements x t  R, t=1,…T. The distance between two points x i and x j is usually defined to be the Euclidean distance: 5

Voronoi Diagram 6

Characteristics of Instance-Based Learning An instance-based learner is a so-called lazy learner which does all the work when the test example is presented. This is as opposed to eager learners, which build a parameterised compact model of the target. It produces local approximation to the target function (different with each test instance) 7

When to consider Nearest Neighbour algorithms? Instances map to points in R n Not more then say 20 attributes per instance Lots of training data Advantages: – Training is very fast – Can learn complex target functions – Don’t lose information Disadvantages: – ? (will see them shortly…) 8

9 two one four three five six seven Eight ?

Training data 10 Test instance

Keep data in normalised form 11 One way to normalise the data a r (x) to a’ r (x) is:

Normalised training data 12 Test instance

Distances of test instance from training data 13 Classification 1-NNYes 3-NNYes 5-NNNo 7-NNNo

What if the target function is real valued? The k-nearest neighbour algorithm would just calculate the mean of the k nearest neighbours 14

Variant of kNN: Distance-Weighted kNN We might want to weight nearer neighbors more heavily: Then it makes sense to use all training examples instead of just k (Stepard’s method) 15

Difficulties with k-nearest neighbour algorithms Have to calculate the distance of the test case from all training cases There may be irrelevant attributes amongst the attributes – curse of dimensionality 16

Case-based reasoning (CBR) CBR is an advanced instance based learning applied to more complex instance objects Objects may include complex structural descriptions of cases & adaptation rules 17

Case-based Reasoning (CBR) CBR cannot use Euclidean distance measures Must define distance measures for those complex objects instead (e.g. semantic nets) CBR tries to model human problem-solving – uses past experience (cases) to solve new problems – retains solutions to new problems CBR is an ongoing area of machine learning research with many applications 18

Applications of CBR Design – landscape, building, mechanical, conceptual design of aircraft sub-systems Planning – repair schedules Diagnosis – medical Adversarial reasoning – legal 19

CBR process 20 New Case matching Matched Cases Retrieve Adapt? No Yes Closest Case Suggest solution Retain Learn Revise Reuse Case Base Knowledge and Adaptation rules

CBR example: Property pricing 21 Test instance

How rules are generated There is no unique way of doing it. Here is one possibility: Examine cases and look for ones that are almost identical – case 1 and case 2 R1: If recep-rooms changes from 2 to 1 then reduce price by £5,000 – case 3 and case 4 R2: If Type changes from semi to terraced then reduce price by £7,000 22

Matching Comparing test instance – matches(5,1) = 3 – matches(5,2) = 3 – matches(5,3) = 2 – matches(5,4) = 1 23 n Estimate price of case 5 is £25,000

Adapting Reverse rule 2 – if type changes from terraced to semi then increase price by £7,000 Apply reversed rule 2 – new estimate of price of property 5 is £32,000 24

Learning So far we have a new case and an estimated price – nothing is added yet to the case base If later we find house sold for £35,000 then the case would be added – could add a new rule if location changes from 8 to 7 increase price by £3,000 25

Problems with CBR How should cases be represented? How should cases be indexed for fast retrieval? How can good adaptation heuristics be developed? When should old cases be removed? 26

Advantages A local approximation is found for each test case Knowledge is in a form understandable to human beings Fast to train 27

Summary K-Nearest Neighbours Case-based reasoning Lazy and eager learning 28

Lazy and Eager Learning Lazy: wait for query before generalizing – k-Nearest Neighbour, Case based reasoning Eager: generalize before seeing query – Radial Basis Function Networks, ID3, … Does it matter? – Eager learner must create global approximation – Lazy learner can create many local approximations 29