Computational Learning An intuitive approach

Human Learning Objects in world –Learning by exploration and who knows? Language –informal training, inputs may be incorrect Programming –A couple of examples of loops or recursions Medicine –See one, do one, teach one People: few complex examples, informal, complex behavioral output

Computational Learning Representation provided Simple inputs: vectors of values Simple outputs: e.g. yes or no, a number, a disease Many examples (thousands to millions) Quantifiable + Useful, e.g. automatic generation of expert systems

Concerns Generalization accuracy –Performance on unseen data –Evaluation Noise and overfitting Biases of representation –You only find what you look for.

Three Learning Problems Classification: from known examples create decision procedure to guess class –Patient data -> guess disease Regression: from known examples create decision procedure to guess real numbers –Stock data -> guess price Clustering: putting data into “meaningful” groups –Patient Data -> new diseases

Simple data attribute-value representation = 1 example Sex, age, smoker, etc are the attributes Values are male, 50, true etc Only data of this form allowed.

The Data: squares and circles ? ? ? ?

Linear Decision Boundaries Perceptron

Learning a (hyper)-line Given data Construct line – the decision boundary –Usually defined by a normal n – data is on one side if dot product data * n >0 –Recall * is x1*y1+x2*y2. What a neuron does

SVM: Optimization approach middle line

Decision Trees: rectangular boundaries

1-Nearest Neighbor Piecewise Linear boundaries

1-Nearest Neighbor classification If x is a example, find the nearest neighbor NN in the data using euclidean distance. Guess the class of c is the class of NN K-nearest neighbor: let the k-nearest neighbors vote Renamed as IB-k in Weka

Neural Nets Smooth boundaries

Neural Net A single perceptron can’t learn some simple concepts, like XOR A multilayered network of perceptrons can learn any boolean function Learning is not biological but follows from multivariable calculus

Gedanken experiments Try ML algorithms on imagined data Ex. Concept: x>y, ie. Data looks like 3,1,+. 2,4,-. etc Which algorithms do best? And how well? Consider the boundaries. My guesses: SMO> Perceptron>NearestN>DT.

Check Guesses with Weka 199 examples. DT= 92.9 (called J48 in weka) NN= 97.5 (called IB1 in weka) SVM = 99.0 (called SMO in weka)