1. Artificial Intelligence 9. Perceptron
Japan Advanced Institute of Science and Technology (JAIST)
Yoshimasa Tsuruoka

2. Outline Feature space Perceptrons The averaged perceptron
Lecture slides

3. Feature space
Instances are represented by vectors in a feature space

4. Feature space Instances are represented by vectors in a feature space正例
<Outlook = sunny, Temperature = cool, Humidity = normal> 負例
<Outlook = rain, Temperature = high, Humidity = high>

5. Separating instances with a hyperplane
Find a hyperplane that separates the positive and negative examples

6. Perceptron learning
Can always find such a hyperplane if the given examples are linearly separable

7. Linear classification
Binary classification with a linear model
：　instance
：　feature vector
：　weight vector
bias
If the inner product of the feature vector with the linear
weights is greater than or equal to zero, then it is classified
as a positive example, otherwise it is classified as a negative
example

8. The Perceptron learning algorithm
Initialize the weight vector
Choose an example (randomly) from the training data
If it is not classified correctly,
If it is a positive example
If it is a negative example
Step 2 and 3 are repeated until all examples are correctly classified.

9. Learning the concept OR
Training data
Negative
Positive
Positive
Positive

10. Iteration 1 x1
Wrong!

11. Iteration 2 x4
Wrong!

12. Iteration 3 x2
OK!

13. Iteration 4 x3
OK!

14. Iteration 5 x1
Wrong!

15. Separating hyperplane
Final weight vector t 1
Separating hyperplane s 1
s and t are the input (the second and
the third elements of the feature vector)

16. Why the update rule works
When a positive example has not been correctly classified
This values was too small
Original value
This is always positive
The update rule makes it less likely for the perceptron
to make the same mistake

17. Convergence
The Perceptron training algorithm converges after a finite number of iterations to a hyperplane that perfectly classifies the training data, provided the training examples are linearly separable.
The number of iterations can be very large
The algorithm does not converge if the training data are not linearly separable

18. Learning the PlayTennis concept
Final weight vector
Feature space
11 binary features
Perceptron learning
Converged in 239 steps　
Bias
Outlook = Sunny -3
Outlook = Overcast 5
Outlook = Rain -2
Temperature = Hot
Temperature = Mild 3
Temperature = Cool
Humidity = High -4
Humidity = Normal 4
Wind = Strong
Wind = Weak

19. Averaged Perceptron A variant of the Perceptron learning algorithm
Output the weight vector which is averaged over iterations rather than the final weight vector
Do not wait until convergence
Determine when to stop by observing the performance on the validation set
Practical and widely used

20. Naive Bayes vs Perceptrons
The naive Bayes model assumes conditional independence between features
Adding informative features does not necessarily improve the performance
Percetrons allow one to incorporate diverse types of features
The training takes longer