Presentation is loading. Please wait.

Presentation is loading. Please wait.

Natural Language Processing COMPSCI 423/723 Rohit Kate 1.

Similar presentations

Presentation on theme: "Natural Language Processing COMPSCI 423/723 Rohit Kate 1."— Presentation transcript:

1 Natural Language Processing COMPSCI 423/723 Rohit Kate 1

2 Classification for NLP: Naïve Bayes Model Maximum Entropy Model Some of these slides have been adapted from Raymond Mooney’s slides from his NLP and Machine Learning courses at UT Austin. Referenes: - Sections 6.6 & 6.7 from Jurafsy & Martin book -Naïve Bayes portions from Word Sense Disambiguation chapters in Jurafsy & Martin and Manning & Schutze books -Generative and Discriminative Classifiers: Naive Bayes and Logistic Regression by Tom Mitchell and Discriminative Classifiers: Naive Bayes and Logistic Regression -A Maximum Entropy approach to Natural Language ProcessingA Maximum Entropy approach to Natural Language Processing by Adam L. Berger, Stephen A. Della Pietra and Vincent J. Della Pietra Computational Linguistics, Vol. 22, No. 1. (1996), pp

3 Naïve Bayes Model 3

4 Classification in NLP Several NLP problems can be formulated as classification problems, a few examples: –Information Extraction Given an entity, is it a person name or not? Given two protein names, does the sentence say they interact or not? –Word Sense Disambiguation I am out of money. I am going to the bank. –Document Classification Given a document, which category does it belong to? –Sentiment Analysis Given a passage (e.g. product or movie review), is it saying positive things or negative things? –Textual Entailment Given two sentences, can the second sentence be inferred from the first? 4

5 Classification Usually the classification output variable is denoted by Y and the input variables by Xs Y: {river bank, money bank, verb bank} X1: Previous word X2: Next word X3: Part-of-speech of previous word X4: Part-of-speech of next word Xs are usually called features in NLP Coming up with good feature sets for NLP problems is a skill: feature engineering –Requires linguistic insights –Grasp of the theory behind the classification method 5

6 Probabilistic Classification Often it is useful to know the probabilities of different output values and not just the best output value –To have confidence in the output ( vs ) –These probabilities may be useful for the next stage of NLP processing Conditional probability: P(Y|X1,X2,..,Xn) 6

7 Probabilistic Classification If the joint probability distribution P(Y,X1,X2,..,Xn) is given then the conditional probability distribution can be easily estimated 7

8 Estimating Conditional Probabilities X1,X2,YP(Y,X1,X2) Circle, Red, Positive0.2 Circle, Red, Negative0.05 Circle, Blue, Positive0.02 Circle, Blue, Negative0.2 Square, Red, Positive0.02 Square, Red, Negative0.3 Square, Blue, Positive0.01 Square, Blue, Negative0.2 Similarly estimate P(Y|X1,X2) for the remaining values. 8

9 Estimating Joint Probability Distributions Not Easy :-( Assuming Y and all X i are binary, we need 2 n entries (parameters) to specify the joint probability distribution This is impossible to accurately estimate from a reasonably-sized training set Note that P(Y|X1,X2,..,Xn) requires fewer entries (2 n -1), why? But they are still too many for even small size of n 9

10 Estimating Joint Probability Distributions Simplification assumptions are made about the joint probability distribution to reduce the number of parameters to estimate Let the random variables be nodes of a graph, there are two major types of simplifications, they are represented as –Directed probabilistic graphical models Simplest: Naïve Bayes model More complex: Hidden Markov Model (HMM) –Undirected probabilistic graphical models Simplest: Maximum entropy model More complex: Conditional Random Field (CRFs) 10

11 Directed Graphical Models Simplification assumption: Some random variables are conditionally independent of others given values for some other random variables 11

12 Conditional Independence Two random variables A and B are conditionally independent given C if P(AПB|C) = P(A|C)P(B|C) Rain and Thunder are not independent (given there was Rain, it increases the probability that there was Thunder). But given that there was Lightning (or no Lighting) they are independent. P(Rain^Thunder|Lightning) = P(Rain|Lightning)P(Thunder|Lightning) 12

13 Directed Graphical Models Also known as Bayesian networks Simplification assumption: Some random variables are conditionally independent of others given values for some other random variables Simplest directed graphical model: Naïve Bayes Naïve Bayes assumption: The features are conditionally independent given the category 13

14 Naïve Bayes Assumption Features are conditionally independent given the category How do we estimate P(Y|X1,X2,..,Xn) from this? Recall the Bayes’ theorem: Lets us calculate P(B|A) in terms of P(A|B) 14

15 Bayes’ Theorem Simple proof from definition of conditional probability: QED: (Def. cond. prob.) 15

16 Naïve Bayes Model From Bayes’ Theorem Naïve Bayes assumption Computing marginals and definition of conditional probability 16

17 Naïve Bayes Model Only need to estimate P(Y) and P(Xi|Y) for all i, that with the naïve Bayes assumption specifies the entire joint probability distribution Assuming all Y and Xis are binary, only 2n+1 parameters instead of 2 n+1 -1 parameters: a dramatic reduction Y P(Y) X1 P(X1|Y) X2 P(X2|Y) X3 ……. P(X3|Y) Xn P(Xn|Y) Directed graphical model representation Lightning RainThunder 17

18 Naïve Bayes Example Probabilitypositivenegative P(Y)0.5 P(small | Y)0.4 P(medium | Y) P(large | Y) P(red | Y) P(blue | Y) P(green | Y) P(square | Y) P(triangle | Y) P(circle | Y) Test Instance: P(Label|Size,Color,Shape) 18

19 Naïve Bayes Example Probabilitypositivenegative P(Y)0.5 P(medium | Y) P(red | Y) P(circle | Y) P(positive |medium,red,circle) = P(positive)*P(medium | positive)*P(red | positive)*P(circle | positive) / P(medium,red,cirlce) 0.5 * 0.1 * 0.9 * 0.9 = / P(medium,red,circle) P(negative |medium,red,circle) = P(negative)*P(medium | negative)*P(red | negative)*P(circle | negative) / P(medium,red,cirlce) 0.5 * 0.2 * 0.3 * 0.3 = / P(medium,red,circle) = / = = / = Test Instance: 19

20 Estimating Probabilities Normally, probabilities are estimated based on observed frequencies in the training data. If D contains n k examples in category y k, and n ijk of these n k examples have the jth value for feature X i, x ij, then: However, estimating such probabilities from small training sets is error-prone. If due only to chance, a rare feature, X i, is always false in the training data,  y k :P(X i =true | Y=y k ) = 0. If X i =true then occurs in a test example, X, the result is that  y k : P(X | Y=y k ) = 0 and  y k : P(Y=y k | X) = 0 20

21 Probability Estimation Example Probabilitypositivenegative P(Y)0.5 P(small | Y)0.5 P(medium | Y)0.0 P(large | Y)0.5 P(red | Y) P(blue | Y) P(green | Y)0.0 P(square | Y)0.0 P(triangle | Y) P(circle | Y) ExSizeColorShapeCategor y 1smallredcirclepositive 2largeredcirclepositive 3smallredtrianglenegitive 4largebluecirclenegitive Test Instance X: P(positive | X) = 0.5 * 0.0 * 1.0 * 1.0 / P(X) = 0 P(negative | X) = 0.5 * 0.0 * 0.5 * 0.5 / P(X) = 0 21

22 Smoothing To account for estimation from small samples, probability estimates are adjusted or smoothed. Laplace smoothing using an m-estimate assumes that each feature is given a prior probability, p, that is assumed to have been previously observed in a “virtual” sample of size m. For binary features, p is simply assumed to be

23 Laplace Smothing Example Assume training set contains 10 positive examples: –4: small –0: medium –6: large Estimate parameters as follows (if m=1, p=1/3) –P(small | positive) = (4 + 1/3) / (10 + 1) = –P(medium | positive) = (0 + 1/3) / (10 + 1) = 0.03 –P(large | positive) = (6 + 1/3) / (10 + 1) = –P(small or medium or large | positive) =

24 Naïve Bayes Model is a Generative Model Models the joint probability distribution P(Y,X1,X2,..,Xn) using P(Y) and P(Xi|Y) An assumed generative process: First generate Y according to P(Y) then generate X1,X2,..,Xn independently according to P(X1|Y), P(X2|Y),.., P(Xn|Y) respectively 24

25 Naïve Bayes Generative Model Size Color Shape Positive Negative pos neg pos neg sm med lg med sm med lg red blue grn circ sqr tri circ sqr tri sm lg med sm lg med lg sm blue red grn blue grn red grn blue circ sqr tri circ sqr circ tri Category 25

26 Naïve Bayes Inference Problem Size Color Shape Positive Negative pos neg pos neg sm med lg med sm med lg red blue grn circ sqr tri circ sqr tri sm lg med sm lg med lg sm blue red grn blue grn red grn blue circ sqr tri circ sqr circ tri Category lg red circ ?? 26

27 Some Comments on Naïve Bayes Model Tends to work well despite strong (or naïve) assumption of conditional independence Experiments show it to be quite competitive with other classification methods on standard UCI datasets Although it does not produce accurate probability estimates when its independence assumptions are violated, it may still pick the correct maximum-probability class in many cases 27

28 Maximum Entropy Model 28

29 Maximum Entropy Models Very popular in NLP Several ways to look at them: –Exponential or log-linear classifiers or multinomial logical regression –Assume a parametric form for conditional distribution –Maximize entropy of the joint distribution given the constraints –Discriminative models instead of generative (directly estimates P(Y|X1,..,Xn) instead of via P(Y,X1,..,Xn)) 29

30 Linear Regression Classification: Predict a discrete values Regression: Predict a real value Linear Regression: Predict a real value using a linear combination of inputs Y = W0 + W1*X1 + W2*X2 + … + Wn*Xn Ws are the weights associated with the features Xs Example: price = *(# vague adjectives) 30

31 Estimating Weights in Linear Regression Find the Ws that minimize the sum-squared error for the given M training examples Statistical packages are available that solve this fast 31

32 Logistic Regression But we are interested in probabilistic classification, that is in predicting P(Y|X1,..,Xn) Can we modify linear regression to do that? –Nothing constrains it to be between [0,1] which is required for a legal probability Predict odds (assume Y is binary) instead of the probability 32

33 Logistic Regression But LHS lies between 0 and infinity, RHS could be between -infinity to infinity Take log of LHS (known as logit function) Logistic function 33

34 Logistic Regression as a Log- Linear Model Logistic regression is basically a linear model, which is demonstrated by taking logs 34

35 Logistic Regression Training Weights are set during training to maximize the conditional data likelihood : where D is the set of training examples and Y d and X i d denote, respectively, the values of Y and X i for example d. Equivalently viewed as maximizing the conditional log likelihood (CLL) 35

36 Logistic Regression Training Use standard gradient descent to find the parameters (weights) that optimize the CLL objective function Many other more advanced training methods are available –Conjugate gradient –Generalized Iterative Scaling (GIS) –Improved Iterative Scaling (IIS) –Limited-memory quasi-Newton (L-BFGS) Packages are available that do these 36

37 Preventing Overfitting in Logistic Regression To prevent overfitting, one can use regularization (smoothing) by penalizing large weights by changing the training objective: This can be shown to be equivalent to assuming a Guassian prior for W with zero mean and a variance related to 1/λ. Where λ is a constant that determines the amount of smoothing 37

38 38 Generative vs. Discriminative Models Generative models and are not directly designed to maximize the performance of classification. They model the complete joint distribution P(Y,X 1,...X n ). But a generative model can also be used to perform any other inference task, e.g. P(X 1 | X 2, …X n, Y) –“Jack of all trades, master of none.” Discriminative models are specifically designed and trained to maximize performance of classification. They only model the conditional distribution P(Y | X 1, …X n ). By focusing on modeling the conditional distribution, they generally perform better on classification than generative models when given a reasonable amount of training data. –Master of one trade: Classification P(Y|X 1,.. Xn) 38

39 Multinomial Logistic Regression (Maximum Entropy or MaxEnt) So far Y was binary, a generalization if Y takes multiple values (classes) Make weights dependent on the class c: W ci instead of W i Normalization term (Z) so that probabilities sum to 1 39

40 Usually features take binary values in NLP Introduce indicator functions (0 or 1 output) that depend on the input and output class Call X as input, features are f i (c,x) Multinomial Logistic Regression (Maximum Entropy or MaxEnt) 40

41 A Small MaxEnt Example Word Sense Disambiguation: Y: {river bank, money bank, verb bank} X: Entire Sentence Features: f 1 (river bank,X) = 1 if “river” is in the sentence, 0 otherwise f 2 (river bank,X) = 1 if “water” is in the sentence, 0 otherwise f 3 (money bank,X) = 1 if “money” is in the sentence, 0 otherwise f 4 (money bank,X) = 1 if “deposit” is in the sentence, 0 otherwise f 5 (verb bank,X) = 1 if previous word was “to”, 0 otherwise Obtain examples of feature values and Y from annotated training data Compute weights W ci to maximize the conditional log- likelihood of the training data For a test example, predict Y using MaxEnt equation 41

42 Why is it Called Maximum Entropy Model? Entropy of a random variable Y: The more uniform distribution, the higher is the entropy It can be shown that standard training for logistic regression gives the distribution with maximum entropy that is empirically consistent with the training data 42

43 Undirected Graphical Model Also called Markov Network, Random Field Undirected graph over a set of random variables, where an edge represents a dependency The Markov blanket of a node, X, in a Markov Net is the set of its neighbors in the graph (nodes that have an edge connecting to X) Every node in a Markov Net is conditionally independent of every other node given its Markov blanket Simplest Markov Network: MaxEnt model 43

44 Relation with Naïve Bayes Y X1X1 X2X2 … XnXn Y X1X1 X2X2 … XnXn Naïve Bayes Logistic Regression Conditional Generative Discriminative 44

45 Simplification Assumption for MaxEnt The probability P(Y|X1..Xn) can be factored as: Note there is no product term that has two or more Xis 45

46 Naïve Bayes and MaxEnt Naïve Bayes can be extended to work with continuous inputs X (like logistic regression) Both make the conditional independence assumption MaxEnt is not rigidly tied with it because it tries to maximize the conditional likelihood of the data even when the data disobeys the assumption It has been observed that with scarce training data Naïve Bayes performs better and with sufficient data MaxEnt performs better 46

47 Classification in General Several other classifiers are also available: perceptron, neural networks, support vector machines, k-nearest neighbors, decision trees… Naïve Bayes and MaxEnt are based on probabilities –Can’t handle combination of features as features –If right features are engineered they work very well Are widely used for tasks other than NLP tasks All this was for one label classification (there was only one Y), extensions to handle multi-label classifications, e.g. sequence labeling with HMMs or CRFs 47

48 HW 2 Write Naïve Bayes (P(Y|f1,f2,f3,f4,f5)) and MaxEnt (P(Y|X)) equations for the example shown on slide #41. 48

49 References for Next Class Chapter 5 (part-of-speech tagging) of Jurafsky & Martin book; Chapter 10 of Manning and Schutze book An Introduction to Conditional Random Fields for Relational Learning By Charles Sutton and Andrew McCallum, Book chapter in Introduction to Statistical Relational Learning. Edited by Lise Getoor and Ben Taskar. MIT Press. 2006An Introduction to Conditional Random Fields for Relational Learning 49

Download ppt "Natural Language Processing COMPSCI 423/723 Rohit Kate 1."

Similar presentations

Ads by Google