1. Text Classification with Belief Augmented Frames Colin Tan Department of Computer Science, School of Computing, National University of Singapore.

2. Outline What are Belief Augmented Frames? Motivation behind Belief Augmented Frames Representing Beliefs in BAFs Some Definitions Belief Augmented Frame Logic (BAF-Logic) Applying BAF-Logic to Text Classification Experiment Protocol and Results Conclusions

3. What are Belief Augmented Frames? Belief Augmented Frames (BAF) combine classical AI frames with belief measures. –Frame-based system to structure knowledge and relations between entities. –Belief measures provide uncertain reasoning on existence of entities and the relationships between them.

4. Motivation behind Belief Augmented Frames Why Belief Measures? –Statistical Measures Standard tool for modeling uncertainty. Essentially, if the probability that a proposition E is true is p, then the probability of that E is false is 1-p. –P(E) = p –P(not E) = 1-p This relationship essentially leaves no room for ignorance. Either the proposition is true with a probability of p, or it is false with a probability of 1-p. This can be counter-intuitive at times.

5. Motivation behind Belief Augmented Frames Why Belief Measures? –[Shortliffe75] cites a study in which, given a set of symptoms, doctors were willing to declare with certainty x that a patient was suffering from a disease D, yet were unwilling to declare with certainty 1-x that the patient was not suffering from D.

6. Motivation behind Belief Augmented Frames Why Belief Measures? –To allow for ignorance our research focuses on belief measures. –The ability to model ignorance is inherent in belief systems. E.g. in Dempster-Shafer Theory [Dempster67], if our belief in E 1 and E 2 are 0.1 and 0.3 respectively, then the ignorance is (1 – (0.1 + 0.3)) = 0.6.

7. Motivation behind Belief Augmented Frames Why Frames? –Frames are a powerful form of representation. Intuitively represents relationships between objects using slot-filler pairs. –Simple to perform reasoning based on relationships. Hierarchical –Can perform generalizations to create general models derived from a set of frames.

8. Example BAF

9. Belief Representation in Belief Augmented Frames Beliefs are represented by two masses: –φ T : Belief mass supporting a proposition. –φ F : Belief mass refuting a proposition. –In general φ T + φ F  1 Room to model ignorance of the facts. Separate belief masses allow us to: –Draw φ T and φ F from different sources. –Have different chains of reasoning for φ T and φ F.

10. Belief Representation in Belief Augmented Frames This ability to derive the refuting masses from different sources and chains of reasoning is unique to BAF. –In Probabilistic Argumentation Systems (the closest competitor to BAF) for example, p(not E) = 1 – p(E).

11. Some Definitions Degree of Inclination –The Degree of Inclination is defined as: DI =  T -  F –DI is in the range of [-1, 1]. –One possible interpretation of DI:

12. Some Definitions Utility Value –The Degree of Inclination DI can be re-mapped to the range [0, 1] through the Utility function: U = (DI + 1) / 2 By normalizing U across all relevant propositions it becomes possible to use U as a statistical measure.

13. Belief Augmented Frame Logic (BAF-Logic) Belief Augmented Frame Logic, or BAF- Logic, is used for reasoning with BAFs. Throughout the remainder of this presentation, we will consider two propositions A and B, with supporting and refuting masses  T A,  F A,  T B, and  F B.

14. Belief Augmented Frame Logic (BAF-Logic) A  B: –  T A  B = min(  T A,  T B ) –  F A  B = max(  F A,  F B ) A  B: –  T A  B = max(  T A,  T B ) –  F A  B = min(  F A,  F B )  A: –  T  A =  F A –  F  A =  T A

15. Belief Augmented Frame Logic (BAF-Logic) BAF-Logic properties that are identical to Propositional Logic: –Associativity, Commutativity, Distributivity, Idempotency, Absorption, De-Morgan’s Theorem,  - elimination. Other properties of Propositional Logic work slightly differently in BAF-Logic. –In particular, some of the properties hold true only if the constituent propositions are at least “probably true” or “probably false” I.e. |DI P |  0.5

16. Belief Augmented Frame Logic (BAF-Logic) An Example: –Given the following propositions in your knowledge base: KB = {(A, 0.7, 0.2), (B, 0.9, 0.1), (C, 0.2, 0.7), (A  B  R,  T ONE,  F ONE, ), (A   B   R,  T ONE,  F ONE )} We want to derive  T R,  F R.

17. Belief Augmented Frame Logic (BAF-Logic) Combining our clauses regarding R, we obtain: –R = (A  B)   (A   B) = A  B  (  A  B) With De-Morgan’s Theorem we can derive  R: –  R=  A   B  (A   B)

18. Belief Augmented Frame Logic (BAF-Logic)  T R = min(  T A,  T B, max(  F A,  T B )) = min(0.7, 0.9, max(0.2, 0.9)) = min(0.7, 0.9, 0.9) = 0.7  F R = max(  F A,  F B, min(  T A,  F B )) = max(0.2, 0.1, min(0.7, 0.1)) = max(0.2, 0.1, 0.1) = 0.2

19. Belief Augmented Frame Logic (BAF-Logic) DI R =  T R -  F R = 0.7 – 0.2 = 0.5 U R = (1 + 0.5) / 2.0 = 0.75 Suppose now it is known that B  C  R

20. Belief Augmented Frame Logic (BAF-Logic) Combining our clauses regarding R, we obtain: –R = (A  B)  (B  C)  (A   B) = A  B  C  (  A  B) With De-Morgan’s Theorem we can derive  R: –  R=  A   B   C  (A   B)

21. Belief Augmented Frame Logic (BAF-Logic)  T R = min(  T A,  T B,  T C, max(  F A,  T B )) = min(0.7, 0.9, 0.2, max(0.2, 0.9)) = min(0.7, 0.9, 0.2, 0.9) = 0.2  F R = max(  F A,  F B,  F C, min(  T A,  F B )) = max(0.2, 0.1, 0.7, min(0.7, 0.1)) = max(0.2, 0.1, 0.7, 0.1) = 0.7

22. Belief Augmented Frame Logic (BAF-Logic) DI R =  T R -  F R = 0.2 – 0.7 = -0.5 U R = (1 - 0.5) / 2.0 = 0.25 Here the new evidence that B  C  R fails to support R, because C is not true (DI C = - 0.5)

23. Text Classification First Approach First Formulation: –Using Individual Word Scores –Assuming that a document d i belongs to a class c k, then for every term t ij the following relation holds: d i  c k  (t i0  c k  t i1  c k  t i2  c k  …  t i,n-1  c k )

24. Text Classification First Approach Likewise, for a document d i not belonging to a class c k, we can derive: d i  c k   m, m  k (t i0  c m  t i1  c m  t i2  c m  …  t i,n-1  c m ) These can be formulated in BAF-Logic:  T di  ck = min(p(c k | t i0 ), p(c k | t i1 ), …, p(c k | t i, n-1 ))  F di  ck = max(min(p(c m | t i0 ), p(c m | t i1 ), …, p(c m | t i, n-1 )), min(p(c n |t i0 ), p(c n |t i1 ),…,p(c n |t i,n-1 )), …)), m, n etc  k

25. Text Classification First Approach The final score of a document d i belong to class c j is given by: Where:

26. Text Classification First Approach Individual term probabilities are derived using Bayesian probabilities:

27. Text Classification Second Approach We classify the entire document using Naïve Bayes assumption: Trivial to derive the supporting score that d i  c k. –It is simply p(c k | d i )

28. Text Classification Second Approach Formulating the Refuting Score is straightforward too: d i  c k  d i  c m  d i  c n ­  d i  c p  …, m, n, p, etc  k We can formulate both supporting and refuting scores in BAF-Logic:

29. Text Classification Second Approach We retain the definitions of DI and U from the first approach.

30. Experiment Protocol Using Andrew McCallum’s “Bag of Words” or BOW library. –Extended “rainbow”, the text-classification front-end, with two BAF classification methods. Methods are called BAF1 and BAF2 –Also extended with two PAS methods (see paper for more details) Methods are called PAS1 and PAS2

31. Experiment Protocol Corpus: –20 Newsgroups –80% (16,000) documents used to generate statistics. –20% (4,000) documents used for testing –Choice of documents for training/testing handled by BOW –Headers removed from all documents

32. Experiment Protocol Trials –10 trials were performed using each classification method. Naïve Bayes, tf.idf, kNN, EMM, Max entropy, Probabilistic Indexing, BAF1, BAF2, PAS1, PAS2 –The average was taken from the 10 trials for each method.

33. Experiment Results

34. Analysis BAF1 performs poorly. –Using individual word scores appears to be a poor idea. BAF2 performs very well. –Better than the other methods attempted. BAF2 Performance slightly better than Naïve Bayers –Appears that considering a document to belong to another class has a positive effect on classification scores.

35. Conclusion Experiment results show that the use of BAF- Logic to classify documents might be a good idea. In addition there are features of BAFs (e.g. daemons attached to slots) that might enhance classification performance further. More work should be done on this. –Understanding better why BAF-Logic works for text classification. –Improving classification performance.