1. Measures to Detect Word Substitution in Intercepted Communication David Skillicorn, SzeWang Fong School of Computing, Queen’s University Dmitri Roussinov W.P. Carey School of Business, Arizona State University

2. What my lab does: 1. Detecting anomalies when their attributes have been chosen to try and make them undetectable; 2. Deception detection in text (currently Enron, Canadian parliament, Canadian election speeches, trial evidence); 3. Markers in text for anomalies and hidden connections.

3. Governments intercept communication as a defensive measure. Increasingly, this requires `domestic’ interception as well as more longstanding external interception – a consequence of asymmetric warfare. A volatile issue! Organizations increasingly intercept communication (e.g. email) to search for intimidation, harassment, fraud, or other malfeasance. This may happen in an online way, e.g. as a response to due diligence requirements of post-Enron financial regulation; or may happen forensically after an incident. There’s no hope of human processing of all of this communication. Indeed, there’s too much for most organizations to afford sophisticated data-mining of all of it. Ways to automatically select the interesting messages are critical.

4. Early-stage filtering of communication traffic must: * be cheap * have a low false negative rate (critical) * but a high false positive rate doesn’t matter (too much) The goal is to be sure about innocence. This is important to communicate to the public – all messages/calls/emails are not examined equally.

5. First technique for message selection: use a list of words whose presence (with the right frequency) in a message indicates an interesting message. This seems like a very weak technique, because the obvious defense is not to use words that might be on the watchlist. However, … * although the existence of the list might be public, it’s much harder to guess what’s on it and where it ends. E.g. `nuclear’ yes `bomb’ yes `ammonium nitrate’ ?? `Strasbourg cathedral’ ?? * the list’s primary role is to provoke a reaction in the guilty (but not in the innocent)

6. One possible reaction: encryption – but this is a poor idea since encryption draws attention to the message. Another reaction: replace words that might be on the watchlist by other, innocuous, words. Which words to choose as replacements? If the filtering is done by humans, then substitutions should ‘make sense’, e.g. Al Qaeda `attack’  `wedding’ works well, because weddings happen at particular places, and require a coordinated group of people to travel and meet.

7. Of course, large-scale interception cannot be handled by human processing. If the filtering is done automatically, substitutions should be syntactically appropriate – e.g. of similar frequency. Can substitutions like this be detected automatically? YES, because they don’t fit as well into the original sentence; The semantic differences can be detected using syntactic markers and oracles for the natural frequency of words, phrases, and bags of words.

8. We define a set of measures that can be applied to a sentence with respect to a particular target word (usually a noun). 1. Sentence oddity (SO), Enhanced sentence oddity (ESO) SO = frequency of bag of words, target word removed frequency of entire bag of words ESO = frequency of bag of words, target word excluded frequency of entire bag of words Intuition: when a contextually appropriate word is removed, the frequency doesn’t change much; when a contextually inappropriate word is removed, the frequency may increase sharply. increase  possible substitution

9. Example original sentence: “we expect that the attack will happen tonight” Substitution: `attack’  `campaign’ “we expect that the campaign will happen tonight” f(we expect that the attack will happen tonight) = 2.42M f(we expect that the will happen tonight) = 5.78M SO = 2.4 f(we expect that the campaign will happen tonight) = 1.63M f(we expect that the will happen tonight) = 5.78M SO = 3.5

10. 2. Left, right and average k-gram frequencies Many short exact (quoted) strings do not occur, even in large repositories!! k-grams estimate frequencies of target words in context, but must keep the context small (or else the estimate is 0). left k-gram = frequency of exact string from closest non-stopword to the left of the target word, up to and including the target word. right k-gram = frequency of exact string from target word up to and including closest non-stopword to the right. average k-gram = average of left and right k-grams. small k-gram  possible substitution

11. Examples of exact string frequencies “the attack will happen tonight” f = 1 even though this seems like a plausible, common phrase Left k-gram: “expect that the attack” f= 50 Right k-gram: “attack will happen” f = 9260 Left k-gram: “expect that the campaign” f = 77 Right k-gram: “campaign will happen” f = 132 This should be smaller than 50, but may be affected by ‘election campaign’

12. 3. Maximum, minimum, average hypernym oddity (HO) The hypernym of a word is the word or phrase above it in a taxonomy of meaning, e.g. `cat’  `feline’. If a word is contextually appropriate, replacing it by its hypernym creates an awkward (pompous) sentence, with lower frequency. If a word is contextually inappropriate, replacing it by its hypernym tends to make the sentence more appropriate, with greater frequency. HO = frequency of bag of words with hypernym – frequency of original bag of words increase  possible substitution

13. Hypernym examples Original sentence: we expect that the attack will happen tonight f = 2.42M we expect that the operation will happen tonight f H = 1.31M Sentence with a substitution: we expect that the campaign will happen tonight f = 1.63M we expect that the race will happen tonight f H = 1.97M

14. Hypernyms are semantic relationships, but we can get them automatically using Wordnet (wordnet.princeton.edu). Most words have more than one hypernym, because of their different senses. We can compute the maximum, minimum and average hypernym oddity over the possible choices of hypernyms.

15. 4. Pointwise mutual information (PMI) PMI = f(target word) f(adjacent region) f(target word + adjacent region) where the adjacent region can be on either side of the target. We use the maximum PMI calculated over all adjacent regions that have non-zero frequency. (Frequency drops to zero with length quickly.) PMI looks for the occurrence of the target word as part of some stable phrase. increase  possible substitution

16. Frequency oracles: We use Google and Yahoo as sources of natural frequencies for words, quoted strings, and bags of words. Some issues: * we use frequency of pages as a surrogate for frequency of words; * we don’t look at how close together words appear in each page, only whether they all occur; * search engines handle stop words in mysterious ways * order of words matters, even in bag of word searches * although Google and Yahoo claim to index about the same number of documents, their reported frequencies for the same word differ by a factor of at least 6 in some cases

17. Test data We want text that is relatively informal, because most intercepted messages will not be polished text (email, phone calls). We selected sentences of length 5-15 from the Enron email corpus. Many of these sentences are informal (some are bizarre). We constructed a set of sentences containing substitutions by replacing the first noun in each original sentence by a frequency- matched substitute. We discarded sentences where the first noun wasn’t in the BNC corpus, or did not have a hypernym known to Wordnet.

18. We built a set of 1714 ‘normal’ sentences, and 1714 sentences with a substitution (but results were very stable for more than about 200 sentences) We also constructed a similar, smaller, set from the Brown corpus (which contains much more formal, and older, texts).

19. For each measure, we built a decision tree predicting normal vs substitution, using the measure value as the single attribute. This gives us insight about the boundary between normal and substitution sentences for each kind of measure. MeasureBoundary: odd if Semantic oddity> 4.6 Enhanced semantic oddity> 0.98 Left k-gram< 155 Right k-gram< 612 Average k-gram< 6173 Min hypernym oddity> -89129 Max hypernym oddity> -6 Average hypernym oddity> -6 Pointwise mutual information> 1.34 These are newer results than those in the paper

20. Individual measures are very weak detectors. (75%/25% split, J48 decision tree, single attribute, Weka) Measure Detection rate % False positive rate % Area under ROC curve Semantic oddity51210.6672 Enhanced semantic oddity72230.6219 Left k-gram56330.6403 Right k-gram84520.6791 Average k-gram56250.6768 Min hypernym oddity66520.5735 Max hypernym oddity57300.6330 Average hypernym oddity43210.6068 Pointwise mutual information 49240.7064

21. Single-measure predictors make their errors on different sentences. Combining them produces much stronger predictors. Combining using a decision tree trained on the full set of measure values: Combining using a random forest (50 trees, Mtry = 4): Surprising this isn’t better. Measure Detection rate % False positive rate % Area under ROC curve Combined decision tree95110.9844 Measure Detection rate % False positive rate % Random forest9011

22. The families of measure are almost completely independent:

23. …and each sentence’s classification is almost completely determined by its score w.r.t one measure, i.e. most sentence have a neutral score on all but one measure (family) – something deeper here.

24. We expected better results for the Brown corpus, reasoning that context should be more helpful in more-careful writing. In fact, the results for the Brown corpus are worse. This may reflect changes in language use, since the 60s. Our oracles are much better representatives of recent writing. But puzzling… Measure Detection rate % False positive rate % Area under ROC curve Combined decision tree84160.9838 Measure Detection rate % False positive rate % Random forest8313

25. Results are similar (within a few percentage points) across different oracles: Google, Yahoo, MSN despite their apparent differences. Results are also similar if the substituted word is much less frequent than the word it replaces. No extra performance from rarity of the replacement word. (cf Skillicorn ISI 2005 where this was critical) But some loss of performance if the substituted word is much more frequent than the word it replaces. This is expected since common words fit into more contexts.

26. Why do the measures make errors? Looking at the first 100 sentences manually… * some of the original sentences are very strange already, email written in a hurry or with strange abbreviations or style * there’s only one non-stopword in the entire sentence, so no real context * the substitution happens to be appropriate in the context There’s some fundamental limit to how well substitutions can be detected because of these phenomena. Both detection rate and false positive rate may be close to their limits.

27. Mapping sentence predictions to message predictions: There’s considerable scope to get nicer properties on a per-message basis by deciding how many sentences should be flagged as suspicious before a message is flagged as suspicious. It’s likely that an interesting message contains more than 1 sentence with a substitution. So a rule like: “select messages with more than 4 suspicious sentences, or more than 10% suspicious sentences” reduces the false positive rate, without decreasing the detection rate much.

28. Summary: A good way to separate ‘bad’ from ‘good’ messages is to deploy a big, visible detection system (whose details, however, remain hidden), and then watch for reaction to the visible system Often this reaction is easier to detect than the innate differences between ‘bad’ and ‘good’. Even knowing this 2-pronged approach, senders of ‘bad’ messages have to react, or else risk being detected by the visible system.

29. For messages, the visible system is a watchlist of suspicious words. The existence of the watchlist can be known, without knowing which words are on it. Senders of ‘bad’ messages are forced to replace any words that might be on the watchlist – so they probably over-react. These substitutions create some kind of discontinuity around the places where they occur. This makes them detectable, although a variety of (very) different measures must be used – and, even then, decent performance requires combining them. So far, detection performance is ~95% with a ~10% false positive rate.

30. ? www.cs.queensu.ca/home/skill skill@cs.queensu.ca www.public.asu.edu/~droussi/ dmitri.roussinov@asu.edu