1. Stylometry System CSIS Stylometry Projects, mostly Fall 2009 Project Seidenberg School of Computer Science and Information Systems

2. Stylometry System CSIS Description of Project Stylometry - is the study of the unique linguistic styles and writing behaviors of individuals in order to determine authorship Part I –Search to determine an interesting and unique application of stylometry for Research Part II –Feasibility study on existing tools/applications for email authorship (250 words or less)

3. Stylometry System CSIS Existing / Potential Uses of Stylometry Music Lyrics Plagiarism Music Melody Social Networking Paintings Electronic Mail Literary Works Instant Messaging Forensic Linguistics - Social networking, electronic mail, and instant messaging are still in early stages of study

4. Stylometry System CSIS Use Cases -Twitter -Used to verify existing Twitter accounts and help mitigate impersonations -Electronic mail -Implemented in a corporate setting helping identify anonymous emails meant to do harm -Chat -Assist in determining authorship of instant messages -Similar to Twitter but needs to be dynamic

5. Stylometry System CSIS Use Cases -Terrorism -Help identify an author of terrorist content or identify terrorist content by using contextual analysis -Applied to blogs, forums, wikis, email, chat and other forms of digital content

6. Stylometry System CSIS Tools discovered -JGAAP (Java Graphical Authorship Attribute Program) -Signature Tool -C# Tool -StyleTool -Blog stylometry tool -Stylometry tool

7. Stylometry System CSIS Tools discovered -JGAAP (Java Graphical Authorship Attribute Program) -Java based tool -Runs on Windows and Linux -Identification tool -1 of n decision – Many known email authors trying to determine the author of one unknown email -One unknown email author compared to 99 known email authors -100 total tests run

8. Stylometry System CSIS Tools discovered -C# Tool -Written in C programming language -Developed by prior Pace CS graduate students -Identification tool -1 of n decision – Many known email authors trying to determine the author of one unknown email -One unknown email author compared to 99 known email authors -100 total tests run

9. Stylometry System CSIS Tools discovered -Signature Tool -Written in C programming language (not confirmed) -Created by Peter Millican from Hartford College -Authentication Tool -Either match / no match -Match testing – 9 known and 1 unknown sample (same author) -No Match – 10 known and 1 unknown (two different authors) -Total of 105 tests were run

10. Stylometry System CSIS Testing methodology -Each team member submitted 20 (or 30) actual emails from 2 (3) different authors. -Total of 100 emails collected from 10 different authors -Removed from native program and saved as text files -Average size (words) of email 195.7 -Different testing for identification and authentication tools -For authentication tool -False Accept Rate - Rate a document is falsely attributed to an author -False Reject Rate - Rate a document is not correctly attributed to an author

11. Stylometry System CSIS Testing Results JGAAP (Levenshtein Distance algorithm) CanonizersOnOff Words 50%30% Word Length 50%30% Characters 60%40% Syllables per Word 40%30% Word Bigrams 70%60% Signature Tool Match Test EventsAccuracyFRR Word Length53.33%46.67% Letters46.67%53.33% Signature Tool No-Match Test EventsAccuracyFAR Word Length53.33%46.67% Letters82.22%17.78% C# Tool Match Test Accuracy 57% Categorizing the result based on the country of the author Tool MatchNo-Match IndiaUSAIndiaUSA JGAAP50%100%NA Signature61.11%75.00%81.48%83.33% C# Tool42%80.00%NA

12. Stylometry System CSIS Earlier Study’s Features – 20 of 55 1. Number of sentences beginning with upper case 2. Number of sentences beginning with lower case 3. Number of Words 4. Average Word Length 5. Number of Sentences 6. Average Number of Words per Sentence 7. Number of Paragraphs 8. Average Number of words per Paragraph 9. Number of Exclamation Marks 10. Number of Number Signs 11. Number of Dollar Signs 12. Number of Ampersands 13. Number of Percent Signs 14. Number of Apostrophes 15. Number of Left parentheses 16. Number of Right parentheses 17. Number of Asterisks 18. Number of Plus Signs 19. Number of Commas 20. Number of Dashes

13. Stylometry System CSIS Conclusion -Overall the moderate accuracy of the test results suggest that none of the tools evaluated are capable of accurate stylometric email author identification -Categorizing email samples by country of origin seems to yield better accuracy results for all three tools tested.

14. Stylometry System CSIS Recommendations -Further testing and research using email from authors of different countries -Continue to refine and add to the stylistic feature set created by prior Pace graduate students -Include new features becoming more prevalent in digital content. Ex. Emoticons, hyperlinks -Internet slang – BRB, LOL, TTYL -Consideration for people who wish to disguise their identity needs to be addressed and researched further