1. Programming for Linguists An Introduction to Python 15/12/2011

2. A sequence of values They are similar to lists: the values can be any type they are indexed by integers Syntactically a tuple is a comma- separated list of values: t = 'a', 'b', 'c', 'd', 'e' Tuples

3. Although it is not necessary, it is common to enclose tuples in parentheses t = ('a', 'b', 'c', 'd', 'e’) To create a tuple with a single element, you have to include a final comma: t1 = 'a’, type(t1)

4. Note: a value in parentheses is not a tuple ! t2 = (‘a’) type(t2) With no argument, the tuple ( ) function creates a new empty tuple t = tuple( )

5. If the argument is a sequence (string, list or tuple), the result is a tuple with the elements of the sequence: t = tuple(‘lupins’) print t Most list operators also work on tuples: print t[0] print t[1:3]

6. BUT if you try to modify one of the elements of the tuple, you get an error message t[0] = ‘A’ You can’t modify the elements of a tuple: a tuple is immutable !

7. You can replace one tuple with another t = ('A',) + t[1:] print t

8. It is often useful to swap the values of two variables, e.g. swap “a” with “b” temp=a a=b b=temp Tuple Assignment

9. More elegant with a tuple assignment a,b = b,a The number of variables on the left and the number of values on the right have to be the same ! a, b = 1,2,3 ValueError: too many values to unpack

10. For example: split an email address into a user name and a domain address = ‘joske@ua.ac.be’ username, domain = address.split('@') print username print domain The return value from split(‘@’) is a list with two elements The first element is assigned to username, the second to domain.

11. Strictly speaking, a function can only return one value If the value is a tuple, the effect is the same as returning multiple values Tuples as Return Values

12. For example: def min_max(t): return min(t), max(t) max( ) and min( ) are built-in functions that find the largest and smallest elements of a sequence min_max(t) computes both and returns a tuple of two values

13. .items ( ) function used on dictionaries we saw last week actually returns a list of tuples, e.g. >>> d = {'a':0, 'b':1, 'c':2} >>> d.items( ) [('a', 0), ('c', 2), ('b', 1)] Dictionaries and Tuples

14. This way you can easily access both keys and values separately: d = {'a':0, 'b':1, 'c':2} for letter, number in d.items( ): print letter print number

15. Example: sorting a list of words by their word length def sort_by_length(words): list1=[ ] for word in words: list1.append((len(word), word)) list1.sort(reverse=True) ordered_list=[ ] for length, word in list1: ordered_list.append(word) return ordered_list

16. NLTK and the Internet A lot of text on the web is in the form of HTML documents To access them, you first need to specify the correct location url = “http://nltk.googlecode.com/svn/trunk/doc/ book/ch03.html” Then use the urlopen( ) function from urllib import urlopen htmltext = urlopen(url).read( )

17. NLTK provides a function nltk.clean_html( ), which takes an HTML string and returns raw text, e.g. rawtext = nltk.clean_html(htmltext) In order to use other NLTK methods, you can then tokenize the raw text tokens=nltk.wordpunct_tokenize(rawtext)

18. NLTK’s WordPunctTokenizer takes as an argument raw text and returns a list of tokens (words + punctuation marks) If you want to use the functions we used on the texts from nltk.book on your own texts, use the nltk.Text( ) function my_text = nltk.Text(tokens) my_text.collocations( )

19. Note: if you are used to working with characters in a particular local encoding (ë, è,…), you need to include the string '# - *- coding: -*-' as the first or second line of your script, e.g. # -*- coding: utf-8 -*-

20. Writing Results to a File It is often useful to write output to files First you have to open/create a file for your output output_file = open(‘(path)/output.txt’,‘w’) output_file = open(‘(path)/output.txt’,‘a’)

21. Now you have to write your output to the file you just opened list = [1, 2, 3] output_file.write(str(list) + "\n”) When you write non-text data to a file you must convert it to a string first Do not forget to close the file when you are done output_file.close( )

22. NLTK and automatic text classification Classification is the computational task of choosing the correct class label for a given input text, e.g. deciding whether an email is spam or not deciding what the topic of a news article is (e.g. sports, politics, financial,…) authorship attribution

23. Framework (1) Gather a training corpus: in which a categorization is possible using metadata, e.g. information about the author(s): name, age, gender, location information about the texts’ genre: sports, humor, romance, scientific

24. Framework (2) Gather a training corpus: for which you need to add the metadata yourself, e.g. annotation of content-specific information: add sentiment labels to utterances annotation of linguistic features: add POS tags to text Result: a dataset with predefined categories

25. Framework (3) Pre-processing of the dataset, e.g. tokenization, removing stop words Feature selection: which features of the text could be informative for your classification task, e.g. lexical features: words, word bigrams,... character features: n-grams syntactic features: POS tags semantic features: role labels others: readability scores, TTR, wl, sl,…

26. Framework (4) Divide your dataset in a training set and a test set (usually 90% vs 10%) Feature selection metrics: based on frequencies: most frequent features based on frequency distributions per category: most informative features in NLTK: Chi-square, Student's t test, Pointwise Mutual Information, Likelihood Ratio, Poisson- Stirling, Jaccard index, Information Gain use them only on training data! (overfitting)

27. Framework (5) For document classification: each document in the dataset is represented by a separate instance containing the features extracted from the training data The format of your instances depends on the classifier you want to use Select your classifier: in NLTK: Naive Bayes, Decision Tree, Maximum Entropy, link to Weka

28. Framework (6) Train the classifier using the training instances you created in the previous step Test your trained model on previously unseen data: the test set Evaluate your classifier’s performance: accuracy, precision, recall and f-scores, confusion matrix Perform error analysis

29. A Case Study Classification task: classifying movie reviews into positive and negative reviews 1. Import the corpus from nltk.corpus import movie_reviews 2. Create a list of categorized documents documents = [(list(movie_reviews.words(fileid)), category) for category in movie_reviews.categories( ) for fileid in movie_reviews.fileids(category)]

30. print documents[:2] 3. Shuffle your list of documents randomly from random import shuffle shuffle(documents) 4. Divide your data in training en test train_docs = documents[:1800] test_docs = documents[1800:] 5. We only consider word unigram features here, so make a dictionary of all (normalized) words from the training data

31. train_words = { } for (wordlist, cat) in train_docs: for w in wordlist: w = w.lower( ) if w not in train_words: train_words[w] = 1 else: train_words[w] += 1 print len(train_words)

32. 6. Define a feature extraction function def extract_features(wordlist): document_words = set(wordlist) features = { } for word in document_words: word = word.lower( ) if word in train_words: features[word] = (word in document_words) return features print extract_features(movie_reviews.words('pos/cv957 _8737.txt'))

33. 7. Use your feature extraction function to extract all features from your training and test set train_feats = [(extract_features(wordlist), cat) for (wordlist,cat) in train_docs] test_feats = [(extract_features(wordlist), cat) for (wordlist,cat) in test_docs]

34. 7. Train e.g. NLTK’s Naïve Bayes classifier on the training set from nltk.classify import NaiveBayesClassifier classifier = NaiveBayesClassifier.train(train_feats) predicted_labels = classifier.batch_classify([fs for (fs, cat) in test_feats]) 8. Evaluate the model on the test set print nltk.classify.accuracy(classifier, test_feats) classifier.show_most_informative_features(20)

35. For Next Week Feedback on the past exercises Some extra exercises If you have additional questions or problems, please e-mail me by Wednesday The evaluation assignment will be announced

36. Ex 1) Choose a website. Read it in in Python using the urlopen function, remove all HTML mark-up and tokenize it. Make a frequency dictionary of all words ending with ‘ing’ and sort it on its values (decreasingly). Ex 2) Write the raw text of the text in the previous exercise to an output file.

37. Ex 3) Write a script that performs the same classification task as we saw today using word bigrams as features instead of single words.

38. Thank you