TWITTER DAY /07/14 LING 3820 & 6820 Natural Language Processing Harry Howard Tulane University

Course organization 07-Nov-2014NLP, Prof. Howard, Tulane University 2   The syllabus is under construction.   Chapter numbering  3.7. How to deal with non-English characters 3.7. How to deal with non-English characters  4.5. How to create a pattern with Unicode characters 4.5. How to create a pattern with Unicode characters  6. Control 6. Control

Open Spyder 07-Nov NLP, Prof. Howard, Tulane University

Conditional frequency distribution Review 07-Nov NLP, Prof. Howard, Tulane University

§10 Twitter 07-Nov NLP, Prof. Howard, Tulane University

What do you know about Twitter?  API = Application Programming Interface 07-Nov-2014NLP, Prof. Howard, Tulane University 6

Twitter ''' Go to and create a new Twitter account, if you haven't already done so. Now you need to set up this account so your script can make calls to Twitter’s API. Go to and sign in with the credentials of your new account. Click on “Create an App”, or just go directly to and fill in the details: I don’t think any of this info actually matters. You don’t need a callback URL. Click “Create your Twitter Application”. On the new app’s web page, and under “Application Type” click on the “Read, Write, and Access direct messages” radio button. Click “Update this Twitter Application’s settings”. Go back to the Details tab, and at the bottom of the screen click “Create my access token”. When the page reloads, you will need to copy the following four pieces of information, “Consumer Key”, “Consumer Secret”, “Access Token”, and “Access Token Secret” into the four places for it below. Be sure to save this information (for example, in this document) because you will have to enter it every time you want to code Twitter, and it is long and complicated.''' 07-Nov-2014NLP, Prof. Howard, Tulane University 7

How to access Twitter's API 07-Nov-2014NLP, Prof. Howard, Tulane University 8 REST (twitter-python) streaming (tweepy)

tweepy In the Terminal: $ pip install –U tweepy 07-Nov-2014NLP, Prof. Howard, Tulane University 9

Test tweepy put this into new script & save as tweepies.py import tweepy from tweepy.api import API CONSUMER_KEY = 'your_info_here' CONSUMER_SECRET = 'your_info_here' ACCESS_TOKEN = 'your_info_here' ACCESS_TOKEN_SECRET = 'your_info_here' key = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET) key.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET) # verify api = tweepy.API(key) print api.me().name # HLT Lab 07-Nov-2014NLP, Prof. Howard, Tulane University 10

A problem  Streaming does not stop …  but you will want to stop it at some point.  You will want to stop it when you have received a certain number of tweets.  How? 07-Nov-2014NLP, Prof. Howard, Tulane University 11

A counter >>> n = 0 >>> m = 20 >>> if n < m: n = n + 1 else: stop! 07-Nov-2014NLP, Prof. Howard, Tulane University 12

Classes in Python class NewStreamListener(tweepy.StreamListener): def __init__(self, api=None): self.api = api or API() def on_status(self, status): 07-Nov-2014NLP, Prof. Howard, Tulane University 13

How to tell StreamListener() to stop listening for tweets  Algorithm  upon initializing StreamListener() set the counter to 0 set the maximum to 20  when a relevant tweet is found if the counter is less than 20, add 1 to it and keep going otherwise, exit StreamListener() 07-Nov-2014NLP, Prof. Howard, Tulane University 14

The code class StopStreamListener(tweepy.StreamListener): def __init__(self, api=None): self.api = api or API() self.n = 0 self.m = 20 def on_status(self, status): self.n = self.n+1 if self.n < self.m: return True else: return False 07-Nov-2014NLP, Prof. Howard, Tulane University 15

Now make it send output to screen (after log-in in tweepies) class Stream2Screen(tweepy.StreamListener): def __init__(self, api=None): self.api = api or API() self.n = 0 self.m = 20 def on_status(self, status): print status.text.encode('utf8') self.n = self.n+1 if self.n < self.m: return True else: print 'tweets = '+str(self.n) return False 07-Nov-2014NLP, Prof. Howard, Tulane University 16

To invoke it (put at end) stream = tweepy.streaming.Stream(key, Stream2Screen()) stream.filter(track=['the,of'], languages=['en']) 07-Nov-2014NLP, Prof. Howard, Tulane University 17

Next quiz More twitter Next time 07-Nov-2014NLP, Prof. Howard, Tulane University 18