Tweets Discrimination Analysis Shuhan Yuan sy005@email.uark.edu

Discrimination Analysis Discrimination is treatment or consideration of, or making a distinction in favor of or against, a person or thing based on the group, class, or category to which that person or thing is perceived to belong to rather than on individual merit. (Wikipedia) Tweets discrimination analysis aims to detect whether a tweet has discrimination against gender, race, age, etc.

Twitter API REST-based API (Representation State Transfer) HTTP-Requests Authentication with OAuth Responses are available in JSON Streaming API Supports long-lived HTTP connection Real-time delivery of tweets API Rate Limits Search will be rate limited at 180 queries per 15 minutes https://dev.twitter.com/overview/documentation

Twitter API https://apps.twitter.com The authentication requires that you get an API key from the Twitter developers site. This just requires that you have a Twitter account. The four keys the site gives you are used as parameters in the programs. The OAuth authentication gives your program permission to make API calls.

Crawler (python+tweepy) Search API Stream API

Twitter JSON

Why choose MongoDB Document Storage Super easy to install and config Documents are stored in BSON (binary JSON) Any valid JSON can be easily imported and queried Schema-less; very flexible Super easy to install and config BSON is a binary serialization of JSON-like objects; This is extremely powerful, b/c it means mongo understands JSON natively http://www.slideshare.net/drumwurzel/intro-to-mongodb

Document-oriented Think of “documents” as database records Documents are basically just JSON objects that Mongo stores in binary Think of “collections” as database tables http://www.slideshare.net/drumwurzel/intro-to-mongodb

Storing Tweets in MongoDB (python + pymongo)

Concept Mapping RDBMS MongoDB Table Collection Row Document JOIN Embedded Document or Reference Queries return record(s) Queries return a cursor https://www.mongodb.com/blog/post/thinking-documents-part-1

https://www.mongodb.com/blog/post/thinking-documents-part-1

Thinking in Documents Blogging Platform RDBMS (JOIN 5 tables) MongoDB (Data Locality) https://www.mongodb.com/blog/post/thinking-documents-part-1

Thinking in Documents Blogging Platform RDBMS (JOIN 5 tables) MongoDB (Data Locality) https://www.mongodb.com/blog/post/thinking-documents-part-1

Queries return “cursors” instead of collections A cursor allows you to iterate through the result set Much more efficient than loading all objects into memory Find() function returns a cursor http://www.slideshare.net/drumwurzel/intro-to-mongodb

Discrimination detecting Assumption: Tweets containing some specific hashtags, like #discrimination, #sexism, #racism, have discrimination. Tweets containing #news hashtag do not have discrimination. Classifier tweets Discrimination or not Naïve Bayes Decision Tree Random Forest SVM Neural Network

One slide intro to machine learning Training (BOW, TFIDF, LSA, EMBEDDINGS, etc…) Testing http://ogrisel.github.io/scikit-learn.org/sklearn-tutorial/tutorial/astronomy/general_concepts.html

Recurrent Neural Networks (RNN) Recurrent Neural Networks (RNN) are capable of conditioning the model on all previous words in the corpus. The sequential information is preserved in the recurrent network’s hidden state, which manages to span many time steps as it cascades forward to affect the processing of each new words. RNNs take as their input not just the current word they see,  but also what they perceived one step back in time.  So recurrent networks have two sources of input, the present and the recent past. Can operate on sequential data of variable length An unrolled recurrent neural network. http://colah.github.io/posts/2015-08-Understanding-LSTMs/

The architecture 𝑦 Logistic Repression ℎ Mean pooling ℎ 0 ℎ 1 ℎ 2 ℎ 𝑛 RNN RNN RNN RNN 𝑤 1 𝑤 2 𝑤 3 𝑤 𝑛 http://deeplearning.net/tutorial/lstm.html

The last step: coding Loading tweets from MongoDB Preprocessing tweets (removing stop words, tokenize, …) Split the dataset into training dataset and test dataset Training the neural network on training dataset and test the accuracy on test dataset.

Resources https://dev.twitter.com/overview/documentation https://github.com/tweepy/tweepy http://www.ark.cs.cmu.edu/TweetNLP/ https://docs.mongodb.org/manual/ http://www.deeplearning.net/tutorial/ https://education.github.com/ http://stats.seandolinar.com/collecting-twitter-data-introduction/ http://adilmoujahid.com/posts/2014/07/twitter-analytics/

Thank you