Twitter Movie Sentiment Using Python, SQL Server, Azure SQL DB, Azure ML, & Power BI Bradley Ball braball@microsoft.com

Speaker Introduction: Bradley Ball Over 15 Years IT Experience, Former Microsoft MVP Previous experience DBA, for the U.S. Army, The Executive Office of the President, Sr. SQL DBA Staff Specialist at Publix, Data Platform Practice Manager for Pragmatic Works Currently Sr. Consultant for Microsoft Premier Support for Developers Pro SQL Server 2012 Practices Author Chapter 14 PAGE & ROW COMPRESSION! Managing Author on Pro Admin 2014 Guide

Agenda Movie Industry Sentiment Architecture How to Collect the Data with Python Storing the Data Scoring the Data Visualize the Data

The Movie Industry The movie industry: Generated 38 billion in revenues in 2016 Employs more than 404,000 people per year https://www.bls.gov/opub/ted/2017/a-look-at-employment-and-wages-in-the-film-industry.htm

What is Sentiment Sentiment – “A view or attitude toward a situation or event; an opinion” Some movies are good

What is Sentiment Sentiment – “A view or attitude toward a situation or event; an opinion” Some movies are…. not

Architecture Collect Twitter Data Store Data Score Data Visualize Data

BeautifulSoup /Selenium/Scrapy Architecture Score Sentiment BeautifulSoup /Selenium/Scrapy Export Table Visualize Data JSON to Pandas to CSV Import CSV

BeautifulSoup -IMDB BeautifulSoup Scrapes rendered html

BeautifulSoup -Twitter Attempt 1 Doesn’t work Tutorials lie Twitter uses AJAX, AJAX does not render to html

Selenium -Twitter Attempt 2 Each element is its own single dimensional array Arrays did not line up Tutorials kinda lie, it works but not how you need it to work

Scrapy - Twitter Attempt 3 Victory! Scrapes AJAX Scrolls Pages Uses Twitter’s Advanced Search URL Doesn’t user Twitter API’s TweetScraper – Github by Jon Bakerfish

JSON to Pandas Files saved in JSON Read in JSON files Insert to Pandas data frame Save to a CSV file

SQL Server Transform table with T-SQL Import Flat File created from Pandas into: SQL Server Transform table with T-SQL

Azure SQL DB Export the data Create Scored table Use this source for Azure ML

Azure ML Import Data Transform it with R – remove punctuation and special characters Calculate Sentiment Export it to Scored table in SQL

Azure SQL DB Transform the data to a time series

Power BI Look for Trends

Demo

What Did We Just Cover Movie Industry Sentiment Architecture How to Collect the Data with Python Storing the Data Scoring the Data Visualize the Data

References Scraping Infinite Scrolling Pages - https://blog.scrapinghub.com/2016/06/22/scrapy-tips-from-the-pros-june-2016/ TweetScraper by Jon Bakerfish - https://github.com/jonbakerfish/TweetScraper Web Scraping with Python and BeautifulSoup - https://www.dataquest.io/blog/web-scraping-beautifulsoup/