Opinion Mining Using Econometrics: A Case Study on Reputation Systems Anindya Ghose, Panagiotis G. Ipeirotis, and Arun Sundararajan Leonard N. Stern School of Business, New York University ACL 2007

Questions/Challenges What makes an opinion positive or negative? Is there an objective measure for this task? How can we rank opinions according to their strength? Can we define an objective measure for ranking opinions? How does the context change the polarity and strength of an opinion and how can we take the context into consideration?

Introduction Reputation systems in electronic markets Pricing power of merchants in Amazon.com Using 9,500 transactions over 180 days Textual feedback and star rank Discover polarity and strength without the need for human annotations or linguistic resources.

Arguments Textual feedback affects the power of merchants to charge higher prices than the competition for the same product and still make a sale.

Reputation Systems A reputation profile –Past transactions for the merchant. –Numerical ratings from buyers who have completed transactions. –Chronological list of textual feedback provided by these buyers.

Price Premiums Price premium/ relative price premium/ relative average price premium

Data Transaction Data: –1,078 merchants –9,487 unique transactions –107,922 price premiums Reputation Data: –4,932 postings per merchant –Numerical ratings: one to five stars –Reconstruct each seller’s exact feedback profile at the time of each transaction

Econometrics-based Opinion Mining Retrieving the dimensions of reputation –Features expressed by noun, noun phrase, verb, verb phrase. –For example, –X 1 might be shipping, X 2 might be packaging.

Reputation dimension example X=(delivery, packaging, service) I was impressed by the speedy delivery! Great service! (post 1) The item arrived in awful packaging and the delivery was slow. (post 2)

Scoring the dimension of reputation Construct an n x p matrix M(s i ) A total of 151 unique dimensions, and a total of 142 modifiers. c is the prob of clicking on the “Next” link. K is the number of postings that appear on each page. Posting–specific weight

Posting – specific weight example Weight is dropped exponentially if the page number is increased. If K = 25, total post = 51, weight of post 1 (page 3)= 1/(25+25c+c 2 ) weight of post 26 (page 2) = c/(25+25c+c 2 ) weight of post 51 (page 1)= c 2 /(25+25c+c 2 ) Score of modifier-dimention (feature) pair:

Reputation Score Modifier-dimention pair score = strength Feature weight = polarity A correlation between the appearance of modifier- dimension opinion phrase ( ) of the merchants and the price premiums observed for each transaction.

Scoring by regression Regressor coefficients Control variables Fixed effects The error term Score: counting appearances and weighting each appearance using the definition of r i Variations:

Regression Settings Predicting Control variables: –The product’s price on Amazon –The average star rating of the merchant –The number of merchant’s past transactions –The number of sellers for the product

Experimental Results Human Recall by two annotators, a random sample of 1,000 posts: Computer Recall: average over two annotators Precision is not an issue, noise will be ruled out by regression

Estimating polarity and strength Good packaging = -$0.58

Price Premiums vs. Ratings Many researches assume that text feedback will not influence the buyers. They used only rating stars as a summary of opinions. Examine R 2 fit of the regression, with and without the use of the text variables. Without: R 2 = 0.35; With: R 2 = 0.63 Text contains significantly more information than the ratings.

Prediction Predict which merchant will make a sale. C4.5 classifier, 4 months data as training and 2 months data as testing.

Possible application Exam the effect of product reviews on product sales and detect the weight that customers put on different product features. The analysis of the effect of news stories on stock prices; how opinion holders and their wordings can cause the market to move up or down. Extract the pragmatic effect of news and blogs on elections or other political events.