Panos Ipeirotis New York University Opinion Mining using Econometrics A Case Study on Reputation Systems Joint work with Anindya Ghose and Arun Sundararajan

Comparative Shopping in e-Marketplaces

Are Customers Irrational?

$11.04 $ $0.61 -$9.00 -$ $1.04 Price Premiums (customers pay more than the minimum price)

Price Amazon Irrational (?)

Average price Amazon Irrational (?)

Why not buying the cheapest? You buy more than a product  Customers do not pay only for the product  Customers also pay for a set of fulfillment characteristics  Delivery  Packaging  Responsiveness  … Reputation Matters!

Reputation Systems Facilitate electronic commerce  Integral part of online marketplaces  Provide information about unobserved fulfillment characteristics (most of which we take for granted in traditional commerce) Reputation in ecommerce is complex  Different buyers value different fulfillment characteristics  Sellers have varying abilities on these characteristics

Example of a reputation profile

Reputation profiles: Observations Reputation profile capture more than “averages”  Well beyond “average score” and “lifetime”  Rich textual content: information about a seller on a variety of dimensions (fulfillment characteristics).  How the seller’s performance (potentially on each of these characteristics) has evolved over time  Buyer-seller networks Reputation in ecommerce is complex  Different buyers value different fulfillment characteristics  Sellers have varying abilities on these characteristics  Previous work studied only effect of “average score” and “lifetime”

Our research questions What are the dimensions of online reputation?  What characteristics comprise the important parts of a seller’s overall reputation? (politeness? packaging? delivery?) How to evaluate the reputation across these dimensions?  How can we measure the reputation across each dimension?  How can we measure polarity and strength of each individual evaluation?  Is good service better than ok service?  Is superfast delivery faster than supersuperfast delivery?  Is good packaging a positive evaluation? Can prior reputation predict marketplace outcomes?  Given a set of sellers, their reputations, and their prices, can one predict which seller will successfully make the sale?

Data Overview  Panel of 280 software products sold by Amazon.com  Data on all “secondary” market transactions  Amazon Web services facilitate capturing transactions  Complete reputation profile for all sellers who completed one or more transactions during this period Summary  280 products X 180 days  1,078 sellers, of which 122 transacted  12,232 transactions  107,922 “observations” (seller-competitor pairs)

Data: Transactions

Sales of (mostly new) software Data: Transactions

Capturing transactions and “price premiums” Data: Transactions Seller ListingItemPrice When item is sold, listing disappears

Capturing transactions and “price premiums” Data: Transactions While listing appears, item is still available time Jan 1 Jan 2 Jan 3 Jan 4 Jan 5 Jan 6 Jan 7 Jan 8 Jan 9 Jan 10

Capturing transactions and “price premiums” Data: Transactions While listing appears, item is still available time Jan 1 Jan 2 Jan 3 Jan 4 Jan 5 Jan 6 Jan 7 Jan 8 Jan 9 Jan 10 Item still not sold on 1/7

Capturing transactions and “price premiums” Data: Transactions When item is sold, listing disappears time Item sold on 1/9 Jan 1 Jan 2 Jan 3 Jan 4 Jan 5 Jan 6 Jan 7 Jan 8 Jan 9 Jan 10

Data: Variables of Interest Price Premium  Difference in the price charged by a seller and the listed price of a competing seller at the time the transaction occurred (Seller Price – Competitor Price)  Calculated for each seller-competitor pair, for each transaction  Each transaction generates M observations, (M: number of competing sellers) Average Price Premium  Difference in the price charged by a seller and the average price of all competing sellers at the time the transaction occurred (Seller Price – Avg. (Competitor Price) )  Calculated for each transaction  Each transaction generates 1 observation

Price Amazon

Average price Amazon

The dimensions of reputation Our conjecture: Price premiums measure reputation

Decomposing reputation Is reputation just a scalar metric?  Previous studies assumed a “monolithic” reputation.  We break down reputation in individual components  Sellers characterized by a set of n fulfillment characteristics

Decomposing reputation What are these characteristics (valued by consumers?)  We think of each characteristic as a dimension, represented by a noun or verb phrase (“shipping”, “packaging”, “delivery”, “arrived”)  We scan the textual feedback to discover these dimensions

Decomposing and scoring reputation  We think of each characteristic as a dimension, represented by a noun or verb phrase (“shipping”, “packaging”, “delivery”, “arrived”)  The sellers are rated on these dimensions by buyers using modifiers (adjectives or adverbs), not numerical scores  “Fast shipping!”  “Great packaging”  “Awesome unresponsiveness”  “Unbelievable delays”  “Unbelievable price”

Dimensions from text: Example Parsing the feedback  P1: I was impressed by the speedy delivery! Great Service!  P2: The item arrived in awful packaging, and the delivery was slow  … Identified modifier-dimension pairs  P1: “speedy – delivery”, “great – service”  P2: “awful – packaging”, “slow – delivery”  … Reducing textual feedback to a n X p matrix  Dimensions: 1-delivery, 2-packaging, 3-service Postings

Decomposing and scoring reputation Scoring reputation  “Fast shipping!”  “Great packaging”  “Awesome unresponsiveness”  “Unbelievable delays”  “Unbelievable price” How can we find out the meaning of these adjectives?

The dimensions of reputation  We assume that each modifier assigns a “score” to each dimension  :score associated with  appearing as the modifier for the k-th dimension  r i : weight of posting that appears on the i-th position (weight down old posts)  w i : weight assigned to the i-th dimension  Thus, the overall (text) reputation score Π(i) is: scores for first dimension scores for n-th dimension Sum of r i weights in which  j modifies dimension i estimated coefficients scores for first posting

The dimensions of reputation Scoring the dimensions  Use price premiums as “true” reputation score  Use regression to assess scores (coefficients) for each dimension- modifier pair Regressions  Control for all variables that affect price premiums  Control for all numeric scores of reputation  Examine effect of text: E.g., seller with “fast delivery” has premium $10 over seller with “slow delivery” estimated coefficients

Some indicative dollar values PositiveNegative Natural method for extraction of sentiment strength and polarity

Results Some dimensions that matter  Delivery and contract fulfillment (extent and speed)  Product quality and appropriate description  Packaging  Customer service  Price (!)  Responsiveness/Communication (speed and quality)  Overall feeling (transaction)

Results Further evidence  Classifier (aka choice model) that predicts sale given set of sellers  Binary decision between seller and competitor  Naïve Bayes and Decision Trees (SVM’s forthcoming)  Only prices and characteristics: 53%  + numerical reputation, lifetime: 74%  + encoded textual information: 89%

Other applications Summarize and query reputation data  Give me all merchants that deliver fast SELECT merchant FROM reputation WHERE delivery > ‘fast’  Summarize reputation of seller XYZ Inc.  Delivery: 3.8/5  Responsiveness: 4.8/5  Packaging: 4.9/5 Pricing reputation  Given the competition, merchant XYZ can charge $20 more and still make the sale (confidence: 83%)

Summary Key contributions  New technique that automatically scores “sentiment” based on economic data  Validation by multiple methods (estimating an econometric model, building classifiers)  New evidence of the extent to which interdisciplinary research can be fun and distracting Broader contribution  Economic data is abundant and there is rich literature on how to handle such data  Economic data can be used for training for MANY applications

Thank you!

Prior studies of reputation Positive feedback significant, negative not  Ba and Pavlou (2002) for CD’s, software, electronics; Bajari and Hortacsu (2003) for collectible coins? Negative feedback significant, positive not  Lee et al. (2000) for computer equipment, Reiley et al. (2000) for collectible coins Nature of price: winning online auction bid (usually eBay) Measure of reputation: average numerical score, # of transactions Both positive and negative feedback significant  Dewan and Hsu (2004) for rare stamps, Melnik and Alm (2002) for gold coins, Houser and Wooders (2005) for Pentium chips