Panos Ipeirotis New York University Opinion Mining using Econometrics A Case Study on Reputation Systems Joint work with Anindya Ghose and Arun Sundararajan

Comparative Shopping in e-Marketplaces

Are Customers Irrational?

$11.04 $ $0.61 -$9.00 -$ $1.04 Price Premiums

Price Amazon Irrational (?)

Average price Amazon Irrational (?)

Why not buying the cheapest? You buy more than a product  Customers do not pay only for the product  Customers also pay for a set of fulfillment characteristics  Delivery  Packaging  Responsiveness  … Reputation Matters!

Reputation Systems Facilitate electronic commerce  Integral part of online marketplaces  Provide information about unobserved fulfillment characteristics (most of which we take for granted in traditional commerce) Reputation in ecommerce is complex  Different buyers value different fulfillment characteristics  Sellers have varying abilities on these characteristics

Example of a reputation profile

Reputation profiles: Observations Reputation profile capture more than “averages”  Well beyond “average score” and “lifetime”  Rich textual content: information about a seller on a variety of dimensions (or fulfillment characteristics).  How the seller’s performance (potentially on each of these characteristics) has evolved over time  Buyer-seller networks Reputation in ecommerce is complex  Different buyers value different fulfillment characteristics  Sellers have varying abilities on these characteristics Previous work studied only effect of “average score” and “lifetime”

Our research agenda What are the dimensions of online reputation?  What characteristics comprise the important parts of a seller’s overall reputation? (politeness? packaging? delivery?) How do these dimensions affect pricing power?  Does a better reputation enable a seller to charge a higher price?  Which dimensions affect this pricing power most significantly?  Average numerical ratings?  Number of prior successful transactions?  Assessments of ability on specific fulfillment characteristics?  Do competitors with better reputations limit a seller’s pricing power? Can prior reputation predict marketplace outcomes?  Given a set of sellers, their reputations, and their prices, can one predict which seller will successfully make the sale?

What’s different about our research? Assessment of reputation based on “realistic” model  Include information contained in text feedback  Place higher weight on more recent postings Identify dimensions of reputation automatically  Discover automatically important dimensions  Modifiers of each dimension scored based on their impact on pricing power amazingserviceunresponsiveness Evidence using multiple techniques  Numeric reputation, text feedback explain variation in pricing power  Numeric reputation, text feedback improve predictions of “making the sale”.

Model Aspects that matter later on  Set of (n) fulfillment characteristics (separate from product quality)  Buyers heterogeneous on value placed on each: w=(w 1,w 2,…,w n )  Buyers assess seller fulfillment quality based on reputation and w

Data Overview  Panel of 280 software products sold by Amazon.com  Data on all “secondary” market transactions  Amazon Web services facilitate capturing transactions  Complete reputation profile for all sellers who completed one or more transactions during this period Summary  280 products X 180 days  1,078 sellers, of which 122 transacted  12,232 transactions  107,922 “observations” (seller-competitor pairs)

Data: Transactions

Sales of (mostly new) software Data: Transactions

Capturing transactions and “price premiums” Data: Transactions Seller ListingItemPrice When item is sold, listing disappears

Capturing transactions and “price premiums” Data: Transactions While listing appears, item is still available time 1/11/21/31/41/51/61/71/81/91/10

Capturing transactions and “price premiums” Data: Transactions While listing appears, item is still available time 1/11/21/31/41/51/61/71/81/91/10 Item still not sold on 1/7

Capturing transactions and “price premiums” Data: Transactions When item is sold, listing disappears time 1/11/21/31/41/51/61/71/81/91/10 Item sold on 1/9

Data: Variables of Interest Regular Price Premium  Difference in the price charged by a seller and the listed price of a competing seller at the time the transaction occurred (Seller Price – Competitor Price)  Calculated for each seller-competitor pair, for each transaction  Each transaction therefore generates N observations, where N is the number of competing sellers Average Price Premium  Difference in the price charged by a seller and the average price of all competing sellers at the time the transaction occurred (Seller Price – Avg. (Competitor Price) )  Calculated for each transaction  Each transaction generates 1 observation

Price Amazon

Average price Amazon

The dimensions of reputation How reputation affects price premiums?

Decomposing reputation Is reputation just a scalar metric?  Previous studies assumed a “monolithic” reputation.  We break down reputation in individual components  Sellers characterized by a set of n fulfillment characteristics What are these characteristics (valued by consumers?)  We think of each characteristic as a dimension, represented by a noun or verb phrase (“shipping”, “packaging”, “delivery”, “arrived”)  We scan the textual feedback to discover these dimensions

seller life seller ranking Data: Reputation Profiles

Decomposing and scoring reputation  We think of each characteristic as a dimension, represented by a noun or verb phrase (“shipping”, “packaging”, “delivery”, “arrived”)  The sellers are rated on these dimensions by buyers using modifiers (adjectives or adverbs), not numerical scores  “Fast shipping!”  “Great packaging”  “Awesome unresponsiveness”  “Unbelievable delays”  “Unbelievable price”

Dimensions from text: Example Parsing the feedback  P1: I was impressed by the speedy delivery! Great Service!  P2: The item arrived in awful packaging, and the delivery was slow  … Identified modifier-dimension pairs  P1: “speedy – delivery”, “great – service”  P2: “awful – packaging”, “slow – delivery”  … Reducing textual feedback to a n X p matrix  Dimensions: 1-delivery, 2-packaging, 3-service Postings

Decomposing and scoring reputation Scoring reputation  “Fast shipping!”  “Great packaging”  “Awesome unresponsiveness”  “Unbelievable delays”  “Unbelievable price” How can we find out the meaning of these adjectives?

The dimensions of reputation  We assume that each modifier assigns a “score” to each dimension  :score associated with  appearing as the modifier for the k-th dimension  r i : weight of posting that appears on the i-th position (weight down old posts)  w i : weight assigned to the i-th dimension  Thus, the overall (text) reputation score Π(i) is: scores for first dimension scores for n-th dimension Sum of r i weights in which  j modifies dimension i estimated coefficients scores for first posting

The dimensions of reputation Scoring the dimensions  Use price premiums as “true” reputation score  Use regression to assess scores (coefficients) for each dimension- modifier pair Regressions  Control for all variables that affect price premiums  Control for all numeric scores of reputation  Examine effect of text: E.g., seller with “fast delivery” has premium $10 over seller with “slow delivery” estimated coefficients

Some indicative dollar values Positive Negative Natural method for extraction of sentiment strength and polarity

Results Some dimensions that matter  Delivery and contract fulfillment (extent and speed)  Product quality and appropriate description  Packaging  Customer service  Price (!)  Responsiveness/Communication (speed and quality)  Overall feeling (transaction)

Results Further evidence  Classifier (aka choice model) that predicts sale given set of sellers  Binary decision between seller and competitor  Naïve Bayes and Decision Trees (SVM’s forthcoming)  Only prices and characteristics: 53%  + numerical reputation, lifetime: 74%  + encoded textual information: 89%

Other applications Summarize and query reputation data  Give me all merchants that deliver fast SELECT merchant FROM reputation WHERE delivery > ‘fast’  Summarize reputation of seller XYZ Inc.  Delivery: 3.8/5  Responsiveness: 4.8/5  Packaging: 4.9/5 Pricing reputation  Given the competition, merchant XYZ can charge $20 more and still make the sale (confidence: 83%)

Summary Key contributions  New technique that automatically scores “sentiment” based on economic data  Validation by multiple methods (estimating an econometric model, building classifiers)  New evidence of the extent to which interdisciplinary research can be fun and distracting Broader contribution  Economic data is abundant and there is rich literature on how to handle such data  Economic data can be used for training for MANY applications

Moving ahead Extensions of current work  Dimensionality reduction, grouping dimensions topics that might correspond more closely to the “true” dimensions of reputation  Latent Dirichlet Allocation, (probabilistic) Latent Semantic Analysis, Non-negative Matrix Factorization, Tensors  Identifying weights for dimensions, using normalized scores  “Correct” game theoretic model of market competition Exploiting network structure  Exploring connection with the “trustrank” literature  Network position as an additional dimension of seller reputation  Buyers as seller/category specific “authorities”

Thank you!

Prior studies of reputation Positive feedback significant, negative not  Ba and Pavlou (2002) for CD’s, software, electronics; Bajari and Hortacsu (2003) for collectible coins? Negative feedback significant, positive not  Lee et al. (2000) for computer equipment, Reiley et al. (2000) for collectible coins Nature of price: winning online auction bid (usually eBay) Measure of reputation: average numerical score, # of transactions Both positive and negative feedback significant  Dewan and Hsu (2004) for rare stamps, Melnik and Alm (2002) for gold coins, Houser and Wooders (2005) for Pentium chips