1. Panos Ipeirotis New York University Opinion Mining using Econometrics A Case Study on Reputation Systems Joint work with Anindya Ghose and Arun Sundararajan

2. Comparative Shopping in e-Marketplaces

3. Are Customers Irrational?

4. $11.04 $18.28 -$0.61 -$9.00 -$11.40 -$1.04 Price Premiums

5. Price premiums @ Amazon Irrational (?)

6. Average price premiums @ Amazon Irrational (?)

7. Why not buying the cheapest? You buy more than a product  Customers do not pay only for the product  Customers also pay for a set of fulfillment characteristics  Delivery  Packaging  Responsiveness  … Reputation Matters!

8. Reputation Systems Facilitate electronic commerce  Integral part of online marketplaces  Provide information about unobserved fulfillment characteristics (most of which we take for granted in traditional commerce) Reputation in ecommerce is complex  Different buyers value different fulfillment characteristics  Sellers have varying abilities on these characteristics

9. Example of a reputation profile

11. Reputation profiles: Observations Reputation profile capture more than “averages”  Well beyond “average score” and “lifetime”  Rich textual content: information about a seller on a variety of dimensions (or fulfillment characteristics).  How the seller’s performance (potentially on each of these characteristics) has evolved over time  Buyer-seller networks Reputation in ecommerce is complex  Different buyers value different fulfillment characteristics  Sellers have varying abilities on these characteristics Previous work studied only effect of “average score” and “lifetime”

12. Our research agenda What are the dimensions of online reputation?  What characteristics comprise the important parts of a seller’s overall reputation? (politeness? packaging? delivery?) How do these dimensions affect pricing power?  Does a better reputation enable a seller to charge a higher price?  Which dimensions affect this pricing power most significantly?  Average numerical ratings?  Number of prior successful transactions?  Assessments of ability on specific fulfillment characteristics?  Do competitors with better reputations limit a seller’s pricing power? Can prior reputation predict marketplace outcomes?  Given a set of sellers, their reputations, and their prices, can one predict which seller will successfully make the sale?

13. What’s different about our research? Assessment of reputation based on “realistic” model  Include information contained in text feedback  Place higher weight on more recent postings Identify dimensions of reputation automatically  Discover automatically important dimensions  Modifiers of each dimension scored based on their impact on pricing power amazingserviceunresponsiveness Evidence using multiple techniques  Numeric reputation, text feedback explain variation in pricing power  Numeric reputation, text feedback improve predictions of “making the sale”.

14. Model Aspects that matter later on  Set of (n) fulfillment characteristics (separate from product quality)  Buyers heterogeneous on value placed on each: w=(w 1,w 2,…,w n )  Buyers assess seller fulfillment quality based on reputation and w

15. Data Overview  Panel of 280 software products sold by Amazon.com  Data on all “secondary” market transactions  Amazon Web services facilitate capturing transactions  Complete reputation profile for all sellers who completed one or more transactions during this period Summary  280 products X 180 days  1,078 sellers, of which 122 transacted  12,232 transactions  107,922 “observations” (seller-competitor pairs)

16. Data: Transactions

17. Sales of (mostly new) software Data: Transactions

18. Capturing transactions and “price premiums” Data: Transactions Seller ListingItemPrice When item is sold, listing disappears

19. Capturing transactions and “price premiums” Data: Transactions While listing appears, item is still available time 1/11/21/31/41/51/61/71/81/91/10

20. Capturing transactions and “price premiums” Data: Transactions While listing appears, item is still available time 1/11/21/31/41/51/61/71/81/91/10 Item still not sold on 1/7

21. Capturing transactions and “price premiums” Data: Transactions When item is sold, listing disappears time 1/11/21/31/41/51/61/71/81/91/10 Item sold on 1/9

22. Data: Variables of Interest Regular Price Premium  Difference in the price charged by a seller and the listed price of a competing seller at the time the transaction occurred (Seller Price – Competitor Price)  Calculated for each seller-competitor pair, for each transaction  Each transaction therefore generates N observations, where N is the number of competing sellers Average Price Premium  Difference in the price charged by a seller and the average price of all competing sellers at the time the transaction occurred (Seller Price – Avg. (Competitor Price) )  Calculated for each transaction  Each transaction generates 1 observation

23. Price premiums @ Amazon

24. Average price premiums @ Amazon

25. The dimensions of reputation How reputation affects price premiums?

26. Decomposing reputation Is reputation just a scalar metric?  Previous studies assumed a “monolithic” reputation.  We break down reputation in individual components  Sellers characterized by a set of n fulfillment characteristics What are these characteristics (valued by consumers?)  We think of each characteristic as a dimension, represented by a noun or verb phrase (“shipping”, “packaging”, “delivery”, “arrived”)  We scan the textual feedback to discover these dimensions

27. seller life seller ranking Data: Reputation Profiles

30. Decomposing and scoring reputation  We think of each characteristic as a dimension, represented by a noun or verb phrase (“shipping”, “packaging”, “delivery”, “arrived”)  The sellers are rated on these dimensions by buyers using modifiers (adjectives or adverbs), not numerical scores  “Fast shipping!”  “Great packaging”  “Awesome unresponsiveness”  “Unbelievable delays”  “Unbelievable price”

31. Dimensions from text: Example Parsing the feedback  P1: I was impressed by the speedy delivery! Great Service!  P2: The item arrived in awful packaging, and the delivery was slow  … Identified modifier-dimension pairs  P1: “speedy – delivery”, “great – service”  P2: “awful – packaging”, “slow – delivery”  … Reducing textual feedback to a n X p matrix  Dimensions: 1-delivery, 2-packaging, 3-service Postings

32. Decomposing and scoring reputation Scoring reputation  “Fast shipping!”  “Great packaging”  “Awesome unresponsiveness”  “Unbelievable delays”  “Unbelievable price” How can we find out the meaning of these adjectives?

33. The dimensions of reputation  We assume that each modifier assigns a “score” to each dimension  :score associated with  appearing as the modifier for the k-th dimension  r i : weight of posting that appears on the i-th position (weight down old posts)  w i : weight assigned to the i-th dimension  Thus, the overall (text) reputation score Π(i) is: scores for first dimension scores for n-th dimension Sum of r i weights in which  j modifies dimension i estimated coefficients scores for first posting

34. The dimensions of reputation Scoring the dimensions  Use price premiums as “true” reputation score  Use regression to assess scores (coefficients) for each dimension- modifier pair Regressions  Control for all variables that affect price premiums  Control for all numeric scores of reputation  Examine effect of text: E.g., seller with “fast delivery” has premium $10 over seller with “slow delivery” estimated coefficients

35. Some indicative dollar values Positive Negative Natural method for extraction of sentiment strength and polarity

36. Results Some dimensions that matter  Delivery and contract fulfillment (extent and speed)  Product quality and appropriate description  Packaging  Customer service  Price (!)  Responsiveness/Communication (speed and quality)  Overall feeling (transaction)

37. Results Further evidence  Classifier (aka choice model) that predicts sale given set of sellers  Binary decision between seller and competitor  Naïve Bayes and Decision Trees (SVM’s forthcoming)  Only prices and characteristics: 53%  + numerical reputation, lifetime: 74%  + encoded textual information: 89%

38. Other applications Summarize and query reputation data  Give me all merchants that deliver fast SELECT merchant FROM reputation WHERE delivery > ‘fast’  Summarize reputation of seller XYZ Inc.  Delivery: 3.8/5  Responsiveness: 4.8/5  Packaging: 4.9/5 Pricing reputation  Given the competition, merchant XYZ can charge $20 more and still make the sale (confidence: 83%)

39. Summary Key contributions  New technique that automatically scores “sentiment” based on economic data  Validation by multiple methods (estimating an econometric model, building classifiers)  New evidence of the extent to which interdisciplinary research can be fun and distracting Broader contribution  Economic data is abundant and there is rich literature on how to handle such data  Economic data can be used for training for MANY applications

40. Moving ahead Extensions of current work  Dimensionality reduction, grouping dimensions topics that might correspond more closely to the “true” dimensions of reputation  Latent Dirichlet Allocation, (probabilistic) Latent Semantic Analysis, Non-negative Matrix Factorization, Tensors  Identifying weights for dimensions, using normalized scores  “Correct” game theoretic model of market competition Exploiting network structure  Exploring connection with the “trustrank” literature  Network position as an additional dimension of seller reputation  Buyers as seller/category specific “authorities”

41. Thank you! http://economining.stern.nyu.edu

42. Prior studies of reputation Positive feedback significant, negative not  Ba and Pavlou (2002) for CD’s, software, electronics; Bajari and Hortacsu (2003) for collectible coins? Negative feedback significant, positive not  Lee et al. (2000) for computer equipment, Reiley et al. (2000) for collectible coins Nature of price: winning online auction bid (usually eBay) Measure of reputation: average numerical score, # of transactions Both positive and negative feedback significant  Dewan and Hsu (2004) for rare stamps, Melnik and Alm (2002) for gold coins, Houser and Wooders (2005) for Pentium chips