1 Rated Aspect Summarization of Short Comments Yue Lu, ChengXiang Zhai, and Neel Sundaresan

2 Web 2.0  Opinions Everywhere Novotel …… Overall Rating iPhone Sushi Kame

Seller’s Feedback on eBay 23,385 Feedback received Very fast shipping and awesome price!!! 3

Seller’s Feedback on eBay 4

Need More Specific Aspects! Fast shipping Is this seller rated high/low mainly because of service? Which seller provides fast shipping? Good service 5

6 Rated Aspect Summarization AspectAspect Rating Representative Phrase Support Information Challenges: –How to identify coherent aspects? with user interest? –How to accurately rate each aspect? –How to get meaningful phrases supporting the ratings? 23,385 Feedback received 6

Overall Approach 7 Step1: Aspect Discovery and Clustering Step2: Aspect Rating PredictionStep3:Extract Representative Phrases 7

8 Preprocessing of Short Comments 2 1 Source businessgreat sellerhonest priceawesome shippingfast Head Term (feature)‏ Modifier (opinion)‏ Very fast shipping and awesome price!!! Great business, honest seller Shallow parsing Comment 1 Comment 2

Step1: Step1: Aspect Discovery & Clustering 9 Step1: Aspect Discovery and Clustering Step2: Aspect Rating PredictionStep3:Extract Representative Phrases 9

10 Method(1) Head Method(1) Head Term Clustering 2 1 Source shippingfast sellerhonest sellerreliable deliveryquick shippingfast Head TermModifier fast:100 speedy:80 slow:50 …Shipping fast:120 speedy:85 slow:70 …Delivery honest:80 reliable:60 …Seller Head TermModifiers Clustering: e.g. k-means Clustering: e.g. k-means Support = Cluster Size

Method(2) Method(2) Unstructured PLSA 2 1 Source shippingfast sellerhonest sellerreliable deliveryquick shippingfast Head TermModifier … 11 22 kk w  d1  d2  dk shiping 0.3 delivery 0.2 service 0.32 exchange comm [Hofmann 99] Topic model = unigram language model = multinomial distribution 11

Method(2) Unstructured PLSA 2 1 Source shippingfast sellerhonest sellerreliable deliveryquick shippingfast Head TermModifier … 11 22 kk w  d1  d2  dk shiping delivery service exchange comm. [Hofmann 99] Topic model = unigram language model = multinomial distribution ? ? ? ? ? ? Estimation: e.g. EM with MLE Estimation: e.g. EM with MLE 12

Method(3) S Method(3) Structured PLSA 2 1 Source deliveryfast Sellerhonest sellerreliable deliveryquick Shippingfast Head TermModifier … 11 22 kk w  d1  d2  dk shiping delivery service exchange comm. ? ? ? ? ? ? shipping: 70 slow delivery: 80 response: 10 delivery: 30 shipping:180fast Head TermModifier 13

Method(2) Method(2) (3): Topics  Aspects … 11 22 kk w  d1  d2  dk shiping 0.3 delivery 0.2 service 0.32 exchange comm Support = Topic Coverage TopicsAspects 14

Method(2) Method(2) (3): Adding Prior to PLSA … 11 22 kk w  d1  d2  dk shiping ? delivery ? service ? exchange ? ? comm. ? a1a1 a2a2 Dirichlet PriorTopics shiping delivery comm. Estimation: e.g. EM with Maximum A Posteriori (MAP) instead of MLE Estimation: e.g. EM with Maximum A Posteriori (MAP) instead of MLE 15

Step2: Step2: Aspect Rating Prediction 16 Step1: Aspect Discovery and Clustering Step2: Aspect Rating PredictionStep3:Extract Representative Phrases 16

Method(1) Method(1) Local Prediction productfine packagedpoorly deliveryslow 2 … 1 Source …… productgreat shippingfast Head TermModifier Shipping Aspects Product slow Shipping Packaging Product What if? 17

Method(2) Method(2) Global Prediction Shipping Aspects Product Shipping Packging Product productfine Packagedpoorly deliveryslow 2 … 1 Source …… productgreat shippingfast Head TermModifier fast, timely, quick, fast, slow, quickly, fast, great, bad Shipping slow, bad, fast, poor, slowly, unbearable, quick, poor Shipping What if? slow shipping What if? slow shipping fast 0.2 timely 0.2 quick 0.2 …… slow 0.01 Shipping slow 0.4 bad 0.2 … … quick 0.02 fast 0.01 Shipping Language Model 18

19 Method(1)(2): Method(1)(2): Rating Aggregation slow shipping Fast delivery quick shipping AVG 2.33 stars badly wrapped poor packaging well packaged AVG 1.67 stars Aspect Rating Shipping Packaging Aspect

Step3: Step3: Representative Phrases 20 Step1: Aspect Discovery and Clustering Step2: Aspect Rating PredictionStep3:Extract Representative Phrases 20

21 Step3: Step3: Top K Frequent Phrases Fast shipping Timely delivery Quickly arrived Slow shipment Bad shipping Slow delivery Step 1Step 2Step 3 slow delivery Fast delivery quick shipping Shipping bad shipping Support = Phrase Freq. (50)‏

22 Experiments: eBay Data Set 28 eBay sellers with high feedback scores for the past year overall rating (positive %)‏ # of phrases/comment # of comments/seller Statistics ,39557,055 STDMean Positive  rating 1 Neutral  rating 0 Negative  rating 0

23 Experiments: Evaluate Step 1 Step1: Aspect Discovery & Clustering Gold standard: human labeled clusters

24 Eval Step 1: Aspect Coverage Aspect Coverage measures the percentage of covered aspects Top K Clusters Aspect Coverage k-means Unstructured PLSA Structured PLSA

25 Eval Step 1: Clustering Accuracy Clustering Accuracy measures the cluster coherence Structured PLSA Unstructured PLSA K-means Method Clustering Accuracy Annot Seller Seller1 AVG Annot1-3 Annot AVGSeller3 Low Agreement; Varies a lot Low Agreement; Varies a lot Still much room for improvement! Human Agreement

26 Experiments: Evaluate Step 2 Step2: Aspect Rating Prediction

27 Detailed Seller Ratings as Gold std Gold standard: user DSR ratings DSR criteria as priors of aspects

28 Eval Step 2: Correlation (-108%)‏ (-58%)‏GlobalK-means (-62%)‏ Kendal’s tau Local Step 2 K-means Baseline Step (-45%)‏ Pearson (+39%)‏ (+76%)‏GlobalUnstr. PLSA LocalUnstr. PLSA (+35%)‏ (+119%)‏GlobalStr. PLSA LocalStr. PLSA Correlation measures the effectiveness of ranking the four DSRs for a given seller

29 Eval Step 2: Ranking Loss (-16%)‏LocalUnstr. PLSA (-11%)‏GlobalUnstr. PLSA (-19%)‏LocalStr. PLSA (+167%)‏GlobalK-means (-35%)‏GlobalStr. PLSA Local Step 2 K-means Baseline Step (-8%)‏ AVG of 3 DSR Ranking Loss measures the distance between the true and predicted ratings (smaller  better)‏

30 Experiments: Evaluate Step 3 Step3: Representative Phrases Questions: –How do previous steps affect the phrase quality?

31 Eval Step 3: Human Labeling Item as Described Communication Shipping time Shipping and Handling Charges Rating 1DSRRating 0 Rating 1: Rating 0: Fast deliveryPrompt Slow shipping… Excessive postageAs promised…

32 Eval Step 3: Measures & Results Prec LocalUnstr. PLSA GlobalUnstr. PLSA LocalStr. PLSA GlobalK-means GlobalStr. PLSA Local Step 2 K-means Step Recall Information Retrieval measures: Human generated phrases  “relevant document“ Computer generated phrases  “retrieved document".

33Summary Novel problem – Rated Aspect Summarization General Methods –Three steps –Effective on eBay Feedback Comments Future Work –Evaluate on other data –Three steps  One optimization framework

34 Thank you!

PLSA & EM Formulas

Structured PLSA & EM Formulas

Incorporated with prior