Stylistics in Customer Reviews of Cultural Objects Xiao Hu, J. Stephen Downie The International Music Information Retrieval Systems Evaluation Lab (IMIRSEL) University of Illinois at Urbana-Champaign THE ANDREW W. MELLON FOUNDATION

Agenda zMotivation zCustomer reviews in epinions.com zExperiments  Genre classification  Rating classification  Usage classification  Feature studies zConclusions & Future Work

Motivation  Online customer reviews on culture objects:  User-generated user-centered retrieval  Detailed descriptions contextual info.  Large amountrich resource  Self-organized ground truth zText mining:  Mature techniques and Handy tools zReview mining: a place to play Stylistics Text Analysis!

Motivation Classify Reviews Identify User Descriptions Connect to Objects Customer Reviews Epinions.com Amazon.com ….. Class 1 Class 2 Description 1 D1D2D3 Prominent Features Genres Ratings Usages D1D2D3 User- centered access points

Customer Reviews  Published on  Focused on the book, movie and music z Each review associated with:  a genre label  a numerical quality rating  a recommended usage (for music reviews)

numerical rating associated full text, to be analyzed recommended usage

Genre Taxonomy (music) Jazz, Rock, Country, Classical, Blues, Gospel, Punk,.… Renaissance, Medieval, Baroque, Romantic, … 28 Major Genre Categories

Experiments zto build and evaluate a prototype system that could automatically :  predict the genre of the work being reviewed  predict the quality rating assigned to the reviewed item  predict the usage recommended by the reviewer  discover distinctive features contributing to each of the above

Models and Methods zPrediction problem:  Naïve Bayesian (NB) Classifier  Computationally efficient  Empirically effective  Hierarchical clustering (for usage prediction only) zFeature analysis:  Frequent pattern mining  Naïve Bayesian feature ranking

Data Preprocessing zHTML tags were stripped out; zStop words were NOT stripped out; zPunctuation was NOT stripped out;  They may contain stylistic information zTokens were stemmed

Genre Classifications zData set Reviews onBookMovieMusic #. Of reviews #. Of genres Mean of review length 1,095 words 1,514 words 1,547 words Std. Dev. of review length 446 words 672 words 784 words Term list size 41,06047,01547,864

Genres Examined BookMovieMusic Action / ThrillerAction /AdventureBlues Juvenile FictionChildrenClassical HumorComediesCountry HorrorHorror/SuspenseElectronic Music & Performing ArtsMusical & Performing ArtsGospel Science Fiction & FantasyScience-Fiction / FantasyHardcore/Punk Biography & AutobiographyDocumentaryHeavy Metal Mystery & CrimeDramasInternational RomanceEducation/General InterestJazz Instrument Japanimation (Anime)Pop Vocal WarR&B Rock & Pop

Genre Classification Results Reviews onBookMovieMusic Number of genres91112 Reviews in each genre Term list size (terms)41,06047,01547,864 Mean of review length (words)1,0951,5141,547 Std Dev of review length (words) Mean of precision72.18%67.70%78.89% Std Dev of precision1.89%3.51%4.11% 5 fold random cross validation for book and movie reviews 3 fold random cross validation for music reviews

Confusion : Book Reviews Classified As  ActionBio.Hor.Hum.Juv.Mus.Mys.Rom.Sci. Action Bio Horror Humor Juvenile Music Mystery Romance Science

Confusion : Movie Reviews Classified As  Act.Ani.Chi.Com.Doc.Dra.Edu.Hor.Mus.Sci.War Action Anime Children Comedy Docu Drama Edu Horror Music Science War

Confusion : Music Reviews Classified As  Blu.Cla.CouEle.Gos.Pun.Met.Int’lJazzPop.RBRoc. Blues Classical Country Electr Gospel Punk Metal Int’l Jazz Pop Vo R&B Rock

Rating Classification zFive-class classification  1 star vs. 2 stars vs. 3 stars vs. 4 stars vs 5 stars zBinary Group classification  1 star + 2 stars vs. 4 stars + 5 stars zad extremis classification  1 star vs. 5 stars 5 fold random cross validation for Book and Movie review experiments 5 fold cross validation for Music review experiments

Rating : Book Reviews Experiments5 classesBinary Group Ad extremis Number of classes522 Reviews in each class Term list size (terms)34,12328,33923,131 Mean of review length (words)1,2401,2281,079 Std Dev of review length (words) Mean of precision36.70%80.13%80.67% Std Dev of precision1.15%4.01%2.16%

Rating : Movie Reviews Experiments5 classesBinary Group Ad extremis Number of classes522 Reviews in each class Term list size (terms)40,23536,62031,277 Mean of review length (words)1,6401,6451,409 Std Dev of review length (words) Mean of precision44.82%82.27%85.75% Std Dev of precision2.27%2.02%1.20%

Rating : Music Reviews Experiments5 classesBinary Group Ad extremis Number of classes522 Reviews in each class Term list size (terms)35,60033,08432,563 Mean of review length (words)1,8752,0321,842 Std Dev of review length (words) Mean of precision44.25%79.75%85.94% Std Dev of precision2.63%3.59%3.58%

Confusion : Book Reviews Classified As  1 star2 stars3 stars4 stars5 stars 1 star stars stars stars stars

Confusion : Movie Reviews Classified As  1 star2 stars3 stars4 stars5 stars 1 star stars stars stars stars

Confusion : Music Reviews Classified As  1 star2 stars3 stars4 stars5 stars 1 star stars stars stars stars

Usage Classification zEach music review has one usage suggested by the reviewer zIt can be chosen from a ready-made list of 13 usages zChose the most popular 11 usages for experiments

Usage Categories and Counts UsageCountUsageCount Driving (DRV)1,349Waking up (WKU)271 Hanging With Friends (HWF)1,215Going to Sleep (GTS)269 Listening (LST)592Cleaning the House (CTH)230 Romancing (ROM)492At Work (AWK)188 Reading or Studying (ROS)447With Family35 Getting ready to go out (GRG)378Sleeping15 Exercising (EXC)291TOTAL5,772

Data and initial result ExperimentsAll classes Number of classes11 Reviews in each class180 Term list size (terms)36,561 Mean of review length (words) Std Dev of review length (words) Mean of precision19.55% Std Dev of precision2.89% 10 fold cross validation

Confusion matrix Classified As  AWKCTHDRVEXCGRGGTSHWFLSTROSROMWKU AWK CTH DRV EXC GRG GTS HWF LIS ROS ROM WKU

Usage super-classes zFrequent confusions: a measure of similarity zHierarchical clustering based on the confusion matrix

Hierarchical clustering Going to sleep Listening Getting ready to go out Driving Reading or studying Romancing Cleaning the house At work Hanging out with friends Waking up Exercising Relaxing Stimulating R1 R2 S1 S2

Classifications on usage super-classes ExperimentsRelaxing, Stimulating R1,R2, S1,S2 Number of classes24 Reviews in each class Term list size (terms)34,75930,637 Mean of review length (words) Std Dev of review length (words) Mean of precision65.72%42.60% Std Dev of precision3.15%4.60% 10 fold cross validation

Feature studies zWhat makes the classes distinguishable? zWhat are important features? zHow important are they? zTwo techniques applied  Frequent Pattern Mining  Naïve Bayesian Feature Ranking zFocus on music reviews

Frequent Pattern Mining (FPM) z Originally used to discover association rules z Finds patterns consisting of items that frequently occur together in individual transactions  Items = candidate words (terms) depending on specific questions zTransactions = review sentences Items Transactions

Positive and negative descriptive patterns zRecall: rating classification on music reviews Experiments5 classesBinary Group Ad extremis Number of classes522 Reviews in each class Term list size (terms)35,60033,08432,563 Mean of review length (words)1,8752,0321,842 Std Dev of review length (words) Mean of precision44.25%79.75%85.94% Std Dev of precision2.63%3.59%3.58%

Positive and negative descriptive patterns Mining frequent descriptive patterns in positive and negative reviews ReviewsPositiveNegative Total Reviews400 Total Sentences Total Words Avg. (STD ) sentences per review (75.49)75.13 (41.62) Avg. (STD) words per sentence16.28 (14.43)14.89 (12.24) adjectives, adverbs and verbs, negatives no nouns, no stopwords

Single term patterns Positive ReviewsNegative Reviews not – 3417 sentences good – 1621 sentences: 1/4 of all sentences not – 1915 sentences good – 1025 sentences: 1/3 of all sentences Good = Bad?! Digging deeper ----

good in a negative context Negation: “Nothing is good.” “It just doesn't sound good.” Song titles: “Good Charlotte, you make me so mad.” “Feels So Good is dated and reprehensibly bad.” Rhetoric: “And this is a good ruiner: …” “What a waste of my good two dollars…” Faint praise: “…the only good thing… is the packaging.” Expressions: “You all have heard … the good old cliché.”

Double term patterns Positive ReviewsNegative Reviews not good not realli realli good not listen not great not good not bad not realli not sound realli good Good  Bad?! Digging deeper and deeper --

Triple term patterns Positive ReviewsNegative Reviews sing open melod sing smooth melod sing fill melod sing smooth open not realli good sing lead melod sound realli good sing plai melod accompani sing melod sing soft melod not realli good not realli listen bad not good bad not sound pretti tight spit bad not don’t realli not don’t realli bad not pretti bad not not sing sound

Noun patterns in genre classification Reviews onMusic Number of genres12 Reviews in each genre150 Term list size (terms)47,864 Mean of review length (words)1,547 Std Dev of review length (words)784 Mean of precision78.89% Std Dev of precision4.11% z Recall: genre classification on music reviews

Noun patterns in genre classification zStudied four popular genres zOnly nouns considered ReviewsClassicalCountryHeavyMetalJazzInstr Total Reviews150 Total Sentences Total Words Avg. (STD ) sentences per review (32.68) (43.77) (71.69) (28.60) Avg. (STD) words per sentence (12.25) (11.79) (12.33) (10.16)

Single term patterns ClassicalCountryHeavy MetalJazz Instrument music record piec cd work song album love music time song album guitar band track song album music solo time

Double term patterns ClassicalCountryHeavy MetalJazz Instrument cd music music piec piec piano piano concerto orchestra symphoni music record piano op music work music time music compos violin concerto cd piec cd record twain shania dixi chick station union guitar steel tim mcgraw cash johnni titl track song titl krauss alison drum guitar countri radio song beat style song album song song guitar riff guitar guitar bass drum guitar song lyric song riff song choru solo guitar song track album track band album band song music jazz liner note drum bass jazz album album song jazz song guitar bass tenor sax solo song piano bass mile davi solo piano section rhythm

Naïve Bayesian Feature Ranking (NBFR) zBased on NB text categorization model Prediction in binary classification cases: > 0, d i is in C j < 0, d i is not in C j

Features in usage super-classes zRecall: classification on usage super-classes ExperimentsRelaxing, Stimulating Number of classes2 Reviews in each class900 Term list size (terms)34,759 Mean of review length (words) Std Dev of review length (words) Mean of precision65.72% Std Dev of precision3.15%

Top-ranked terms in super-classes RelaxingStimulating Botti (Chris) Shelby (Lynne) Bethany (Joy) Debelah (Morgan) Mckennitt (Loreena) Pontiy(Jean Luc) Shabazz (lyricist) Tru nightwish Tarja (Turunen) Dio (Ronnie James) Roca (Zach De La Roca) Slade (British band) Incubus (band) Edan (rap artist) Twiztid (band) KJ (KJ52) blue Serj (Tankian) Stooges (The) Terms in ()’s were manually added for clarity

Artist-usage relationship zBinomial exact test on artists with >10 reviews (p < 0.05) ArtistUsagep value AFIWaking Up Black SabbathAt Work Celine DionRomancing Dream TheaterListening MetallicaWaking Up Nirvana_(USA)Going to Sleep

Implementation & T2K (demo) zText-to-Knowledge (T2K) Toolkit A text mining framework Ready-to-use modules and itineraries Natural Language Processing tools integrated Supporting fast prototyping of text mining Data Preprocessing NB Classifier

Conclusions zText analysis of user-generated reviews on culture objects  NB on genre, rating, and usage classification  Feature studies: FPM and NBFR  Customer reviews are good resources for connecting users’ opinions to cultural objects and thus facilitating information access via novel, user-oriented facets.

Future work zMore text mining techniques zOther critical text  blogs, wikis, etc zFeature studies  other kinds of features

Questions? IMIRSEL Thank you! THE ANDREW W. MELLON FOUNDATION