1. 1 Web Usage Mining Modelling: frequent-pattern mining I (sequence mining with WUM), classification and clustering) Prof. Dr. Bettina Berendt Humboldt Univ. Berlin, Germany www.berendt.de

2. 2 Please note These slides use and/or refer to a lot of material available on the Internet. To reduce clutter, credits and hyperlinks are given in the following ways: n Slides adapted from other people‘s materials: at bottom of slide n Pictures, screenshots etc.: URL visible in screenshot or given in PPT „Comments“ field n Literature, software: On the accompanying Web site http://vasarely.wiwi.hu-berlin.de/WebMining07/ http://vasarely.wiwi.hu-berlin.de/WebMining07/ Thanks to the Internet community! You are invited to re-use these materials, but please give the proper credit.

3. 3 Stages of knowledge discovery discussed in this lecture Application understanding

4. 4 An addendum to the association rules: main interestingness measures of association rules (and a recommendation for postprocessing the result set) n Support of a rule A  B = no. of instances with A and B / no. of all instances n Confidence of a rule A  B = no. of instances with A and B / no. of instances with A = support (A & B) / support (A) n Lift of a rule A  B = support (A & B) / [ support (A) * support (B) ] l What does this measure, and in what numerical interval can it be? n Deleting redundant rules from the result set: l If you have A  B and A & C  B, the second rule is redundant.

5. 5 Agenda Sequence mining: tool WUM (case study “school search”) Classification: method Naïve Bayes (case study “happiness”) Clustering: tool DocumentAtlas (case study “EU proposals”) A very short note on other uses of clustering (e.g. in query mining) Some observations on privacy... Best-practice „design patterns“with open-source tools

6. 6 Demonstration of WUM

7. 7 The site Business understanding / problem definition: * How do users search in this online catalog? * Which search criteria are popular? * Which are efficient? [Berendt & Spiliopoulou, VLDB Journal 2000]

8. 8 The concept hierarchies / site ontology (excerpt) SEITE1-...LI (1st page of a list) or SEITEn-...LI (further page) LA („Land“)SA („Schulart“)SU („Suche“)

9. 9 Sequence mining – one result pattern: successful search for a school in Germany a refinement a repetition a continuation one example pattern select t from node a b, template a * b as t where a.url startswith "SEITE1-" and a.occurrence = 1 and b.url contains "1SCHULE" and b.occurrence = 1 and (b.support / a.support) >= 0.2 (Berendt & Spiliopoulou, VLDB J. 2000) /liste.html?offset=920&ze ilen=20&anzahl=1323&sprac he=de&sw_kategorie=de&ers cheint=&suchfeld=&suchwer t=&staat=de&region=by&sch ultyp=

10. 10 Sequences

11. 11 Generalized sequences, navigation patterns, hits in WUM

12. 12 Aggregated Logs: The basic internal representation in WUM

13. 13 The confi- dence measure for genera- lized sequences

14. 14 Templates in the query language MINT, g-sequences, and navigation patterns

15. 15 Interestingness measures: Support (hits) and confidence

16. 16 Aggregated Logs, queries, and query results

17. 17 The basic idea of the WUM algorithm

18. 18 MINT can express 3 types of constraints (“predicates“)

19. 19 The WUM gseqm algorithm (B predicates)

20. 20 Agenda Sequence mining: tool WUM (case study “school search”) Classification: method Naïve Bayes (case study “happiness”) Clustering: tool DocumentAtlas (case study “EU proposals”) A very short note on other uses of clustering (e.g. in query mining) Some observations on privacy... Best-practice „design patterns“with open-source tools

21. 21 “What makes people happy?” – a corpus-based approach to finding happiness

22. 22 Bayes‘ formula and its use for classification 1. Joint probabilities and conditional probabilities: basics n P(A & B) = P(A|B) * P(B) = P(B|A) * P(A) n  P(A|B) = ( P(B|A) * P(A) ) / P(B) (Bayes´ formula) n P(A) : prior probability of A (a hypothesis, e.g. that an object belongs to a certain class) n P(A|B) : posterior probability of A (given the evidence B) 2. Estimation: n Estimate P(A) by the frequency of A in the training set (i.e., the number of A instances divided by the total number of instances) n Estimate P(B|A) by the frequency of B within the class-A instances (i.e., the number of A instances that have B divided by the total number of class-A instances) 3. Decision rule for classifying an instance: n If there are two possible hypotheses/classes (A and ~A), choose the one that is more probable given the evidence n (~A is „not A“) n If P(A|B) > P(~A|B), choose A n The denominators are equal  If ( P(B|A) * P(A) ) > ( P(B|~A) * P(~A) ), choose A

23. 23 Simplifications and Naive Bayes 4. Simplify by setting the priors equal (i.e., by using as many instances of class A as of class ~A) n  If P(B|A) > P(B|~A), choose A 5. More than one kind of evidence n General formula: n P(A | B 1 & B 2 ) = P(A & B 1 & B 2 ) / P(B 1 & B 2 ) = P(B 1 & B 2 | A) * P(A) / P(B 1 & B 2 ) = P(B 1 | B 2 & A) * P(B 2 | A) * P(A) / P(B 1 & B 2 ) n Enter the „naive“ assumption: B 1 and B 2 are independent given A n  P(A | B 1 & B 2 ) = P(B 1 |A) * P(B 2 |A) * P(A) / P(B 1 & B 2 ) n By reasoning as in 3. and 4. above, the last two terms can be omitted n  If (P(B 1 |A) * P(B 2 |A) ) > (P(B 1 |~A) * P(B 2 |~A) ), choose A n The generalization to n kinds of evidence is straightforward. n These kinds of evidence are often called features in machine learning.

24. 24 Example: Texts as bags of words Common representations of texts n Set: can contain each element (word) at most once n Bag (aka multiset): can contain each word multiple times (most common representation used in text mining) Hypotheses and evidence n A = The blog is a happy blog, the email is a spam email, etc. n ~A = The blog is a sad blog, the email is a proper email, etc. n B i refers to the i th word occurring in the whole corpus of texts Estimation for the bag-of-words representation: n Example estimation of P(B 1 |A) : l number of occurrences of the first word in all happy blogs, divided by the total number of words in happy blogs (etc.)

25. 25 WEKA – NaiveBayes and NaiveBayesMultinomial n The WEKA classifier learning scheme NaiveBayesMultinomial implements this model of „the probability that a word occurs in a document given that the document is in that classs“. l Its output is a table giving these probabilities n The WEKA classifier learning scheme NaiveBayes assumes that the attributes are normally distributed. l Needed when the attributes are numerical and not necessarily 0 | 1 l Its output describes the parameters of these normal distributions l Explanation of the annotations of the attributes: http://grb.mnsu.edu/grbts/doc/manual/Naive_Bayes.html n Explanation of the error measures: l http://grb.mnsu.edu/grbts/doc/manual/Error_Measurements.html#sec:error http://grb.mnsu.edu/grbts/doc/manual/Error_Measurements.html#sec:error

26. 26 The „happiness factor“ of Mihalcea & Liu (2006) “Starting with the features identified as important by the Naïve Bayes classifier (a threshold of 0.3 was used in the feature selection process), we selected all those features that had a total corpus frequency higher than 150, and consequently calculate the happiness factor of a word as the ratio between the number of occurrences in the happy blogposts and the total frequency in the corpus.”  What is the relation to the Naïve Bayes estimators?

27. 27 Agenda Sequence mining: tool WUM (case study “school search”) Classification: method Naïve Bayes (case study “happiness”) Clustering: tool DocumentAtlas (case study “EU proposals”) A very short note on other uses of clustering (e.g. in query mining) Some observations on privacy... Best-practice „design patterns“with open-source tools

28. 28 Clustering by information contained in the objects to be clustered (here: documents contain text) – www.kartoo.com

29. 29 The basic idea of clustering: group similar things Group 1 Group 2 Attribute 1 Attribute 2 Based on http://rakaposhi.eas.asu.edu/cse494/notes/f02-clustering.ppthttp://rakaposhi.eas.asu.edu/cse494/notes/f02-clustering.ppt

30. 30 Idea and Applications Clustering is the process of grouping a set of physical or abstract objects into classes of similar objects. n It is also called unsupervised learning. n It is a common and important task that finds many applications. Applications in text analysis / Web content mining, e.g. for search engines: n Structuring search results n Suggesting related pages n Automatic directory construction/update n Finding near identical/duplicate pages Applications in Web usage mining n Customer/user segmentation n User segmentation for recommender systems / personalization Based on http://rakaposhi.eas.asu.edu/cse494/notes/f02-clustering.ppthttp://rakaposhi.eas.asu.edu/cse494/notes/f02-clustering.ppt

31. 31 Concepts in Clustering n “Defining distance between points l Cosine distance (which you already know) l Overlap distance n A good clustering is one where l (Intra-cluster distance) the sum of distances between objects in the same cluster are minimized, l (Inter-cluster distance) while the distances between different clusters are maximized l Objective to minimize: F(Intra,Inter) n Clusters can be evaluated with “internal” as well as “external” measures l Internal measures are related to the inter/intra cluster distance l External measures are related to how representative are the current clusters to “true” classes –See entropy and F-measure Based on http://rakaposhi.eas.asu.edu/cse494/notes/f02-clustering.ppthttp://rakaposhi.eas.asu.edu/cse494/notes/f02-clustering.ppt

32. 32 Inter/Intra Cluster Distances Intra-cluster distance (Sum/Min/Max/Avg) the (absolute/squared) distance between - All pairs of points in the cluster OR - Between the centroid and all points in the cluster OR - Between the “medoid” and all points in the cluster Inter-cluster distance Sum the (squared) distance between all pairs of clusters Where distance between two clusters is defined as: - distance between their centroids/medoids - (Spherical clusters) - Distance between the closest pair of points belonging to the clusters - (Chain shaped clusters) From http://rakaposhi.eas.asu.edu/cse494/notes/f02-clustering.ppthttp://rakaposhi.eas.asu.edu/cse494/notes/f02-clustering.ppt

33. 33 How hard is clustering? One idea is to consider all possible clusterings, and pick the one that has best inter and intra cluster distance properties Suppose we are given n points, and would like to cluster them into k-clusters n How many possible clusterings? Too hard to do it brute force or optimally Solution: Iterative optimization algorithms –Start with a clustering, iteratively improve it (eg. K-means) From http://rakaposhi.eas.asu.edu/cse494/notes/f02-clustering.ppthttp://rakaposhi.eas.asu.edu/cse494/notes/f02-clustering.ppt

34. 34 Classical clustering methods Partitioning methods n k-Means (and EM), k-Medoids Hierarchical methods n agglomerative, divisive, BIRCH Model-based clustering methods From http://rakaposhi.eas.asu.edu/cse494/notes/f02-clustering.ppthttp://rakaposhi.eas.asu.edu/cse494/notes/f02-clustering.ppt

35. 35 K-means Works when we know k, the number of clusters we want to find Idea: n Randomly pick k points as the “centroids” of the k clusters n Loop: l For each point, put the point in the cluster to whose centroid it is closest l Recompute the cluster centroids l Repeat loop (until there is no change in clusters between two consecutive iterations.) Iterative improvement of the objective function: Sum of the squared distance from each point to the centroid of its cluster From http://rakaposhi.eas.asu.edu/cse494/notes/f02-clustering.ppthttp://rakaposhi.eas.asu.edu/cse494/notes/f02-clustering.ppt

36. 36 K Means Example ( K=2) For a more complex simulation, see http://www.cs.tu-bs.de/rob/lehre/bv/Kmeans/Kmeans.html Pick seeds Reassign clusters Compute centroids x x Reasssign clusters x x x x Compute centroids Reassign clusters Converged! Based on http://rakaposhi.eas.asu.edu/cse494/notes/f02-clustering.ppthttp://rakaposhi.eas.asu.edu/cse494/notes/f02-clustering.ppt

37. 37 A map of documents, grouped by their topics

38. 38 DocumentAtlas: A two-step procedure 1. Latent semantic indexing: Project documents into a semantic space (dimensionality reduction and identification of commonalities even if vocabulary is different) 2. Multidimensional scaling: Project that space into 2D, preserving the distances as well as possible n Input: a set of documents n Output: a „document map“

39. 39 Agenda Sequence mining: tool WUM (case study “school search”) Classification: method Naïve Bayes (case study “happiness”) Clustering: tool DocumentAtlas (case study “EU proposals”) A very short note on other uses of clustering (e.g. in query mining) Some observations on privacy... Best-practice „design patterns“with open-source tools

40. 40 Clustering by information contained in the objects to be clustered (here: documents contain text) – www.kartoo.com

41. 41 Clustering by information associated with the objects to be clustered (here: photos are associated with tags) – www.flickr.com

42. 42 Clustering by information associated with the objects to be clustered (here: queries are associated with document texts) – (1)

43. 43 Clustering by information associated with the objects to be clustered... (2) – Baeza-Yates, Query Mining, ECIR 2005 1. Create instances of past ( query – result set ) combinations 2. Cluster them by the textual similarity of the (viewed) result documents 3. Use this to recommend a better / an additional query Result set 1 Query 1 Result set 2 Query 2 New user recommend

44. 44 Ranking by similarity and popularity: Examples

45. 45 Agenda Sequence mining: tool WUM (case study “school search”) Classification: method Naïve Bayes (case study “happiness”) Clustering: tool DocumentAtlas (case study “EU proposals”) A very short note on other uses of clustering (e.g. in query mining) Some observations on privacy... Best-practice „design patterns“with open-source tools

46. 46 Internet users are worried about their privacy... (results from a meta- study of 30 questionnaire-based studies [TK03])

47. 47... but are they really? An online shop with a difference [Berendt, Günther, & Spiekermann, Communications of the ACM,2005]

48. 48 Privacy-related behaviour Shopping for jackets Shopping for cameras [Berendt, Data Mining and Knowledge Discovery, 2002], [Berendt, Postproc. WebKDD 2002]

49. 49 Finding: People are willing to exchange privacy for personalization benefits n Users would provide, in return for personalized content, information on their name (88%), education (88%), age (86%), hobbies (83%), salary (59%), or credit card number (13%). n 27% of Internet users think tracking allows the site to provide information tailored to specific users. n 73% of online users find it useful if site remembers basic information such as name and address. n People are willing to give information to receive a personalized online experience: 51% or 40%, depending on the study. [TK03]

50. 50 User-centric evaluation: An experimental investigation of the effect of explaining the personalization-privacy tradeoff [KT05] compared the effects of traditional privacy statements with that of a contextualized explanation on users ’ willingness to answer questions about themselves and their (product) preferences. In the contextualized-explanation condition, participants n answered 8.3% more questions (gave at least one answer) (p<0.001), n gave 19.6% more answers (p<0.001), n purchased 33% more often (p<0.07), stated that their data had helped the Web store to select better books (p<0.035) – even though the recommendations were static and identical for both groups. [KT05] compared the effects of traditional privacy statements with that of a contextualized explanation on users ’ willingness to answer questions about themselves and their (product) preferences. In the contextualized-explanation condition, participants n answered 8.3% more questions (gave at least one answer) (p<0.001), n gave 19.6% more answers (p<0.001), n purchased 33% more often (p<0.07), stated that their data had helped the Web store to select better books (p<0.035) – even though the recommendations were static and identical for both groups. (screenshot from Teltzrow, M. & Kobsa, A. (2004). Communication of Privacy and Personalization in E-Business. In Proceedings of the Workshop “WHOLES: A Multiple View of Individual Privacy in a Networked World”, Stockholm, Sweden.

51. 51 But what is privacy? Is it only about data protection? Phillips, D.J. 2004. “Privacy Policy and PETs: The Influence of Policy Regimes on the Development and Social Implications of Privacy Enhancing Technologies.” New Media & Society 6(6): 691-706 n freedom from intrusion n construction of the public/private divide n separation of identities n protection from surveillance (the right to choose belonging)

52. 52 Also: whose privacy? Stakeholders and privacy interests: a (partially) fictitious example users of the system: n passengers n system administrators Other stakeholders: n airport administration n airport security n airlines n duty-free shop

53. 53 Different privacy interests of the different stakeholders

54. 54 Agenda Sequence mining: tool WUM (case study “school search”) Classification: method Naïve Bayes (case study “happiness”) Clustering: tool DocumentAtlas (case study “EU proposals”) A very short note on other uses of clustering (e.g. in query mining) Some observations on privacy... Best-practice „design patterns“with open-source tools

55. 55 In the preparation of a log file (recommendations for open-source tools are shown in green) 1. Use qualitative methods for application understanding (read!) 2. Inspect the site and the URLs for data understanding 1. Generate Analog reports for getting base statistics of usage 2. Build concept system / hierarchy and mapping: URLs  concepts (notation: WUMprep regex) 3. Use WUMprep for data preparation 1. Remove unwanted entries (pictures etc.) 2. Sessionize 3. Remove robots 4. Replace URLs by concepts 5. (Build a database) 4. Use WEKA for modelling 1. Transform log file into ARFF (WUMprep4WEKA) 2. Cluster, classify, find association rules,... 5. Use WUM for modelling 6. Select patterns based on objective interestingness measures (support, confidence, lift,...) and on subjective interestingness measures (unexpected? Application-relevant?) 7. Present results in tabular, textual and graphical form (use Excel,...) 8. Interpret the results 9. Make recommendations for site improvement etc.

56. 56 In the case study: 1. Use qualitative methods for application understanding (read!) 2. Inspect the site and the URLs for data understanding 1. Generate Analog reports for getting base statistics of usage 2. Build concept system / hierarchy and mapping: URLs  concepts (notation: WUMprep regex) 3. Use WUMprep for data preparation 1. Remove unwanted entries (pictures etc.) 2. Sessionize 3. Remove robots 4. Replace URLs by concepts 5. (Build a database) 4. Use WEKA for modelling 1. Transform log file into ARFF (WUMprep4WEKA) 2. Cluster, classify, find association rules,... 5. Use WUM for modelling 6. Select patterns based on objective interestingness measures (support, confidence, lift,...) and on subjective interestingness measures (unexpected? Application-relevant?) 7. Present results in tabular, textual and graphical form (use Excel,...) 8. Interpret the results 9. Make recommendations for site improvement etc. done

57. 57 The preparation of texts (e.g., for an automatic version of step 2.2.) n Is quite involved when done properly (a good introduction to preprocessing for text mining can be found in Grobelnik, M., & Mladenic, D. Text Mining Tutorial. http://eprints.pascal-network.org/archive/00000017/01/Tutorial_Marko.pdf http://eprints.pascal-network.org/archive/00000017/01/Tutorial_Marko.pdf ) n However, as a first step, you can also use the raw text of documents (generated with only a few of the tools in the TextGarden library).

58. 58 Thank you!