1. Overview on Web Mining and Recommendation June 13, 20161 CENG 770

2.  Web Mining is the use of the data mining techniques to automatically discover and extract information from web documents/services Web Mining

3. Examples of Discovered Patterns  Association rules  75% of Facebook users also have FourSquare accounts  Classification  People with age less than 40 and salary > 40k trade on-line  Clustering  Users A and B access similar URLs  Outlier Detection  User A spends more than twice the average amount of time surfing on the Web

4. Why is Web Mining Different?  The Web is a huge collection of documents except for  Hyper-link information  Access and usage information  The Web is very dynamic  New pages are constantly being generated  Challenge: Develop new Web mining algorithms and adapt traditional data mining algorithms to  Exploit hyper-links and access patterns  Be incremental

5. Web Mining Web Content Mining – Web page content mining – Search result mining Web Structure Mining – Search Web Usage Mining – Access patterns – Customized Usage patterns June 13, 20165

6. Web Content Mining Crawler – A program that traverses the hypertext structure in the Web – Seed URL: page/set of pages that the crawler starts with – Links from visited page saved in a queue – Build an index Focused crawlers June 13, 20166

7. Basic Measures for Text Retrieval Precision: the percentage of retrieved documents that are in fact relevant to the query (i.e., “correct” responses) Recall: the percentage of documents that are relevant to the query and were, in fact, retrieved 6/13/20167 Relevant Relevant & Retrieved Retrieved All Documents

8. Information Retrieval Techniques Basic Concepts – A document can be described by a set of representative keywords called index terms. – Different index terms have varying relevance when used to describe document contents. – This effect is captured through the assignment of numerical weights to each index term of a document. (e.g.: frequency, tf-idf) DBMS Analogy – Index Terms  Attributes – Weights  Attribute Values 6/13/20168

9. Indexing Inverted index – A data structure for supporting text queries, similar to index in a book document_table: a set of document records term_table: a set of term records, – Answer query: Find all docs associated with one or a set of terms – + easy to implement – – do not handle well synonymy and polysemy, and posting lists could be too long (storage could be very large) 6/13/20169

10. Inverted index inverted index aalborg 3452, 11437, …... arm 4, 19, 29, 98, 143,... armada 145, 457, 789,... armadillo 678, 2134, 3970,... armani 90, 256, 372, 511,.... zz 602, 1189, 3209,... disks with documents indexing

11. Vector Space Model Documents and user queries are represented as m- dimensional vectors, where m is the total number of index terms in the document collection. The degree of similarity of the document d with regard to the query q is calculated as the correlation between the vectors that represent them, using measures such as the Euclidian distance or the cosine of the angle between these two vectors. 6/13/201611

12. Vector Space Model Represent a doc by a term vector – Term: basic concept, e.g., word or phrase – Each term defines one dimension – N terms define a N-dimensional space – Element of vector corresponds to term weight – E.g., d = (x 1,…,x N ), x i is “importance” of term i 6/13/201612

13. VS Model: Illustration 6/13/201613 Java Microsoft Starbucks C2C2 Category 2 C1C1 Category 1 C3C3 Category 3 new doc New document is assigned to the most likely category based on vector similarity.

14. Issues to be handled How to select terms to capture “basic concepts” – Word stopping e.g. “a”, “the”, “always”, “along” – Word stemming e.g. “computer”, “computing”, “computerize” => “compute” – Latent semantic indexing How to assign weights – Not all words are equally important: Some are more indicative than others e.g. “algebra” vs. “science” How to measure the similarity 6/13/201614

15. Latent Semantic Indexing Basic idea – Similar documents have similar word frequencies – Difficulty: the size of the term frequency matrix is very large – Use a singular value decomposition (SVD) techniques to reduce the size of frequency table – Retain the K most significant rows of the frequency table Method – Create a term x document weighted frequency matrix A – SVD construction: A = U * S * V’ – Define K and obtain U k,, S k, and V k. – Create query vector q’. – Project q’ into the term-document space: Dq = q’ * U k * S k -1 – Calculate similarities: cos α = Dq. D / ||Dq|| * ||D|| 6/13/201615

16. How to Assign Weights Two-fold heuristics based on frequency – TF (Term frequency) More frequent within a document  more relevant to semantics – IDF (Inverse document frequency) Less frequent among documents  more discriminative 6/13/201616

17. TF Weighting Weighting: – More frequent => more relevant to topic Raw TF= f(t,d): how many times term t appears in doc d Normalization: – Document length varies => relative frequency preferred e.g., Maximum frequency normalization 6/13/201617

18. IDF Weighting Ideas: – Less frequent among documents  more discriminative Formula: IDF(t) = 1+ log (n/k) n: total number of docs k: # docs with term t appearing 6/13/201618

19. TF-IDF Weighting TF-IDF weighting : weight(t, d) = TF(t, d) * IDF(t) – Frequent within doc  high tf  high weight – Selective among docs  high idf  high weight Recall VS model – Each selected term represents one dimension – Each doc is represented by a feature vector – Its t-term coordinate of document d is the TF-IDF weight Many complex and more effective weighting variants exist in practice 6/13/201619

20. How to Measure Similarity? Given two documents Similarity definition – dot product – normalized dot product (or cosine) 6/13/201620

21. Illustrative Example 6/13/201621 text mining travel map search engine govern president congress IDF 2.4 4.5 2.8 3.3 2.1 5.4 2.2 3.2 4.3 doc12(4.8) 1(4.5) 1(2.1) 1(5.4) doc21(2.4 ) 2 (5.6) 1(3.3) doc3 1 (2.2) 1(3.2) 1(4.3) newdoc1(2.4) 1(4.5) doc3 text mining search engine text travel text map travel government president congress doc1 doc2 …… Sim(newdoc,doc1)=2*4.8*2.4+4.5*4.5 Sim(newdoc,doc2)=2.4*2.4 Sim(newdoc,doc3)=0

22. Web Structure Mining PageRank (Google ’00) Clever (IBM ’99) June 13, 201622

23. Search Engine – Two Rank Functions 6/13/201623 Ranking based on link structure analysis Similarity based on content or text

24. The PageRank Algorithm PR(A) = p + (1-p)(PR(T1)/C(T1) + … + PR(Tn)/C(Tn)) where, C(Ti) = # out-links of page i Parameter p is probability that the surfer gets bored and starts on a new random page (1-p) is the probability that the random surfer follows a link on current page 6/13/2016 24 Intuition: PageRank can be seen as the probability that a “random surfer” visits a page Brin, S. and Page, L. (1998). The anatomy of a large-scale hypertextual web search engine. In Proc. WWW Conference, pages 107–117 Basic idea: significance of a page is determined by the significance of the pages linking to it Link i → j : i considers j important. the more important i, the more important j becomes. if i has many out-links: links are less important. Initially: all importances pi = 1. Iteratively, pi is refined.

25. 6/13/2016 25 The HITS Algorithm  Hyperlink-induced topic search (HITS) Kleinberg, J. M. (1999). Authoritative sources in a hyperlinked environment. Journal of the ACM, 46(5):604–632.  Basic idea: Sufficiently broad topics contain communities consisting of two types of hyperlinked pages: Authority: best source for requested info, highly- referenced pages on a topic Hub: contains links to authoritative pages

26. 6/13/2016 26 The HITS Algorithm Collect seed set of pages S (returned by search engine) Expand seed set to contain pages that point to or are pointed to by pages in seed set (removes links inside a site) Iteratively update hub weight h(p) and authority weight a(p) for each page: a (p )= ∑ h (q ), for all q  p h (p )= ∑ a (q), for all p  q After a fixed number of iterations, pages with highest hub/authority weights form core of community

27. Problems with Web Search Today Today’s search engines are plagued by problems: – the abundance problem (99% of info of no interest to 99% of people) – limited coverage of the Web (internet sources hidden behind search interfaces) Largest crawlers cover < 18% of all web pages – limited query interface based on keyword- oriented search – limited customization to individual users

28. Problems with Web Search Today Today’s search engines are plagued by problems: – Web is highly dynamic Lot of pages added, removed, and updated every day – Very high dimensionality

29. Web Usage Mining Pages contain information Links are ‘roads’ How do people navigate the Internet –  Web Usage Mining (clickstream analysis) Information on navigation paths available in log files Logs can be mined from a client or a server perspective

30. Website Usage Analysis Why analyze Website usage? Knowledge about how visitors use Website could – Provide guidelines to web site reorganization; Help prevent disorientation – Help designers place important information where the visitors look for it – Pre-fetching and caching web pages – Provide adaptive Website (Personalization) – Questions which could be answered What are the differences in usage and access patterns among users? What user behaviors change over time? How usage patterns change with quality of service (slow/fast)? What is the distribution of network traffic over time?

31. Website Usage Analysis

32. There are analysis services such as Analog (http://www.analog.cx/(http://www.analog.cx/), Google analytics Gives basic statistics such as number of hits average hits per time period what are the popular pages in your site who is visiting your site what keywords are users searching for to get to you what is being downloaded

33. Web Usage Mining Process

34. June 13, 2016Data Mining: Concepts and Techniques34 Data cleaning By checking the suffix of the URL name, for example, all log entries with filename suffixes such as, gif, jpeg, etc User identification If a page is requested that is not directly linked to the previous pages, multiple users are assumed to exist on the same machine Other heuristics involve using a combination of IP address, machine name, browser agent, and temporal information to identify users Transaction identification All of the page references made by a user during a single visit to a site Size of a transaction can range from a single page reference to all of the page references Data Preparation

35. June 13, 2016Data Mining: Concepts and Techniques35 Main Questions: how to identify unique users how to identify/define a user transaction Problems: user ids are often suppressed due to security concerns individual IP addresses are sometimes hidden behind proxy servers client-side & proxy caching makes server log data less reliable Standard Solutions/Practices: user registration – practical ???? client-side cookies – not fool proof cache busting - increases network traffic Sessionizing

36. June 13, 2016Data Mining: Concepts and Techniques36 Time oriented By total duration of session not more than 30 minutes By page stay times (good for short sessions) not more than 10 minutes per page Navigation oriented (good for short sessions and when timestamps unreliable) Referrer is previous page in session, or Referrer is undefined but request within 10 secs, or Link from previous to current page in web site Sessionizing

37. Web Usage Mining Different Types of Traversal Patterns Association Rules – Which pages are accessed together – Support(X) = freq(X) / no of transactions Episodes – Frequent partially order set of pages – Support(X) = freq(X) / no of time windows Sequential Patterns – Frequent ordered set of pages – Support(X) = freq(X) / no of sessions/customers Forward Sequences – Removes backward traversals, reloads, refreshes – E.g.  and – Support(X) = freq(X) / no of forward sequences Maximal Forward Sequences – Support(X) = freq(X) / no of clicks Clustering – User clusters (similar navigational behaviour) – Page clusters (grouping conceptually related pages) June 13, 2016Data Mining: Concepts and Techniques37

38. Recommender Systems June 13, 2016Data Mining: Concepts and Techniques38

39. Recommender Systems RS – problem of information filtering RS – problem of machine learning seeks to predict the 'rating' that a user would give to an item she/he had not yet considered. Enhance user experience – Assist users in finding information – Reduce search and navigation time

40. Types of RS Three broad types: 1.Content based RS 2.Collaborative RS 3.Hybrid RS

41. Types of RS – Content based RS Content based RS highlights – Recommend items similar to those users preferred in the past – User profiling is the key – Items/content usually denoted by keywords – Matching “user preferences” with “item characteristics” … works for textual information – Vector Space Model widely used

42. Types of RS – Content based RS Content based RS - Limitations – Not all content is well represented by keywords, e.g. images – Items represented by the same set of features are indistinguishable – Users with thousands of purchases is a problem – New user: No history available

43. Types of RS – Collaborative RS Collaborative RS highlights – Use other users recommendations (ratings) to judge item’s utility – Key is to find users/user groups whose interests match with the current user – Vector Space model widely used (directions of vectors are user specified ratings) – More users, more ratings: better results – Can account for items dissimilar to the ones seen in the past too – Example: Movielens.orgMovielens.org

44. Types of Collaborative Filtering User-based collaborative filtering Item-based collaborative filtering

45. User-based Collaborative Filtering Idea: People who agreed in the past are likely to agree again To predict a user’s opinion for an item, use the opinion of similar users Similarity between users is decided by looking at their overlap in opinions for other items

46. Example: User-based Collaborative Filtering Item 1Item 2Item 3Item 4Item 5 User 181 ? 27 User 22 ? 575 User 354747 User 471738 User 517465 User 683837

47. Similarity between users Item 1Item 2Item 3Item 4Item 5 User 181?27 User 22?575 User 471738 How similar are users 1 and 2? How similar are users 1 and 5? How do you calculate similarity?

48. Similarity between users: simple way Item 1Item 2Item 3Item 4Item 5 User 181?27 User 22?575 Only consider items both users have rated For each item: Calculate difference in the users’ ratings Take the average of this difference over the items Average j : Item j rated by User 1 and User 2: | rating (User 1, Item j) – rating (User 2, Item j) |

49. Algorithm 1: using entire matrix 5 4 77 8 Aggregation function: often weighted sum Weight depends on similarity

50. Algorithm 2: K-Nearest-Neighbour 5 4 77 8 Aggregation function: often weighted sum Weight depends on similarity Neighbours are people who have historically had the same taste as our user

51. Item-based Collaborative Filtering Idea: a user is likely to have the same opinion for similar items [same idea as in Content-Based Filtering] Similarity between items is decided by looking at how other users have rated them [different from Content-based, where item features are used] Advantage (compared to user-based CF): – Prevents User Cold-Start problem – Improves scalability (similarity between items is more stable than between users)

52. Example: Item-based Collaborative Filtering Item 1Item 2Item 3Item 4Item 5 User 181 ? 27 User 22 ? 575 User 354747 User 471738 User 517465 User 683837

53. Similarity between items Item 3Item 4Item 5 ? 27 575 747 738 465 837 How similar are items 3 and 4? How similar are items 3 and 5? How do you calculate similarity?

54. Similarity between items: simple way Item 3Item 4 ? 2 57 74 73 46 83 Only consider users who have rated both items For each user: Calculate difference in ratings for the two items Take the average of this difference over the users Average i : User i has rated Items 3 and 4: | rating (User i, Item 3) – rating (User i, Item 4) |

55. Algorithms As User-Based: can use nearest-neighbours or all Item 3 2 1 8 7 Aggregation function: often weighted sum Weight depends on similarity Item 5 Item 4 Item 2 Item 1

56. Types of RS – Collaborative RS Collaborative RS - Limitations – Different users might use different scales. Possible solution: weighted ratings, i.e. deviations from average rating – Finding similar users/user groups isn’t very easy – New user: No preferences available (user cold start problem) – New item: No ratings available (item cold start problem) – Demographic filtering is required

57. Some ways to make a Hybrid RS Weighted. Ratings of several recommendation techniques are combined together to produce a single recommendation Switching. The system switches between recommendation techniques depending on the current situation Mixed. Recommendations from several different recommenders are presented simultaneously (e.g. Amazon) Cascade. One recommender refines the recommendations given by another

58. Model-based collaborative filtering Instead of using ratings directly, develop a model of user ratings Use the model to predict ratings for new items To build the model: – Bayesian network (probabilistic) – Clustering (classification) – Rule-based approaches (e.g., association rules between co-purchased items)