1. Web Mining Research: A Survey
By Raymond Kosala & Hendrik Blockeel, Katholieke Universitat Leuven, July 2000
Presented 4/22/2004 by Arifa Mannan

2. Arifa Mannan, University of Vermont
Overview
Introduction
Web Mining
Web Content Mining
Web Structure Mining
Web Usage Mining
Conclusions
Questions/Answers
4/22/2004
Arifa Mannan, University of Vermont

3. Arifa Mannan, University of Vermont
Introduction
The Web is huge, dynamic & diverse, and thus raises the scalability, multimedia data and temporal issues respectively.
Thus we are drowning in information and facing information overload. Information users can encounter problems when interacting with the Web
4/22/2004
Arifa Mannan, University of Vermont

4. Arifa Mannan, University of Vermont
More Introduction
PROBLEMS:
Finding Relevant information: irrelevance of many of the search results, inability to index all the information available on the web.
Creating new knowledge out of the information available on the web: presumes that we already have a collection of web data and we want to extract potentially useful knowledge out of it.
4/22/2004
Arifa Mannan, University of Vermont

5. Arifa Mannan, University of Vermont
More Introduction
Personalization of the information: This problem is often associated with the type and presentation of the information.
Learning about consumers or individual users: This problem is about knowing what the customers do and want.
4/22/2004
Arifa Mannan, University of Vermont

6. Arifa Mannan, University of Vermont
More Introduction
Web mining techniques could be directly or indirectly used to solve the information overload problems described before.
directly - application of web mining techniques directly addresses the problem
indirectly- web mining techniques are used as a part of a bigger application that addresses the problems mentioned before.
Web mining NOT only useful tool: other useful techniques include
DB database
IR Information Retrieval
NLP Natural Language Processing
Web document community
4/22/2004
Arifa Mannan, University of Vermont

7. Arifa Mannan, University of Vermont
Web Mining: Outline
Overview of Web Mining
Describe some confusion in use of the term “Web Mining”
Provide a Classification
Relate Classification to the agent paradigm
Describe some research in their respective categories
4/22/2004
Arifa Mannan, University of Vermont

8. Arifa Mannan, University of Vermont
Web Mining: Overview
Web mining is the use of data mining techniques to automatically discover and extract information from web documents and services.
We suggest decomposing Web mining into these subtasks:
1 Resource finding: the task of retrieving intended web documents
2 Information selection and pre-processing: automatically selecting and pre-processing specific information from retrieved Web resources
3 Generalization: automatically discovers general patterns at individual Web sites as well as across multiple sites.
4 Analysis: validation and/or interpretation of the mined patterns.
We’ll call this pattern , as we’ll later see, sometimes is also used.
4/22/2004
Arifa Mannan, University of Vermont

9. Arifa Mannan, University of Vermont
Web Mining: Confusion
Web mining is often associated with Information Retrieval or Information Extraction, but it is different from both.
IR is the automatic retrieval of all relevant documents while at the same time retrieving as few non-relevant ones as possible. [views documents as bag-of-words]
IE has the goal of transforming a collection of documents, usually with the help of an IR system, into information that is more readily digested and analyzed. [interested in the structure or representation of a document]
We argue that Web mining intersects with the application of machine learning on the web.
4/22/2004
Arifa Mannan, University of Vermont

10. Web Mining: Classification
Web mining is categorized into three areas of interest based on which part of the web to mine:
Web content mining
Web structure mining
Web usage mining
4/22/2004
Arifa Mannan, University of Vermont

11. Web Mining: Classification
Web content mining: describes the discovery of useful information from Web contents/data/documents.
Web content data – the data the Web page was designed to convey to the users,
consists of textual, image, audio, video
4/22/2004
Arifa Mannan, University of Vermont

12. Web Mining: Classification
Web structure mining: tries to discover the model underlying the link structures of the Web.
Web structure data – which describes the organization of the content. Intra-page structure information includes the arrangement of various HTML or XML tags within a given page. Inter-page structure information is hyper-links connecting one page to another.
4/22/2004
Arifa Mannan, University of Vermont

13. Web Mining: Classification
Web usage mining: tries to make sense of the data generated by the Web surfer’s sessions or behaviors.
Web usage data are secondary data.
Web usage data includes data from web server access logs, proxy server logs, browser logs, registration data, cookies, mouse clicks and scrolls, and any other data as the results of interactions.
4/22/2004
Arifa Mannan, University of Vermont

14. Web Mining: Classification
In practice, the three Web mining tasks could be used in isolation or combined in an application, especially in Web content and structure mining since the Web documents might also contain links.
4/22/2004
Arifa Mannan, University of Vermont

15. Web Mining & the Agent Paradigm
Web mining is often viewed from or implemented within an agent paradigm. Thus, web mining has a close relationship with software agents or intelligent agents.
Two relevant types of software agents:
User interface agents : information retrieval agents, information filtering agents, & personal assistant agents.
Distributed agents : distributed agents for knowledge discovery or data mining .
4/22/2004
Arifa Mannan, University of Vermont

16. Web Mining & the Agent Paradigm
Two relevant types of intelligent agents:
Content-based approach: the system searches for items that match based on an analysis of the content using the user preference.
Collaborative approach: the system tries to find users with similar interests to give recommendations to.
4/22/2004
Arifa Mannan, University of Vermont

17. Web Content Mining: IR view
The goal of Web content mining is to improve the information finding or filtering the information to the users.
Information retrieval view for unstructured documents:
most of the research uses “bag of words” to represent unstructured documents.
See the table that follows
4/22/2004
Arifa Mannan, University of Vermont

18. Arifa Mannan, University of Vermont
1998
1999
1995
1998
1995
1998
1999
1999
1999
1997
1999
1997
2000
1999
1999
1996
1999
1995
1999
1999
4/22/2004
Arifa Mannan, University of Vermont

19. Web Content Mining: IR view
4/22/2004
Arifa Mannan, University of Vermont

20. Web Content Mining: DB View
The database techniques on the web are related to the problem of managing and querying the information on the web.
Three classes of tasks: modeling and querying the web, information extraction and integration, and web site construction and restructuring.
Tries to model the data on the web and to integrate them so that more sophisticated queries other than the keywords based search can be performed.
Research in this area mainly deals with semi-structured data
4/22/2004
Arifa Mannan, University of Vermont

21. Web Content Mining: DB view
4/22/2004
Arifa Mannan, University of Vermont

22. Arifa Mannan, University of Vermont
Web Structure Mining
In Web structure mining we are interested in the structure of the hyperlinks within the Web itself. (inter-document structure)
This line of research is inspired by the study of social networks and citation analysis.
We could discover specific types of pages (hubs, authorities etc.) based on incoming and outgoing links.
4/22/2004
Arifa Mannan, University of Vermont

23. Arifa Mannan, University of Vermont
Web Structure Mining
Web structure mining utilizes the hyperlinks structure to apply social network analysis to model the underlying link structure of the web.
A few different algorithms have been proposed to do this such as HITS, PageRank, improved HITS using content info & outlier filtering
Applied as a method to calculate the quality rank or relevancy of each Web page.
4/22/2004
Arifa Mannan, University of Vermont

24. Arifa Mannan, University of Vermont
Web Usage Mining
Web usage mining focuses on techniques that could predict user behavior while the user interacts with the web.
Two commonly used approaches: 1) mapping the usage data of the web server into relational tables before an adapted data mining technique is performed, 2) uses the log data directly by using special preprocessing techniques.
4/22/2004
Arifa Mannan, University of Vermont

25. Arifa Mannan, University of Vermont
Web Usage Mining
Applications of web usage mining fall into two main categories: learning a user profile/user modeling in adaptive interfaces [personalized] and learning user navigation patterns [impersonalized]
Web users would be interested in techniques that could learn their information needs and preferences, which is user modeling
Information providers would be interested in techniques that could improve the effectiveness of the information on their websites.
4/22/2004
Arifa Mannan, University of Vermont

26. Arifa Mannan, University of Vermont
Conclusions
We surveyed research in Web Mining,
clarified some confusion in the use of the term Web mining,
suggested three Web mining categories and situated some current research with respect to these categories.
explored the connection between Web mining categories and the agent paradigm,
The Web presents new challenges to the traditional data mining algorithms that work on flat data. We have seen that some of the traditional data mining algorithms have been extended or new algorithms have been used to work on the Web data.
4/22/2004
Arifa Mannan, University of Vermont

27. Arifa Mannan, University of Vermont
Questions and Answers
Q1. Outline the main characteristics of web information.
Ans: Huge
Diverse
Dynamic
4/22/2004
Arifa Mannan, University of Vermont

28. Arifa Mannan, University of Vermont
Questions and Answers
Q2. How data mining techniques can be used in web information analysis? Give at least two examples.
Ans:
Classification: classification on server logs using decision tree, Naïve-Bayes classifier to discover the profiles of users belonging to a particular class
Clustering: Clustering can be used to group users exhibiting similar browsing patterns.
Association Analysis: association analysis can be used to relate pages that are most often referenced together in a single server session.
4/22/2004
Arifa Mannan, University of Vermont

29. Arifa Mannan, University of Vermont
Questions and Answers
Q3. What are the three main areas of interest for web mining?
Ans: Web content mining
Web structure mining
Web usage mining
4/22/2004
Arifa Mannan, University of Vermont