1. Web Mining Research: A survey
Revised and Presented by
Narine Manukyan
Authors:
Raymond Kosala and Hendrik Blockeel
ACM SIGKDD, July 2000
Presented by
Shan Huang, 4/24/2007
Fan Min, 4/22/2009
Nima, 12/06/2011
Course: Data Mining[CS332]
Pr. Xindong Wu
Computer Science Department
University Of Vermont
Complex Systems Center
Achieving insight, innovative design, and informed decision making through systems thinking

2. minding our own business To mining other’s business on the web
From
minding our own business To
mining other’s business
on the web

5. Social network mining Data on human Interactions 2001 2010 over time…
Facebook
Twitter
Data on human Interactions
What they read
Who they meet
Personal data
What they do
2010
2001
over time

6. Dynamic Datasets Includes time series component
people do things over time …
tn=2010
t1=2001 … t1 tn … t1 tn … t1 tn

7. The Web is huge, diverse, and dynamic.…
tn=2010
t1=2001
Scalability issues
multimedia data issues
temporal data issues … t1 tn … t1 tn … t1 tn

8. Finding patterns … t1 tn
tn=2010
t1=2001
Yey!

9. Fishing for Data

10. Outline Introduction Web Mining Web Content Mining
Web Structure Mining
Web Usage Mining
Conclusion & Exam Questions

11. Four Problems Finding relevant information
Low precision and not indexed information
A query-triggered process
Creating new knowledge out of available information on the web
Intelligent tools are necessary
A data-triggered process
Personalizing the information
Personal preference in content and presentation of the information
Learning about the consumers
What does the customer want to do?

12. Direct vs. Indirect Web Mining
Web mining techniques can be used to solve the information overload problems:
Directly
Address the problem with web mining techniques
E.g. newsgroup agent classifies whether the news as relevant
Indirectly
Used as part of a bigger application that addresses problems
E.g. used to create index terms for a web search service
Web Mining Research: A Survey

13. Web Mining
“Web mining refers to the overall process of discovering potentially useful and previously unknown information or knowledge from the Web data.”
Goyal’s Definition: “Using data mining techniques to make the web more useful and more profitable (for some) and to increase the efficiency of our interaction with the web”

14. Web Mining by Etzioni Resource finding
Retrieving intended Web Documents (query triggered)
Information selection and pre-processing
Automatically selecting and pre-processing specific information from retrieved Web recourses
Generalization
Automatically discovering general patterns (data-triggerd)
Analysis
Validation and/or interpretation of the mined patterns

15. Web Mining and Information Retrieval
Information retrieval (IR) is the automatic retrieval of all relevant documents while at the same time retrieving as few of the non-relevant documents as possible
Goal: Indexing text and searching for useful documents in a collection.
Research in IR: modeling, document classification and categorization, user interfaces, data visualization, filtering etc.
Web document classification, which is a Web Mining task, could be part of an IR system (e.g. indexing for a search engine)
Viewed in this respect, Web mining is part of the (Web) IR process.
In my opinion IR is missing the Analysis (interpretation) step of the Web mining
“Some have claimed that resource or document discovery (IR) on
the Web is an instance of Web (content) mining and others associate
Web mining with intelligent IR.”
Web Mining Research: A Survey

16. Web Mining and Information Extraction
Information Extraction (IE): Transforming a collection of documents, into information that is more easily understood and analyzed.
IE is a kind of pre-processing stage, a step after IR and before the data mining techniques are applied.
Building IE systems manually for the general Web are not feasible
According to authors Web mining is a part of IE.
Web Mining Research: A Survey

17. Information Extraction (IE)
Classical: Relies on linguistic pre-processing like syntactic analysis, semantic analysis and discourse analysis.
Structural IE: Utilizes the meta information (e.g. HTML tags, delimiters)
These systems can use data mining techniques to learn extraction rules from the annotated corpora.

18. Web Mining Research: A Survey
Compare IR and IE VS
IR aims to select relevant documents from web
IR views the text in a document just as a bag of unordered words
IE aims to extract the relevant facts from given documents
IE interested in structure or representation of a document (finer level)
Web Mining Research: A Survey

19. Web Mining and Machine Learning
Web mining is not the same as learning from the Web or machine learning techniques applied on the Web
There are applications of machine learning applied on the Web that are not instances of Web mining
There are methods used for web mining that are not Machine Learning
Web Mining Research: A Survey

20. Web Mining and The Agent Paradigm
Web mining is often viewed from or implemented within an agent paradigm.
User Interface Agents
information retrieval agents, information filtering agents, & personal assistant agents.
Distributed Agents
Concerned with problem solving by a group of agents.
distributed agents for knowledge discovery or data
Mobile Agents
not relevant for data mining tasks
Web Mining Research: A Survey

21. Web Mining and The Agent Paradigm (contd.)
Two frequently used approaches for developing intelligent agents:
Content-based approach (User interface Agents)
The system searches for items that match based on an analysis of the content using the user preferences.
Collaborative approach (Distributed Agents)
The system tries to find users with similar interests to give recommendations to.
Analyze the user profiles and sessions or transactions.
Web Mining Research: A Survey

22. Outline Introduction Web Mining Web Content Mining
Web Structure Mining
Web Usage Mining
Conclusion & Exam Questions

23. Web Mining Research: A Survey
Web Mining Categories
Web Mining
Web Usage
Mining
Web Content
Web Structure
Web Content Mining
Discovering useful information from web page contents/data/documents.
Web Structure Mining
Discovering the model underlying link structures (topology) on the Web. E.g. discovering authorities and hubs
Web Usage Mining
Extraction of interesting knowledge from logging information produced by web servers.
Usage data from logs, user profiles, user sessions, cookies, user queries, bookmarks, mouse clicks and scrolls, etc.
Web Mining Research: A Survey

24. Outline Introduction Web Mining Web Content Mining
Web Structure Mining
Web Usage Mining
Conclusion & Exam Questions

25. Web Content Data Structure
Web content consists of several types of data
Text, image, audio, video, hyperlinks, metadata.
Unstructured – free text
Semi-structured – HTML
More structured – Data in the tables or database generated HTML pages
Note: much of the Web content data is unstructured text data.
Web Mining Research: A Survey 25 25

26. Web Content Mining: IR View
Unstructured Documents
Bag of words to represent unstructured documents
Takes single word as feature
Ignores the sequence in which words occur
Features could be
Boolean
Word either occurs or does not occur in a document
Frequency based
Frequency of the word in a document
Variations of the feature selection include
Removing the case, punctuation, infrequent words and stop words
Features can be reduced using different feature selection techniques:
Information gain, mutual information, cross entropy.
Stemming: which reduces words to their morphological roots.
Web Mining Research: A Survey

27. Web Content Mining: IR View
Semi-Structured Documents
Uses richer representations for features
Due to the additional structural information in the hypertext document (typically HTML and hyperlinks)
Uses common data mining methods (whereas unstructured might use more text mining methods)
Application:
Hypertext classification or categorization and clustering,
learning relations between web documents,
learning extraction patterns or rules, and
finding patterns in semi-structured data.
Web Mining Research: A Survey

28. Web Content Mining: DB View
The database techniques on the Web are related to the problems of managing and querying the information on the Web.
DB view tries to infer the structure of a Web site or transform a Web site to become a database
Better information management
Better querying on the Web
Can be achieved by:
Finding the schema of Web documents
Building a Web warehouse
Building a Web knowledge base
Building a virtual database
Web Mining Research: A Survey

29. Web Content Mining: DB View
DB view mainly uses the Object Exchange Model (OEM)
Represents semi-structured data by a labeled graph
The data in the OEM is viewed as a graph, with objects as the vertices and labels on the edges
Each object is identified by an object identifier [oid] and
Value is either atomic or complex
Process typically starts with manual selection of Web sites for doing Web content mining
Main application:
The task of finding frequent substructures in semi-structured data
The task of creating multi-layered database
Web Mining Research: A Survey

30. Outline Introduction Web Mining Web Content Mining
Web Structure Mining
Web Usage Mining
Conclusion & Exam Questions

31. Web Mining Research: A Survey
Web Structure Mining
Interested in the structure of the hyperlinks within the Web
Inspired by the study of social networks and citation analysis
Can discover specific types of pages(such as hubs, authorities, etc.) based on the incoming and outgoing links.
Application:
Discovering micro-communities in the Web ,
measuring the “completeness” of a Web site
Web Mining Research: A Survey

32. Topic oriented community detection through social objects and link analysis in social networks

33. Outline Introduction Web Mining Web Content Mining
Web Structure Mining
Web Usage Mining
Conclusion & Exam Questions

34. Website Usage Analysis

35. Web Mining Research: A Survey
Web Usage Mining
Tries to predict user behavior from interaction with the Web
Wide range of data (logs)
Web client data
Proxy server data
Web server data
Two common approaches
Maps the usage data of Web server into relational tables before an adapted data mining techniques
Uses the log data directly by utilizing special pre-processing techniques
Web Mining Research: A Survey

36. Web Mining Research: A Survey
Web Usage Mining
Typical problems:
Distinguishing among unique users, server sessions, episodes, etc. in the presence of caching and proxy servers
Often Usage Mining uses some background or domain knowledge
E.g. site topology, Web content, etc.
Web Mining Research: A Survey

37. Web Mining Research: A Survey
Web Usage Mining
Applications:
Two main categories:
Learning a user profile (personalized)
Web users would be interested in techniques that learn their needs and preferences automatically
Learning user navigation patterns (impersonalized)Information providers would be interested in techniques that improve the effectiveness of their Web site
Web Mining Research: A Survey

38. Web-Usage Mining Example
Data Mining Techniques – Navigation Patterns
Analysis:
Example:
70% of users who accessed /company/product2 did so by starting at /company and proceeding through /company/new, /company/products and company/product1
80% of users who accessed the site started from /company/products
65% of users left the site after
four or less page references

39. Visual Representation of Web usage
This is a visualization of the
web graph of the Computer
Science department of
Rensselaer Polytechnic Institute(
Strahler numbers are used for
assigning colors to edges.
One can see user access paths scattering from first page of website
(the node in center) to cluster of web pages corresponding to
faculty pages, course home pages, etc.

40. Visual Representation of Web usage
Adding third dimension enables visualization of more information and clarifies user behavior in and between clusters. Center node of circular basement is first page of web site from which users scatter to different clusters of web pages. Color spectrum from Red
(entry point into clusters) to Blue (exit points) clarifies behavior of users.
This is a 3D visualization of web usage for above site.The cylinder like part of this figure is visualization of web usage of surfers as they browse a long HTML document.

41. Visual Representation of Web usage
User’s browsing access pattern is amplified by a different
coloring. Depending on link structure of underlying
pages, we can see vertical access patterns of a user drilling down the cluster, making a cylinder shape (bottom-left corner of the figure). Also users following links going down a hierarchy of webpages makes a cone shape and users going up hierarchies,e.g., back to main page of website makes a funnel shape
(top-right corner of the figure).

42. Visual Representation of Web usage
Right: One can observe long user sessions as strings falling off clusters. Those are special type of long sessions when user navigates sequence of web pages which come one after the other under a cluster, e.g., sections of a long document. In many cases we found web pages with many nodes connected with Next/Up/Previous hyperlinks.
Left: A zoom view of the same visualization

43. Visual Representation of Web usage
Frequent access patterns extracted by web mining
process are visualized as a
white graph on top of embedded and colorful graph of web usage.

44. Web Mining Research: A Survey
Conclusions
Survey the research in the area of Web mining.
Clarify differences between Web mining, Information Retrieval (IR) and Information Extraction (IE)
Suggest three Web mining categories
Content, Structure, and Usage Mining
Explored connection between Web mining categories and related agent paradigm
Web Mining Research: A Survey

45. Web Mining Research: A Survey
Exam Question #1
Question: What is multimedia data mining?
Answer: Research on minding multi types of data (e.g., textual, image, audio, video, metadata, etc.)
Web Mining Research: A Survey

46. Web Mining Research: A Survey
Exam Question #2
Question: Which one of the following is a data-triggered process:
1. Information Retrieval (IR)
2. Information Extraction (IE)
Answer: 2. Information Extraction
Web Mining Research: A Survey

47. Web Mining Research: A Survey
Exam Question #3
Question: What are the three Web mining categories ?
Answer: (1) Web Content
(2) Web Structure
(3) Web Usage
Web Mining Research: A Survey

48. Something to think about
Can web mining techniques be useful if Web disappears?

49. Questions?

50. Example of Web Mining from my research

51. Web Mining Research: A Survey
Other Approaches
Web mining is NOT the only approach
Database approach (DB)
Information retrieval (IR)
Natural language processing (NLP)
Web document community
Web Mining Research: A Survey