Václav Snášel, Miloš Kudělka VSB-Technical University of Ostrava

Václav Snášel, Miloš Kudělka VSB-Technical University of Ostrava
Web Mining Václav Snášel, Miloš Kudělka VSB-Technical University of Ostrava Czech Republic

Knowledge Engineering Group Praha 2009
Outline... Introduction Classification of recent approaches. Web page segmentation. Genre detection, Table extraction. Opinion, News, and Discussion extraction. Product details and Technical features extraction Knowledge Engineering Group Praha 2009

Web mining Web content mining describes the discovery of useful information from Web contents. The goal of Web content mining is to improve finding information or filtering information for the users. Web structure mining tries to discover the model underlying the link structures of the Web. This model can be used to categorize Web pages and can be useful to generate the relationship between Web sites. Web usage mining tries to make sense of the data generated by the Web surfer's sessions or behaviors. Web usage mining mines the data derived from the interactions of the users. Knowledge Engineering Group Praha 2009

Web mining Web mining - the application of data mining techniques to extract knowledge from Web content, structure, and usage. Knowledge Engineering Group Praha 2009

Web Content Mining Web Content Mining is the process of extracting useful information from the contents of Web documents. It may consist of text, images, audio, video, or structured records such as lists and tables. Web Content mining refers to the overall process of discovering potentially useful and previously unknown information or knowledge from the Web data. Knowledge Engineering Group Praha 2009

Web Structure Mining Web structure mining tries to discover the model underlying the link structures of the Web. The model is based on the topology of the hyperlinks with or without the description of the links. This model can be used to categorize Web pages and is useful to generate information such as the similarity and relationship between different Web sites. Web structure mining could be used to discover authority sites for the subjects (authorities) and overview sites for the subjects that point to many authorities (hubs). Knowledge Engineering Group Praha 2009

Web Usage Mining Web 2.0, enables individuals to create and share content on the Web. One of the important distinguishing features of Web 2.0 is the creation of communities of users, i.e., social networks with new demands on data management. In social content sites, both content and user interest are dynamic: people review and tag new content every day. Knowledge Engineering Group Praha 2009

Web Usage Mining Web usage mining, which aims to discover interesting and frequent user access patterns from web usage data, can be used to model past web access behavior of users. The acquired model can then be used for analyzing and predicting the future user access behavior. Knowledge Engineering Group Praha 2009

Opportunities and Challenges
Web offers an unprecedented opportunity and challenge to data mining The amount of information on the Web is huge, and easily accessible. The coverage of Web information is very wide and diverse. One can find information about almost anything. Information/data of almost all types exist on the Web, e.g., structured tables, texts, multimedia data, etc. Much of the Web information is semi-structured due to the nested structure of HTML code. Much of the Web information is linked. There are hyperlinks among pages within a site, and across different sites. Knowledge Engineering Group Praha 2009

Opportunities and Challenges
The Web is noisy. A Web page typically contains a mixture of many kinds of information, e.g., main contents, advertisements, navigation panels, copyright notices, etc. Above all, the Web is a virtual society. It is not only about data, information and services, but also about interactions among people, organizations and automatic systems, i.e., communities. The Web is also about services. Many Web sites and pages enable people to perform operations with input parameters, i.e., they provide services. The Web is dynamic. Information on the Web changes constantly. Keeping up with the changes and monitoring the changes are important issues. Knowledge Engineering Group Praha 2009

Web content mining Knowledge Engineering Group Praha 2009

Web content mining algorithm is like a blindfolded person...
Algorithms for the detection of page type (Genre detection). Algorithms for the detection of page parts (on a domain dependent or domain independent level). Algorithms for the extraction of information content (Web information extraction). Knowledge Engineering Group Praha 2009

Visual layout based Web page analysis...
The trend is evolving towards visual layout based Web page analysis... A Web page is represented by various individuals’ formats (VIPS, MDR, m-tree, zone-tree,...). The purpose is to find data records (or sub Web pages with a useful content). The aim can be a comparison of two Web pages or sub Web pages. Knowledge Engineering Group Praha 2009

Genre detection methods
The goal of Genre detection methods is to assign the Web page to a known type... Methods are based on existing (manually identified) classifications. In traditional genre classification, one page belongs to a single genre. There is a need of multi genre classification schemes. Known approaches are focused on home pages, e-shopping, academic Web pages, news, and blogs. Knowledge Engineering Group Praha 2009

Tables Tables are an important element for structuring related data... Domain independent Named Web object Tables are analyzed along four aspects: Physical - a description in terms of inter-cell relative location Structural - the topology of cells as an indicator of their navigational relationship Functional - the purpose of areas of the tables in terms of data access Semantic - the meaning of text in the table and the relationship between the interpretation of cell content Knowledge Engineering Group Praha 2009

Opinion extraction Opinion extraction is about how to summarize customer opinions on product features... Domain dependent Named Web objects The main source for analysis: Opinions of customers on product Web pages Discussions on thematic forums Individual reviews in the form of articles Knowledge Engineering Group Praha 2009

Product details Product details and features usually contain a picture, product name, price information... Domain dependent Named Web object The main source for analysis is a Product page How to extract information and save into a database and then use it How to extract product technical features (the aim is to be able to compare similar products) Knowledge Engineering Group Praha 2009

DynamicMining Current methods of Web content mining focus on analyzing static web sites and cannot deal with constantly changing web sites, such as news sites. Dynamic Mining propose a method for mining online news sites. This method applies dynamic schemes for exploring these web sites and extracting news reports, and uses domain independent statistical analysis for trend analysis. The overall method is an application of web mining that goes beyond straightforward news analysis, trying to understand current society interests and to measure the social importance of ongoing events. Knowledge Engineering Group Praha 2009

Web page is like a family house
Knowledge Engineering Group Praha 2009

Web page is like a family house Each of its sections has its significance, determined by the function which it serves. Every section can be named so that everybody imagines the same thing under that name. Three tasks for a blindfolded person: what sections the building contains the purpose of the building furnishings of individual sections Knowledge Engineering Group Praha 2009

Usability principles are a good foundation
Web usability received renewed attention as many early e-commerce Web sites started failing in 2000 (Wikipedia). User Centered Design - corresponds to what users are used to and does not make the user change their way of working. In which way does the visual organization of the Web pages help to lead the visual exploration for information retrieval? Knowledge Engineering Group Praha 2009

Usability principles are a good foundation
Eye-tracking conclusion: It must be compatible with the set of the designer's intentions. It must be compatible with the set of the user's potentials Knowledge Engineering Group Praha 2009

Task for web mining User Technology Intent User interface Software design intent concept user Interaction Technology Implementation Application domain User Interaction Technology solution Knowledge Engineering Group Praha 2009

Patterns, Patterns language Knowledge Engineering Group Praha 2009

Patterns in Architecture
Does this room makes you feel happy? Why? Light (direction) Proportions Symmetry Furniture And more… Some studies showed that certain patterns in architecture are connected with well-being They solve hard problems, which are not always trivial Notice that the room direction is to the south, letting a lot of light to enter Notice the symmetry Notice that the height of the top glass tiles is exactly as the width of the tiles on both sides of the door Notice the chair by the door, a place for dumping stuff when you get home Knowledge Engineering Group Praha 2009

Patterns - LIGHT ON TWO SIDES OF EVERY ROOM
Architecture, Design Patterns, … “When they have a choice, people will always gravitate to those rooms which have light on two sides, and leave the rooms which are lit only from one side unused and empty.” (Alexander et al., 1977 pattern 159) Knowledge Engineering Group Praha 2009

Patterns - LIGHT ON TWO SIDES OF EVERY ROOM
The solution is then included: “Locate each room so that it has outdoor space outside it on at least two sides, and then place windows in these outdoor walls so that natural light falls into every room from more than one direction. “ (Alexander et al., 1977 pattern 159) Knowledge Engineering Group Praha 2009

Patterns Architecture, Design Patterns, … In essence, patterns are structural and behavioral features that improve the applicability of software architecture, a user interface, a Web site or something another in some domain. J. Tidwell, Designing Interfaces: Patterns for Effective Interaction Design, O'Reilly Media, Inc., 2006. Knowledge Engineering Group Praha 2009

What is a Design Pattern?
A description of a recurrent problem and of the core of possible solutions. In Short, a solution for a typical problem Knowledge Engineering Group Praha 2009

Why do we need Patterns? Reusing design knowledge Problems are not always unique. Reusing existing experience might be useful. Patterns give us hints to “where to look for problems”. Establish common terminology Easier to say, "We need a Facade here“. Provide a higher level prospective Frees us from dealing with the details too early In short, it’s a “reference” A reference according to the Technion undergrad notion, not the regular notion! Knowledge Engineering Group Praha 2009

History of Design Patterns
Architecture Christopher Alexander The Timeless Way of Building A Pattern Language: Towns, Buildings, Construction 1970’ Object Oriented Software Design Gang of Four (GoF) Design Patterns: Elements of Reusable Object-Oriented Software 1995’ Other Areas: HCI, Organizational Behavior, Education, Concurent Programming… Many Authors 2007’ Knowledge Engineering Group Praha 2009

Structure of a design pattern*
Pattern Name and Classification Intent a Short statement about what the pattern does Motivation A scenario that illustrates where the pattern would be useful Applicability Situations where the pattern can be used *According to GoF Knowledge Engineering Group Praha 2009

Structure of a design pattern
A graphical representation of the pattern Participants The classes and objects participating in the pattern Collaborations How to do the participants interact to carry out their responsibilities? Consequences What are the pros and cons of using the pattern? Implementation Hints and techniques for implementing the pattern Consequences are important, they require us to focus on the pros and cons of patterns. There are always pros and cons! Every pattern creates a problem. Knowledge Engineering Group Praha 2009

Patterns Architecture, Design Patterns, … In essence, patterns are structural and behavioral features that improve the applicability of software architecture, a user interface, a Web site or something another in some domain. J. Tidwell, Designing Interfaces: Patterns for Effective Interaction Design, O'Reilly Media, Inc., 2006. Knowledge Engineering Group Praha 2009

Patterns – Catalogue 1/1 There are catalogs of patterns. For example: Tidwell, Designing Interfaces: Patterns for Effective Interaction Design. O'Reilly Media, Inc., For pattern description we use the structure originated by Kent Beck Knowledge Engineering Group Praha 2009

Patterns catalogue 1/2 Context of design Site types · Web-based Application · Artist Site · Automotive Site · Branded Promotion Site · Campaign Site · E-commerce Site · Community Site · Corporate Site · Multinational Site · Museum Site · Personalized 'My' Site · News Site · Portal Site · Travel Site Experiences · Community Building · Information Management · Fun · Information Seeking · Learning · Assistence · Shopping · Story Telling Page types · Article Page · Blog Page · Case Study · Contact Page · Event Calendar · Forum · Guest Book · Help Page · Homepage · Newsletter · Printer-friendly Page · Product Page · Tutorial Knowledge Engineering Group Praha 2009

Pattern Title – appropriate pattern name Problem: A single brief sentence describing the problem which pattern solves. Context: A list of situations where the pattern occurs. Forces: A list of details which influence the pattern identification. We are focusing especially on features useful for automatic detection. Solution: Description of the solution with examples. Knowledge Engineering Group Praha 2009

Patterns - Example Knowledge Engineering Group Praha 2009

Pattern – (Toy) Example
<?xml version="1.0" encoding="utf-8" ?> - <PATTERN> <ID>0</ID> <NAME>Information about price</NAME> <PROXIMITY>8</PROXIMITY> <BASE_WEIGHT>1</BASE_WEIGHT> <PROMINENCE_WEIGHT>1</PROMINENCE_WEIGHT> <COMPOSITE_WEIGHT>2</COMPOSITE_WEIGHT> <RECURRENT_WEIGHT>0,25</RECURRENT_WEIGHT> <TEXTUAL_WEIGHT>0</TEXTUAL_WEIGHT> <SYNERGY_WEIGHT>2</SYNERGY_WEIGHT> - <PRIMARY_KEYWORDS> <WORD>EU</WORD> <WORD>Dollar</WORD> <WORD>Price</WORD> </PRIMARY_KEYWORDS> - <SECONDARY_KEYWORDS> <WORD>Price</WORD> <WORD>Prices</WORD> <WORD>monetary value</WORD> <WORD> guarantee </WORD> <WORD> warranty </WORD> <WORD> guaranty </WORD> <WORD> goods </WORD> <WORD> commodity </WORD> </SECONDARY_KEYWORDS> - <PRIMARY_ONTOLOGIES> <WORD><price_token></WORD> </PRIMARY_ONTOLOGIES> - <SECONDARY_ONTOLOGIES> <WORD><percentage_token></WORD> </SECONDARY_ONTOLOGIES> </PATTERN> Knowledge Engineering Group Praha 2009

Gestalt principles Knowledge Engineering Group Praha 2009

Gestalt principles Gestalt is also known as the "Law of Simplicity" or the "Law of Prägnanz" (the entire figure or configuration), which states that every stimulus is perceived in its most simple form. Gestalt theorists followed the basic principle that the whole is greater than the sum of its parts. Knowledge Engineering Group Praha 2009

Gestalt principles In other words, the whole (a picture, a car) carried a different and altogether greater meaning than its individual components (paint, canvas, brush; or tire, paint, metal, respectively). In viewing the "whole," a cognitive process takes place – the mind makes a leap from comprehending the parts to realizing the whole. Knowledge Engineering Group Praha 2009

Gestalt principles can provide a theoretical base...
Proximity: If things are close together viewers will associate them with one another. Similarity: Similar elements tend to be perceived as a group. Continuity: Our eyes want to see continuous lines and curves formed by the alignment of smaller elements. Closure: Elements are not completely enclosed in a space. If enough information is provided, elements tend to be perceived as a group. Knowledge Engineering Group Praha 2009

Gestalt principles Visual systems usually implement the four basic principles: Proximity - Similar information are close. Similarity – Similar things have silmilar meanin. Continuity- Each information follow one by one. Closure – Related information are grouping. Knowledge Engineering Group Praha 2009

(proximity, similarity, continuity, closer) Knowledge Engineering Group Praha 2009

Gestalt principles – Web page
We want to buy mobile phone Knowledge Engineering Group Praha 2009

Gestalt principles – Web page

Patterns - Gestalt principles
Following the Gestalt principles we can suppose a page pattern as a group of characteristic technical elements (whose are based on GUI patterns such as lists, tables, continuous texts) and group of domain specific elements for the domain we are involved in (typical keywords related to given pattern and other entities such as the price, date, percent etc.). The key aspect of the pattern manifestation is that the introduced elements are close to each other. Knowledge Engineering Group Praha 2009

Object retrieval Object-level Information Extraction – A Web object is constructed by collecting related data records extracted from multiple Web sources. The sources for holding object information could be HTML pages, documents put on the Web (e.g. PDF, PS, Word, and other formats.), and deep contents hidden in Web databases. (In previuos Figure.) Knowledge Engineering Group Praha 2009

Object retrieval There is already extensive research to explore algorithms for extraction of objects from Web sources. Object Identification and Integration – Each extracted instance of a Web object needs to be mapped to a real world object and stored into the Web data warehouse. To do so, we need techniques to integrate information about the same object and disambiguate different objects. Knowledge Engineering Group Praha 2009

Motivation Web object retrieval – After information extraction and integration, we should provide retrieval mechanism to satisfy users’ information needs. Basically, the retrieval should be conducted at the object level, which means that the extracted objects should be indexed, ranked and clustered against user queries. Knowledge Engineering Group Praha 2009

Algorithm 1. For proximity we defined method how to measure closeness (distance) between entities in searched text segments. 2. For similarity we defined method for measuring similarity of two searched text segments (for Discussion we are able to identify repetition of replies). We work with comparison of trees representing text segments. 3. For continuity we defined method how to find out whether two or more found text segments make together instance of pattern. We assume that two or more little-similar text segments (trees of entities from one pattern) match together. 4. For closure we defined a method for computation of weight of one single searched text segment. In essence we used two criteria. We rated shape of the segment tree (particularly ratio of height and entity count) and quantity of all words and paragraphs in text segment. On the overall computation of weight also the proximity rate participates. Knowledge Engineering Group Praha 2009

Algorithm – membership computation
FOR each page entity in all page entities IF page entity is pattern entity THEN IF does not exist snippet to add page entity to THEN create new snippet in list of snippets END IF add page entity to snippet END FOR FOR each snippet in list of snippets compute proximity of snippet compute closure of snippet compute value(proximity, closure) of snippet IF value is not good enough THEN remove snippet from list of snippets compute similarity of list of snippets compute continuity of list of snippets compute value(similarity, continuity) of pattern RETURN value Knowledge Engineering Group Praha 2009

Experiments We collected 31,738 various web pages which we got from the Google search engine using queries on products. After the analysis we discovered that on the 11,038 web pages there was not any extracted patterns. There were more than 200 searches of products tested (cellular phones, computers, components and peripheries, electronics, sport equipment, cosmetics, books, CDs, DVDs, etc.). Knowledge Engineering Group Praha 2009

Experiments - Re-ranking

Experiments - Retrieval Accuracy

Experiments - Re-ranking

Implementation Knowledge Engineering Group Praha 2009

Pattrio: Inspired by Patterns and Objects...
Web design patterns and patterns languages Named Web object as a Web design pattern projection Catalog of Named Web objects Detection of Named Web objects Use of Named Web Objects Knowledge Engineering Group Praha 2009

Named Web objects can provide a simple description for SERP...

Patterns are intended for developers and they do not contain technical details... Design pattern is a text description about how to solve an existing problem. Technical details are important for the recognition by the user (and by the algorithm). A different description has to be used (Pattrio catalog). The Named object is a projection of a Web design pattern (or Genre) to a concrete part of Web page. Knowledge Engineering Group Praha 2009

Pattern Extraction In our experiment we were searching web pages in sets of thirty using very precise query. The query contained product identification (ex. Nokia 9300) and group of six words from the pattern dictionary connected in OR relation for making query more accurate. From the searched pages our algorithm extracted nine patterns (Price Information, Purchasing Possibility, Special Offer, Annuity Selling, Product information, Discussion, Review, Sign on possibility, Advertising). For evaluation of each pattern we used seven criterions. Each criterion was rated using three-degree scale. In all it is expressed using 21 Boolean values. Knowledge Engineering Group Praha 2009

SOM — web pages from selling product domain

Web Communities Defined by Web Page Content

Web Communities Defined by Web Page Content
In this part we are looking for a relationship between the intent of Web pages, their architecture and the communities who take part in their usage and creation. For us, the Web page is entity carrying information about these communities. We present an experiment which proves the feasibility of our approach. Knowledge Engineering Group Praha 2009

Typical Web site aimed at information sharing

Typical university Web site

Vision The crucial aspect of our approach is that we do not need to analyze page’s HTML code. Our algorithm is based on analysis of plain text of the page. For page evaluation we do not use any meta‑information about page (such as title, hyperlinks, meta‑tags and so on). We also confirmed that key characteristics of web patterns are independent of language environment. We tested our method in English and Arabic and Czech language environment. The only thing we had to do was to change patterns dictionaries. Knowledge Engineering Group Praha 2009

Thank you Knowledge Engineering Group Praha 2009

Václav Snášel, Miloš Kudělka VSB-Technical University of Ostrava

Similar presentations

Presentation on theme: "Václav Snášel, Miloš Kudělka VSB-Technical University of Ostrava"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Václav Snášel, Miloš Kudělka VSB-Technical University of Ostrava

Similar presentations

Presentation on theme: "Václav Snášel, Miloš Kudělka VSB-Technical University of Ostrava"— Presentation transcript:

Similar presentations

About project

Feedback