Presentation is loading. Please wait.

Presentation is loading. Please wait.

Web Mining Václav Snášel, Miloš Kudělka VSB-Technical University of Ostrava Czech Republic.

Similar presentations

Presentation on theme: "Web Mining Václav Snášel, Miloš Kudělka VSB-Technical University of Ostrava Czech Republic."— Presentation transcript:

1 Web Mining Václav Snášel, Miloš Kudělka VSB-Technical University of Ostrava Czech Republic

2 Outline... Introduction Classification of recent approaches. Web page segmentation. Genre detection, Table extraction. Opinion, News, and Discussion extraction. Product details and Technical features extraction Knowledge Engineering Group Praha 20092

3 Web content mining describes the discovery of useful information from Web contents. The goal of Web content mining is to improve finding information or filtering information for the users. Web structure mining tries to discover the model underlying the link structures of the Web. This model can be used to categorize Web pages and can be useful to generate the relationship between Web sites. Web usage mining tries to make sense of the data generated by the Web surfer's sessions or behaviors. Web usage mining mines the data derived from the interactions of the users. Web mining Knowledge Engineering Group Praha 20093

4 Web mining Web mining - the application of data mining techniques to extract knowledge from Web content, structure, and usage. 4 Knowledge Engineering Group Praha 2009

5 Web Content Mining Web Content Mining is the process of extracting useful information from the contents of Web documents. It may consist of text, images, audio, video, or structured records such as lists and tables. Web Content mining refers to the overall process of discovering potentially useful and previously unknown information or knowledge from the Web data. Knowledge Engineering Group Praha 20095

6 Web Structure Mining Web structure mining tries to discover the model underlying the link structures of the Web. The model is based on the topology of the hyperlinks with or without the description of the links. This model can be used to categorize Web pages and is useful to generate information such as the similarity and relationship between different Web sites. Web structure mining could be used to discover authority sites for the subjects (authorities) and overview sites for the subjects that point to many authorities (hubs). Knowledge Engineering Group Praha 20096

7 Web Usage Mining Web 2.0, enables individuals to create and share content on the Web. One of the important distinguishing features of Web 2.0 is the creation of communities of users, i.e., social networks with new demands on data management. In social content sites, both content and user interest are dynamic: people review and tag new content every day. Knowledge Engineering Group Praha 20097

8 Web Usage Mining Web usage mining, which aims to discover interesting and frequent user access patterns from web usage data, can be used to model past web access behavior of users. The acquired model can then be used for analyzing and predicting the future user access behavior. Knowledge Engineering Group Praha 20098

9 Opportunities and Challenges Web offers an unprecedented opportunity and challenge to data mining The amount of information on the Web is huge, and easily accessible. The coverage of Web information is very wide and diverse. One can find information about almost anything. Information/data of almost all types exist on the Web, e.g., structured tables, texts, multimedia data, etc. Much of the Web information is semi-structured due to the nested structure of HTML code. Much of the Web information is linked. There are hyperlinks among pages within a site, and across different sites. Knowledge Engineering Group Praha 20099

10 Opportunities and Challenges The Web is noisy. A Web page typically contains a mixture of many kinds of information, e.g., main contents, advertisements, navigation panels, copyright notices, etc. Above all, the Web is a virtual society. It is not only about data, information and services, but also about interactions among people, organizations and automatic systems, i.e., communities. The Web is also about services. Many Web sites and pages enable people to perform operations with input parameters, i.e., they provide services. The Web is dynamic. Information on the Web changes constantly. Keeping up with the changes and monitoring the changes are important issues. Knowledge Engineering Group Praha

11 Knowledge Engineering Group Praha Web content mining

12 12 Web content mining algorithm is like a blindfolded person... Algorithms for the detection of page type (Genre detection). Algorithms for the detection of page parts (on a domain dependent or domain independent level). Algorithms for the extraction of information content (Web information extraction). Knowledge Engineering Group Praha 2009

13 13 Visual layout based Web page analysis... The trend is evolving towards visual layout based Web page analysis... A Web page is represented by various individuals formats (VIPS, MDR, m-tree, zone-tree,...). The purpose is to find data records (or sub Web pages with a useful content). The aim can be a comparison of two Web pages or sub Web pages. Knowledge Engineering Group Praha 2009

14 Genre detection methods The goal of Genre detection methods is to assign the Web page to a known type... Methods are based on existing (manually identified) classifications. In traditional genre classification, one page belongs to a single genre. There is a need of multi genre classification schemes. Known approaches are focused on home pages, e-shopping, academic Web pages, news, and blogs. Knowledge Engineering Group Praha

15 Tables Tables are an important element for structuring related data... Domain independent Named Web object Tables are analyzed along four aspects: Physical - a description in terms of inter-cell relative location Structural - the topology of cells as an indicator of their navigational relationship Functional - the purpose of areas of the tables in terms of data access Semantic - the meaning of text in the table and the relationship between the interpretation of cell content Knowledge Engineering Group Praha

16 16 Opinion extraction Opinion extraction is about how to summarize customer opinions on product features... Domain dependent Named Web objects The main source for analysis: Opinions of customers on product Web pages Discussions on thematic forums Individual reviews in the form of articles Knowledge Engineering Group Praha 2009

17 17 Product details Product details and features usually contain a picture, product name, price information... Domain dependent Named Web object The main source for analysis is a Product page How to extract information and save into a database and then use it How to extract product technical features (the aim is to be able to compare similar products) Knowledge Engineering Group Praha 2009

18 DynamicMining Current methods of Web content mining focus on analyzing static web sites and cannot deal with constantly changing web sites, such as news sites. Dynamic Mining propose a method for mining online news sites. This method applies dynamic schemes for exploring these web sites and extracting news reports, and uses domain independent statistical analysis for trend analysis. The overall method is an application of web mining that goes beyond straightforward news analysis, trying to understand current society interests and to measure the social importance of ongoing events. Knowledge Engineering Group Praha

19 Web page is like a family house Knowledge Engineering Group Praha

20 Each of its sections has its significance, determined by the function which it serves. Every section can be named so that everybody imagines the same thing under that name. Three tasks for a blindfolded person: what sections the building contains the purpose of the building furnishings of individual sections Web page is like a family house Knowledge Engineering Group Praha

21 Usability principles are a good foundation Web usability received renewed attention as many early e-commerce Web sites started failing in 2000 (Wikipedia). User Centered Design - corresponds to what users are used to and does not make the user change their way of working. In which way does the visual organization of the Web pages help to lead the visual exploration for information retrieval? Knowledge Engineering Group Praha

22 Usability principles are a good foundation Eye-tracking conclusion: It must be compatible with the set of the designer's intentions. It must be compatible with the set of the user's potentials Knowledge Engineering Group Praha

23 Task for web mining Knowledge Engineering Group Praha Application domainUser InteractionTechnology solution User interface Software design

24 Patterns, Patterns language Knowledge Engineering Group Praha

25 Patterns in Architecture Does this room makes you feel happy? Why? Light (direction) Proportions Symmetry Furniture And more… Knowledge Engineering Group Praha

26 Patterns - LIGHT ON TWO SIDES OF EVERY ROOM Architecture, Design Patterns, … When they have a choice, people will always gravitate to those rooms which have light on two sides, and leave the rooms which are lit only from one side unused and empty. (Alexander et al., 1977 pattern 159) Knowledge Engineering Group Praha

27 Patterns - LIGHT ON TWO SIDES OF EVERY ROOM The solution is then included: Locate each room so that it has outdoor space outside it on at least two sides, and then place windows in these outdoor walls so that natural light falls into every room from more than one direction. (Alexander et al., 1977 pattern 159) Knowledge Engineering Group Praha

28 Patterns Architecture, Design Patterns, … In essence, patterns are structural and behavioral features that improve the applicability of software architecture, a user interface, a Web site or something another in some domain. J. Tidwell, Designing Interfaces: Patterns for Effective Interaction Design, O'Reilly Media, Inc., Knowledge Engineering Group Praha

29 What is a Design Pattern? In Short, a solution for a typical problem A description of a recurrent problem and of the core of possible solutions. Knowledge Engineering Group Praha

30 Why do we need Patterns? Reusing design knowledge Problems are not always unique. Reusing existing experience might be useful. Patterns give us hints to where to look for problems. Establish common terminology Easier to say, "We need a Facade here. Provide a higher level prospective Frees us from dealing with the details too early In short, its a reference Knowledge Engineering Group Praha

31 History of Design Patterns Christopher Alexander The Timeless Way of Building A Pattern Language: Towns, Buildings, Construction Architecture Object Oriented Software Design Other Areas: HCI, Organizational Behavior, Education, Concurent Programming… Gang of Four (GoF) Design Patterns: Elements of Reusable Object-Oriented Software Many Authors Knowledge Engineering Group Praha

32 Structure of a design pattern* Pattern Name and Classification Intent a Short statement about what the pattern does Motivation A scenario that illustrates where the pattern would be useful Applicability Situations where the pattern can be used *According to GoF Knowledge Engineering Group Praha

33 Structure of a design pattern Structure A graphical representation of the pattern Participants The classes and objects participating in the pattern Collaborations How to do the participants interact to carry out their responsibilities? Consequences What are the pros and cons of using the pattern? Implementation Hints and techniques for implementing the pattern Knowledge Engineering Group Praha

34 Patterns Architecture, Design Patterns, … In essence, patterns are structural and behavioral features that improve the applicability of software architecture, a user interface, a Web site or something another in some domain. J. Tidwell, Designing Interfaces: Patterns for Effective Interaction Design, O'Reilly Media, Inc., Knowledge Engineering Group Praha

35 Patterns – Catalogue 1/1 There are catalogs of patterns. For example: Tidwell, Designing Interfaces: Patterns for Effective Interaction Design. O'Reilly Media, Inc., For pattern description we use the structure originated by Kent Beck Knowledge Engineering Group Praha

36 Patterns catalogue 1/2 Site types · Web-based Application · Artist Site · Automotive Site · Branded Promotion Site · Campaign Site · E-commerce Site · Community Site · Corporate Site · Multinational Site · Museum Site · Personalized 'My' Site · News Site · Portal Site · Travel SiteWeb-based ApplicationArtist SiteAutomotive SiteBranded Promotion SiteCampaign SiteE-commerce SiteCommunity SiteCorporate SiteMultinational SiteMuseum SitePersonalized 'My' SiteNews SitePortal SiteTravel Site Experiences · Community Building · Information Management · Fun · Information Seeking · Learning · Assistence · Shopping · Story TellingCommunity BuildingInformation ManagementFunInformation SeekingLearningAssistenceShoppingStory Telling Page types · Article Page · Blog Page · Case Study · Contact Page · Event Calendar · Forum · Guest Book · Help Page · Homepage · Newsletter · Printer-friendly Page · Product Page · TutorialArticle PageBlog PageCase StudyContact PageEvent CalendarForumGuest BookHelp PageHomepageNewsletterPrinter-friendly PageProduct PageTutorial Knowledge Engineering Group Praha Context of design

37 Pattern Title – appropriate pattern name Problem: A single brief sentence describing the problem which pattern solves. Context: A list of situations where the pattern occurs. Forces: A list of details which influence the pattern identification. We are focusing especially on features useful for automatic detection. Solution: Description of the solution with examples. Knowledge Engineering Group Praha

38 Patterns - Example Knowledge Engineering Group Praha

39 Pattern – (Toy) Example - 0 Information about price , EU Dollar Price - Price Prices monetary value guarantee warranty guaranty goods commodity - Knowledge Engineering Group Praha

40 Gestalt principles Knowledge Engineering Group Praha

41 Gestalt principles Gestalt is also known as the "Law of Simplicity" or the "Law of Prägnanz" (the entire figure or configuration), which states that every stimulus is perceived in its most simple form. Gestalt theorists followed the basic principle that the whole is greater than the sum of its parts. Knowledge Engineering Group Praha

42 Gestalt principles In other words, the whole (a picture, a car) carried a different and altogether greater meaning than its individual components (paint, canvas, brush; or tire, paint, metal, respectively). In viewing the "whole," a cognitive process takes place – the mind makes a leap from comprehending the parts to realizing the whole. Knowledge Engineering Group Praha

43 Gestalt principles can provide a theoretical base... Proximity: If things are close together viewers will associate them with one another. Similarity: Similar elements tend to be perceived as a group. Continuity: Our eyes want to see continuous lines and curves formed by the alignment of smaller elements. Closure: Elements are not completely enclosed in a space. If enough information is provided, elements tend to be perceived as a group. Knowledge Engineering Group Praha

44 Gestalt principles Visual systems usually implement the four basic principles: Proximity - Similar information are close. Similarity – Similar things have silmilar meanin. Continuity- Each information follow one by one. Closure – Related information are grouping. Knowledge Engineering Group Praha

45 Knowledge Engineering Group Praha (proximity, similarity, continuity, closer)

46 Gestalt principles – Web page We want to buy mobile phone Knowledge Engineering Group Praha

47 Gestalt principles – Web page Knowledge Engineering Group Praha

48 Patterns - Gestalt principles Following the Gestalt principles we can suppose a page pattern as a group of characteristic technical elements (whose are based on GUI patterns such as lists, tables, continuous texts) and group of domain specific elements for the domain we are involved in (typical keywords related to given pattern and other entities such as the price, date, percent etc.). The key aspect of the pattern manifestation is that the introduced elements are close to each other. Knowledge Engineering Group Praha

49 Object retrieval Object-level Information Extraction – A Web object is constructed by collecting related data records extracted from multiple Web sources. The sources for holding object information could be HTML pages, documents put on the Web (e.g. PDF, PS, Word, and other formats.), and deep contents hidden in Web databases. (In previuos Figure.) Knowledge Engineering Group Praha

50 Object retrieval There is already extensive research to explore algorithms for extraction of objects from Web sources. Object Identification and Integration – Each extracted instance of a Web object needs to be mapped to a real world object and stored into the Web data warehouse. To do so, we need techniques to integrate information about the same object and disambiguate different objects. Knowledge Engineering Group Praha

51 Motivation Web object retrieval – After information extraction and integration, we should provide retrieval mechanism to satisfy users information needs. Basically, the retrieval should be conducted at the object level, which means that the extracted objects should be indexed, ranked and clustered against user queries. Knowledge Engineering Group Praha

52 Algorithm 1. For proximity we defined method how to measure closeness (distance) between entities in searched text segments. 2. For similarity we defined method for measuring similarity of two searched text segments (for Discussion we are able to identify repetition of replies). We work with comparison of trees representing text segments. 3. For continuity we defined method how to find out whether two or more found text segments make together instance of pattern. We assume that two or more little-similar text segments (trees of entities from one pattern) match together. 4. For closure we defined a method for computation of weight of one single searched text segment. In essence we used two criteria. We rated shape of the segment tree (particularly ratio of height and entity count) and quantity of all words and paragraphs in text segment. On the overall computation of weight also the proximity rate participates. Knowledge Engineering Group Praha

53 Algorithm – membership computation FOR each page entity in all page entities IF page entity is pattern entity THEN IF does not exist snippet to add page entity to THEN create new snippet in list of snippets END IF add page entity to snippet END IF END FOR FOR each snippet in list of snippets compute proximity of snippet compute closure of snippet compute value(proximity, closure) of snippet IF value is not good enough THEN remove snippet from list of snippets END IF END FOR compute similarity of list of snippets compute continuity of list of snippets compute value(similarity, continuity) of pattern RETURN value Knowledge Engineering Group Praha

54 Experiments We collected 31,738 various web pages which we got from the Google search engine using queries on products. After the analysis we discovered that on the 11,038 web pages there was not any extracted patterns. There were more than 200 searches of products tested (cellular phones, computers, components and peripheries, electronics, sport equipment, cosmetics, books, CDs, DVDs, etc.). Knowledge Engineering Group Praha

55 Experiments - Re-ranking Knowledge Engineering Group Praha

56 Experiments - Retrieval Accuracy Knowledge Engineering Group Praha

57 Experiments - Re-ranking Knowledge Engineering Group Praha

58 Implementation Knowledge Engineering Group Praha

59 Pattrio: Inspired by Patterns and Objects... Web design patterns and patterns languages Named Web object as a Web design pattern projection Catalog of Named Web objects Detection of Named Web objects Use of Named Web Objects Knowledge Engineering Group Praha

60 Named Web objects can provide a simple description for SERP... Knowledge Engineering Group Praha

61 Patterns are intended for developers and they do not contain technical details... Design pattern is a text description about how to solve an existing problem. Technical details are important for the recognition by the user (and by the algorithm). A different description has to be used (Pattrio catalog). The Named object is a projection of a Web design pattern (or Genre) to a concrete part of Web page. Knowledge Engineering Group Praha

62 Pattern Extraction In our experiment we were searching web pages in sets of thirty using very precise query. The query contained product identification (ex. Nokia 9300) and group of six words from the pattern dictionary connected in OR relation for making query more accurate. From the searched pages our algorithm extracted nine patterns (Price Information, Purchasing Possibility, Special Offer, Annuity Selling, Product information, Discussion, Review, Sign on possibility, Advertising). For evaluation of each pattern we used seven criterions. Each criterion was rated using three-degree scale. In all it is expressed using 21 Boolean values. Knowledge Engineering Group Praha

63 SOM web pages from selling product domain Knowledge Engineering Group Praha

64 SOM web pages from selling product domain Knowledge Engineering Group Praha

65 Web Communities Defined by Web Page Content Knowledge Engineering Group Praha

66 Web Communities Defined by Web Page Content Knowledge Engineering Group Praha In this part we are looking for a relationship between the intent of Web pages, their architecture and the communities who take part in their usage and creation. For us, the Web page is entity carrying information about these communities. We present an experiment which proves the feasibility of our approach.

67 Typical Web site aimed at information sharing Knowledge Engineering Group Praha

68 Typical university Web site Knowledge Engineering Group Praha

69 Vision The crucial aspect of our approach is that we do not need to analyze pages HTML code. Our algorithm is based on analysis of plain text of the page. For page evaluation we do not use any meta information about page (such as title, hyperlinks, meta tags and so on). We also confirmed that key characteristics of web patterns are independent of language environment. We tested our method in English and Arabic and Czech language environment. The only thing we had to do was to change patterns dictionaries. Knowledge Engineering Group Praha

70 Knowledge Engineering Group Praha Thank you

Download ppt "Web Mining Václav Snášel, Miloš Kudělka VSB-Technical University of Ostrava Czech Republic."

Similar presentations

Ads by Google