Presented by: Apeksha Khabia Guided by: Dr. M. B. Chandak A Combined Approach for Classification of Web Results based on Ranking & Clustering Presented by: Apeksha Khabia Guided by: Dr. M. B. Chandak

Contents Introduction Page rank, weighted page rank Document clustering Algorithm for clustering and ranking Conclusion Future scope

Introduction The web is most precious place for Information retrieval and Knowledge Discovery Retrieving information through queries from a search engine is tedious Solution is Web Mining web content mining web structure mining web uses mining

How to Generate Web Results?

Page Rank (PR) Order the search results such that important documents move up and less important move down in the list If a page has some important incoming links then its outgoing link also becomes important

Page Rank (PR) Rank score of a page p is evenly divided among outgoing links Modified PR in view Random Surfer Model – not all the users follow direct Links on WWW

Example of Page Rank (PR) PR(A)= (1-d)+d((PR(B)/2+PR(C)/2 ) PR(B)= (1-d)+d( PR(A)/1+PR(C)/2 ) PR(C)= (1-d)+d( PR(B)/2) IF d=0.5 PR(A)=1.2, PR(B)=1.2, PR(C)=0.8 Table : Iterative method of page rank

Weighted Page Rank (WPR) Assign larger rank values to more important pages instead of evenly dividing among its outgoing links. Outlink page gets value according to its popularity

Document Clustering  Automatic document organization, topic extraction and fast information retrieval or filtering Documents are grouped together based upon measure of similarity of content or of hyperlinked structure Clustering divides the results of a search for "cell" into groups like "biology," "battery," and "prison."

Document Clustering Examples : K-means, hierarchical Clustering may be based on content alone, or both on contents and links or only on links Two ways to define content based similarity between the documents Resemblance Containment

Limitations of Ranking Approach They give emphasis to links of the resultant pages No algorithm exists to combine the link score and content score of the page into a single score Existing approaches return millions of documents in an ordered format Rank based approaches give equal emphasis to inlinks as well as outlinks of pages

Introduction of Combined Approach This mechanism takes advantage of importance of inlinks over outlinks With the use of this user can put search results into hierarchy of query related clusters Also the documents in each cluster can be ranked to represent them according to their relevancy Such organization enables the user to effectively limit his search area

Algorithm for Clustering and Ranking

Algorithm: Steps Step 1: Get the URLs of the pages Step 2: Provide a similarity value sim(q, p) to each returned document Step 3: Use sim(q, p) to cluster the documents Step 4: Provide a rank score WSR(p) to the documents of each cluster:

Output Clusters of web pages documents are formed based on the similarity Also the documents in each cluster are ranked to represent them according to their relevancy

Similarity Calculation between Web Pages Similarity of the document with the query means: what query terms are present in the document? where they are present? how many times? Calculated using cosine between vector of query terms and vector of documents

Rank Calculation of Web pages WSR- Weight and Similarity based Rank Back-links contribute more towards the importance of a page rather than forward links WSR gives more importance to the inlinks of a page Importance of the backlink page v of a page u, given by

Rank Calculation of Web pages Redefined formula for rank is given by

Clustering of Web pages The clustering is purely based on the similarity values of the pages with respect to the user query The number of clusters is not predefined The maximum number of pages that can be in a cluster should be decided

Clustering of Web pages Lower and upper value of similarity is identified from range of similarity values Complete page set is divided into number of sets according to the similarity values lying within the partitioned ranges. Rank(p)= WSR(p) + sim(q, p)

Applications of Clustering and Ranking Readability assessment - automatically determining the degree of readability of a text, either to find suitable materials for different age groups or reader types or as part of a larger text simplification system Genre classification - automatically determining the genre of a text

Tools Available Weka Rapid Miner KNIME Orange

Conclusion Ranking and clustering gives a way to organize the search results in the form of clusters, the pages in each cluster are further ranked to provide the most relevant and important pages on the top of the cluster User search space decreases and he can get required content in short time

Future Ideas Different data mining tools (KNIME, Rapid Miner, Weka) can be used to analyze the result for classification Search query results for classification can be incorporated from multiple search engines

References Parneet Kaur, Sawtantar Singh Khurmi and Gurpreet Singh Josan, " Analysis for Classification of Similar Documents among Various Websites using Rapid Miner“. In the proceedings of IEEE International Conference on Issues and Challenges in Intelligent Computing Techniques (ICICT), 2014 Neelam Duhan and A.K. Shanna, "A Novel Approach for Organizing Web Search Results using Ranking and Clustering". In the Proceedings of International Journal of Computer Applications, vol. 5, No. 10, pp. 1-9, August 2010. O. Zamir, O. Etzioni. “Web document clustering: A feasibility demonstration”. Proceedings of the 19th International ACM SIGIR Conference on Research and Development of Information Retrieval (SIGIR'98), 46-54,1998.

References Miguel Gomes da Costa Júnior, Zhiguo Gong, “Web Structure Mining: An introduction”. Proceedings of the IEEE International Conference on Information Acquisition, 2005, China. Taher H. Haveliwala, Aristides Gionis, Dan Klein, Piotr Indyk, “Evaluating strategies for similarity search on the Web”. WWW2002, May, 2002, Honolulu, Hawaii, USA.ACM 1-58113-449-5/02/0005.

Thank You!!