Presentation is loading. Please wait.

Presentation is loading. Please wait.

Probe, Count, and Classify: Categorizing Hidden Web Databases

Similar presentations


Presentation on theme: "Probe, Count, and Classify: Categorizing Hidden Web Databases"— Presentation transcript:

1 Probe, Count, and Classify: Categorizing Hidden Web Databases
Panagiotis G. Ipeirotis Luis Gravano Columbia University Mehran Sahami E.piphany Inc. Frequently, presenters must deliver material of a technical nature to an audience unfamiliar with the topic or vocabulary. The material may be complex or heavy with detail. To present technical material effectively, use the following guidelines from Dale Carnegie Training®. Consider the amount of time available and prepare to organize your material. Narrow your topic. Divide your presentation into clear segments. Follow a logical progression. Maintain your focus throughout. Close the presentation with a summary, repetition of the key steps, or a logical conclusion. Keep your audience in mind at all times. For example, be sure data is clear and information is relevant. Keep the level of detail and vocabulary appropriate for the audience. Use visuals to support key points or steps. Keep alert to the needs of your listeners, and you will have a more receptive audience.

2 Surface Web vs. Hidden Web
In your opening, establish the relevancy of the topic to the audience. Give a brief preview of the presentation and establish value for the listeners. Take into account your audience’s interest and expertise in the topic when choosing your vocabulary, examples, and illustrations. Focus on the importance of the topic to your audience, and you will have more attentive listeners. Surface Web Link structure Crawlable Hidden Web No link structure Documents “hidden” behind search forms

3 Do We Need the Hidden Web?
Surface Web Hidden Web 2 billion pages 500 billion pages (?) Example: PubMed/MEDLINE PubMed: (www.ncbi.nlm.nih.gov/PubMed) search: “cancer”  1,341,586 matches AltaVista: “cancer site:www.ncbi.nlm.nih.gov”  21,830 matches In your opening, establish the relevancy of the topic to the audience. Give a brief preview of the presentation and establish value for the listeners. Take into account your audience’s interest and expertise in the topic when choosing your vocabulary, examples, and illustrations. Focus on the importance of the topic to your audience, and you will have more attentive listeners.

4 Interacting With Searchable Text Databases
Searching: Metasearchers Browsing: Yahoo!-like web directories: InvisibleWeb.com SearchEngineGuide.com Example from InvisibleWeb.com Health > Publications > PubMED Created Manually! In your opening, establish the relevancy of the topic to the audience. Give a brief preview of the presentation and establish value for the listeners. Take into account your audience’s interest and expertise in the topic when choosing your vocabulary, examples, and illustrations. Focus on the importance of the topic to your audience, and you will have more attentive listeners.

5 Classifying Text Databases Automatically: Outline
Definition of classification Classification through query probing Experiments If you have several points, steps, or key ideas use multiple slides. Determine if your audience is to understand a new idea, learn a process, or receive greater depth to a familiar concept. Back up each point with adequate explanation. As appropriate, supplement your presentation with technical support data in hard copy or on disc, , or the Internet. Develop each point adequately to communicate with your audience.

6 Database Classification: Two Definitions
Coverage-based classification: Database contains many documents about a category Coverage: #docs about this category Specificity-based classification: Database contains mainly documents about a category Specificity: #docs/|DB| If you have several points, steps, or key ideas use multiple slides. Determine if your audience is to understand a new idea, learn a process, or receive greater depth to a familiar concept. Back up each point with adequate explanation. As appropriate, supplement your presentation with technical support data in hard copy or on disc, , or the Internet. Develop each point adequately to communicate with your audience.

7 Database Classification: An Example
Category: Basketball Coverage-based classification ESPN.com, NBA.com, not KnicksTerritory.com Specificity-based classification NBA.com, KnicksTerritory.com, not ESPN.com If you have several points, steps, or key ideas use multiple slides. Determine if your audience is to understand a new idea, learn a process, or receive greater depth to a familiar concept. Back up each point with adequate explanation. As appropriate, supplement your presentation with technical support data in hard copy or on disc, , or the Internet. Develop each point adequately to communicate with your audience.

8 Database Classification: More Details
Thresholds for coverage and specificity Tc: coverage threshold (e.g., 100) Ts: specificity threshold (e.g., 0.5) Tc, Ts “editorial” choices Ideal(D) Root Ideal(D): set of classes for database D Class C is in Ideal(D) if: D has “enough” coverage and specificity (Tc, Ts) for C and all of C’s ancestors and D fails to have both “enough” coverage and specificity for each child of C SPORTS C=800 S=0.8 HEALTH C=200 S=0.2 BASKETBALL S=0.5 BASEBALL S=0.5

9 From Document to Database Classification
If we know the categories of all documents inside the database, we are done! We do not have direct access to the documents. Databases do not export such data! How can we extract this information? If you have several points, steps, or key ideas use multiple slides. Determine if your audience is to understand a new idea, learn a process, or receive greater depth to a familiar concept. Back up each point with adequate explanation. As appropriate, supplement your presentation with technical support data in hard copy or on disc, , or the Internet. Develop each point adequately to communicate with your audience.

10 Our Approach: Query Probing
Train a rule-based document classifier. Transform classifier rules into queries. Adaptively send queries to databases. Categorize the databases based on adjusted number of query matches. If you have several points, steps, or key ideas use multiple slides. Determine if your audience is to understand a new idea, learn a process, or receive greater depth to a familiar concept. Back up each point with adequate explanation. As appropriate, supplement your presentation with technical support data in hard copy or on disc, , or the Internet. Develop each point adequately to communicate with your audience.

11 Training a Rule-based Document Classifier
Feature Selection: Zipf’s law pruning, followed by information-theoretic feature selection [Koller & Sahami’96] Classifier Learning: AT&T’s RIPPER [Cohen 1995] Input: A set of pre-classified, labeled documents Output: A set of classification rules IF linux THEN Computers IF jordan AND bulls THEN Sports IF lung AND cancer THEN Health If you have several points, steps, or key ideas use multiple slides. Determine if your audience is to understand a new idea, learn a process, or receive greater depth to a familiar concept. Back up each point with adequate explanation. As appropriate, supplement your presentation with technical support data in hard copy or on disc, , or the Internet. Develop each point adequately to communicate with your audience.

12 Constructing Query Probes
Transform each rule into a query IF lung AND cancer THEN health  +lung +cancer IF linux THEN computers  +linux Send the queries to the database Get number of matches for each query, NOT the documents (i.e., number of documents that match each rule) These documents would have been classified by the rule under its associated category! If you have several points, steps, or key ideas use multiple slides. Determine if your audience is to understand a new idea, learn a process, or receive greater depth to a familiar concept. Back up each point with adequate explanation. As appropriate, supplement your presentation with technical support data in hard copy or on disc, , or the Internet. Develop each point adequately to communicate with your audience.

13 Adjusting Query Results
Classifiers are not perfect! Queries do not “retrieve” all the documents in a category Queries for one category “match” documents not in this category From the classifier’s training phase we know its “confusion matrix” If you have several points, steps, or key ideas use multiple slides. Determine if your audience is to understand a new idea, learn a process, or receive greater depth to a familiar concept. Back up each point with adequate explanation. As appropriate, supplement your presentation with technical support data in hard copy or on disc, , or the Internet. Develop each point adequately to communicate with your audience.

14 X = Confusion Matrix M . Coverage(D) ~ ECoverage(D) Correct class
10% of “Sports” classified as “Computers” comp sports health 0.70 0.10 0.00 0.18 0.65 0.04 0.02 0.05 0.86 DB-real 1000 5000 50 Probing results = 1200 3432 313 X = If you have several points, steps, or key ideas use multiple slides. Determine if your audience is to understand a new idea, learn a process, or receive greater depth to a familiar concept. Back up each point with adequate explanation. As appropriate, supplement your presentation with technical support data in hard copy or on disc, , or the Internet. Develop each point adequately to communicate with your audience. 10% of the 5000 “Sports” docs to “Computers” Classified into M . Coverage(D) ~ ECoverage(D)

15 Confusion Matrix Adjustment: Compensating for Classifier’s Errors
-1 DB-real 1000 5000 50 comp sports health 0.70 0.10 0.00 0.18 0.65 0.04 0.02 0.05 0.86 Probing results 1200 3432 313 = X M is diagonally dominant, hence invertible If you have several points, steps, or key ideas use multiple slides. Determine if your audience is to understand a new idea, learn a process, or receive greater depth to a familiar concept. Back up each point with adequate explanation. As appropriate, supplement your presentation with technical support data in hard copy or on disc, , or the Internet. Develop each point adequately to communicate with your audience. Coverage(D) ~ M-1 . ECoverage(D) Multiplication better approximates the correct result

16 Classifying a Database
Send the query probes for the top-level categories Get the number of matches for each probe Calculate Specificity and Coverage for each category “Push” the database to the qualifying categories (with Specificity>Ts and Coverage>Tc) Repeat for each of the qualifying categories Return the classes that satisfy the coverage/specificity conditions The result is the Approximation of the Ideal classification If you have several points, steps, or key ideas use multiple slides. Determine if your audience is to understand a new idea, learn a process, or receive greater depth to a familiar concept. Back up each point with adequate explanation. As appropriate, supplement your presentation with technical support data in hard copy or on disc, , or the Internet. Develop each point adequately to communicate with your audience.

17 Real Example: ACM Digital Library (Tc=100, Ts=0.5)

18 Experiments: Data 72-node 4-level topic hierarchy from InvisibleWeb/Yahoo! (54 leaf nodes) 500,000 Usenet articles (April-May 2000): Newsgroups assigned by hand to hierarchy nodes RIPPER trained with 54,000 articles (1,000 articles per leaf) 27,000 articles used to construct estimations of the confusion matrices Remaining 419,000 articles used to build 500 Controlled Databases of varying category mixes, size If you have several points, steps, or key ideas use multiple slides. Determine if your audience is to understand a new idea, learn a process, or receive greater depth to a familiar concept. Back up each point with adequate explanation. As appropriate, supplement your presentation with technical support data in hard copy or on disc, , or the Internet. Develop each point adequately to communicate with your audience.

19 Comparison With Alternatives
DS: Random sampling of documents via query probes Callan et al., SIGMOD’99 Different task: Gather vocabulary statistics We adapted it for database classification TQ: Title-based Probing Yu et al., WISE 2000 Query probes are simply the category names

20 Experiments: Metrics Expanded(N) Accuracy of classification results: Expanded(N) = N and all descendants Correct = Expanded(Ideal(D)) Classified = Expanded(Approximate(D)) Precision = |Correct /\ Classified|/|Classified| Recall = |Correct /\ Classified|/|Correct| F-measure = 2.Precision.Recall/(Precision + Recall) Cost of classification: Number of queries to database N

21 Average F-measure, Controlled Databases
PnC =Probe & Count, DS=Document Sampling, TQ=Title-based probing

22 Experimental Results: Controlled Databases
Feature selection helps. Confusion-matrix adjustment helps. F-measure above 0.8 for most <Tc, Ts> combinations. Results degrade gracefully with hierarchy depth. Relatively small number of probes needed for most <Tc, Ts> combinations tried. Also, probes are short: 1.5 words on average; 4 words maximum. Both better performance and lower cost than DS [Callan et al. adaptation] and TQ [Yu et al.]

23 Web Databases 130 real databases classified from InvisibleWeb™.
Used InvisibleWeb’s categorization as correct. Simple “wrappers” for querying (only # of matches needed). The Ts, Tc thresholds are not known (unlike with the Controlled databases) but implicit in the IWeb categorization. We can learn/validate the thresholds (tricky but easy!). More details in the paper!

24 Web Databases: Learning Thresholds

25 Experimental Results: Web Databases
130 Real Web Databases. F-measure above 0.7 for best <Tc, Ts> combination learned. 185 query probes per database on average needed for classification. Also, probes are short: 1.5 words on average; 4 words maximum.

26 Conclusions Accurate classification using only a small number of short queries No need for document retrieval Only need a result like: “X matches found” No need for any cooperation or special metadata from databases If you have several points, steps, or key ideas use multiple slides. Determine if your audience is to understand a new idea, learn a process, or receive greater depth to a familiar concept. Back up each point with adequate explanation. As appropriate, supplement your presentation with technical support data in hard copy or on disc, , or the Internet. Develop each point adequately to communicate with your audience.

27 Current and Future Work
Build “wrappers” automatically Extend to non-topical categories Evaluate impact of varying search interfaces (e.g., Boolean vs. ranked) Extend to other classifiers (e.g., SVMs or Bayesian models) Integrate with searching (connection with database selection?)

28 Questions?

29 Contributions Easy, inexpensive method for database classification
Uses results from document classification “Indirect” classification of the documents in a database Does not inspect documents, only number of matches Adjustment of results according to classifier’s performance Easy wrapper construction No need for any metadata from the database

30 Related Work Callan et al., SIGMOD 1999 Gauch et al., Profusion
Dolin et al., Pharos Yu et al., WISE 2000 Raghavan and Garcia Molina, VLDB 2001

31 Controlled Databases 500 databases built using 419,000 newsgroup articles One label per document 350 databases with single (not necessarily leaf) category 150 databases with varying category mixes Database size ranges from 25 to 25,000 articles Indexed and queries using SMART

32 F-measure for Different Hierarchy Depths
PnC =Probe & Count, DS=Document Sampling, TQ=Title-based probing Tc=8, Ts=0.3

33 Query Probes Per Controlled Database

34 Web Databases: Number of Query Probes

35 3-fold Cross-validation
These charts are not included in the paper and I am not quite sure whether they can be useful or not. Actually they are the F measure values for the three disjoint sets of the Web set. Their behavior is exactly the same for Varying thresholds, confirming strongly the fact that we are not overfitting the data

36 Real Confusion Matrix for Top Node of Hierarchy
Health Sports Science Computers Arts 0.753 0.018 0.124 0.021 0.017 0.006 0.171 0.016 0.064 0.024 0.255 0.047 0.004 0.042 0.080 0.610 0.031 0.027 0.298


Download ppt "Probe, Count, and Classify: Categorizing Hidden Web Databases"

Similar presentations


Ads by Google