Focused Crawling for both Topical Relevance and Quality of Medical Information By Tim Tang, David Hawking, Nick Craswell, Kathy Griffiths CIKM ’05 November, 2005

2 Outlines  Problems and Motivation  The experiment –Focused crawling –Relevance and quality prediction –The three crawlers –Measures for relevance and quality –Results, findings  Future work

3 Why Health Information on the Web?  The Internet is a free medium  Health information of various quality  Incorrect health advice may be dangerous  High user demand for Web health information

4 Problems  Relevance (in IR): –Topical relevance based on text –Navigational and distillation relevance based on links  None of these techniques guarantee quality  Our previous study ( Tang et al., JIR ‘05 ) showed Google returns a lot of low-quality health results -> PageRank does not guarantee quality

5 Problems: Quality of Health Info  Quality of health information is often measured by evidence-based medicine which are Interventions supported by a systematic review of the evidence as effective.  Low quality health information originate from untrusted sites: personal home pages, commercial sites, chat sites, web forums, and even some published materials,…

6 Wrong Advice from an Article

7 Dangerous Information from Personal Web Pages

8 Commercial Promotion

9 Why Domain-specific Search?  Impose domain restriction  Results from previous work ( Tang et. al, JIR ‘05 )  Quality: Domain-specific engines performed much better than Google  Relevance: GoogleD was best  Coverage analysis: BPS & 4sites are poor EngineRelevanceQuality GoogleD BPS sites Google

10 The Problems of Domain-specific Engines  The current method to build domain-specific engines is very expensive: manual, rule-based.  Example: BluePages Search – A depression portal at the ANU ( ) –Manual judgments of health sites by domain experts for two weeks to decide what to include in the index. –Low coverage: only 207 Web sites in the index –Tedious maintenance process: Web pages change, cease to exist, new pages come out, etc. -> A quality focused crawler may be a cheaper approach, maintaining high quality while improving coverage

11 The FC Process  Designed to selectively fetch content relevant to a specified topic of interest using the Web’s hyperlink structure. URL Frontier Link extractorDownload Classifier {URLs, link info} dequeue {URLs, scores} enqueue Link info = anchor text, URL, source page’s content, so on.

12 Relevance Prediction  anchor text: text appearing in a hyperlink  text around the link: 50 bytes before and after the link  URL words: Words formed by parsing the URL address

13 Relevance Indicators  URL: herapy.html => URL words: depression, com, psychotherapy  Anchor text: psychotherapy  Text around the link: –50 bytes before: section, learn –50 bytes after: talk, therapy, standard, treatment

14 Methods  Machine learning approach: Train and test relevant and irrelevant Web pages using the discussed indicators.  Evaluated different learning algorithms: k-nearest neighbor, Naïve Bayes, C4.5, Perceptron.  Result: The C4.5 decision tree was the best to predict relevance.  A Laplace correction formula ( Margineantu et al., LNS, ‘02 ) was used to produce a confidence score ( confidence_level ) at each leaf node of the tree.  The same method applied to predict quality but not successful!!! -> Link anchor context cannot predict quality

15 Quality Prediction  Using evidence-based medicine, and  Using Relevance Feedback (RF) technique

16 Evidence-based Medicine  Evidence-based treatments were divided into single and 2-word terms.  Example: –Cognitive behavioral therapy -> cognitive, behavioral, therapy, cognitive behavioral, behavioral therapy

17 Relevance Feedback  Well-known IR approach of query by examples.  Basic idea: Do an initial query, get feedback from users about what documents are relevant, then add words from relevant document to the query.  Goal: Add terms to the query in order to get more relevant results.  Usually, 20 terms are added into the query in total

18 Our RF Approach  Not for relevance, but Quality  Not only single terms, but also phrases  Generate a list of single terms and 2-word phrases and their associated weights  Select the top weighted terms and phrases  Cut-off points at the lowest-ranked term that appears in the evidence-based treatment list  20 phrases and 29 single words form a ‘quality query’

19 Predicting Quality  For downloaded pages, quality score (QScore) is computed using a modification of the BM25 formula, taking into account term weights.  Quality of a new page is then predicted based on the quality of all the downloaded pages linking to it. (Assumption: There is quality locality, pages with similar content are inter-connected (Davison, SIGIR ‘00))  Predicted quality score of a page with n downloaded source pages: PScore = Σ QScore/n … Downloaded sources P1 P2 Pn Target

20 Combining Relevance and Quality  We need to balance between relevance and quality  Quality and relevance score combination is new  Our method uses a product of the two scores: URLScore = confidence_level * PScore  Other ways to combine these scores will be explored in future work  A quality focused crawler will rely on this combined score to order its crawl queue

21 The Three Crawlers  The Breadth-first crawler: Traverses the link graph in a FIFO fashion (serves as baseline for comparison)  The Relevance crawler: For topical relevance, ordering the crawl queue using the C4.5 decision tree  The Quality crawler: For both relevance and quality, ordering the crawl queue using the combination of the C4.5 decision tree and RF techniques.

22 Measures  Relevance: The relevance performance of the three crawlers were evaluated using a relevance classifier.  Quality: were judged by domain experts using the evidence-based guidelines from the Centre for Evidence Based Mental Health (CEBMH). –Overall quality: taking into account all pages –High and low quality categories: the top 25%, and the bottom 25% results in each crawl were compared.

23 Results

24 Relevance

25 Quality

26 High Quality Pages AAQ = Above Average Quality: top 25%

27 Low Quality Pages BAQ = Below Average Quality: bottom 25%

28 Findings  Topical-relevance can be predicted using link anchor context.  Relevance feedback technique proved its usefulness in quality prediction.  Domain-specific search portals can be successfully built using focused crawling techniques.

29 Future Work  We only experimented in one health topic. Our plan is to repeat the same experiments with another topic, and generalise the technique to another domain.  Other ways of combining relevance and quality should be explored.  Experiments to compare our quality crawl with other health portals is necessary.