Presentation is loading. Please wait.

Presentation is loading. Please wait.

Topic-Sensitive SourceRank: Extending SourceRank for Performing Context-Sensitive Search over Deep-Web MS Thesis Defense Manishkumar Jha Committee Members.

Similar presentations


Presentation on theme: "Topic-Sensitive SourceRank: Extending SourceRank for Performing Context-Sensitive Search over Deep-Web MS Thesis Defense Manishkumar Jha Committee Members."— Presentation transcript:

1 Topic-Sensitive SourceRank: Extending SourceRank for Performing Context-Sensitive Search over Deep-Web MS Thesis Defense Manishkumar Jha Committee Members Dr. Subbarao Kambhampati Dr. Huan Liu Dr. Hasan Davulcu

2 2 Deep Web Integration Scenario Web DB Mediator ← query Web DB answer tuples → ← answer tuples ← query query → Deep Web Web DB Millions of sources containing structured tuples Autonomous Uncontrolled collection Access is limited to query-forms Contains information spanning multiple topics

3 Source quality and SourceRank Source quality Deep-Web is adversarial Source quality is a major issue over deep-web SourceRank SourceRank [1] provides a measure for assessing source quality based on source trustworthiness and result importance 3 [1] SourceRank:Relevance and Trust Assessment for Deep Web Sources Based on Inter-Source Agreement, WWW, 2011

4 … But Source quality is topic-sensitive Sources might have data corresponding to multiple topics. Importance may vary across topics – Example: Barnes & Noble might be quite good as a book source but not be as good a movie source SourceRank will fail to capture this fact Issues were noted for surface-web. But are much more critical for deep-web as sources are even more likely to cross topics 4 bookbook moviemovie

5 5 Deep Web Integration Scenario Web DB Mediator ← query Web DB answer tuples → ← answer tuples ← query query → Deep Web Web DB ` Movie Books Music Camera

6 Problem Definition Problem Definition: Performing effective multi-topic source selection sensitive to trustworthiness for deep-web 6

7 Our solution – Topic sensitive- SourceRank Compute multiple topic-sensitive SourceRanks At query-time, using query-topic combine these rankings into composite importance ranking Challenges – Computing topic-sensitive SourceRanks – Identifying query-topic – Combining topic-sensitive SourceRanks 7

8 Agenda SourceRank Topic-sensitive SourceRank Experiments and Results Conclusion 8

9 Agenda SourceRank Topic-sensitive SourceRank Experiments and Results Conclusion 9

10 SourceRank Computation Assesses source quality based on trustworthiness and result importance Introduces a domain-agnostic agreement based technique for implicitly creating an endorsement structure between deep-web sources Agreement of answer sets returned in response to same queries manifests as a form of implicit endorsement 10

11 SourceRank Computation contd. Endorsement is modeled as directed weighted agreement graph Nodes represent sources Edge weights represent agreement between the sources SourceRank of a source is computed as the stationary visit probability of a Markov random walk performed on this agreement graph 11

12 Agenda SourceRank Topic-sensitive SourceRank Experiments and Results Conclusion 12

13 Trust-based measure for multi-topic deep-web Issues with SourceRank for multi-topic deep-web – Single importance ranking – Is query-agnostic We propose Topic-sensitive SourceRank, TSR for effectively performing multi-topic selection sensitive to trustworthiness TSR overcomes the drawbacks of SourceRank 13

14 Topic-sensitive SourceRank Overview Instead of creating a single importance ranking, multiple importance rankings are created – Each importance ranking is biased towards a particular topic At query-time, using query information and individual topic-specific importance rankings, compute a composite importance ranking biased towards the query 14

15 15

16 Challenges for TSR Computing topic-specific importance rankings is not trivial Inferring query information – Identifying query-topic – Computing composite importance ranking 16

17 Computing topic-specific SourceRank For a deep-web source, its SourceRank score for a topic, will depend on the answers to queries of same topic Using topic-specific sampling queries for a topic, will result in an endorsement structure, biased towards the same topic – Example: If movie-related sampling queries are used, then movie sources are more likely to agree on the answer sets than other topic sources. This will result in the endorsement structure biased towards the movie-topic 17

18 Computing topic-specific SourceRank contd. SourceRank computed on biased agreement graph for a topic will capture topic-specific source importance ranking for the same topic 18

19 Topic-specific sampling queries Publicly available online directories such as ODP, Yahoo Directory provide hand-constructed topic hierarchies These directories along with the links posted under each topic are a good source for obtaining topic- specific sampling queries 19

20 Computing Topic-specific SourceRanks Partial topic-specific sampling queries are used for obtaining source crawls Topic-specific source crawls are used for computing biased agreement graphs Topic-specific SourceRanks, TSR’s are obtained by performing a weighted random walk on the biased agreement graphs 20

21 Query Processing Query processing involves – Computing query-topic – Computing query-topic sensitive importance scores – Source selection 21

22 Computing query-topic Query-topic – Likelihood of the query belonging to topics – Soft classification problem: For user query q and a set of topics c i  C, goal is to find fractional topic membership of q with each topic c i 22 CameraBookMovieMusic query-topic For Query=“godfather” topic

23 Computing query-topic – Training Data Training data – Description of topics – Use complete topic-specific sampling queries to obtain topic-specific source crawls – Topic descriptions are treated as bag of words 23

24 Computing query-topic – Classifier Classifier – Naïve Bayes Classifier (NBC) is used with parameters set to maximum likelihood estimates – For a user query q, NBC uses topic-description to estimate topic probability conditioned on q i.e. for topic c i, NBC uses topic-description for c i to estimate P(c i |q) 24

25 Computing query-topic – Classifier contd. Computing P(c i |q) where q j is the j th term of query q P(c i ) can be set based on domain knowledge, but for our computations, we use uniform probabilities for topics 25

26 Computing query-topic sensitive importance scores Query-topic sensitive importance scores – Topic-specific SourceRanks are linearly combined using query-topic as weights – Query-topic sensitive or composite SourceRank score for source s k is computed as where TSR ki is the topic-specific SourceRank score of source s k for topic c i 26

27 Source selection Linearly combines relevance-scores with importance scores Overall score of a source s k is computed as where R k : relevancy score of s k CSR k : query-topic sensitive score of s k 27

28 28

29 Agenda SourceRank Topic-sensitive SourceRank Experiments and Results Conclusion 29

30 Experimental setup Experiments were conducted on a multi-topic deep- web environment consisting of four-representative topics – camera, book, movie and music Source DataSet – Sources were collected via Google Base – Google Base was probed with 40 queries containing a mix of camera names, book, movie and music album titles – Total of 1440 sources were collected: 276 camera, 556 book, 572 movie and 281 music sources 30

31 Sampling queries Generated using publicly available online listings Used 200 titles or names in each topic Randomly selected cameras from pbase.com, book from New York Times best sellers, movies from ODP and music albums from Wikipedia’s top-100,

32 Test queries Contained a mix of queries from all four topics Do not overlap with the sampling queries Generated by randomly removing words from camera names, book, movie and music album titles with 0.5 probability Number of test queries varied for different topics to obtain the required (0.95) statistical significance 32

33 Query similarity based measure- CORI CORI – Source statistics were collected using highest document frequency terms – Sources were selected using the same parameters as found optimal in CORI paper 33

34 Query similarity based measure- Google Base Google Base – Two-versions of Google Base were used – Gbase on dataset: Google Base search is restricted to our crawled sources – Gbase: Google Base search with no restrictions i.e. considers all sources in Google Base 34

35 Agreement based measures - USR Undifferentiated SourceRank, USR – SourceRank extended to multi-topic deep-web – Single agreement graph is computed using entire set of sampling queries – USR of sources is computed based on a random walk on this graph 35

36 Agreement based measures - DSR Oracular source selection, DSR – Assumes a perfect classification of sources and user queries are available i.e. each source and test query is manually labeled with its domain association – Creates agreement graphs and SourceRanks for a domain including only sources in that domain – For each test query, sources ranking high in the domain corresponding to the test query are used 36

37 Result merging, ranking and relevance evaluation Top-k sources are selected Google Base is made to query only on these to top-k sources Experimented with different values of k and found k=10 to be optimal Google Base’s tuple ranking was used for ranking resulting tuples and return top-5 results in response to test queries 37

38 Result merging, ranking and relevance evaluation contd. Top-5 results returned were manually classified as relevant or irrelevant Result classification was rule based – Example- if the test query is “pirates caribbean chest” and original movie name is “Pirates of Caribbean and Dead Man’s Chest”, then if the result entity refers to the same movie (dvd, blue-ray etc.) then the result is classified as relevant and otherwise irrelevant To avoid author bias, results from different source selection methods were merged in a single file so that the evaluator does not know which method each result came from while he does the classification 38

39 Results TSR was compared with the baseline source selection methods Agreement based measures (TSR, USR and DSR) were combined with query-similarity based CORI measure. The combination is represented by agreement based measure name and the weight assigned to agreement based measure, 1-  – Example: TSR(0.1) represents 0.9xCORI + 0.1xTSR We experimented with different values of  and found that  =0.9 gives best precision for TSR-based source selection i.e. TSR(0.1) Higher weightage of CORI compared to TSR is to compensate the fact that TSR scores have higher dispersion compared to CORI scores 39

40 40 Comparison of top-5 precision of TSR(0.1) and query similarity based methods: CORI and Google Base TSR precision exceeds that of similarity-based measures by 85%

41 41 Comparison of topic-wise top-5 precision of TSR(0.1) and query similarity based methods: CORI and Google Base TSR significantly out-performs all query-similarity based measures for all topics

42 42 Comparison of top-5 precision of TSR(0.1) and agreement based methods: USR(0.1) and USR(1.0) TSR precision exceeds USR(0.1) by 18% and USR(1.0) by 40%

43 43 Comparison of topic-wise top-5 precision of TSR(0.1) and agreement based methods: USR(0.1) and USR(1.0) For three out of the four topics, TSR(0.1) out-performs USR(0.1) and USR(1.0) with confidence levels 0.95 or more

44 44 Comparison of top-5 precision of TSR(0.1) and oracular DSR(0.1) TSR(0.1) is able to match DSR(0.1)’s performance

45 45 Comparison of topic-wise top-5 precision of TSR(0.1) and oracular DSR(0.1) TSR(0.1) matches DSR(0.1) performance across all topics indicating its effectiveness in identifying important sources across all topics

46 Agenda SourceRank Topic-sensitive SourceRank Experiments and Results Conclusion 46

47 Conclusion We attempted multi-topic source selection sensitive to trustworthiness and importance for the deep-web We introduced topic-sensitive SourceRank (TSR) Our experiments on more than a thousand deep- web sources show that a TSR-based approach is highly effective in extending SourceRank to multi- topic deep-web 47

48 Conclusion contd. TSR out-performs query-similarity based measures by around 85% in precision TSR results in statistically significant precision improvements over other baseline agreement- based methods Comparison with oracular DSR approach reveals effectiveness of TSR for topic-specific query and source classification and subsequent source selection 48

49 Paper submitted to Comad’11 49

50 Questions? 50


Download ppt "Topic-Sensitive SourceRank: Extending SourceRank for Performing Context-Sensitive Search over Deep-Web MS Thesis Defense Manishkumar Jha Committee Members."

Similar presentations


Ads by Google