Presentation is loading. Please wait.

Presentation is loading. Please wait.

When worlds collide Metasearching meets central indexes Mike Taylor – Index Data –

Similar presentations


Presentation on theme: "When worlds collide Metasearching meets central indexes Mike Taylor – Index Data –"— Presentation transcript:

1 When worlds collide Metasearching meets central indexes Mike Taylor – Index Data –

2 Search When worlds collide : metasearching and central indexes Mike Taylor –

3 Search When worlds collide : metasearching and central indexes Mike Taylor –

4 Search When worlds collide : metasearching and central indexes Mike Taylor – Data

5 Search When worlds collide : metasearching and central indexes Mike Taylor – Data Problem solved!

6 Search When worlds collide : metasearching and central indexes Mike Taylor – Data ??

7 Metasearch When worlds collide : metasearching and central indexes Mike Taylor – Magic box Data Searching

8 Metasearch When worlds collide : metasearching and central indexes Mike Taylor – Magic box Data Searching 360 Search EHIS (EBSCO) MetaLib

9 Metasearch When worlds collide : metasearching and central indexes Mike Taylor – Magic box Data Searching 360 Search EHIS (EBSCO) MetaLib Pazpar2 (Open source)

10 Metasearch When worlds collide : metasearching and central indexes Mike Taylor – Magic box Data Searching

11 Metasearch When worlds collide : metasearching and central indexes Mike Taylor – Magic box Data A.K.A. federated search Searching

12 Metasearch When worlds collide : metasearching and central indexes Mike Taylor – Magic box Data A.K.A. federated search A.K.A. distributed search Searching

13 Metasearch When worlds collide : metasearching and central indexes Mike Taylor – Magic box Data A.K.A. federated search A.K.A. broadcast search A.K.A. distributed search Searching ?

14 Back to the sad searcher When worlds collide : metasearching and central indexes Mike Taylor – Data ??

15 Central index When worlds collide : metasearching and central indexes Mike Taylor – Data Fat database Harvesting

16 Central index When worlds collide : metasearching and central indexes Mike Taylor – Data Fat database Harvesting Summon WorldCat Primo Central

17 Central index When worlds collide : metasearching and central indexes Mike Taylor – Data Fat database Harvesting Summon WorldCat Primo Central MasterKey

18 Central index When worlds collide : metasearching and central indexes Mike Taylor – Data Fat database Harvesting A.K.A. local index

19 Central index When worlds collide : metasearching and central indexes Mike Taylor – Data Fat database Harvesting A.K.A. local index A.K.A. discovery services

20 Central index When worlds collide : metasearching and central indexes Mike Taylor – Data Fat database Harvesting A.K.A. local index A.K.A. vertical search A.K.A. discovery services ?

21 We need a controlled vocabulary! When worlds collide : metasearching and central indexes Mike Taylor – Metasearch = Federated search = Distributed search = Broadcast search Central index = Local index = Discovery services = Vertical search (if you ever heard anything so dumb)

22 Which approach is better? When worlds collide : metasearching and central indexes Mike Taylor – Central indexing compared with metasearching: - requires harvesting infrastructure - requires lots of local storage - requires co-operation from services to be harvested - does not have access to all searchable data - will always be somewhat out of date - is faster at search time (or SHOULD be) - allows data to be normalised (e.g. dates extracted) - allows for better relevance ranking - can provide pre-baked facets - may have access to some data that not searchable

23 Which approach is better? When worlds collide : metasearching and central indexes Mike Taylor –

24 Which approach is better? When worlds collide : metasearching and central indexes Mike Taylor –

25 Which approach is better? When worlds collide : metasearching and central indexes Mike Taylor –

26 Which approach is better? When worlds collide : metasearching and central indexes Mike Taylor – Let's do both!

27 When worlds collide : metasearching and central indexes Mike Taylor – Magic box Data Searching Data Fat database Harvesting ! Integrated Search

28 When worlds collide : metasearching and central indexes Mike Taylor – Magic box Data Searching Data Fat database Harvesting ! Integrated Search

29 When worlds collide : metasearching and central indexes Mike Taylor – Magic box Data Searching Data Fat database Harvesting ! Integrated Search

30 When worlds collide : metasearching and central indexes Mike Taylor – Magic box Data Searching Data Fat database Harvesting ! Integrated Search

31 Metasearch hides the complexity When worlds collide : metasearching and central indexes Mike Taylor – Magic box Data Searching

32 Metasearch Nine tenths under The surface When worlds collide : metasearching and central indexes Mike Taylor – Magic box Data Searching

33 Metasearch What you see looks beautiful When worlds collide : metasearching and central indexes Mike Taylor – Magic box Data Searching

34 Problems that need solving When worlds collide : metasearching and central indexes Mike Taylor – A. Problems with pure metasearching B. How those problems change when you add a central index

35 Problems with metasearching When worlds collide : metasearching and central indexes Mike Taylor – Examples based on Index Data's suite: Pazpar2 is a free metasearching engine with a stupid name MasterKey is a non-open suite that wraps it MasterKey is only one way to use Pazpar2 Also integrated into other vendors' UIs.

36 Problems with metasearching #1: No data server at all! When worlds collide : metasearching and central indexes Mike Taylor – Data is often only in a user-facing Web UI Must be made available via a standard protocol

37 Problems with metasearching #1: No data server at all! When worlds collide : metasearching and central indexes Mike Taylor – Data is often only in a user-facing Web UI Must be made available via a standard protocol Option 1: build a gateway in Perl

38 Problems with metasearching #1: No data server at all! When worlds collide : metasearching and central indexes Mike Taylor – Data is often only in a user-facing Web UI Must be made available via a standard protocol Option 1: build a gateway in Perl Option 2: MasterKey Connect (non-open)

39 Problems with metasearching #2: data server is crap^H^H^H^Hsuboptimal When worlds collide : metasearching and central indexes Mike Taylor – Catalogs searchable using ANSI/NISO Z39.50 Support is very nominal in some cases

40 Problems with metasearching #2: data server is crap^H^H^H^Hsuboptimal When worlds collide : metasearching and central indexes Mike Taylor – Catalogs searchable using ANSI/NISO Z39.50 Support is very nominal in some cases IRSpy probes behaviour MasterKey target profiles describe behaviour

41 Problems with metasearching #3: Data servers don't support relevance When worlds collide : metasearching and central indexes Mike Taylor –

42 Problems with metasearching #3: Data servers don't support relevance When worlds collide : metasearching and central indexes Mike Taylor – Pazpar2 does its own relevance ranking (Part of merging/deduplication)

43 Problems with metasearching #4: Data servers don't return facets When worlds collide : metasearching and central indexes Mike Taylor –

44 Problems with metasearching #4: Data servers don't return facets When worlds collide : metasearching and central indexes Mike Taylor – Pazpar2 calculates its own facets

45 There is a lot of magic in the magic box Searching Sorting Merging Deduplication Relevance Facet generation Time travel... When worlds collide : metasearching and central indexes Mike Taylor – Magic box Data

46 There is a lot of magic in the magic box Searching Sorting Merging Deduplication Relevance Facet generation Time travel... When worlds collide : metasearching and central indexes Mike Taylor – Pazpar2 Data Remember, our engine is free:

47 When worlds collide : metasearching and central indexes Mike Taylor – Magic box Data Searching Data Fat database Harvesting ! What happens when we add a central index?

48 Problems with integrated search #1: No data server at all! When worlds collide : metasearching and central indexes Mike Taylor – Data is often only in a user-facing Web UI

49 Problems with integrated search #1: No data server at all! When worlds collide : metasearching and central indexes Mike Taylor – Data is often only in a user-facing Web UI

50 Problems with integrated search #1: No data server at all! When worlds collide : metasearching and central indexes Mike Taylor – Data is often only in a user-facing Web UI You can't harvest Google

51 Problems with integrated search #1: No data server at all! When worlds collide : metasearching and central indexes Mike Taylor – Data is often only in a user-facing Web UI You can't harvest Google You just can't

52 Problems with integrated search #2: data server is crap^H^H^H^Hsuboptimal When worlds collide : metasearching and central indexes Mike Taylor – Repositories harvestable using OAI-PMH (an even worse name than pazpar2) Support is very nominal in some cases

53 Problems with integrated search #2: data server is crap^H^H^H^Hsuboptimal When worlds collide : metasearching and central indexes Mike Taylor – Repositories harvestable using OAI-PMH (an even worse name than pazpar2) Support is very nominal in some cases OAI-PMH client must be very tolerant Extensive data-cleaning is usually required

54 Problems with integrated search #3: Central index does support relevance When worlds collide : metasearching and central indexes Mike Taylor – Returned records carry relevance scores Must be merged with records scored by engine Requires score normalisation into same range Existing ordering may be used in merge

55 Problems with integrated search #3: Central index does support relevance When worlds collide : metasearching and central indexes Mike Taylor – Unranked #1 Ranked #1 Ranked #2 Solr Sort Merged Unranked #2 Sort

56 Problems with integrated search #4: Central index does return facets When worlds collide : metasearching and central indexes Mike Taylor – Lists of field values with occurrence counts: Author Kernighan 27 Pike 13 Ritchie 7 Thompson 4 Title C 7 Unix 35 Programming 16 Date

57 Problems with integrated search #4: Central index does return facets When worlds collide : metasearching and central indexes Mike Taylor – Lists are returned or calculated for each server: Server 1 (central index) (all facets from 2000 hits) Cat 68 Dinosaur 162 Fish 145 Frog 19 Server 2 (metasearch) (1000 hits, 100 records) Cat 7 Dog 10 Dinosaur 87 Fish 23

58 Problems with integrated search #4: Central index does return facets When worlds collide : metasearching and central indexes Mike Taylor – Metasearched counts normalised by total hit-count Server 1 (central index) (all facets from 2000 hits) Cat 68 Dinosaur 162 Fish 145 Frog 19 Server 2 (metasearch) (normalised to 1000 hits) Cat 70 Dog 100 Dinosaur 870 Fish 230

59 Problems with integrated search #4: Central index does return facets When worlds collide : metasearching and central indexes Mike Taylor – Facet lists are merged Servers 1+2 (integrated) (as though for all records in result sets) Cat = 138 Dog = 100 Dinosaur = 1032 Fish = 375 Frog 19+0 = 19

60 Problems with integrated search #4: Central index does return facets When worlds collide : metasearching and central indexes Mike Taylor – Fringe benefit: facet-count normalisation is also useful when doing pure metasearching. Servers 1+2 (as though for all records in result sets) Cat = 138 Dog = 100 Dinosaur = 1032 Fish = 375 Frog 19+0 = 19

61 Summary of search issues When worlds collide : metasearching and central indexes Mike Taylor – Issue Metasearch solution Central index solution No data server Build gateways MasterKey Connect --- Bad data server Probe capabilities Profile targets Tolerant harvester Data-cleaning Relevance scores Magic engine Normalise scores Ingest from server Facets Magic engine Normalise counts Ingest from server

62 When worlds collide : metasearching and central indexes Mike Taylor – Magic box Data Searching Data Fat database Harvesting

63 When worlds collide Metasearching meets central indexes Mike Taylor – Index Data –


Download ppt "When worlds collide Metasearching meets central indexes Mike Taylor – Index Data –"

Similar presentations


Ads by Google