December 16, 2003 INEX Ray R. Larson Cheshire II at INEX 2003: Component and Algorithm Fusion Ray R. Larson School of Information Management and Systems University of California, Berkeley

December 16, 2003 INEX Ray R. Larson OverviewOverview Cheshire II feature overviewCheshire II feature overview –Logistic Regression Ranking and Boolean Operations Additions from INEX ‘02Additions from INEX ‘02 –XML Schemas and Element Retrieval –CORI, Okapi BM-25 ranking algorithms –Result Set sorting, merging and ranking operators Evaluation ResultsEvaluation Results

December 16, 2003 INEX Ray R. Larson Overview of Cheshire II It supports SGML and XML with components and component indexesIt supports SGML and XML with components and component indexes It is a client/server applicationIt is a client/server application Uses the Z39.50 Information Retrieval Protocol, support for SRW, OAI, SOAP, SDLIP also implementedUses the Z39.50 Information Retrieval Protocol, support for SRW, OAI, SOAP, SDLIP also implemented Server supports a Relational Database GatewayServer supports a Relational Database Gateway Supports Boolean searching of all serversSupports Boolean searching of all servers Supports probabilistic ranked retrieval in the Cheshire search engine as well as Boolean and proximity searchSupports probabilistic ranked retrieval in the Cheshire search engine as well as Boolean and proximity search Search engine supports ``nearest neighbor'' searches and relevance feedbackSearch engine supports ``nearest neighbor'' searches and relevance feedback GUI interface on X window displays and Windows NTGUI interface on X window displays and Windows NT WWW/CGI forms interface for DL, using combined client/server CGI scripting via WebCheshireWWW/CGI forms interface for DL, using combined client/server CGI scripting via WebCheshire Scriptable clients using Tcl and (new) PythonScriptable clients using Tcl and (new) Python Store SGML/XML as files or “Datastore” databaseStore SGML/XML as files or “Datastore” database

December 16, 2003 INEX Ray R. Larson XML Element Extraction A new search “ElementSetName” is XML_ELEMENT_A new search “ElementSetName” is XML_ELEMENT_ Any Xpath, element name, or regular expression can be included following the final underscore when submitting a present requestAny Xpath, element name, or regular expression can be included following the final underscore when submitting a present request The matching elements are extracted from the records matching the search and delivered in a simple format..The matching elements are extracted from the records matching the search and delivered in a simple format..

December 16, 2003 INEX Ray R. Larson XML Extraction % zselect sherlock 372 {Connection with SHERLOCK (sherlock.berkeley.edu) database 'bibfile' at port 2100 is open as connection #372} % zfind topic mathematics {OK {Status 1} {Hits 26} {Received 0} {Set Default} {RecordSyntax UNKNOWN}} % zset recsyntax XML % zset elementset XML_ELEMENT_Fld245 % zdisplay {OK {Status 0} {Received 10} {Position 1} {Set Default} {NextPosition 11} {RecordSyntax XML }} { Singularitâes áa Cargáese … etc…

December 16, 2003 INEX Ray R. Larson Boolean Search Capability All Boolean operations are supportedAll Boolean operations are supported –“zfind author x and (title y or subject z) not subject A” Named sets are supported and stored on the serverNamed sets are supported and stored on the server Boolean operations between stored sets are supportedBoolean operations between stored sets are supported –“zfind SET1 and subject widgets or SET2” Nested parentheses and truncation are supportedNested parentheses and truncation are supported –“zfind xtitle Alice#”

December 16, 2003 INEX Ray R. Larson Probabilistic Retrieval Uses Logistic Regression ranking method developed at Berkeley (W. Cooper, F. Gey, D. Dabney, A. Chen) with new algorithm for weigh calculation at retrieval timeUses Logistic Regression ranking method developed at Berkeley (W. Cooper, F. Gey, D. Dabney, A. Chen) with new algorithm for weigh calculation at retrieval time Z39.50 “relevance” operator used to indicate probabilistic searchZ39.50 “relevance” operator used to indicate probabilistic search Any index can have Probabilistic searching performed:Any index can have Probabilistic searching performed: –zfind “cheshire cats, looking glasses, march hares and other such things” –zfind caucus races Boolean and Probabilistic elements can be combined:Boolean and Probabilistic elements can be combined: –zfind government documents and title guidebooks

December 16, 2003 INEX Ray R. Larson Probability of relevance is based on Logistic regression from a sample set of documents to determine values of the coefficients (TREC). At retrieval the probability estimate is obtained by: For the 6 X attribute measures shown on the next slide Note that we did NOT retrain the coefficients this year Probabilistic Retrieval: Logistic Regression

December 16, 2003 INEX Ray R. Larson Probabilistic Retrieval: Logistic Regression attributes Average Absolute Query Frequency Query Length Average Absolute Component Frequency Document Length Average Inverse Component Frequency Inverse Component Frequency Number of Terms in common between query and Component -- logged

December 16, 2003 INEX Ray R. Larson Combining Boolean and Probabilistic Search Elements Two original approaches:Two original approaches: –Boolean Approach –Non-probabilistic “Fusion Search” Set merger approach is a weighted merger of document scores from separate Boolean and Probabilistic queries

December 16, 2003 INEX Ray R. Larson Ranking Methods added since INEX ‘02 CORI -- From Jamie Callan: Simple implementation of a weighting scheme for distributed search. Very effective for distributed search collection selection. Not used for official INEX runs.CORI -- From Jamie Callan: Simple implementation of a weighting scheme for distributed search. Very effective for distributed search collection selection. Not used for official INEX runs. OKAPI BM From Steve Robertson. This is now seems to be the “default” retrieval algorithm in experimental IROKAPI BM From Steve Robertson. This is now seems to be the “default” retrieval algorithm in experimental IR New operators (later) let us mix and match ranking methods and Boolean operationsNew operators (later) let us mix and match ranking methods and Boolean operations

December 16, 2003 INEX Ray R. Larson Okapi BM25 Where:Where: Q is a query containing terms TQ is a query containing terms T K is k 1 ((1-b) + b.dl/avdl)K is k 1 ((1-b) + b.dl/avdl) k 1, b and k 3 are parameters, usually 1.2, 0.75 and k 1, b and k 3 are parameters, usually 1.2, 0.75 and tf is the frequency of the term in a specific documenttf is the frequency of the term in a specific document qtf is the frequency of the term in a topic from which Q was derivedqtf is the frequency of the term in a topic from which Q was derived dl and avdl are the document length and the average document length measured in some convenient unitdl and avdl are the document length and the average document length measured in some convenient unit w (1) is the Robertson-Sparck Jones weight.w (1) is the Robertson-Sparck Jones weight.

December 16, 2003 INEX Ray R. Larson INEX ‘02 Fusion Search Query Results Sort/ Merge Final Ranked List Merge multiple resultsets and sort new setMerge multiple resultsets and sort new set –Sort by index name/key (ATTRIBUTE) –Sort by rank (ELEMENTS) Merges ranked results and Boolean resultsMerges ranked results and Boolean results –Sort by XML/SGML Tag contents (TAG)

December 16, 2003 INEX Ray R. Larson Merging and Ranking Operators Extends the capabilities of merging to include merger operations in queries like Boolean operatorsExtends the capabilities of merging to include merger operations in queries like Boolean operators Fuzzy Logic Operators (not used for INEX)Fuzzy Logic Operators (not used for INEX) –!FUZZY_AND –!FUZZY_OR –!FUZZY_NOT Containment operators: Restrict components to or with a particular parentContainment operators: Restrict components to or with a particular parent –!RESTRICT_FROM –!RESTRICT_TO Merge OperatorsMerge Operators –!MERGE_SUM –!MERGE_MEAN –!MERGE_NORM

December 16, 2003 INEX Ray R. Larson Query Generation - CO # 91 TITLE = Internet traffic# 91 TITLE = Internet traffic {Internet traffic internet, web, traffic, measurement, congestion}) !MERGE_NORM {Internet traffic}) !MERGE_NORM {Internet traffic}) !MERGE_NORM {Internet traffic internet, web, traffic, measurement, congestion}) !MERGE_NORM {Internet traffic}) !MERGE_NORM {Internet {Internet traffic internet, web, traffic, measurement, congestion}) !MERGE_NORM {Internet traffic}) !MERGE_NORM {Internet traffic}) !MERGE_NORM {Internet traffic internet, web, traffic, measurement, congestion}) !MERGE_NORM {Internet traffic}) !MERGE_NORM {Internet traffic}) TARGETPATH = XML_ELEMENT_articleTARGETPATH = XML_ELEMENT_article

December 16, 2003 INEX Ray R. Larson INEX CO Runs

December 16, 2003 INEX Ray R. Larson Query Generation - SCAS #66 TITLE = /article[./fm//yr < '2000’] //sec[about(.,'"search engines"')]#66 TITLE = /article[./fm//yr < '2000’] //sec[about(.,'"search engines"')] ((date < '2000')) !RESTRICT_FROM {"search engines"} !MERGE_MEAN (sec_words {$search engines$})))((date < '2000')) !RESTRICT_FROM {"search engines"} !MERGE_MEAN (sec_words {$search engines$}))) TARGETPATH = XML_ELEMENT_secTARGETPATH = XML_ELEMENT_sec

December 16, 2003 INEX Ray R. Larson Query Generation -- SCAS This run uses Logistic regression matching combined with Boolean phrase matching and MERGE_MEAN partial result combinations FUZZY_AND and FUZZY_OR operators were used in combining AND and OR elements within an "about" predicate. Containment operators were used to constrain component searches within ancestor elements, E.g.: This run uses Logistic regression matching combined with Boolean phrase matching and MERGE_MEAN partial result combinations FUZZY_AND and FUZZY_OR operators were used in combining AND and OR elements within an "about" predicate. Containment operators were used to constrain component searches within ancestor elements, E.g.:

December 16, 2003 INEX Ray R. Larson INEX SCAS Runs

December 16, 2003 INEX Ray R. Larson Future Plans Bug fixes -- incorrect query generation for some SCAS queries, for example…Bug fixes -- incorrect query generation for some SCAS queries, for example… –TITLE = //article[about(.,'security +biometrics') AND about(.//sec,'"facial recognition"')] –Submitted : {security biometrics} !MERGE_MEAN {biometrics biometrics biometrics biometrics}) ) !FUZZY_AND {"facial recognition"} !MERGE_MEAN (sec_title {$facial recognition$})) –Should have included sec_words and Boolean subquery for biometrics merged with ranked subquery

December 16, 2003 INEX Ray R. Larson Future Plans Add Language Model ranking for componentsAdd Language Model ranking for components Retrain Logistic Regression coefficients on INEX assessment data -- and experiment with including new variables, such as relative component sizeRetrain Logistic Regression coefficients on INEX assessment data -- and experiment with including new variables, such as relative component size Find bugs in Okapi BM-25Find bugs in Okapi BM-25 Find more bugs ahead of time, and be more consistent in runs!Find more bugs ahead of time, and be more consistent in runs!