1. Martin Theobald Max-Planck-Institut Informatik Stanford University
TopX
Efficient & Versatile
Top-k Query Processing for Text,
Semistructured & Structured Data
Martin Theobald
Max-Planck-Institut Informatik
Stanford University

2. RANKING PRUNING VAGUENESS //article[.//bib[about(.//item, “W3C”)]
]//sec[about(.//, “XML retrieval”)]
//par[about(.//, “native XML databases”)]
RANKING
“Native XML data base systems can store schemaless
data ... ”
“Data management systems control data acquisition, storage, and retrieval. Systems evolved from flat files … ”
“XML-QL:
A Query
Language
for XML.”
“Native XML
Data Bases.”
“Proc. Query
Languages
Workshop,
W3C,1998.”
“XML queries
with an expres-
sive power
similar to that
of Datalog …”
sec
article
par
bib
title
“Current
Approaches
to XML Data Manage-
ment”
item
inproc
“What does XML
add for retrieval?
It adds formal ways …”
“w3c.org/xml”
sec
article
par
“Sophisticated
technologies
developed by
smart people.”
title
“The
XML Files”
Ontology
Game”
Dirty Little
Secret”
bib
“There, I've said
it - the "O" word. If anyone is thinking along ontology lines, I would like to break some old news …”
item
url
“XML”
VAGUENESS
PRUNING

3. Unified Text & XML Schema
Frontends
Web Interface
Web Service
API
TopX
Query Processor
Probabilistic
Index Access
Scheduling
Candidate
Queue
Candidate
Cache SA
Scan Threads
Top-k
Queue
Probabilistic
Candidate
Pruning
Query Processing Time
Random Access
Dynamic
Query
Expansion
Sequential Access
Incremental
XPath Engine
Auxiliary
Predicates RA
Thesaurus
WordNet,
OpenCyc, etc.
Index Metadata
Selectivities
Histograms
Correlations
DBMS / Inverted Lists
Unified Text & XML Schema
Indexing Time RA
Indexer
/Crawler

4. “xml manage system vary wide expressive
Data Model
“xml data manage xml manage system vary wide expressive power native xml native xml data base system store
schemaless data“
“xml data manage xml manage system vary wide expressive power native xml data base native xml data base system
store schemaless data“
“xml data manage”
article
title
abs
sec
“xml manage system vary wide expressive
power“
“native xml data base”
“native xml data base system store schemaless data“
par 1 6 2 3 4 5
<article>
<title>XML Data Management
</title>
<abs>XML management systems vary
widely in their expressive power.
</abs>
<sec>
<title>Native XML Data Bases.
<par>Native XML data base systems
can store schemaless data.
</par>
</sec>
</article>
ftf (“xml”,
article1 ) = 4
“native xml data base
native xml data base
system store
schemaless data“
XML trees (no XLinks or ID/IDref attributes)
Pre-/postorder node labels
Redundant full-content text nodes

5. Scoring Model [INEX ’06/’07]
XML-specific extension to Okapi BM25
(originating from probabilistic text IR)
ftf instead of tf
ef instead of df
Element type-specific
length normalization
Tunable parameters k1 and b
bib[“transactions”]
vs.
par[“transactions”]

6. TopX Query Processing [VLDB ’05]
//sec[about(.//, “XML”) and
about(.//title, “native”]
//par[about(.//, “retrieval”)]
sec[“xml”]
title[“native”]
par[“retrieval”]
1.0
1.0
1.0
0.9
eid
docid
score
pre
post 46 2
0.9 15 9
0.5 10 8
171 5
0.85 1 20 84 3
0.1 12
0.9
eid
docid
score
pre
post
216 17
0.9 2 15 72 3
0.8 14 10 51
0.5 4 12
1.0
eid
docid
score
pre
post 3 1
1.0 21 28 2
0.8 8 14
182 5
0.75 7
0.8
0.8
0.85
0.5
0.75
0.1 … 16 11
0.4 5 89 … 14 8
0.8 19 … 20 3
0.04 8 21 5
0.05 4 35 1
0.09 32
Top-2
Candidate
Queue
worst=1.6
171
182
worst=1.0 3
worst=0.9 46
worst=2.2 46 28 51
worst=1.7 46 28
worst=0.5 9
worst=0.9
216
max-q=2.75
max-q=2.15
max-q=2.55
max-q=2.45
max-q=2.8
max-q=1.6
max-q=3.0
max-q=2.9
max-q=2.7
min-2=1.0
min-2=0.5
min-2=0.0
min-2=0.9
min-2=1.6

7. Index Access Scheduling [VLDB ’06]
Inverted
Block Index
SA Scheduling
Look-ahead Δi through precomputed score histograms
Knapsack-based optimization of Score Reduction
RA Scheduling
2-phase probing:
Schedule RAs “late & last”
Extended probabilistic cost model for integrating SA & RA scheduling SA SA SA
1.0
0.9
0.8
0.2
0.7
0.6 …
Δ1,3 = 0.8
Δ3,3 = 0.2 RA

8. Probabilistic Pruning [VLDB ’04]
Convolutions of score distributions (assuming independence)
P [d gets in the final top-k] =
title[“native”] f1 1
high1 f2
high2
eid …
max-
score
216
0.9 72
0.8 51
0.5 2
δ(d)
par[“retrieval”]
sampling
eid …
max-
score 3
1.0 28
0.8
182
0.75
Probabilistic candidate pruning:
Drop d from the candidate queue if
P [d gets in the final top-k] < ε
With probabilistic guarantees for precision & recall
Indexing Time
Query Processing Time

9. Dynamic Query Expansion [SIGIR ’05]
TREC Robust
Topic #363
Top-k
(transport, tunnel,
~disaster)
Incrementally merge
inverted lists for expansion ti,1...ti,m
in descending order of s(tij, d)
Best-match score aggregation
Specialized expansion operators
Incremental Merge operator
Nested Top-k operator
(efficient phrase matching)
Boolean (but ranked) retrieval mode
Supports any sorted inverted index for text, structured records & XML SA SA
transport
d66
d93
d95
...
d101
tunnel
d17
d11
d99
d42
d11
d92
d37 …
~disaster SA
d42
d11
d92
...
d21
d78
d10 d1
d37
d32
d87
disaster
accident
fire
Incr. Merge

10. Incremental Merge Operator
Thesaurus lookups/
Relevance feedback
Index list metadata
(e.g., histograms)
Initial high-scores
Expansion terms
~t = { t1, t2, t3 }
Large corpus
term correlations
sim(t, t1 ) = 1.0 t1
...
d78
0.9 d1
0.4
d88
0.3
d23
0.8
d10
0.9
0.4
...
d12
0.2
d78
0.1
d64
0.8
d23
d10
0.7 t2
sim(t, t2 ) = 0.9
Expansion similarities
0.72
0.18
sim(t, t3 ) = 0.5 t3
...
d99
0.7
d34
0.6
d11
0.9
d78
d64 SA
0.45
0.35 ~t
d78
0.9
d23
0.8
d10
0.8
d64
0.72
d23
0.72
d10
0.63
d11
0.45
d78
0.45 d1
0.4
d88
0.3
...
Meta histograms
seamlessly integrate Incremental Merge
into probabilistic scheduling and candidate pruning

11. Some Experiments New XML-ified Wikipedia corpus (INEX 2006)
660,000 documents w/ 130,000,000 elements
125 INEX queries, each as content-only (CO)
and content-and-structure (CAS) formulation
CO: +“state machine” figure Mealy Moore
CAS: //article[about(., “state machine” )]
//figure[about(., Mealy ) or about(., Moore )]
Primary cost metric: Cost = #SA + cR/cS #RA

12. TopX vs. Full-Merge Significant cost savings for large ranges of k
CAS cheaper than CO !

13. Efficiency vs. Effectiveness
Very good precision/runtime ratio for
probabilistic pruning

14. Static vs. Dynamic Expansions
Query expansions with up to m=292 keywords & phrases
Balanced amount of sorted vs. random disk access
Adaptive scheduling wrt.
cR/cS cost ratio
Dynamic expansions superior to static expansions & full-merge in both efficiency & effectiveness

15. Thanks… Gerhard Weikum Ralf Schenkel
Norbert Fuhr, Michalis Vazirgiannis
Holger Bast, Debapriyo Majumdar
All the MPI & INEX folks

16. topx.sourceforge.net
See our Sigmod’07 demo!