1. UPI: A Primary Index for Uncertain Databases (VLDB 10)
Hideaki Kimura (BrownU)
Samuel Madden (MIT)
Stanley B. Zdonik (BrownU)
Speaker: Yinuo Zhang
Supervisor: Dr. Reynold Cheng

2. Outline Introduction Uncertain Primary Index (UPI)
Secondary Index on UPI
Experiments
Conclusion and Future Work

3. Introduction Table Author A Possible World Name Institutionp Existence
Alice
Brown: 80%, MIT: 20%
90%
Bob
MIT: 95%, UCB: 5%
100%
Carol
Brown: 60%, U. Tokyo: 40%
80%
A Possible World
Name
Institution
Alice
Brown
Bob
MIT
SELECT * FROM Author WHERE Institution=MIT
Threshold: confidence QT
Query answering over Possible World Semantics
The probability of such world: 90%*80% * 100%*95% * 20% = 13.7%

4. Ex) DBLP with Uncertain Affiliation
DBLP: 1.3M Papers and 0.7M Authors
Complemented Author Affiliation
q=“David DeWitt”
Rank
URL 1
Wisc.edu/… 2
Microsoft.com/… 3
Columbia.edu/…
Name
Inst.
David DeWitt ?
Google
API
Zipfian Distribution
Name
Institutionp
Countryp
David DeWitt
Wisconsin: 40%, MS: 20%,
Columbia: 13%, …
US: 100%

5. Introduction
Achieving an efficient implementation using possible world semantics is difficult.
Probabilistic Inverted Index [Singh07] – a secondary index
Heap
Institution
Pointer
Brown
[Alice] 0.3
[Carol] 0.2
[Bob] 0.4
MIT
[Bob] 0.3
[Alice] 0.8
Disk
Seeking

6. Over Uncertain Attributes
Introduction
Goal
Build A Primary Index
Over Uncertain Attributes
Primary Index
Seq.
Read

7. Challenges on Building PI over Uncertain Data
Name
Institutionp
Existence
Alice
Brown: 80%, MIT: 20%
90%
Bob
MIT: 95%, UCB: 5%
100%
Carol
Brown: 60%, U. Tokyo: 40%
80%
Cluster on most probable
possible value?
Replicate tuples
into inverted index?
Cluster
Tuples
Brown
Alice, Carol
MIT
Bob
Cluster
Tuples
Brown
Alice, Carol, …
MIT
Alice, Bob, …
Too Large for
Long-tail distribution
(e.g., 100 values with 0.1%)
Alice?
SELECT …WHERE Inst.=MIT

8. UPI: Heap + Cutoff Index
Name
Institutionp
Existence
Alice
Brown: 80%, MIT: 20%
90%
Bob
MIT: 95%, UCB: 5%
100%
Carol
Brown: 60%, U. Tokyo: 40%
80%
Heap: Sorted by (Inst., Prob)
Cutoff Index:
Sorted by (Inst., Prob)
Institution
Tuple
Brown (72%)
Alice
Brown (48%)
Carol
MIT (95%)
Bob
MIT (18%)
UCB (5%)
U. Tokyo (32%)
Institution
TupleID
Pointer
UCB (5%)
Bob
MIT
Cutoff Entries with
Less than C probability
(Cutoff Threshold)

9. Answering Queries with UPI
Probabilistic Threshold Query (PTQ)
SELECT * FROM Author WHERE Inst.=UCB With: Probability ≥ QT (Query Threshold)
Institution
Tuple
MIT (95%)
Bob …
UCB (90%)
Dan
UCB (20%)
Emily
C=10%
Seek
Institution
TupleID
Pointer
UCB (5%)
Bob
MIT
If QT<C (e.g., QT=5%),
follow Cutoff pointers
If QT≥C (e.g., QT=20%),
Sequentially Read

10. Choosing Cutoff Threshold
Faster, but
Larger
Slower, but
Smaller
Cutoff Threshold C
SELECT * FROM Author WHERE Institution=Ishikawa U
Threshold: confidence QT (QT is given at runtime)

11. Determining C Based on Value/Probability Histograms Histograms (Inst.)
#Keys …
Br*
30,000
Bs*
31,000
Bt*
30,500
Prob.
#Keys …
10%-15%
15,000
15%-25%
28,000
25%-40%
33,000
Histograms (Inst.)
Tolerable average query runtime
Available Disk Capability -> UPI Size
= Costfullscan * Selectivity+ Costseek * # Pointers ?
#Pointers C C

12. #Pointers and Query Cost
(replace Ishikawa U with Stanford)
Saturation
Cost
Model
Logistic function

13. Secondary Index on UPI SELECT * FROM Author WHERE Country=US
Name
Institutionp
Countryp
Existence
Alice
Brown: 80%, MIT: 20%
US: 100%
90%
Bob
MIT: 95%, UCB: 5%
100%
Carol
Brown: 60%, U. Tokyo: 40%
US: 60%, Japan: 40%
80%
SELECT * FROM Author WHERE Country=US
Secondary Index on (Country)
Institution
Tuple
Brown (72%)
Alice …
MIT (95%)
Bob
MIT (18%)
Country
TulpleID
Pointers
US (100%)
Bob
MIT
US (90%)
Alice
Brown, MIT
Store Multiple Pointers
Tailored Access

14. Experiments Environments C++ & BerkeleyDB 4.7 on Fedora Core 11
Quad-Core, 4GB RAM, 10k RPM SATA HDD
Dataset: DBLP w/ Uncertain Affiliation
700k authors and 1.3M publications
SwetoDblp, Google API (institutions up to ten per author)
Compared With PII

15. Query Runtime: PII vs. UPI
Q1: SELECT * FROM Author WHERE Institution=x
Elapsed
Read PII
5 [ms]
Sort Pointer
30 [µs]
Read Heap
5,200 [ms]
Elapsed
Read UPI
47 [ms]
UPI Causes Much Fewer Disk Seeks

16. Secondary Index Access
Q2: SELECT Journal, COUNT (*) FROM Publication WHERE Country=x GROUP BY Journal
Elapsed
Read PII
110 [ms]
Read UPI
3,200 [ms]
Elapsed
Read PII
110 [ms]
Tailor
33 [ms]
Read UPI
500 [ms]

17. Conclusion and Future Work
UPI
Heap + Cutoff Index
Tailored Secondary Index Access
Fractured UPI (not presented here)
Applying to other types of queries
Top-k Query: UPI as Tuple Access Layer

18. Thanks!

19. Fractured UPI New Fracture Main Fracture Fracture 1 Query Dump
Heap File
Cutoff Index
Delete Set
Delete Set
2ndary Index
2ndary Index
Query
Independently
Dump
Insert Buffer
(On RAM)
SELECT
INSERT
DELETE

20. Fractured UPI 8 sec 75 sec 650 sec 212 sec 4 sec 0.03 sec Insert 10%
Delete 1%
Unclustered Heap
8 sec
75 sec
UPI
650 sec
212 sec
Fractured UPI
4 sec
0.03 sec
Fragmentation
More Fractures

21. Cutoff Index Cost Model (1)
Selective Case (Q1, #Pointers=300)
Real Runtime
Estimated Runtime

22. Cutoff Index Cost Model (2)
Non-Selective Case (Q1, #Pointers=37000)
Real Runtime
Estimated Runtime