1. Indexing Correlated Probabilistic Databases Bhargav Kanagal & Amol Deshpande University of Maryland

2. Introduction Correlated Probabilistic data generated in many scenarios Data Integration [AFM06]: Conflicting information best captured using “mutual exclusivity” Information Extraction [JK+06]: Annotations on consecutive text segments are strongly correlated Bill can be reached at 3-4057 annotation: first name prob: 80% annotation: phone number prob: 90% High positive correlation Sensor Networks [KD08]: Very strong spatio-temporal correlations TIMEUSERMODE (INFERRED) Location (INFERRED) 5pmJohn Walking: 0.9 Car: 0.1 5pmJane Walking: 0.9 Car : 0.1 5:05pmJohn Walking: 0 Car: 1 Mystiq/Lahar, PrDB, MayBMS, MCDB, BayeStore... Represent complex correlations and query them But, do not scale to large numbers of arbitrary correlations

3. Challenges with arbitrary correlations The problem with chains of correlations – Simple queries involving few variables might need to access and manipulate the entire database Searching through the vast numbers of correlations is inefficient 2. What is the average temperature measured by the sensor ? (AGGREGATION query) 2. What is the average temperature measured by the sensor ? (AGGREGATION query) 1. What is the likelihood that the sensor is working at time t 100 given that it was working at time t 0 ? (WHAT-IF/INFERENCE query) TIMETEMPERATUREHUMIDITYSTATUS (inferred) t0t0 WORKING: 0.9 FAILED: 0.1 t1t1 WORKING: 0.9 FAILED: 0.1 ……… … ……… … t 100 WORKING: 0.4 FAILED: 0.6

4. Problem Given: Probabilistic database with millions of tuples and a large number of arbitrary correlations Need to support scalable execution of inference queries and aggregation queries – accessing as few data tuples from disk as possible – doing as few probability computations as possible We build a novel data structure – INDSEP that provides an index and a caching mechanism for efficient query processing over probabilistic databases

5. Outline 1.Background 1.Probabilistic Databases as Junction Trees 2.Querying Junction Trees 2.INDSEP 1.Shortcut Potentials 2.Overview of Data structure 3.Querying using INDSEP 1.Inference queries 2.Aggregation queries 4.Building INDSEP 5.Handling Updates 6.Results

6. Background: ProbDBs as Junction Trees (SD07) idXexists ? 1? 2? 31 4? 51 61 Tuple Uncertainty Attribute Uncertainty idYZ 1 2 3 4 5 Correlations Perfectly correlated idXexists ? 1a 2a 31 4e 51 61 idXexists ? 1ba 2ca 3k1 4de 5n1 6o1 idYZ 1hi 2hf 3gj 4lj 5lm Junction Tree Concise encoding of joint probability distribution

7. Background: Junction trees Clique Separator p(a,b) p(a)p(c)p(a,c) p(c,d) Each clique and separator stores joint pdf Tree structure reflects Markov property Given a, b independent of c Given c, a independent of d CAVEAT: Our approach inherits the disadvantages of the junction tree technique: Restricted to ProbDBs that can be efficiently represented using a junction tree [i.e., low tree width models]

8. Query Processing on Junction trees Inference Queries  Determine probability distribution of the set of query variables p(a,b) p(a)p(c)p(a,c) p(c,d) Push summation inside and smartly eliminate

9. (Naïve) Hugin’s Algorithm {b, e, o} Keep query variables Keep correlations Remove others Keep query variables Keep correlations Remove others PIVOT Steiner tree + Send messages toward a given pivot node For ProbDBs ≈ 1 million tuples, not scalable (1)Searching for cliques is expensive: Linear scan over all the nodes is inefficient (2)Span of the query can be very large – almost the complete database accessed even for a 3 variable query

10. Outline 1.Background 1.Probabilistic Databases as Junction Trees 2.Querying Junction Trees 2.INDSEP 1.Shortcut Potentials 2.Overview of Data structure 3.Querying using INDSEP 1.Inference queries 2.Aggregation queries 4.Building INDSEP 5.Handling Updates 6.Results

11. INDSEP (Shortcut potentials) How can we make the inference query computation scalable ? Shortcut Potential (1)Boundary separators (2)Distribution required to completely shortcut the partition (3)Which to build ? (1)Boundary separators (2)Distribution required to completely shortcut the partition (3)Which to build ? Junction tree on set variables {a, b, d, e, l, n, o} Junction tree on set variables {a, b, d, e, l, n, o}

12. Root I1I1 I1I1 I2I2 I2I2 I3I3 I3I3 P1P1 P1P1 P2P2 P2P2 P6P6 P6P6 P5P5 P5P5 P4P4 P4P4 P3P3 P3P3 1.Variables: {c, f, g, j, k}, {f, h, i} 2.Child Separators: p(f) 3.Tree induced on the children 4.Shortcut potentials of children: {p(c,j,f), p(f)} 1.Variables: {c, f, g, j, k}, {f, h, i} 2.Child Separators: p(f) 3.Tree induced on the children 4.Shortcut potentials of children: {p(c,j,f), p(f)} INDSEP - Overview 1.Variables: {a,b,..} {c,f,..} {j,n..o} 2.Child Separators: p(c), p(j) 3.Tree induced on the children 4.Shortcut potentials of children: {p(c), p(c,j), p(j)} 1.Variables: {a,b,..} {c,f,..} {j,n..o} 2.Child Separators: p(c), p(j) 3.Tree induced on the children 4.Shortcut potentials of children: {p(c), p(c,j), p(j)} Obtained by hierarchical partitioning of the junction tree

13. Root I1I1 I1I1 I2I2 I2I2 I3I3 I3I3 P1P1 P1P1 P2P2 P2P2 P6P6 P6P6 P5P5 P5P5 P4P4 P4P4 P3P3 P3P3 Query Processing using INDSEP {g} Search for ‘g’ using the variable set Extract the partition P 3 from disk Determine p(g) from the junction tree stored in P 3 P3P3 P3P3 {g} Number of disk accesses is only 3 Logarithmic (like) number of disk accesses for query processing Number of disk accesses is only 3 Logarithmic (like) number of disk accesses for query processing Recursion on the hierarchical index structure Isn’t this expensive ?

14. Query Processing using INDSEP Root I1I1 I1I1 I2I2 I2I2 I3I3 I3I3 P1P1 P1P1 P2P2 P2P2 P6P6 P6P6 P5P5 P5P5 P4P4 P4P4 P3P3 P3P3 {b, e, o} {b} {b, e} {o} {b, e, c} {c, j} {j, o} 1.Construct Steiner tree on query rvs 2.Is shortcut potential sufficient ? 3.Recursively proceed along its children 4.Assemble the probabilities obtained from the children 1.Construct Steiner tree on query rvs 2.Is shortcut potential sufficient ? 3.Recursively proceed along its children 4.Assemble the probabilities obtained from the children {b, e, o}

15. Aggregation Root I1I1 I1I1 I2I2 I2I2 I3I3 I3I3 P1P1 P1P1 P2P2 P2P2 P6P6 P6P6 P5P5 P5P5 P4P4 P4P4 P3P3 P3P3 Q = b + c + d + k + n + o {Q} {b, c, d} k k {n, o} Y 1 = b + c + d Y 2 = k Y 3 = n + o {Y 1, c} {c, Y 2, j} {j, Y 3 } {b+c, c, a} {d, a} “Push” the aggregate inside the index and exploit decomposability Currently working on supporting more complex SQL style queries {c, Y 2, j} {j, n, l} {l, o}

16. Outline 1.Background 1.Probabilistic Databases as Junction Trees 2.Querying Junction Trees 2.INDSEP 1.Shortcut Potentials 2.Overview of Data structure 3.Querying using INDSEP 1.Inference queries 2.Aggregation queries 4.Building INDSEP 5.Handling Updates 6.Results

17. Building INDSEP: Partitioning Subroutine: we use the linear tree partitioning algo of Kundu, Misra et al. [KM77] Input: node weighted tree, block size B Output: A Partitioning into connected components such that: (1)each component has size <= B (2)number of components is minimum. Linear in size of tree, pseudo-linear in B Linear in size of tree, pseudo-linear in B BOTTOM-UP ALGORITHM weight = size(p(cfg)) = 8 weight = size(p(a)) = 2 weight = size(p(jl)) = 4+ weight = size(p(cjf)) = 8+

18. Building INDSEP: Variable Renaming Each node stores all the variables present in its subtree - inefficient Rename variables so that we can store a range (not sets) Each partition will have – (min, max) e.g. [4, 7] – add-list e.g. {2} Mapping is stored in another relation, indexed by hash table Updates ? [Paper]  Starting from the leftmost partition assign numbers from 0 to the variables,  Increment the counter when new variables are seen  Overlapping variables fall into the add-list.

19. Update Processing [very briefly] We support two kinds of updates: – Updates to existing probabilities – Inserting new data [discussed in paper] On modifying a certain probability: – Every clique in the junction tree needs to be modified [Inefficient] – We develop a lazy algorithm: future queries take the burden of updating the index Root I1I1 I1I1 I2I2 I2I2 I3I3 I3I3 P1P1 P1P1 P2P2 P2P2 P6P6 P6P6 P5P5 P5P5 P4P4 P4P4 P3P3 P3P3 Suppose update at P 6 Load P 6 and update it Load I 3 and update shortcuts Load root and update shortcuts {k}

20. Outline 1.Background 1.Probabilistic Databases as Junction Trees 2.Querying Junction Trees 2.INDSEP 1.Shortcut Potentials 2.Overview of Data structure 3.Querying using INDSEP 1.Inference queries 2.Aggregation queries 4.Building INDSEP 5.Handling Updates 6.Results

21. Implementation [PrDB] System implemented in Java Disk: simulated in memory, array of disk blocks Each disk block is a serialized byte array with size BLOCK_SIZE Data is read and written in units of disk block insert into table1 values (‘t1’, 1, 2) uncertain(‘0 0.5; 1 0.5’); Tuple Uncertainty Tuple Uncertainty insert into table2 values (‘r1’, uncertain(‘xy 0.5; ab 0.5’),‘t1’); Attribute Uncertainty Attribute Uncertainty Correlations insert factor ‘0 0 0.5; 1 1 0.5’ in table1 on ‘t1.e’,‘t2.e’; Language for inserting new tuples and correlations into the probabilistic database

22. Experimental Setup Markov sequence DB [LR09, KD09] -2 million time steps ⇒ 4 million nodes -Periodic update at each time instant Event Monitoring: – 500,000 nodes – Generate arbitrary correlations in the database – Each node connected to k others k ∈ [1,5] – Random updates Queries & Updates: 4 Workloads based on span [Short: 20%, 40%, 60%, Long: 80%] (100 queries per workload) Between 2 and 5 random variables Similar workloads for Updates SPAN

23. Results INDSEP makes query processing highly efficient and scalable I/O cost for different workloads (Event) I/O cost for different workloads (Event) CPU cost for different workloads (Event) CPU cost for different workloads (Event) Notice that having just the index is not sufficient. We also need shortcut potentials Comparison Systems (1)No Index (2)With Index but no shortcut potentials (3)INDSEP NOTE: LOG scale

24. Results % inconsistent shortcut potentials (Markov Sequence) % inconsistent shortcut potentials (Markov Sequence) Time to build INDSEP (Event) Time to build INDSEP (Event) INDSEP construction is efficient Query Performance Results Note that the disk is currently simulated in memory.

25. Related Work DTrees (Darwiche et al.) – Design index for directed PGMs – Limited querying support Markov Chain Index (Letchner et al., [LR+09]) – Design index for a restricted correlation structure: single attribute Markov chains Indexing spatial probabilistic data (Tao et al., Singh et al.) – Based on R-trees – Attribute uncertainty with independence assumptions, i.e. no correlations

26. Future work Multiple queries – Sharing computation across queries Supporting approximations for high tree width ProbDBs in the index data structure Disconnections – We assume a connected junction tree, we would like to also support disconnected PGMs Thank you !!!