Presentation is loading. Please wait.

Presentation is loading. Please wait.

Scalable RDF Data Management & SPARQL Query Processing

Similar presentations


Presentation on theme: "Scalable RDF Data Management & SPARQL Query Processing"— Presentation transcript:

1 Scalable RDF Data Management & SPARQL Query Processing
Martin Theobald1, Katja Hose2, Ralf Schenkel3 1University of Antwerp, Belgium 2University of Aalborg, Denmark 3University of Passau, Germany

2 Outline of this Tutorial
Part I RDF in Centralized Relational Databases Part II RDF in Distributed Settings Part III Managing Uncertain RDF Data

3 Outline for Part I Part I.1: Foundations Part I.2: Rowstore Solutions
Introduction to RDF and Linked Open Data A short overview of SPARQL Part I.2: Rowstore Solutions Part I.3: Columnstore Solutions Part I.4: Other Solutions and Outlook

4 Information Extraction
YAGO/DBpedia et al. bornOn(Jeff, 09/22/42) gradFrom(Jeff, Columbia) hasAdvisor(Jeff, Arthur) hasAdvisor(Surajit, Jeff) knownFor(Jeff, Theory) >120 M facts for YAGO2 (mostly from Wikipedia infoboxes & categories)

5 YAGO2 Knowledge Base 3 M entities, 120 M facts
100 relations, 200k classes Entity accuracy  95% subclass subclass subclass Organization Person Location subclass subclass subclass subclass subclass Country Scientist Politician subclass subclass State instanceOf instanceOf Biologist instanceOf Physicist City instanceOf Germany instanceOf instanceOf locatedIn Oct 23, 1944 Erwin_Planck diedOn locatedIn Kiel Schleswig-Holstein FatherOf bornIn Nobel Prize hasWon instanceOf citizenOf diedOn Oct 4, 1947 Max_Planck Society Max_Planck Angela Merkel Apr 23, 1858 bornOn means means means means means “Max Planck” “Max Karl Ernst Ludwig Planck” “Angela Merkel” “Angela Dorothea Merkel”

6 Why care about scalability?
Sources: linkeddata.org wikipedia.org Rapid growth of available semantic data

7 Why care about scalability?
As of Sept. 2011: > 5 million owl:sameAs links between DBpedia/YAGO/Freebase Sources: linkeddata.org wikipedia.org Rapid growth of available semantic data More than 30 billion triples in more than 200 sources across the LOD cloud DBPedia: 3.4 million entities, 1 billion triples

8 … and still growing Billion triple challenge 2008: 1B triples
| War stories from BigOWLIM: 12B triples in Jun 2009 Garlik 4Store: 15B triples in Oct 2009 OpenLink Virtuoso: 15.4B+ triples AllegroGraph: 1+ Trillion triples

9 Queries can be complex, too
SELECT DISTINCT ?a ?b ?lat ?long WHERE { ?a dbpedia:spouse ?b. ?a dbpedia:wikilink dbpediares:actor. ?b dbpedia:wikilink dbpediares:actor. ?a dbpedia:placeOfBirth ?c. ?b dbpedia:placeOfBirth ?c. ?c owl:sameAs ?c2. ?c2 pos:lat ?lat. ?c2 pos:long ?long. } Q7 on BTC2008 in [Neumann & Weikum, 2009]

10 What effects does the financial crisis have on migration rates in the US?

11 Is there a significant increase of serious weather conditions in Europe over the past 20 years?

12 Which glutamic-acid proteases are inhibitors of HIV?

13 Question Answering (QA) Systems
KB from Wikipedia and user edits 600 million facts, 25 million entities KB of curated, structured data 10 trillion (!) facts, 50k algorithms

14 IBM Watson: Deep Question Answering
William Wilkinson's "An Account of the Principalities of Wallachia and Moldavia" inspired this author's most famous novel This town is known as "Sin City" & its downtown is "Glitter Gulch" As of 2010, this is the only former Yugoslav republic in the EU YAGO knowledge back-ends question classification & decomposition 99 cents got me a 4-pack of Ytterlig coasters from this Swedish chain D. Ferrucci et al.: Building Watson: An Overview of the DeepQA Project. AI Magazine, Fall 2010.

15 SPARQL 1.0 / 1.1 Query language for RDF suggested by the W3C.
3 ways to interpret RDF data: Instances of logical predicates (“facts”) Graphs (subjects/objects as nodes, predicates as directed and labeled edges) Relations (either multiple binary relations or a single, large ternary relation) SPARQL main building block: select-project-join combination of relational triple patterns equivalent to graph isomorphism queries over a potentially very large RDF graph

16 SPARQL – Example Example query: Find all actors from Ontario (that are in the knowledge base) vegetarian Albert_Einstein physicist Jim_Carrey actor Ontario Canada Ulm Germany scientist chemist Otto_Hahn Frankfurt Mike_Myers Newmarket Scarborough Europe isA bornIn locatedIn

17 SPARQL – Example Example query: Find all actors from Ontario (that are in the knowledge base) SELECT ?person WHERE { ?person isA actor. ?person bornIn ?loc ?loc locatedIn Ontario . } scientist Find subgraphs of this form: isA isA actor Ontario ?person ?loc bornIn locatedIn isA variables constants actor vegetarian physicist chemist isA isA isA isA isA isA Mike_Myers Jim_Carrey Albert_Einstein Otto_Hahn bornIn bornIn bornIn bornIn Scarborough Newmarket Ulm Frankfurt locatedIn locatedIn locatedIn locatedIn Ontario Germany locatedIn locatedIn Canada Europe

18 SPARQL 1.0 – More Features Eliminate duplicates in results
Return results in some order with optional LIMIT n clause Optional matches and filters on bounded var’s More operators: ASK, DESCRIBE, CONSTRUCT See: SELECT DISTINCT ?c WHERE {?person isA actor. ?person bornIn ?loc ?loc locatedIn ?c} SELECT ?person WHERE {?person isA actor. ?person bornIn ?loc ?loc locatedIn Ontario} ORDER BY DESC(?person) SELECT ?person WHERE {?person isA actor OPTIONAL{?person bornIn ?loc} FILTER (!BOUND(?loc))}

19 SPARQL 1.1 Extensions of the W3C
W3C SPARQL 1.1: Aggregations (COUNT, AVG, …) and grouping Subqueries in the WHERE clause Safe negation: FILTER NOT EXISTS {?x …} Syntactic sugar for OPTIONAL {?x … } FILTER(!BOUND(?x)) Expressions in the SELECT clause: SELECT (?a+?b) AS ?sum Label constraints on paths: ?x foaf:knows/foaf:knows/foaf:name ?name More functions and operators …

20 RDF+SPARQL: Centralized Engines
BigOWLIM (now ontotext.com) OpenLink Virtuoso OntoBroker (now semafora-systems.com) Apache Jena (different main-memory/relational backends) Sesame (now openRDF.org) SW-Store, Hexastore, 3Store, RDF-3X (no reasoning) System deployments with >1011 triples ( see

21 SPARQL: Extensions from Research (1)
More complex graph patterns: Transitive paths [Anyanwu et al., WWW’07] SELECT ?p, ?c WHERE { ?p isA scientist . ?p ??r ?c. ?c isA Country . ?c locatedIn Europe . PathFilter(cost(??r) < 5) . PathFilter(containsAny(??r,?t ). ?t isA City . } Regular expressions [Kasneci et al., ICDE’08] SELECT ?p, ?c WHERE { ?p isA ?s. ?s isA scientist. ?p (bornIn | livesIn | citizenOf) locatedIn* Europe.} Meanwhile mostly covered by the SPARQL 1.1 query proposal.

22 SPARQL: Extensions from Research (2)
Queries over federated RDF sources: Determine distribution of triple patterns as part of query (for example in Jena ARQ) Automatically route triple predicates to useful sources

23 SPARQL: Extensions from Research (2)
Queries over federated RDF sources: Determine distribution of triple patterns as part of query (for example in Jena ARQ) Automatically route triple predicates to useful sources Potentially requires mapping of identifiers from different sources SPARQL 1.1 explicitly supports federation of sources

24 Ranking is Essential! Queries often have a huge number of results:
“scientists from Canada” “publications in databases” “actors from the U.S.” Queries may have no matches at all: “Laboratoire d'informatique de Paris 6” “most beautiful railway stations” Ranking is an integral part of search Huge number of app-specific ranking methods: paper/citation count, impact, salary, … Need for generic ranking of 1) entities and 2) facts

25 Extending Entities with Keywords
Remember: entities occur in facts & in documents Associate entities with terms in those documents, keywords in URIs, literals, … (context of entity) chancellor Germany scientist election Stuttgart21 Guido Westerwelle France Nicolas Sarkozy

26 Extensions: Keywords Problem: not everything is triplified!
Consider witnesses/sources (provenance meta-facts) Allow text predicates with each triple pattern (à la XQ-FT) Semantics: triples match struct. pred. witnesses match text pred. European composers who have won the Oscar, whose music appeared in dramatic western scenes, and who also wrote classical pieces ? Select ?p Where { ?p instanceOf Composer . ?p bornIn ?t . ?t inCountry ?c . ?c locatedIn Europe . ?p hasWon ?a .?a Name AcademyAward . ?p contributedTo ?movie [western, gunfight, duel, sunset] . ?p composed ?music [classical, orchestra, cantata, opera] . }

27 Extensions: Keywords Problem: not everything is triplified!
Consider witnesses/sources (provenance meta-facts) Allow text predicates with each triple pattern (à la XQ-FT) Proximity of keywords or phrases boosts expressiveness French politicians married to Italian singers? Select ?p1, ?p2 Where { ?p1 instanceOf ?c1 [France, politics] . ?p2 instanceOf ?c2 [Italy, singer] . ?p1 marriedTo ?p2 . } CS researchers whose advisors worked on the Manhattan project? Select ?r, ?a Where { ?r instOf researcher [“computer science“] . ?a workedOn ?x [“Manhattan project“] . ?r hasAdvisor ?a . } Select ?r, ?a Where { ?r ?p1 ?o1 [“computer science“] . ?a ?p2 ?o2 [“Manhattan project“] . ?r ?p3 ?a . }

28 Extensions: Keywords CLEF/INEX 2012-13 Linked Data Track
Problem: not everything is triplified! CLEF/INEX Linked Data Track https://inex.mmci.uni-saarland.de/tracks/lod/ <topic id=" " category="Politics"> <jeopardy_clue>Which German politician is a successor of another politician who stepped down before his or her actual term was over, and what is the name of their political ancestor?</jeopardy_clue> <keyword_title>German politicians successor other stepped down before actual term name ancestor</keyword_title> <sparql_ft> SELECT ?s ?s1 WHERE { ?s rdf:type <http://dbpedia.org/class/yago/GermanPoliticians> . ?s1 <http://dbpedia.org/property/successor> ?s . FILTER FTContains (?s, "stepped down early") . } </sparql_ft> </topic>

29 Extensions: Keywords / Multiple Languages
Problem: not everything is triplified! Multilingual Question Answering over Linked Data (QALD-3), CLEF <question id="4" answertype="resource" aggregation="false" onlydbo="true"> <string lang="en">Which river does the Brooklyn Bridge cross?</string> <string lang="de">Welchen Fluss überspannt die Brooklyn Bridge?</string> <string lang="es">¿Por qué río cruza la Brooklyn Bridge?</string> <string lang="it">Quale fiume attraversa il ponte di Brooklyn?</string> <string lang="fr">Quelle cours d'eau est traversé par le pont de Brooklyn?</string> <string lang="nl">Welke rivier overspant de Brooklyn Bridge?</string> <keywords lang="en">river, cross, Brooklyn Bridge</keywords> <keywords lang="de">Fluss, überspannen, Brooklyn Bridge</keywords> <keywords lang="es">río, cruza, Brooklyn Bridge</keywords> <keywords lang="it">fiume, attraversare, ponte di Brooklyn</keywords> <keywords lang="fr">cours d'eau, pont de Brooklyn</keywords> <keywords lang="nl">rivier, Brooklyn Bridge, overspant</keywords> <query> PREFIX dbo: <http://dbpedia.org/ontology/> PREFIX res: <http://dbpedia.org/resource/> SELECT DISTINCT ?uri WHERE { res:Brooklyn_Bridge dbo:crosses ?uri . } </query> </question>

30 What Makes a Fact “Good”?
Confidence: Prefer results that are likely correct accuracy of info extraction trust in sources (authenticity, authority) bornIn (Jim Gray, San Francisco) from “Jim Gray was born in San Francisco” (en.wikipedia.org) livesIn (Michael Jackson, Tibet) from “Fans believe Jacko hides in Tibet” (www.michaeljacksonsightings.com) Informativeness: Prefer results with salient facts Statistical estimation from: frequency in answer frequency on Web frequency in query log q: Einstein isa ? Einstein isa scientist Einstein isa vegetarian q: ?x isa vegetarian Whocares isa vegetarian Diversity: Prefer variety of facts E won … E discovered … E played … E won … E won … E won … E won … Conciseness: Prefer results that are tightly connected size of answer graph weight of Steiner tree Einstein won NobelPrize Bohr won NobelPrize Einstein isa vegetarian Cruise isa vegetarian Cruise born Bohr died 1962

31 How Can We Implement This?
Confidence: Prefer results that are likely correct accuracy of info extraction trust in sources (authenticity, authority) Empirical accuracy of Information Extraction PageRank-style estimate of trust combine into: max { accuracy (f,s) * trust(s) | s  witnesses(f) } Informativeness: Prefer results with salient facts Statistical estimation from: frequency in answer frequency on Web frequency in query log PageRank-style entity/fact ranking [V. Hristidis et al., S.Chakrabarti, …] IR models: tf*idf … [K.Chang et al., …] Statistical Language Models [de Rijke et al.] or Diversity: Prefer variety of facts Statistical Language Models [Zhai et al., Elbassuoni et al.] Conciseness: Prefer results that are tightly connected size of answer graph weight of Steiner tree Graph algorithms (BANKS, STAR, …) [S.Chakrabarti et al., G.Kasneci et al., …]

32 Outline for Part I Part I.1: Foundations Part I.2: Rowstore Solutions
Introduction to RDF A short overview of SPARQL Part I.2: Rowstore Solutions Part I.3: Columnstore Solutions Part I.4: Other Solutions and Outlook

33 RDF in Rowstores Strings often mapped to unique integer IDs
Rowstore: general relational database, storing relations (incl. facts) as complete rows (MySQL, PostgreSQL, Oracle, DB2, SQLServer, …) General principles: store triples in one giant three-attribute table (subject, predicate, object) convert SPARQL to equivalent SQL The database will do the rest Strings often mapped to unique integer IDs Used by many TripleStores, including 3Store, Jena, HexaStore, RDF-3X, … Simple extension to quadruples (with graphid): (graph,subject,predicate,object) We consider only triples for simplicity!

34 Example: Single Triple Table
ex:Katja ex:teaches ex:Databases; ex:works_for ex:MPI_Informatics; ex:PhD_from ex:TU_Ilmenau. ex:Martin ex:teaches ex:Databases; ex:PhD_from ex:Saarland_University. ex:Ralf ex:teaches ex:Information_Retrieval; ex:PhD_from ex:Saarland_University; ex:works_for ex:Saarland_University, ex:MPI_Informatics. subject predicate object ex:Katja ex:teaches ex:Databases ex:Katja ex:works_for ex:MPI_Informatics ex:Katja ex:PhD_from ex:TU_Ilmenau ex:Martin ex:teaches ex:Databases ex:Martin ex:works_for ex:MPI_Informatics ex:Martin ex:PhD_from ex:Saarland_University ex:Ralf ex:teaches ex:Information_Retrieval ex:Ralf ex:PhD_from ex:Saarland_University ex:Ralf ex:works_for ex:Saarland_University ex:Ralf ex:works_for ex:MPI_Informatics

35 Conversion of SPARQL to SQL
General approach to translate SPARQL into SQL: (1) Each triple pattern is translated into a (self-) JOIN over the triple table (2) Shared variables create JOIN conditions (3) Constants create WHERE conditions (4) FILTER conditions create WHERE conditions (5) OPTIONAL clauses create OUTER JOINS (6) UNION clauses create UNION expressions

36 Example: Conversion to SQL Query
P1 P2 P3 P4 Filter regex(?u,“Saar“) Projection SELECT ?a ?b ?t WHERE {?a works_for ?u. ?b works_for ?u. ?a phd_from ?u. } OPTIONAL {?a teaches ?t} FILTER (regex(?u, “Saar”)) ?a ?a,?u SELECT R1.A, R1.B, R2.T FROM ( SELECT P1.subject as A, P2.subject as B FROM Triples P1, Triples P2, Triples P3 WHERE P1.predicate=“works_for” AND P2.predicate=“works_for” AND P3.predicate=“phd_from” AND P1.object=P2.object AND P1.subject=P3.subject AND P1.object=P3.object AND REGEXP_LIKE(P1.object, “Saar”) ) R1 LEFT OUTER JOIN ( SELECT P4.subject as A, P4.object as T FROM Triples P4 WHERE P4.predicate=“teaches”) AS R2 ) ON (R1.A=R2.A) SELECT FROM Triples P1, Triples P2, Triples P3 WHERE P1.predicate=“works_for” AND P2.predicate=“works_for” AND P3.predicate=“phd_from” AND P1.object=P2.object AND P1.subject=P3.subject AND P1.object=P3.object AND REGEXP_LIKE(P1.object, “Saar”) SELECT FROM Triples P1, Triples P2, Triples P3 WHERE P1.predicate=“works_for” AND P2.predicate=“works_for” AND P3.predicate=“phd_from” AND P1.object=P2.object AND P1.subject=P3.subject AND P1.object=P3.object SELECT P1.subject as A, P2.subject as B FROM Triples P1, Triples P2, Triples P3 WHERE P1.predicate=“works_for” AND P2.predicate=“works_for” AND P3.predicate=“phd_from” AND P1.object=P2.object AND P1.subject=P3.subject AND P1.object=P3.object AND REGEXP_LIKE(P1.object, “Saar”) SELECT FROM Triples P1, Triples P2, Triples P3 SELECT FROM Triples P1, Triples P2, Triples P3 WHERE P1.predicate=“works_for” AND P2.predicate=“works_for” AND P3.predicate=“phd_from” ?u

37 Is that all? Well, no. Which indexes should be built? (to support efficient evaluation of triple patterns) How can we reduce storage space? How can we find the best execution plan? Existing databases need modifications: flexible, extensible, generic storage not needed here cannot deal with multiple self-joins of a single table often generate bad execution plans

38 Dictionary for Strings
Map all strings to unique integers (e.g., via hashing) Regular size (4-8 bytes), much easier to handle Dictionary usually small, can be kept in main memory <http://example.de/Katja>  <http://example.de/Martin>  <http://example.de/Ralf>  This may break original lexicographic sorting order  RANGE conditions (not in SPARQL) are difficult!  FILTER conditions may be more expensive!

39 Indexes for Commonly Used Triple Patterns
Patterns with a single variable are frequent Example: Albert_Einstein invented ?x Build clustered index over (s,p,o) Can also be used for pattern like Albert_Einstein ?p ?x Build similar clustered indexes for all six permutations (3 x 2 x 1 = 6) SPO, POS, OSP to cover all possible triplet patterns SOP, OPS, PSO to have all sort orders for patterns with two var’s (16,19,5356) (16,24,567) (16,24,876) (27,19,643) (27,48,10486) (50,10,10456) … Lookup ids for constants: Albert_Einstein  16, invented  24 Lookup known prefix in index: (16,24,0) Read results while prefix matches: (16,24,567), (16,24,876) come already sorted! All triples in (s,p,o) order B+ tree for easy access Triple table no longer needed, all triples in each index

40 Why Sort Order Matters for Joins
When inputs sorted by join attribute, use Merge Join: sequentially scan both inputs immediately join matching triples skip over parts without matches allows pipelining When inputs are unsorted/sorted by wrong attribute, use Hash Join: build hash table from one input scan other input, probe hash table needs to touch every input triple breaks pipelining MJ HJ (16,19,5356) (16,24,567) (16,24,876) (27,19,643) (27,48,10486) (50,10,10456) (16,33,46578) (16,56,1345) (24,16,1353) (27,18,133) (47,37,20495) (50,134,1056) (16,19,5356) (16,24,567) (16,24,876) (27,19,643) (27,48,10486) (50,10,10456) (27,18,133) (50,134,1056) (16,56,1345) (24,16,1353) (47,37,20495) (16,33,46578) In general, Merge Joins are more preferable: small memory footprint, pipelining

41 RDF-3x: Even More Indexes!
SPARQL 1.0 considers duplicates (unless removed with DISTINCT) but does not (yet) support aggregates/counting often queries with many duplicates like SELECT ?x WHERE ?x ?y Germany. to retrieve entities related to Germany (but counts may be important in the application!) this materializes many identical intermediate results Solution: even more redundancy! Pre-compute aggregated indexes SP,SO,PO,PS,OP,OS,S,P,O Example: SO contains, for each pair (s,o), the number of triples with subject s and object o Do not materialize identical bindings, but keep counts Example: ?x=Albert_Einstein:4; ?x=Angela_Merkel:10 15 indexes overall (all SPO permutations + their unique subsets)

42 RDF-3x: Compression Scheme for Triplets
Compress sequences of triples in lexicographic order (v1;v2;v3); for SPO: v1=S, v2=P, v3=O Step 1: compute per-attribute deltas Step 2: variable-byte encoding for each delta triple 1-13 bytes (16,19,5356) (16,24,567) (16,24,676) (27,19,643) (27,48,10486) (50,10,10456) (16,19,5356) (0,5,-4798) (0,0,109) (11,-5,-34) (0,29,9843) (23,-38,-30) gap bit header (7 bits) Delta of value 1 (0-4 bytes) Delta of value 2 (0-4 bytes) Delta of value 3 (0-4 bytes) When gap=1, the delta of value3 is included in header, all others are 0 Otherwise, header contains length of encoding for each of the three deltas (5*5*5=125 combinations stored in 7 bits) Many variants exist; this one is designed for triplets…

43 Compression Effectiveness vs. Efficiency
Byte-level encoding almost as effective as bit-level encoding techniques (Gamma, Golomb, Rice, etc.) Much faster (10x) for decompressing Example for Barton dataset [Neumann & Weikum: VLDB’10]: Raw data 51 million triples, 7GB uncompressed (as N-Triples) All 6 main indexes: 1.1GB size, 3.2s decompression with byte-level encoding Optionally: additional compression with LZ77 2x more compact, but much slower to decompress Compression always on page level

44 Back to the Example Query
SELECT ?a ?b ?t WHERE {?a works_for ?u. ?b works_for ?u. ?a phd_from ?u. } OPTIONAL {?a teaches ?t} FILTER (regex(?u, “Saar”)) POS(works_for,?u,?a) POS(works_for,?u,?b) PSO(phd_from,?a,?u) POS(teaches,?a,?t) Projection ?u ?a,?u ?a MJ HJ Filter regex(?u,“Saar“) 250 POS(works_for,?u,?a) POS(pdh_from,?u,?a) PSO(works_for,?u,?b) Projection ?u,?a ?u ?a MJ Filter regex(?u,“Saar“) POS(teaches,?a,?t) 250 250 2500 250 100 50 5 1000 1000 50 1000 100 1000 Core ingredients of a good query optimizer are selectivity estimators for triple patterns (index scans) and joins Which of the two plans is better? How many intermediate results?

45 RDF-3x: Selectivity Estimation
How many results will a triple pattern have? Standard databases: Per-attribute histograms Assume independence of attributes  Use aggregated indexes for exact count Additional join statistics for triple blocks (pages): too simplistic and inexact … (16,19,5356) (16,24,567) (16,24,876) (27,19,643) (27,48,10486) (50,10,10456) … Assume independence between triple patterns; additionally precompute exact statistics for frequent paths in the data

46 Solution: Differential Indexing
Handling Updates What should we do when our data changes? (SPARQL 1.1 has updates!) Assumptions: Queries far more frequent than updates Updates mostly insertions, hardly any deletions Different applications may update concurrently Solution: Differential Indexing

47 RDF-3x: Differential Updates
Staging architecture for updates in RDF-3X Workspace A: Triples inserted by application A completion of A Workspace B: Triples inserted by application B completion of B on-demand indexes at query time kept in main memory Deletions: Insert the same tuple again with “deleted” flag Modify scan/join operators: merge differential indexes with main index

48 Outline for Part I Part I.1: Foundations Part I.2: Rowstore Solutions
Introduction to RDF A short overview of SPARQL Part I.2: Rowstore Solutions Part I.3: Columnstore Solutions Part I.4: Other Solutions and Outlook

49 Principles Observations and assumptions:
Not too many different predicates Triple patterns usually have fixed predicate Need to access all triples with one predicate Design consequence: Use one two-attribute table for each predicate

50 Example: Columnstores
ex:Katja ex:teaches ex:Databases; ex:works_for ex:MPI_Informatics; ex:PhD_from ex:TU_Ilmenau. ex:Martin ex:teaches ex:Databases; ex:PhD_from ex:Saarland_University. ex:Ralf ex:teaches ex:Information_Retrieval; ex:PhD_from ex:Saarland_University; ex:works_for ex:Saarland_University, ex:MPI_Informatics. subject object ex:Katja ex:MPI_Informatics ex:Martin ex:MPI_Informtatics ex:Ralf ex:Saarland_University ex:Ralf ex:MPI_Informatics works_for subject object ex:Katja ex:Databases ex:Martin ex:Databases ex:Ralf ex:Information_Retrieval teaches subject object ex:Katja ex:TU_Ilmenau ex:Martin ex:Saarland_University ex:Ralf ex:Saarland_University PhD_from

51 Simplified Example: Query Conversion
SELECT ?a ?b ?t WHERE {?a works_for ?u. ?b works_for ?u. ?a phd_from ?u. } SELECT W1.subject as A, W2.subject as B FROM works_for W1, works_for W2, phd_from P3 WHERE W1.object=W2.object AND W1.subject=P3.subject AND W1.object=P3.object So far, this is yet another relational representation of RDF. So, what is a columnstore?

52 Columnstores and RDF Columnstores store all columns of a table separately. PhD_from:subject ex:Katja ex:Martin ex:Ralf subject object ex:Katja ex:TU_Ilmenau ex:Martin ex:Saarland_University ex:Ralf ex:Saarland_University PhD_from PhD_from:object ex:TU_Ilmenau ex:Saarland_University Advantages: Fast if only subject or object are accessed, not both Allows for a very compact representation Problems: Need to recombine columns if subject and object are accessed Inefficient for triple patterns with predicate variable

53 Compression in Columnstores
General ideas: Store subject only once Use same order of subjects for all columns, including NULL values when necessary Additional compression to get rid of NULL values subject ex:Katja ex:Martin ex:Ralf PhD_from ex:TU_Ilmenau ex:Saarland_University NULL teaches ex:Databases ex:Databases ex:Information_Retrieval NULL works_for ex:MPI_Informatics ex:Saarland_University PhD_from: bit[1110] ex:TU_Ilmenau ex:Saarland_University Teaches: range[1-3] ex:Databases ex:Databases ex:Information_Retrieval

54 Outline for Part I Part I.1: Foundations Part I.2: Rowstore Solutions
Introduction to RDF A short overview of SPARQL Part I.2: Rowstore Solutions Part I.3: Columnstore Solutions Part I.4: Other Solutions and Outlook

55 Property Tables Group entities with similar predicates into a relational table (for example using RDF types or a clustering algorithm). ex:Katja ex:teaches ex:Databases; ex:works_for ex:MPI_Informatics; ex:PhD_from ex:TU_Ilmenau. ex:Martin ex:teaches ex:Databases; ex:PhD_from ex:Saarland_University. ex:Ralf ex:teaches ex:Information_Retrieval; ex:PhD_from ex:Saarland_University; ex:works_for ex:Saarland_University, ex:MPI_Informatics. subject teaches PhD_from ex:Katja ex:Databases ex:TU_Ilmenau ex:Martin ex:Databases ex:Saarland_University ex:Ralf ex:IR ex:Saarland_University subject teaches PhD_from ex:Katja ex:Databases ex:TU_Ilmenau ex:Martin ex:Databases ex:Saarland_University ex:Ralf ex:IR ex:Saarland_University ex:Axel NULL ex:TU_Vienna subject predicate object ex:Katja ex:works_for ex:MPI_Informatics ex:Martin ex:works_for ex:MPI_Informatics ex:Ralf ex:works_for ex:Saarland_University ex:Ralf ex:works_for ex:MPI_Informatics “Leftover triples”

56 Property Tables: Pros and Cons
Advantages: More in the spirit of existing relational systems Saves many self-joins over triple tables etc. Disadvantages: Potentially many NULL values Multi-value attributes problematic Query mapping depends on schema Schema changes very expensive

57  See our list of readings.
Even More Systems… Store RDF data as sparse matrix with bit-vector compression [BitMat, Hendler at al.: ISWC’09] Convert RDF into XML and use XML methods (XPath, XQuery, …) Store RDF data in graph databases and perform bi-simulation [Fletcher at al.: ESWC’12] or employ specialized graph index structures [gStore, Zou et al.: PVLDB’11] And many more …  See our list of readings.

58 Which Technique is Best?
Performance depends a lot on precomputation, optimization, implementation, fine-tuning … Comparative results on BTC 2008: (from [Neumann & Weikum, 2009]) RDF-3X RDF-3X (2008) COLSTORE ROWSTORE RDF-3X RDF-3X (2008) COLSTORE ROWSTORE

59 Challenges and Opportunities
SPARQL with different entailment regimes New SPARQL 1.1 features (grouping, aggregation, updates) User-oriented ranking of query results Efficient top-k operators Effective scoring methods for structured queries What are the limits of a centralized RDF engine? Dealing with uncertain RDF data – what is the most likely query answer? Triples with probabilities  probabilistic databases

60 Outline of this Tutorial
Part I RDF in Centralized Relational Databases Part II RDF in Distributed Settings Part III Managing Uncertain RDF Data

61 Outline for Part II Part II.1: Search Engines for the Semantic Web
Part II.2: Mediator-based and Federated Architectures

62 Semantic Web Search Engines
Querying RDF data collections started by adapting existing search engines to RDF data. Crawling for .rdf files, and HTML documents with embedded RDF content (see: RDFa microformat). Indexing & search based on keywords extracted from entity- and property names. Usually generate a virtual document for an entity (string literals and human-readable names). Swoogle [Ding et al., CIKM’04] (University of Maryland) Falcons [Cheng at al., WWW’08] (Nanjing University)

63

64

65 Outline for Part II Part II.1: Search Engines for the Semantic Web
Part II.2: Mediator-based and Federated Architectures

66 Classification of Distributed Approaches
Approaches for querying distributed and potentially heterogeneous (RDF) data sources Virtually materialized approaches Materialization-based approaches (data-warehousing) Mediator-based systems Federated systems MapReduce/ Hadoop Shared-nothing architectures DARQ FedEx YARS2 Partout 4Store Eagre Shard, Jena-HBase [Abadi et al. PVLDB’11] Peer-2-Peer Shared-memory architectures (Message-Passing, RMI, etc.) Gridvine RDFPeers Trinity (MSR)

67 How to Integrate Data Sources?
Ship and integrate data from different sources to the client. Three common approaches: Query-driven (single mediator) Database federations (exported schemas) Warehousing (fully integrated & centrally managed) ? RDF Source RDF Source

68 Query-Driven Approach
SPARQL Client SPARQL Client Mediator Wrapper Wrapper Wrapper query result query result query result RDF Source RDF Source RDF Source List of SPARQL endpoints: DBpedia: YAGO: https://d5gate.ag5.mpi-sb.mpg.de/webyagospo/Browser 7

69 Advantages of Query-Driven Integration
No need to copy data no or little own storage costs no need to purchase data Potentially more up-to-date data Mediator holds catalog (statistics, etc.) and may optimize queries Only generic query interface needed at sources (SPARQL endpoints) May be less draining on sources Sources often even unaware of participation

70 Federation-based Approach
SPARQL Client SPARQL Client Federated Schema Exported Schema Exported Schema Exported Schema query result query result query result Local Schema Local Schema Local Schema RDF Source RDF Source RDF Source Source 1 Source … Source n

71 Advantages of Federation-Based Integration
Very similar to query-driven integration, except that the sources know that they are part of a federation; and they export their local schemas into a federated schema. Intermediate step toward full integration of the data in a single “warehouse”.

72 Warehousing Architecture
SPARQL Client SPARQL Client Query & Analysis Warehouse Metadata Integration RDF Source RDF Source RDF Source Integrated LOD index:

73 Advantages of Warehousing
Perform Extract-Transform-Load (ETL) processes with periodic updates over the source High query performance Local processing at sources unaffected Can operate even when sources are offline Can query data that is no longer stored at sources More detailed statistics and metadata available at warehouse Modify, summarize (store aggregates), analyse Add historical information, provenance, timestamps, etc.

74 Classification of Distributed Approaches
Approaches for querying distributed and potentially heterogeneous (RDF) data sources Virtually materialized approaches Materialization-based approaches (data-warehousing) Mediator-based systems Federated systems MapReduce/ Hadoop Shared-nothing architectures DARQ FedEx YARS2 Partout 4Store Eagre Shard, Jena-HBase [Abadi et al. PVLDB’11] Peer-2-Peer Shared-memory architectures (Message-Passing, RMI, etc.) Gridvine RDFPeers Trinity (MSR)

75 DARQ [Leser et al., Humbold University Berlin, ISWC’08]
Classical mediator-based architecture connecting a given SPARQL endpoint to other endpoints via a combination of wrappers and service descriptions. Service descriptions RDF data descriptions Statistical information Binding constraints Query optimizer based on rewriting rules and cost estimations for physical join operators.

76 FedEx [fluid Op’s & MPI-INF: ISWC’11]
Online query optimization over federations of SPARQL endpoints. Cost estimates based on result sizes of SPARQL ASK queries. “Bound nested-loop joins” by grouping sets of variable bindings into SPARQL UNION queries (instead of using FILTER conditions): SELECT ?drug ?title WHERE { ?drug drugbank:drugCategory drugbank-category:micronutrient . ?drug drugbank:casRegistryNumber ?id . ?keggDrug rdf:type kegg:Drug . ?keggDrug bio2rdf:xRef ?id . ?keggDrug purl:title ?title . }

77 Partout [Galaraga, Hose, Schenkel: PVLDB’13]
Materialization-based, distributed & workload-aware SPARQL engine. Distribution helps to scale-out query processing via parallel join executions. Global query workload (aka. “query log”) Global query graph Triple fragments are distributed over hosts H1…Hn by (1) maximizing query locality, and (2) balancing the hosts workload. H1…Hn run local RDF-3x instances. (1) local S,P,O statistics by RDF-3x, (2) global (cached) statistics.

78 Partout Example Query Plan
H1, H2, H3 hold triplets for ?s rdf:type db:city H1 has triplets for ?s db:located db:Germany H2 has triplets for ?s db:name ?name

79 More Distributed RDF Engines
Shard TripleStore (Hadoop + Hash-Partitioning) RDFPeers (P2P/Chord architecture) [Cai et al., WWW’04] Gridvine (P2P/Chord architecture) [Aberer et al., VLDB’07] YARS2 (federated architecture) [Decker at al., ISWC’07] Jena-HBase (Hadoop & HBase) [Khadilkar et al., ISWC’12] SW-Store (Hadoop/RDF-3x) [Abadi et al., PVLDB’11] 4Store (materialized, shared-nothing) [Harris et al., SSWS’09] Eagre (materialized, shared-nothing) [HKUST & HP Labs, ICDE’13] Trinity (materialized, shared-memory, message passing) [MSR, SIGMOD’13]  more in Zoi’s tutorial in the afternoon…

80 Outline of this Tutorial
Part I RDF in Centralized Relational Databases Part II RDF in Distributed Settings Part III Managing Uncertain RDF Data

81 Outline for Part III Part III.1: Motivation
What is uncertain data, and where does it come from? Part III.2: Possible Worlds & Beyond Part III.3: Probabilistic Database Engines Stanford Trio Project MystiQ @ U Washington Part III.4: Managing Uncertain RDF Data Max Planck Institute

82 What is “Uncertain” Data?
Temperature is F Sensor reported 75 ± 0.5 F Bob works for Yahoo Bob works for either Yahoo or Microsoft Mary sighted a Finch Mary sighted either a Finch (60%) or a Sparrow (40%) It always rains in Galway There is a 89% chance of rain in Galway tomorrow Yahoo stocks will be at 100 in a month Yahoo stock will be between 60 and 120 in a month John’s age is 23 John’s age is in [20,30]

83 … And Why Does It Arise? “Certain” Data “Uncertain” Data
Temperature is F Sensor reported 75 ± 0.5 F Bob works for Yahoo Bob works for either Yahoo or Microsoft Mary sighted a Finch Mary sighted either a Finch (60%) or a Sparrow (40%) It always rains in Galway There is a 89% chance of rain in Galway tomorrow Yahoo stocks will be at 100 in a month Yahoo stock will be between 60 and 120 in a month John’s age is 23 John’s age is in [20,30] Precision of devices Lack of exact information (alternatives and missing values) Uncertainty about future events Anonymization

84 Applications: Deduplication
Name John Doe J. Doe ? 80% match

85 Applications: Information Integration
name, hPhone, oPhone, hAddr, oAddr name, phone, address at the schema level: “schema integration” at the instance level: “record linkage” Combined View

86 Applications: Information Extraction (I)
Restaurant Zip Hard Rock Cafe 94109

87 Applications: Information Extraction (II)
Subj. Pred. Obj. Galway type City locatedIn Ireland hasPopulation 75,414 areaCode 091 namedAfter Gaillimh_River What is Uncertain Data and Why Does It Arise?

88 Applications: Information Extraction (III)
YAGO/DBpedia et al. bornOn(Jeff, 09/22/42) gradFrom(Jeff, Columbia) hasAdvisor(Jeff, Arthur) hasAdvisor(Surajit, Jeff) knownFor(Jeff, Theory) >120 M facts for YAGO2 (mostly from Wikipedia infoboxes) New fact candidates type(Jeff, Author)[0.9] author(Jeff, Drag_Book)[0.8] author(Jeff,Cind_Book)[0.6] worksAt(Jeff, Bell_Labs)[0.7] type(Jeff, CEO)[0.4] 100’s M additional facts from Wikipedia text

89 How do current database management systems (DBMS) handle uncertainty?
They don’t 

90 What Do (Most) Applications Do?
Clean: turn into data that DBMSs can handle Observer Bird-1 Mary Finch: 80% Sparrow: 20% Susan Dove: 70% Sparrow: 30% Jane Hummingbird: 65% Sparrow: 35% Bird-1 Finch Dove Hummingbird Loss of information Errors compound and propagate insidiously

91 Outline for Part III Part III.1: Motivation
What is uncertain data, and where does it come from? Part III.2: Possible Worlds & Beyond Part III.3: Probabilistic Database Engines Stanford Trio Project MystiQ @ U Washington Part III.4: Managing Uncertain RDF Data Max Planck Institute Dan Suciu, Dan Olteanu, Christopher Ré, Christoph Koch: Probabilistic Databases (Synthesis Lectures on Data Management) Morgan & Claypool Publishers, 2012

92 Databases Today are Deterministic
An item either is in the database or it is not. A tuple either is in the query answer or it is not. This applies to all variety of data models: Relational, E/R, hierarchical, XML, …

93 What is a Probabilistic Database ?
“An tuple belongs to the database” is a probabilistic event. “A tuple is an answer to the query” is a probabilistic event. Can be extended to all possible kinds of data models; we consider only  probabilistic relational data.

94 Sample Spaces & Venn Diagrams
“Tuple t2 is an answer to a query.” “Tuple t1 is in the database.” Sample space : all possible events that can be observed. Pr() = 1. Random variable χt assigns a probability to an event s.t. 0 ≤ Pr( χt ) ≤ 1. As a convention, we will use tuple identifiers in the place of random variables to denote probabilistic events.

95 Possible Worlds Semantics
Attribute domains: int, varchar(55), datetime # values: 232, , Relational schema: Employee(ID:int, name:varchar(55), dob:datetime, salary:int) # of possible tuples: × 2440 × 264 × 232 # of possible relation instances: × 2440 × 264 × 232 Database schema: Employee(. . .), Projects( ), Groups( . . .), WorksFor( . . .) # of possible database instances: N (= big but finite)

96 The Definition Given a finite set of all possible database instances:
INST = {I1, I2, I3, . . ., IN} Definition: A probabilistic database Ip is a probability distribution on INST s.t. åi=1,…,N Pr(Ii) = 1 Pr : INST → [0,1] Definition: A possible world is I  INST s.t. Pr(I) > 0

97 Example Ip = Possible worlds = {I1, I2, I3, I4} Customer Address
Product John Seattle Gizmo Camera Sue Denver Customer Address Product John Boston Gadget Sue Denver Gizmo Pr(I2) = 1/12 Pr(I1) = 1/3 Customer Address Product John Seattle Gizmo Camera Sue Customer Address Product John Boston Gadget Sue Seattle Camera Pr(I4) = 1/12 Pr(I3) = 1/2 Possible worlds = {I1, I2, I3, I4}

98 Tuples as Events One tuple t  event “t  I” Pr(t) = åI: t  I Pr(I)
Marginal probability of t One tuple t  event “t  I” Pr(t) = åI: t  I Pr(I) Two tuples t1, t2  event “t1  I Λ t2  I” Pr(t1 Λ t2) = åI: t1I Λ t2I Pr(I) Marginal probability of t1 Λ t2

99 Tuple Correlations NOT Pr(⌐t1) = 1 - Pr(t1) Independent-AND
Pr(t1 Λ t2) = Pr(t1) Pr(t2) I Λ Independent-OR Pr(t1 V t2) = 1-(1-Pr(t1))(1-Pr(t2)) I V Disjoint-AND Pr(t1 Λ t2) = 0 Disjoint-OR Pr(t1 V t2) = Pr(t1)+Pr(t2) D V Negatively correlated Pr(t1 Λ t2) < Pr(t1) Pr(t2) N Positively correlated Pr(t1 Λ t2) > Pr(t1) Pr(t2) P Identical Pr(t1 Λ t2) = Pr(t1) = Pr(t2) =

100 Example with Correlations
Ip = Customer Address Product John Seattle Gizmo Camera Sue Denver Customer Address Product John Boston Gadget Sue Denver Gizmo = D N Pr(I2) = 1/12 Pr(I1) = 1/3 D Customer Address Product John Seattle Gizmo Camera Sue Customer Address Product John Boston Gadget Sue Seattle Camera P Pr(I4) = 1/12 Pr(I3) = 1/2

101 Special Case! Tuple-independent probabilistic database
INST = P(TUP) N = 2M TUP = {t1, t2, …, tM} = all tuples No restrictions w.r.t. other tuples pr : TUP → (0,1] Pr(I) = Õt  I pr(t) × Õt Ï I (1-pr(t))

102 … back to the Venn Diagram (I)
Sample Space  “Tuple t2 is in the database.” “Tuple t1 is in the database.” Pr(“Tuple t1 is in the database and tuple t2 is in the database”) := Pr(t1) x Pr(t2) = pr(t1) x pr(t2) If t1 and t2 are independent (per assumption!): 4 possible worlds = 4 subsets of events

103 … back to the Venn Diagram (II)
Sample Space  “Tuple t2 is in the database.” “Tuple t1 is in the database.” Pr(“Tuple t1 is in the database and tuple t2 is in the database”) := 0 If t1 and t2 are disjoint (per assumption!): 3 possible worlds = 3 subsets of events

104 Tuple Prob.  Possible Worlds
Assumption: Tuples are independent! Name City pr John Seattle p1 = 0.8 Sue Boston p2 = 0.6 Fred p3 = 0.9 J = å = 1 Ip = Name City Name City John Seattl Name City Sue Bosto Name City Fred Bosto Name City John Seattl Sue Bosto Name City John Seattl Fred Bosto Name City Sue Bosto Fred Name City John Seattl Sue Bosto Fred I1 (1-p1) (1-p2) (1-p3) I2 p1(1-p2)(1-p3) I3 (1-p1)p2(1-p3) I4 (1-p1)(1-p2)p3 I5 p1p2(1-p3) I6 p1(1-p2)p3 I7 (1-p1)p2p3 I8 p1p2p3

105 Tuple Prob.  Query Evaluation
Customer Product Date pr John Gizmo . . . q1 Gadget q2 q3 Sue Camera q4 q5 q6 Fred q7 Name City pr John Seattle p1 Sue Boston p2 Fred p3 Marginals SELECT DISTINCT x.city FROM Personp x, Purchasep y WHERE x.Name = y.Customer and y.Product = ‘Gadget’ Tuple Probability Seattle Boston p1( ) 1-(1-q2)(1-q3) 1- ( ) × ( ) p2( ) p3 1-(1-q5)(1-q6 ) q7

106 Summary of Data Model Possible Worlds Semantics Very powerful model:
Can capture any tuple correlations. Needs separate representation formalism: (“just tables” are generally not enough)  Boolean event expressions to capture complex tuple- dependencies: “provenance”, “lineage”, “views”, etc. But: query evaluation may be very expensive. Need to find good cases, otherwise must approximate.

107 Outline for Part III Part III.1: Motivation
What is uncertain data, and where does it come from? Part III.2: Possible Worlds & Beyond Part III.3: Probabilistic Database Engines Stanford Trio Project MystiQ @ U Washington Part III.4: Managing Uncertain RDF Data Max Planck Institute

108 Uncertainty-Lineage Databases (ULDBs)
Trio’s Data Model [Widom et al.: 2008] Uncertainty-Lineage Databases (ULDBs) Alternatives ‘?’ (Maybe) Annotations Confidence values Lineage

109 red, Honda ∥ red, Toyota ∥ orange, Mazda
Trio’s Data Model 1. Alternatives: uncertainty about value Saw (witness, color, car) Amy red, Honda ∥ red, Toyota ∥ orange, Mazda Three possible instances

110 red, Honda ∥ red, Toyota ∥ orange, Mazda
Trio’s Data Model 1. Alternatives 2. ‘?’ (Maybe): uncertainty about presence Saw (witness, color, car) Amy red, Honda ∥ red, Toyota ∥ orange, Mazda Betty blue, Acura ? Six possible instances

111 red, Honda 0.5 ∥ red, Toyota 0.3 ∥ orange, Mazda 0.2
Trio’s Data Model 1. Alternatives 2. ‘?’ (Maybe) Annotations 3. Confidences: weighted uncertainty Saw (witness, color, car) Amy red, Honda 0.5 ∥ red, Toyota 0.3 ∥ orange, Mazda 0.2 Betty blue, Acura 0.6 ? Six possible instances, each with a probability

112 So Far: Model is Not Closed
Saw (witness, car) Cathy Honda ∥ Mazda Drives (person, car) Jimmy, Toyota ∥ Jimmy, Mazda Billy, Honda ∥ Frank, Honda Hank, Honda Suspects = πperson(Saw ⋈ Drives) Suspects Jimmy Billy ∥ Frank Hank CANNOT Does not correctly capture possible instances in the result ? ? ?

113 Jimmy, Toyota ∥ Jimmy, Mazda Billy, Honda ∥ Frank, Honda
Example with Lineage ID Drives (person, car) 21 Jimmy, Toyota ∥ Jimmy, Mazda 22 Billy, Honda ∥ Frank, Honda 23 Hank, Honda ID Saw (witness, car) 11 Cathy Honda ∥ Mazda Suspects = πperson(Saw ⋈ Drives) ID Suspects 31 Jimmy 32 Billy ∥ Frank 33 Hank ? ? ? λ(31) = (11,2) Λ (21,2) λ(32,1) = (11,1) Λ (22,1); λ(32,2) = (11,1) Λ (22,2) λ(33) = (11,1) Λ 23

114 Jimmy, Toyota ∥ Jimmy, Mazda Billy, Honda ∥ Frank, Honda
Example with Lineage ID Drives (person, car) 21 Jimmy, Toyota ∥ Jimmy, Mazda 22 Billy, Honda ∥ Frank, Honda 23 Hank, Honda ID Saw (witness, car) 11 Cathy Honda ∥ Mazda Suspects = πperson(Saw ⋈ Drives) ID Suspects 31 Jimmy 32 Billy ∥ Frank 33 Hank λ(31) = (11,2) Λ (21,2) ? λ(32,1) = (11,1) Λ (22,1); λ(32,2) = (11,1) Λ (22,2) ? ? λ(33) = (11,1) Λ 23 Correctly captures possible instances in the result (7)

115 Operational Semantics
D D′ Closure: up-arrow always exists direct implementation possible instances rep. of instances Q on each instance I1, I2, …, In J1, J2, …, Jm Completeness: any (finite) set of possible instances can be represented

116 Summary on Trio’s Data Model
Uncertainty-Lineage Databases (ULDBs) Alternatives ‘?’ (Maybe) Annotations Confidence values Lineage Theorem: ULDBs are closed and complete. Formally studied properties like minimization, equivalence, approximation and membership based on lineage. [Benjelloun, Widom, et al.: VLDB J. 08]

117 MYSTIQ: Query Complexity
Data complexity of a query Q: Compute Q(Ip), for probabilistic database J Extensional query evaluation: Works for “safe” query plans with PTIME data complexity Intensional query evaluation: Works for any plan but has #P-complete data complexity in the general case Assume independent tuples in J Compute marginal probabilities for tuples in Q Boolean event expressions for intensional query evaluation

118 Extensional Query Evaluation
[Fuhr&Roellke:1997, Dalvi&Suciu:2004] Relational op’s compute probabilities or: p1 + p2 + … v p v 1-(1-p1)(1-p2)… v1 v2 p1 p2 v p1(1-p2) P - s × v p1 p2 v p v1 p1 v2 p2 v p1 v p2 Data complexity: PTIME

119 “Safe Plans” × × P P Depends on plan !!! [Dalvi&Suciu:2004]
SELECT DISTINCT x.City FROM Personp x, Purchasep y WHERE x.Name = y.Customer and y.Product = ‘Gadget’ “Safe Plans” Jon Sea p1(1-(1-q1)(1-q2)(1-q3)) Sea 1-(1-p1q1)(1- p1q2)(1- p1q3) × P Correct ! × P Jon Sea p1q1 p1q2 p1q3 Wrong ! Jon 1-(1-q1)(1-q2)(1-q3) Jon q1 q2 q3 Jon q1 q2 q3 Jon Sea p1 Jon Sea p1 Depends on plan !!!

120 Data complexity is #P-complete
Query Complexity [Dalvi&Suciu:2004] Sometimes there exists a correct extensional plan, but consider the following: Data complexity is #P-complete Qbad :- R(x), S(x,y), T(y) NP = class of problems of the form “is there a witness ?” #P = class of problems of the form “how many witnesses ?” (will be coming back to this…)

121 Intensional Database Atomic event ids e1, e2, e3, … Probabilities:
[Fuhr&Roellke:1997] Atomic event ids e1, e2, e3, … Probabilities: p1, p2, p3, …  [0,1] Event expressions: Λ, V, ⌐ e3 Λ (e5 V ⌐e2) Intensional probabilistic database J  each tuple t has an event attribute t.E

122 Probability of Boolean Expressions
Needed for query evaluation! E = X1X3 v X1X4 v X2X5 v X2X6 Sampling: Randomly make each variable true with the following probabilities Pr(X1) = p1, Pr(X2) = p2, , Pr(X6) = p6 “Read once” formula What is Pr(E) ??? Answer: Re-group cleverly E = X1 (X3 v X4 ) v X2 (X5 v X6) Pr(E) = 1 - (1-p1(1-(1-p3)(1-p4))) (1-p2(1-(1-p5)(1-p6)))

123 Complexity Issues Theorem [Valiant:1979] For a Boolean expression E, computing Pr(E) is #P-complete NP = class of problems of the form “is there a witness ?”  SAT #P = class of problems of the form “how many witnesses ?”  #SAT The decision problem for 2CNF is in PTIME The counting problem for 2CNF is #P-complete

124 Probabilistic Query Engine
MYSTIQ: [Re, Suciu: VLDB’04] Probabilistic Query Evaluation on Top of a Deterministic Database Engine (Top-k) Answers 1. Sampling Probabilistic Query Engine 2. Extensional joins SQL Query Deterministic Database 3. Indexes

125 Outline for Part III Part III.1: Motivation
What is uncertain data, and where does it come from? Part III.2: Possible Worlds & Beyond Part III.3: Probabilistic Database Engines Stanford Trio Project MystiQ @ U Washington Part III.4: Uncertain RDF Data URDF Max Planck Institute

126 Uncertain RDF (URDF) Data Model
Extensional Layer (information extraction & integration) High-confidence facts: existing knowledge base (“ground truth”) New fact candidates: extracted facts with confidence values Integration of different knowledge sources: Ontology merging or explicit Linked Data (owl:sameAs, owl:equivProp.)  Large “Probabilistic Database” of RDF facts Intensional Layer (query-time inference) Soft rules: deductive grounding & lineage (Datalog/SLD resolution) Hard rules: consistency constraints (more general FOL rules) Propositional & probabilistic consistency reasoning

127 Soft Rules vs. Hard Rules
(Soft) Deduction Rules vs. (Hard) Consistency Constraints People may live in more than one place livesIn(x,y)  marriedTo(x,z)  livesIn(z,y) livesIn(x,y)  hasChild(x,z)  livesIn(z,y) People are not born in different places/on different dates bornIn(x,y)  bornIn(x,z)  y=z bornOn(x,y)  bornOn(x,z)  y=z People are not married to more than one person (at the same time, in most countries?) marriedTo(x,y,t1)  marriedTo(x,z,t2)  y≠z  disjoint(t1,t2) [0.8] [0.5]

128 Soft Rules vs. Hard Rules
Deductive database: Datalog, core of SQL & relational algebra, RDF/S, OWL2-RL, etc. (Soft) Deduction Rules vs. (Hard) Consistency Constraints People may live in more than one place livesIn(x,y)  marriedTo(x,z)  livesIn(z,y) livesIn(x,y)  hasChild(x,z)  livesIn(z,y) People are not born in different places/on different dates bornIn(x,y)  bornIn(x,z)  y=z bornOn(x,y)  bornOn(x,z)  y=z People are not married to more than one person (at the same time, in most countries?) marriedTo(x,y,t1)  marriedTo(x,z,t2)  y≠z  disjoint(t1,t2) [0.8] [0.5] More general FOL constraints: Datalog with constraints, X-Tuples in Prob.-DB’s owl:FunctionalProperty, etc.

129 URDF: Running Example KB: Base Facts Jeff Surajit David Rules
hasAdvisor(x,y)  worksAt(y,z)  graduatedFrom(x,z) [0.4] graduatedFrom(x,y)  graduatedFrom(x,z)  y=z Computer Scientist type[1.0] type[1.0] type[1.0] hasAdvisor[0.7] hasAdvisor[0.8] Jeff Surajit David graduatedFrom[0.9] graduatedFrom[0.6] graduatedFrom[?] graduatedFrom[?] graduatedFrom[0.7] Stanford Princeton Derived Facts gradFr(Surajit,Stanford) gradFr(David,Stanford) worksAt[0.9] type[1.0] type[1.0] University

130 Basic Types of Inference
MAP Inference Find the most likely assignment to query variables y under a given evidence x. Compute: arg max y P( y | x) (NP-hard for MaxSAT) Marginal/Success Probabilities Probability that query y is true in a random world Compute: ∑y P( y | x) (#P-hard already for conjunctive queries)

131 General Route: Grounding & MaxSAT Solving
Query graduatedFrom(x, y) 1) Grounding Consider only facts (and rules) which are relevant for answering the query 2) Propositional formula in CNF, consisting of Grounded hard & soft rules Weighted base facts 3) Propositional Reasoning Find truth assignment to facts such that the total weight of the satisfied clauses is maximized  MAP inference: compute “most likely” possible world CNF (graduatedFrom(Surajit, Stanford)  graduatedFrom(Surajit, Princeton))  (graduatedFrom(David, Stanford)  graduatedFrom(David, Princeton))  (hasAdvisor(Surajit, Jeff)  worksAt(Jeff, Stanford)  graduatedFrom(Surajit, Stanford))  (hasAdvisor(David, Jeff)  graduatedFrom(David, Stanford))  worksAt(Jeff, Stanford)  hasAdvisor(Surajit, Jeff)  hasAdvisor(David, Jeff)  graduatedFrom(Surajit, Princeton)  graduatedFrom(Surajit, Stanford)  graduatedFrom(David, Princeton) 1000 0.4 0.9 0.8 0.7 0.6

132 URDF: MaxSAT Solving with Soft & Hard Rules
Special case: Horn-clauses as soft rules & mutex-constraints as hard rules [Theobald,Sozio,Suchanek,Nakashole: VLDS‘12] { graduatedFrom(Surajit, Stanford), graduatedFrom(Surajit, Princeton) } { graduatedFrom(David, Stanford), graduatedFrom(David, Princeton) } 0.4 0.9 0.8 0.7 0.6 Find: arg max y P( y | x) Resolves to a variant of MaxSAT for propositional formulas S: Mutex-const. Compute W0 = ∑clauses C w(C) P(C is satisfied); For each hard constraint S { For each fact f in St { Wf+t = ∑clauses C w(C) P(C is sat. | f = true); } WS-t = ∑clauses C w(C) P(C is sat. | St = false); Choose truth assignment to f in St that maximizes Wf+t , WS-t ; Remove satisfied clauses C; t++; MaxSAT Alg. (hasAdvisor(Surajit, Jeff)  worksAt(Jeff, Stanford)  graduatedFrom(Surajit, Stanford))  (hasAdvisor(David, Jeff)  graduatedFrom(David, Stanford))  worksAt(Jeff, Stanford)  hasAdvisor(Surajit, Jeff)  hasAdvisor(David, Jeff)  graduatedFrom(Surajit, Princeton)  graduatedFrom(Surajit, Stanford)  graduatedFrom(David, Princeton) C: Weighted Horn clauses (CNF) Runtime: O(|S||C|) Approximation guarantee of 1/2

133 Deductive Grounding with Lineage (SLD Resolution/Datalog)
Query graduatedFrom(Surajit, y) Rules hasAdvisor(x,y)  worksAt(y,z)  graduatedFrom(x,z) [0.4] graduatedFrom(x,y)  graduatedFrom(x,z)  y=z graduatedFrom (Surajit, Princeton) graduatedFrom (Surajit, Stanford) Q1 Q2 A(B (CD))  A(B (CD)) Base Facts graduatedFrom(Surajit, Princeton) [0.7] graduatedFrom(Surajit, Stanford) [0.6] graduatedFrom(David, Princeton) [0.9] hasAdvisor(Surajit, Jeff) [0.8] hasAdvisor(David, Jeff) [0.7] worksAt(Jeff, Stanford) [0.9] type(Princeton, University) [1.0] type(Stanford, University) [1.0] type(Jeff, Computer_Scientist) [1.0] type(Surajit, Computer_Scientist) [1.0] type(David, Computer_Scientist) [1.0] \/ C D A B graduatedFrom (Surajit, Princeton)[0.7] graduatedFrom (Surajit, Stanford)[0.6] /\ hasAdvisor (Surajit,Jeff)[0.8] worksAt (Jeff,Stanford)[0.9]

134 Lineage & Possible Worlds
[Das Sarma,Theobald,Widom: ICDE‘08 Dylla, Miliaraki,Theobald: CIKM‘11] Query graduatedFrom(Surajit, y) 1) Deductive Grounding Dependency graph of the query Trace lineage of individual query answers 2) Lineage DAG (not in CNF), consisting of Grounded hard & soft rules Weighted base facts Plus: entire derivation history! 3) Probabilistic Inference  Compute marginals: P(Q): aggregate probabilities of all possible worlds where the lineage of the query evaluates to “true” P(Q|H): drop “impossible worlds” 0.7x( )=0.078 (1-0.7)x0.888=0.266 1-(1-0.72)x(1-0.6) =0.888 0.8x0.9 =0.72 graduatedFrom (Surajit, Princeton) graduatedFrom (Surajit, Stanford) Q1 Q2 A(B (CD))  A(B (CD)) \/ C D A B graduatedFrom (Surajit, Princeton)[0.7] graduatedFrom (Surajit, Stanford)[0.6] /\ hasAdvisor (Surajit,Jeff)[0.8] worksAt (Jeff,Stanford)[0.9]

135 Possible Worlds Semantics
P(Q1)=0.0784 P(Q1|H)= / 0.412 = P(Q2|H)= / 0.412 = P(Q2)=0.2664 A:0.7 B:0.6 C:0.8 D:0.9 Q2:  A(B(CD)) P(W) 1 0.7x0.6x0.8x0.9 = 0.7x0.6x0.8x0.1 = … = … = … = … = … = … = 0.3x0.6x0.8x0.9 = 0.3x0.6x0.8x0.1 = 0.3x0.6x0.2x0.9 = 0.3x0.6x0.2x0.1 = 0.3x0.4x0.8x0.9 = … = … = … = 0.0784 0.2664 1.0 0.412 Hard rule H:  A   (B  (CD))

136 More Probabilistic Approaches
Propositional Stochastic MaxSat solvers: MaxWalkSat (MAP-Inference) URDF: constrained weighted MaxSat solver for soft & hard rules Lineage & Possible Worlds (tuple-independent database) Exact probabilistic inference: junction trees, variable elimination Approximate inference: decision diagrams/Shannon expansions, sampling Combining First-Order Logic & Probabilistic Graphical Models Markov Logic Networks* [Richardson & Domingos: Machine Learning 2006] Factor Graphs [FactorIE, McCallum et al.: NIPS 2008] Variety of MCMC sampling techniques for probabilistic inference (e.g., Gibbs sampling, MC-SAT, etc.) *Alchemy – Open-Source AI:

137 Experiments YAGO Knowledge Base: 2 Mio entities, 20 Mio facts
Basic query answering: SLD grounding & MaxSat solving of 10 queries over 16 soft rules (partly recursive) & 5 hard rules (bornIn, diedIn, marriedTo, …) Asymptotic runtime checks: runtime comparisons for synthetic rule expansions URDF: SLD grounding & MaxSat solving URDF MaxSat vs. Markov Logic (MAP inference & MC-SAT) |C| - # literals in soft rules |S| - # literals in hard rules

138 UViz: URDF Visualization Frontend
[Meiser, Dylla, Theobald: CIKM’11 Demo] System components: Flash Player client Tomcat server (JRE) Relational backend (JDBC) Remote Method Invocation & Object Serialization (BlazeDS)

139 UViz: URDF Visualization Frontend
[Meiser, Dylla, Theobald: CIKM’11 Demo] Demo!

140 Recommended Readings PART I
SPARQL Query Language for RDF, W3C Recommendation, 15 January 2008, SPARQL 1.1 Query Language, W3C Working Draft, 21 March 2013, SPARQL 1.1 Federated Query, W3C Working Draft, 21 March 2013, Kemafor Anyanwu, Angela Maduko, Amit P. Sheth: SPARQ2L: towards support for subgraph extraction queries in RDF databases. WWW Conference, 2007 Krisztian Balog, Edgar Meij, Maarten de Rijke: Entity Search: Building Bridges between Two Worlds. WWW, 2010 Gaurav Bhalotia, Arvind Hulgeri, Charuta Nakhe, Soumen Chakrabarti, S. Sudarshan: Keyword Searching and Browsing in Databases using BANKS. ICDE, 2002 Tao Cheng , Xifeng Yan , Kevin Chen-Chuan Chang: EntityRank: searching entities directly and holistically. VLDB, 2007 Shady Elbassuoni, Maya Ramanath, Ralf Schenkel, Marcin Sydow, Gerhard Weikum: Language-model-based ranking for queries on RDF-graphs. CIKM, 2009 Vagelis Hristidis, Heasoo Hwang, Yannis Papakonstantinou: Authority-based keyword search in databases. ACM Transactions on Database Systems 33(1), 2008 Gjergji Kasneci, Maya Ramanath, Mauro Sozio, Fabian M. Suchanek, Gerhard Weikum: STAR: Steiner-Tree Approximation in Relationship Graphs. ICDE, 2009 Gjergji Kasneci, Fabian M. Suchanek, Georgiana Ifrim, Maya Ramanath, Gerhard Weikum: NAGA: Searching and Ranking Knowledge. ICDE, 2008 Thomas Neumann, Gerhard Weikum: Scalable join processing on very large RDF graphs. SIGMOD Conference, 2009 Thomas Neumann, Gerhard Weikum: The RDF-3X engine for scalable management of RDF data. VLDB Journal 19(1), 2010 François Picalausa, Yongming Luo, George H. L. Fletcher, Jan Hidders, Stijn Vansummeren: A Structural Approach to Indexing Triples. ESWC 2012 Nicoleta Preda, Gjergji Kasneci, Fabian M. Suchanek, Thomas Neumann, Wenjun Yuan, Gerhard Weikum: Active knowledge: dynamically enriching RDF knowledge bases by Web Services. SIGMOD Conference, 2010 Cheng Xiang Zhai: Statistical Language Models for Information Retrieval. Morgan & Claypool Publishers, 2008 Lei Zou, Jinghui Mo, Lei Chen, M. Tamer Özsu, Dongyan Zhao: gStore: Answering SPARQL Queries via Subgraph Matching. PVLDB 4(8), 2011 PART II Min Cai, Martin R. Frank: RDFPeers: a scalable distributed RDF repository based on a structured peer-to-peer network. WWW, 2004 Gong Cheng, Weiyi Ge, Yuzhong Qu: Falcons: searching and browsing entities on the semantic web. WWW, 2008 Li Ding, Tim Finin, Anupam Joshi, Rong Pan, R. Scott Cost, Yun Peng, Pavan Reddivari, Vishal C Doshi, Joel Sachs: Swoogle: A Search and Metadata Engine for the Semantic Web. CIKM, 2004 Luis Galárraga, Katja Hose, Ralf Schenkel: Partout: A Distributed Engine for Efficient RDF Processing. To appear in PVLDB, 2013 Steve Harris, Nick Lamb, Nigel Shadbolt: 4store: The Design and Implementation of a Clustered RDF Store. SSWS, 2009 Jiewen Huang, Daniel J. Abadi, Kun Ren: Scalable SPARQL Querying of Large RDF Graphs. PVLDB 4(11), 2011 Vaibhav Khadilkar, Murat Kantarcioglu, Bhavani Thuraisingham, Paolo Castagna: Jena-HBase: A Distributed, Scalable and Efficient RDF Triple Store. ISWC, 2012 Bastian Quilitz, Ulf Leser: Querying Distributed RDF Data Sources with SPARQL. ISWC, 2008 Andreas Schwarte, Peter Haase, Katja Hose, Ralf Schenkel, Michael Schmidt: FedX: A Federation Layer for Distributed Query Processing on Linked Open Data. ESWC, 2011 Bin Shao, Haixun Wang, Yatao Li: Trinity: A Distributed Graph Engine on a Memory Cloud. To appear in SIGMOD, 2013 Xiaofei Zhang, Lei Chen, Yongxin Tong, Min Wang: EAGRE: Towards Scalable I/O Efficient SPARQL Query Evaluation on the Cloud. ICDE, 2013 PART III Omar Benjelloun, Anish Das Sarma, Alon Y. Halevy, Martin Theobald, Jennifer Widom: Databases with uncertainty and lineage. VLDB J. 17(2), 2008 Jihad Boulos, Nilesh N. Dalvi, Bhushan Mandhani, Shobhit Mathur, Christopher Ré, Dan Suciu: MYSTIQ: a system for finding more answers by using probabilities. SIGMOD Conference, 2005 Nilesh N. Dalvi, Dan Suciu: Efficient Query Evaluation on Probabilistic Databases. VLDB, 2004 Maximilian Dylla, Iris Miliaraki, Martin Theobald: Top-k Query Processing in Probabilistic Databases with Non-Materialized Views. ICDE, 2013 Norbert Fuhr, Thomas Rölleke: A Probabilistic Relational Algebra for the Integration of Information Retrieval and Database Systems. ACM Trans. Inf. Syst. 15(1), 1997 Timm Meiser, Maximilian Dylla, Martin Theobald: Interactive reasoning in uncertain RDF knowledge bases. CIKM, 2011 Ndapandula Nakashole, Mauro Sozio, Fabian Suchanek, Martin Theobald: Query-Time Reasoning in Uncertain RDF Knowledge Bases with Soft and Hard Rules. VLDS, 2012 Dan Suciu, Dan Olteanu, Christopher Ré, Christoph Koch: Probabilistic Databases (Synthesis Lectures on Data Management), Morgan & Claypool Publishers, 2012


Download ppt "Scalable RDF Data Management & SPARQL Query Processing"

Similar presentations


Ads by Google