Datalog and Data Integration Zachary G. Ives University of Pennsylvania CIS 550 – Database & Information Systems November 12, 2007 LSD Slides courtesy AnHai Doan

2 An Important Set of Questions Views are incredibly powerful formalisms for describing how data relates: fn: rel  …  rel  rel (Or XML  XML  XML, or rel  rel  XML,...) Can I define a view recursively?  Why might this be useful in the XML construction case? When should the recursion stop? Suppose we have two views, v 1 and v 2  How do I know whether they represent the same data?  If v 1 is materialized, can we use it to compute v 2 ?  This is fundamental to query optimization and data integration, as we’ll see later

3 Reasoning about Queries and Views SQL or XQuery are a bit too complex to reason about directly  Some aspects of it make reasoning about SQL queries undecidable We need an elegant way of describing views (let’s assume a relational model for now)  Should be declarative  Should be less complex than SQL  Doesn’t need to support all of SQL – aggregation, for instance, may be more than we need

4 Let’s Go Back a Few Weeks… Domain Relational Calculus Queries have form: { | p } Predicate: boolean expression over x1,x2, …, xn  We have the following operations:  Rx i op x j x i op constconst op x i  x i. p  x j. p p  q, p  q  p, p  q where op is , , , , ,  and x i,x j,… are domain variables; p,q are predicates  Recall that this captures the same expressiveness as the relational algebra domain variables predicate

5 A Similar Logic-Based Language: Datalog Borrows the flavor of the relational calculus but is a “real” query language  Based on the Prolog logic-programming language  A “datalog program” will be a series of if-then rules (Horn rules) that define relations from predicates

6 Datalog Terminology  An example datalog rule: idb(x,y)  r1(x,z), r2(z,y), z < 10  Irrelevant variables can be replaced by _ (anonymous var)  Extensional relations or database schemas (edbs) are relations only occurring in rules’ bodies, or as base relations:  Ground facts only have constants, e.g., r1(“abc”, 123)  Intensional relations (idbs) appear in the heads – these are basically views  Distinguished variables are the ones output in the head headsubgoals body

7 Datalog in Action  As in DRC, the output (head) consists of a tuple for each possible assignment of variables that satisfies the predicate  We typically avoid “  ” in Datalog queries: variables in the body are existential, ranging over all possible values  Multiple rules with the same relation in the head represent a union  We often try to avoid disjunction (“  ”) within rules  Let’s see some examples of datalog queries (which consist of 1 or more rules):  Given Professor(fid, name), Teaches(fid, serno, sem), Courses(serno, cid, desc), Student(sid, name)  Return course names other than CIS 550  Return the names of all people (professors or students)

8 Datalog is Relationally Complete We can map RA  Datalog:  Selection  p : p becomes a datalog subgoal  Projection  A : we drop projected-out variables from head  Cross-product r  s: q(A,B,C,D)  r(A,B),s(C,D)  Join r ⋈ s : q(A,B,C,D)  r(A,B),s(C,D), condition  Union r U s: q(A,B)  r(A,B) ; q(C, D) :- s(C,D)  Difference r – s: q(A,B)  r(A,B), : s(A,B) (If you think about it, DRC  Datalog is even easier) Great… But then why do we care about Datalog?

9 A Query We Can’t Answer in RA/TRC/DRC… Recall our example of a binary relation for graphs or trees (similar to an XML Edge relation): edge(from, to) If we want to know what nodes are reachable: reachable(F, T, 1) :- edge(F, T)distance 1 reachable(F, T, 2) :- edge(F, X), edge(X, T)dist. 2 reachable(F, T, 3) :- reachable(F, X, 2), edge(X, T)dist. 3 But how about all reachable paths? (Note this was easy in XPath over an XML representation -- //edge) (another way of writing  )

10 Recursive Datalog Queries Define a recursive query in datalog: reachable(F, T, 1) :- edge(F, T)distance 1 reachable(F, T, D + 1) :- reachable(F, X, D), edge(X, T)distance >1 What does this mean, exactly, in terms of logic?  There are actually three different (equivalent) definitions of semantics; we will use that of fixpoint:  We start with an instance of data, then derive all immediate consequences  We repeat as long as we derive new facts  In the RA, this requires a (restricted) while loop!  “Inflationary semantics” (which terminates in time polynomial in the size of the database!)

11 Our Query in RA + while (inflationary semantics, no negation) Datalog: reachable(F, T, 1) :- edge(F, T) reachable(F, T, D+1) :- reachable(F, X, D), edge(X, T) RA procedure with while: reachable += edge ⋈ literal1 while change { reachable +=  F, T, D (  F  X (edge) ⋈  T  X,D  D0 (reachable) ⋈ add1 ) } Note literal1(F,1) and add1(D0,D) are actually arithmetic and literal functions modeled here as relations.

12 A Special Type of Query: Conjunctive Queries A single Datalog rule with no “ ,” “ ,” “  ” can express select, project, and join – a conjunctive query  Conjunctive queries are possible to reason about statically  (Note that we can write CQ’s in other languages, e.g., SQL!) We know how to “minimize” conjunctive queries An important simplification that can’t be done for general SQL We can test whether one conjunctive query’s answers always contain another conjunctive query’s answers (for ANY instance)  Why might this be useful?

13 Example of Containment Suppose we have two queries: q1(S,C) :- Student(S, N), Takes(S, C), Course(C, X), inCIS(C), Course(C, “DB & Info Systems”) q2(S,C) :- Student(S, N), Takes(S, C), Course(C, X) Intuitively, q1 must contain the same or fewer answers vs. q2:  It has all of the same conditions, except one extra conjunction (i.e., it’s more restricted)  There’s no union or any other way it can add more data We can say that q2 contains q1 because this holds for any instance of our DB {Student, Takes, Course}

14 Wrapping up Datalog… We’ve seen a new language, Datalog  It’s basically a glorified DRC with a special feature, recursion  It’s much cleaner than SQL for reasoning about  … But negation (as in the DRC) poses some challenges We’ve seen that a particular kind of query, the conjunctive query, is written naturally in Datalog  Conjunctive queries are possible to reason about  We can minimize them, or check containment  Conjunctive queries are very commonly used in our next problem, data integration

15 A Problem  We’ve seen that even with normalization and the same needs, different people will arrive at different schemas  In fact, most people also have different needs!  Often people build databases in isolation, then want to share their data  Different systems within an enterprise  Different information brokers on the Web  Scientific collaborators  Researchers who want to publish their data for others to use  This is the goal of data integration: tie together different sources, controlled by many people, under a common schema

16 Building a Data Integration System Create a middleware “mediator” or “data integration system” over the sources  Can be warehoused (a data warehouse) or virtual  Presents a uniform query interface and schema  Abstracts away multitude of sources; consults them for relevant data  Unifies different source data formats (and possibly schemas)  Sources are generally autonomous, not designed to be integrated  Sources may be local DBs or remote web sources/services  Sources may require certain input to return output (e.g., web forms): “binding patterns” describe these

17 Data Integration System / Mediator Typical Data Integration Components Mediated Schema Wrapper Source Relations Mappings in Catalog Source Catalog QueryResults

18 Typical Data Integration Architecture Reformulator Query Processor Source Catalog Wrapper Query Query over sources Source Descrs. Queries + bindings Data in mediated format Results

19 Challenges of Mapping Schemas In a perfect world, it would be easy to match up items from one schema with another  Every table would have a similar table in the other schema  Every attribute would have an identical attribute in the other schema  Every value would clearly map to a value in the other schema Real world: as with human languages, things don’t map clearly!  May have different numbers of tables – different decompositions  Metadata in one relation may be data in another  Values may not exactly correspond  It may be unclear whether a value is the same

20 Different Aspects to Mapping Schema matching / ontology alignment How do we find correspondences between attributes? Entity matching / deduplication / record linking / etc. How do we know when two records refer to the same thing? Mapping definition  How do we specify the constraints or transformations that let us reason about when to create an entry in one schema, given an entry in another schema? Let’s see one influential approach to schema matching…

21 The LSD (Learning Source Descriptions) System Suppose user wants to integrate 100 data sources 1.User:  manually creates mappings for a few sources, say 3  shows LSD these mappings 2.LSD learns from the mappings  “Multi-strategy” learning incorporates many types of info in a general way  Knowledge of constraints further helps 3.LSD proposes mappings for remaining 97 sources

22 listed-price $250,000 $110, address price agent-phone description Example location Miami, FL Boston, MA... phone (305) (617) comments Fantastic house Great location... realestate.com location listed-price phone comments Schema of realestate.com If “fantastic” & “great” occur frequently in data values => description Learned hypotheses price $550,000 $320, contact-phone (278) (617) extra-info Beautiful yard Great beach... homes.com If “phone” occurs in the name => agent-phone Mediated schema

23 LSD’s Multi-Strategy Learning Use a set of base learners  Each exploits well certain types of information:  Name learner looks at words in the attribute names  Naïve Bayes learner looks at patterns in the data values  Etc. Match schema elements of a new source  Apply the base learners  Each returns a score  For different attributes one learner is more useful than another  Combine their predictions using a meta-learner Meta-learner  Uses training sources to measure base learner accuracy  Weighs each learner based on its accuracy

24 Boston, MA $110,000 (617) Great location Miami, FL $250,000 (305) Fantastic house Training the Learners Naive Bayes Learner (location, address) (listed-price, price) (phone, agent-phone) (comments, description)... (“Miami, FL”, address) (“$ 250,000”, price) (“(305) ”, agent-phone) (“Fantastic house”, description)... realestate.com Name Learner address price agent-phone description Schema of realestate.com Mediated schema location listed-price phone comments

25 Beautiful yard Great beach Close to Seattle (278) (617) (512) Seattle, WA Kent, WA Austin, TX Applying the Learners Name Learner Naive Bayes Meta-Learner (address,0.8), (description,0.2) (address,0.6), (description,0.4) (address,0.7), (description,0.3) (address,0.6), (description,0.4) Meta-Learner Name Learner Naive Bayes (address,0.7), (description,0.3) (agent-phone,0.9), (description,0.1) address price agent-phone description Schema of homes.com Mediated schema area day-phone extra-info

26 Putting It All Together: LSD System L1L1 L2L2 LkLk Mediated schema Source schemas Data listings Training data for base learners Constraint Handler Mapping Combination User Feedback Domain Constraints Matching PhaseTraining Phase

27 Mappings between Schemas LSD provides attribute correspondences, but not complete mappings Mappings generally are posed as views: define relations in one schema (typically either the mediated schema or the source schema), given data in the other schema  This allows us to “restructure” or “recompose + decompose” our data in a new way We can also define mappings between values in a view  We use an intermediate table defining correspondences – a “concordance table”  It can be filled in using some type of code, and corrected by hand

28 A Few Mapping Examples  Movie(Title, Year, Director, Editor, Star1, Star2)  PieceOfArt(ID, Artist, Subject, Title, TypeOfArt)  MotionPicture(ID, Title, Year) Participant(ID, Name, Role) CustIDCustName 1234Smith, J. PennIDEmpName 46732John Smith PieceOfArt(I, A, S, T, “Movie”) :- Movie(T, Y, A, _, S1, S2), ID = T || Y, S = S1 || S2 Movie(T, Y, D, E, S1, S2) :- MotionPicture(I, T, Y), Participant(I, D, “Dir”), Participant(I, E, “Editor”), Participant(I, S1, “Star1”), Participant(I, S2, “Star2”) T1 T2 Need a concordance table from CustIDs to PennIDs

29 Two Important Approaches  TSIMMIS [Garcia-Molina+97] – Stanford  Focus: semistructured data (OEM), OQL-based language (Lorel)  Creates a mediated schema as a view over the sources  Spawned a UCSD project called MIX, which led to a company now owned by BEA Systems  Other important systems of this vein: Penn  Information Manifold [Levy+96] – AT&T Research  Focus: local-as-view mappings, relational model  Sources defined as views over mediated schema  Requires a special  Led to peer-to-peer integration approaches (Piazza, etc.)  Focus: Web-based queriable sources

30 TSIMMIS  One of the first systems to support semi-structured data, which predated XML by several years: “OEM”  An instance of a “global-as-view” mediation system  We define our global schema as views over the sources

31 XML vs. Object Exchange Model Bernstein Newcomer Principles of TP Chamberlin DB2 UDB O1: book { O2: author { Bernstein } O3: author { Newcomer } O4: title { Principles of TP } } O5: book { O6: author { Chamberlin } O7: title { DB2 UDB } }

32 Queries in TSIMMIS Specified in OQL-style language called Lorel  OQL was an object-oriented query language that looks like SQL  Lorel is, in many ways, a predecessor to XQuery Based on path expressions over OEM structures: select book where book.title = “DB2 UDB” and book.author = “Chamberlin” This is basically like XQuery, which we’ll use in place of Lorel and the MSL template language. Previous query restated = for $b in AllData()/book where $b/title/text() = “DB2 UDB” and $b/author/text() = “Chamberlin” return $b

33 Query Answering in TSIMMIS Basically, it’s view unfolding, i.e., composing a query with a view  The query is the one being asked  The views are the MSL templates for the wrappers  Some of the views may actually require parameters, e.g., an author name, before they’ll return answers  Common for web forms (see Amazon, Google, …)  XQuery functions (XQuery’s version of views) support parameters as well, so we’ll see these in action

34 A Wrapper Definition in MSL Wrappers have templates and binding patterns ($X) in MSL: B :- B: }> // $$ = “select * from book where author=“ $X //  This reformats a SQL query over Book(author, year, title) In XQuery, this might look like: define function GetBook($x AS xsd:string) as book { for $b in sql(“Amazon.DB”, “select * from book where author=‘” + $x +”’”) return {$b/title} $x } book title author … … … The union of GetBook’s results is unioned with others to form the view Mediator()

35 How to Answer the Query Given our query: for $b in Mediator()/book where $b/title/text() = “DB2 UDB” and $b/author/text() = “Chamberlin” return $b Find all wrapper definitions that:  Contain output enough “structure” to match the conditions of the query  Or have already tested the conditions for us!

36 Query Composition with Views We find all views that define book with author and title, and we compose the query with each: define function GetBook($x AS xsd:string) as book { for $b in sql(“Amazon.DB”, “select * from book where author=‘” + $x + “’”) return {$b/title} {$x} } for $b in Mediator()/book where $b/title/text() = “DB2 UDB” and $b/author/text() = “Chamberlin” return $b book title author … …

37 Matching View Output to Our Query’s Conditions  Determine that $b/book/author/text()  $x by matching the pattern on the function’s output: define function GetBook($x AS xsd:string) as book { for $b in sql(“Amazon.DB”, “select * from book where author=‘” + $x + “’”) return { $b/title } {$x} } let $x := “Chamberlin” for $b in GetBook($x)/book where $b/title/text() = “DB2 UDB” return $b book title author … …

38 The Final Step: Unfolding let $x := “Chamberlin” for $b in ( for $b’ in sql(“Amazon.com”, “select * from book where author=‘” + $x + “’”) return { $b/title } {$x} )/book where $b/title/text() = “DB2 UDB” return $b  How do we simplify further to get to here? for $b in sql(“Amazon.com”, “select * from book where author=‘Chamberlin’”) where $b/title/text() = “DB2 UDB” return $b

39 Virtues of TSIMMIS  Early adopter of semistructured data, greatly predating XML  Can support data from many different kinds of sources  Obviously, doesn’t fully solve heterogeneity problem  Presents a mediated schema that is the union of multiple views  Query answering based on view unfolding  Easily composed in a hierarchy of mediators

40 Limitations of TSIMMIS’ Approach Some data sources may contain data with certain ranges or properties  “Books by Aho”, “Students at UPenn”, …  If we ask a query for students at Columbia, don’t want to bother querying students at Penn…  How do we express these? Mediated schema is basically the union of the various MSL templates – as they change, so may the mediated schema