Submitted by: Deepti Kundu Submitted to: Dr.T.Y.Lin Summarization – CS 257 Chapter – 21 (Information Integration) Database Systems: The Complete Book Submitted by: Deepti Kundu Submitted to: Dr.T.Y.Lin

21.1 Introduction to Information Integration Need for Information Integration All the data in the world could put in a single database (ideal database system) In the real world (impossible for a single database): databases are created independently hard to design a database to support future use 2

University Database Registrar: to record student and grade Bursar: to record tuition payments by students Human Resources Department: to record employees Other department….

Inconvenient Record grades for students who pay tuition Want to swim in SJSU aquatic center for free in summer vacation? (all the cases above cannot achieve the function by a single database) Solution: one database

How to integrate Start over build one database: contains all the legacy databases; rewrite all the applications result: painful Build a layer of abstraction (middleware) on top of all the legacy databases this layer is often defined by a collection of classes BUT…

Heterogeneity Problem What is Heterogeneity Problem Aardvark Automobile Co. 1000 dealers has 1000 databases to find a model at another dealer can we use this command: SELECT * FROM CARS WHERE MODEL=“A6”;

Type of Heterogeneity Communication Heterogeneity Query-Language Heterogeneity Schema Heterogeneity Data type difference Value Heterogeneity Semantic Heterogeneity

21.2 Modes of Information Integration Federations The simplest architecture for integrating several DBs One to one connections between all pairs of DBs n DBs talk to each other, n(n-1) wrappers are needed Good when communications between DBs are limited Wrapper a software translates incoming queries and outgoing answers. In a result, it allows information sources to conform to some shared schema.

Federations Diagram DB1 DB2 2 Wrappers 2 Wrappers 2 Wrappers A federated collection of 4 DBs needs 12 components to translate queries from one to another.

Data Warehouse Sources are translated from their local schema to a global schema and copied to a central DB. User transparent: user uses Data Warehouse just like an ordinary DB User is not allowed to update Data Warehouse

Warehouse Diagram Warehouse Combiner Extractor Extractor Source 1 User query result Warehouse Combiner Extractor Extractor Source 1 Source 2

Construct Data Warehouse There are mainly 3 ways to constructing the data in the warehouse: 1) Periodically reconstructed from the current data in the sources, once a night or at even longer intervals. Advantages: simple algorithms. Disadvantages: need to shut down the warehouse; data can become out of date. 2) Updated periodically based on the changes (i.e. each night) of the sources. involve smaller amounts of data. (important when warehouse is large and needs to be modified in a short period) the process to calculate changes to the warehouse is complex.

3) Changed immediately, in response to each change or a small set of changes at one or more of the sources. Advantages: data won’t become out of date. Disadvantages: requires too much communication, therefore, it is generally too expensive. (practical for warehouses whose underlying sources changes slowly.)

Mediators Virtual warehouse, which supports a virtual view or a collection of views, that integrates several sources. Mediator doesn’t store any data. Mediators’ tasks: 1)receive user’s query, 2)send queries to wrappers, 3)combine results from wrappers, 4)send the final result to user.

A Mediator diagram Mediator Wrapper Wrapper Source 1 Source 2 User query Result Mediator Query Result Wrapper Wrapper Source 1 Source 2

21.3 Wrappers in Mediator-Based Systems Intro Templates for Query patterns Wrapper Generator Filter

Wrappers in Mediator-based Systems More complicated than that in most data warehouse system. Able to accept a variety of queries from the mediator and translate them to the terms of the source. Communicate the result to the mediator. How to design a wrapper? Classify the possible queries that the mediator can ask into templates, which are queries with parameters that represent constants.

Wrapper Generators Filter Templates for Query Patterns: Use notation T=>S to express the idea that the template T is turned by the wrapper into the source query S. The wrapper generator creates a table holds the various query patterns contained in the templates. The source queries that are associated with each. Filter Have a wrapper filter to supporting more queries.

A driver is used in each wrapper, the task of the driver is to: Accept a query from the mediator. Search the table for a template that matches the query. The source query is sent to the source, again using a “plug-in” communication mechanism. The response is processed by the wrapper.

21.4 Capability Based Optimization Introduction Typical DBMS estimates the cost of each query plan and picks what it believes to be the best Mediator – has knowledge of how long its sources will take to answer Optimization of mediator queries cannot rely on cost measure alone to select a query plan Optimization by mediator follows capability based optimization 20

21.4.1 The Problem of Limited Source Capabilities Many sources have only Web Based interfaces Web sources usually allow querying through a query form E.g. Amazon.com interface allows us to query about books in many different ways. But we cannot ask questions that are too general E.g. Select * from books; 21

(con’t) Reasons why a source may limit the ways in which queries can be asked Earliest database did not use relational DBMS that supports SQL queries Indexes on large database may make certain queries feasible, while others are too expensive to execute Security reasons 22

21.4.2 A Notation for Describing Source Capabilities For relational data, the legal forms of queries are described by adornments Adornments – Sequences of codes that represent the requirements for the attributes of the relation, in their standard order f(free) – attribute can be specified or not b(bound) – must specify a value for an attribute but any value is allowed u(unspecified) – not permitted to specify a value for a attribute 23

(cont’d) c[S](choice from set S) means that a value must be specified and value must be from finite set S. o[S](optional from set S) means either do not specify a value or we specify a value from finite set S A prime (f’) specifies that an attribute is not a part of the output of the query A capabilities specification is a set of adornments A query must match one of the adornments in its capabilities specification 24

21.4.3 Capability-Based Query-Plan Selection Given a query at the mediator, a capability based query optimizer first considers what queries it can ask at the sources to help answer the query The process is repeated until: Enough queries are asked at the sources to resolve all the conditions of the mediator query and therefore query is answered. Such a plan is called feasible. We can construct no more valid forms of source queries, yet still cannot answer the mediator query. It has been an impossible query. 25

(cont’d) The simplest form of mediator query where we need to apply the above strategy is join relations E.g we have sources for dealer 2 Autos(serial, model, color) Options(serial, option) Suppose that ubf is the sole adornment for Auto and Options have two adornments, bu and uc[autoTrans, navi] Query is – find the serial numbers and colors of Gobi models with a navigation system 26

21.4.4 Adding Cost-Based Optimization Mediator’s Query optimizer is not done when the capabilities of the sources are examined Having found feasible plans, it must choose among them Making an intelligent, cost based query optimization requires that the mediator knows a great deal about the costs of queries involved Sources are independent of the mediator, so it is difficult to estimate the cost 27

21.5 Optimizing Mediator Queries Chain algorithm – a greed algorithm that finds a way to answer the query by sending a sequence of requests to its sources. Will always find a solution assuming at least one solution exists. The solution may not be optimal.

21.5.1 Simplified Adornment Notation A query at the mediator is limited to b (bound) and f (free) adornments. We use the following convention for describing adornments: Nameadornments (attributes) where: name is the name of the relation the number of adornments = the number of attributes

21.5.2 Obtaining Answers for Subgoals Rules for subgoals and sources: Suppose we have the following subgoal: Rx1x2…xn(a1, a2, …, an), and source adornments for R are: y1y2…yn. If yi is b or c[S], then xi = b. If xi = f, then yi is not output restricted. The adornment on the subgoal matches the adornment at the source: If yi is f, u, or o[S] and xi is either b or f.

21.5.3 The Chain Algorithm Maintains 2 types of information: An adornment for each subgoal. A relation X that is the join of the relations for all the subgoals that have been resolved. Initially, the adornment for a subgoal is b iff the mediator query provides a constant binding for the corresponding argument of that subgoal. Initially, X is a relation over no attributes, containing just an empty tuple.

(cont’d) First, initialize adornments of subgoals and X. Then, repeatedly select a subgoal that can be resolved. Let Rα(a1, a2, …, an) be the subgoal: Wherever α has a b, we shall find the argument in R is a constant, or a variable in the schema of R. Project X onto its variables that appear in R.

(cont’d) For each tuple t in the project of X, issue a query to the source as follows (β is a source adornment). If a component of β is b, then the corresponding component of α is b, and we can use the corresponding component of t for source query. If a component of β is c[S], and the corresponding component of t is in S, then the corresponding component of α is b, and we can use the corresponding component of t for the source query. If a component of β is f, and the corresponding component of α is b, provide a constant value for source query.

(cont’d) If a component of β is u, then provide no binding for this component in the source query. If a component of β is o[S], and the corresponding component of α is f, then treat it as if it was a f. If a component of β is o[S], and the corresponding component of α is b, then treat it as if it was c[S]. Every variable among a1, a2, …, an is now bound. For each remaining unresolved subgoal, change its adornment so any position holding one of these variables is b.

(cont’d) Replace X with X πs(R), where S is all of the variables among: a1, a2, …, an. Project out of X all components that correspond to variables that do not appear in the head or in any unresolved subgoal. If every subgoal is resolved, then X is the answer. If every subgoal is not resolved, then the algorithm fails. α

21.5.4 Incorporating Union Views at the Mediator This implementation of the Chain Algorithm does not consider that several sources can contribute tuples to a relation. If specific sources have tuples to contribute that other sources may not have, it adds complexity. To resolve this, we can consult all sources, or make best efforts to return all the answers.

(cont’d) Consulting All Sources We can only resolve a subgoal when each source for its relation has an adornment matched by the current adornment of the subgoal. Less practical because it makes queries harder to answer and impossible if any source is down. Best Efforts We need only 1 source with a matching adornment to resolve a subgoal. Need to modify chain algorithm to revisit each subgoal when that subgoal has new bound requirements.

21.6 Local-as-View Mediators GAV: Global as view mediators are like view, it doesn’t exist physically, but piece of it are constructed by the mediator by asking queries LAV: Local as view mediators, defines the global predicates at the mediator, but we do not define these predicates as views of the source of data Global expressions are defined for each source involving global predicates that describe the tuple that source is able to produce and queries are answered at mediator by discovering all possible ways to construct the query using the views provided by sources

Motivation for LAV Mediators LAV mediators help us to discover how and when to use that source in a given query Example: Par(c,p)-> GAV of Par(c,p) gives information about the child and parent but does not give information of grandparents LAV Par(c,p) will help to get information of chlid-parent and even grandparent

Terminology for LAV Mediation It is in form of logic that serves as the language for defining views. Datalog is used which will remain common for the queries of mediator and source which is known as Conjunctive query. LAV has global predicates which are the subgoals of mediator queries Conjunctive queries defines the views which has unique view predicate and that view has Global predicates and associated with particular view.

Containment of Conjunctive Queries Conjunctive query S be the solution to the mediator Q, Expansion of S->E, produces same answers that Q produces, so, E subset Q. A containment mapping from Q to E is function Γ(x) is the ith argument of the head E. Add to Γ the rule that Γ(c) =c for any constant c. IF P(x1,x2,..xn) is a subgoal of Q, then P(Γ(x1), Γ(x2),.., Γ(xn)) is a subgoal of E.

Why Containment Mapping Test Works: Questions: If there is containment mapping, why must there be a containment of conjunctive queries? If there is containment, why must there be a containment mapping?

Finding Solutions to a Mediator Query Query Q, solutions S, Expansion E of S is contained in Q. “If a query Q has n subgoals, then any answer produced by any solution is also produced by a solution that has at most n subgoals. This is known by LMSS Theorem

Why the LMSS Theorem Holds Query Q with n subgoals and S with n subgoals, E of S must be contained in query Q, E is expansion of Q. S’ must be the solution got after removing all subgoals from S those are not the target of Q. E subset or equal to Q and also E’ is the expansion of S’. So, S is subser of S’ : identity mapping. Thus there is no need for solution s among the solution S among the solutions to query Q.

21.7 Entity Resolution Determining whether two records or tuples do or do not represent the same person, organization, place or other entity is called ENTITY RESOLUTION.

Deciding whether Records represent a Common Entity Two records represent the same individual if the two records have similar values for each of the fields associated with those records. It is not sufficient that the values of corresponding fields be identical because of following reasons: 1. Misspellings 2. Variant Names 3. Misunderstanding of Names 4. Evolution of Values 5. Abbreviations

Deciding Whether Records Represents a Common Entity - Edit Distance First approach to measure the similarity of records is Edit Distance. Values that are strings can be compared by counting the number of insertions and deletions of characters it takes to turn one string into another. So the records represent the same entity if their similarity measure is below a given threshold.

Deciding Whether Records Represents a Common Entity - Normalization To normalize records by replacing certain substrings by others. For instance: we can use the table of abbreviations and replace abbreviations by what they normally stand for. Once normalize we can use the edit distance to measure the difference between normalized values in the fields.

Merging Similar Records Merging means replacing two records that are similar enough to merge and replace by one single record which contain information of both. There are many merge rules: 1. Set the field in which the records disagree to the empty string. 2. (i) Merge by taking the union of the values in each field (ii) Declare two records similar if at least two of the three fields have a nonempty intersection.

Useful Properties of Similarity and Merge Functions The following properties say that the merge operation is a semi lattice : Idempotence : That is, the merge of a record with itself should surely be that record. Commutativity : If we merge two records, the order in which we list them should not matter. Associativity : The order in which we group records for a merger should not matter.

There are some other properties that we expect similarity relationship to have: Idempotence for similarity : A record is always similar to itself Commutativity of similarity : In deciding whether two records are similar it does not matter in which order we list them Representability : If r is similar to some other record s, but s is instead merged with some other record t, then r remains similar to the merger of s and t and can be merged with that record.

R-swoosh Algorithm for ICAR Records Input: A set of records I, similarity function and a merge function. Output: A set of merged records O. Method: O:= emptyset; WHILE I is not empty DO BEGIN Let r be any record in I; Find, if possible, some record s in O that is similar to r; IF no record s exists THEN move r from I to O ELSE BEGIN delete r from I; delete s from O; add the merger of r and s to I; END;

Other Approaches to Entity Resolution The other approaches to entity resolution are : Non ICAR Datasets : We can define a dominance relation r<=s that means record s contains all the information contained in record r.If so, then we can eliminate record r from further consideration. Clustering : Some time we group the records into clusters such that members of a cluster are in some sense similar to each other and members of different clusters are not similar. Partitioning : We can group the records, perhaps several times, into groups that are likely to contain similar records and look only within each group for pairs of similar records.