1. Planning for the Web I Data Integration Dan Weld University of Washington June, 2003

2. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 2 Acknowledgements Oren Etzioni Alon Halevy Zachary Ives Rao Kambhampati Caveat

3. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 3 My Two Talks for Today Data Integration Providing uniform access to disparate data srcs AI meets DB Answering queries using views Execution in the face of uncertainty, latency Service Integration Invoking and composing web services Query and update Planning with incomplete information

4. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 4 Overview: Data Integration Motivation / intro Wrappers / information extraction Database review Integrating data sources Content, completeness, capabilities Reformulation algorithms: Bucket

5. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 5 What is Data Integration? Uniform (same query interface to all sources) Access to (queries; eventually updates too) Multiple (we want many, but 2 is hard too) Autonomous (DBA doesn’t report to you) Heterogeneous (data models are different) Structured (or at least semi-structured) Data Sources (not only databases). A system providing:

6. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 6 User enters query Formulate queries LycosExcite... Collate results Remove duplicates Post-process + rank Download? Present to user

7. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 7 Meta-? Web Search Shopping Product Reviews Chat Finder Columnists (e.g. jokes, sports, ….) Email Lookup Event Finder People Finder Restaurant Reviews Job Listings Classifieds Apartment + Real Estate

8. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration Intuition: Info Integration Info aggregation … on Steroids! Want agent such that User says what she wants Agent decides how & when to achieve it Example: Show me all reviews of movies starring Matt Damon that are currently playing in Seattle EbertIMDB Fandango Sidewalk

9. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 9 Info. Aggregation vs. Integration More complex queries Dynamic generation/optimization of execution plan Applicable to wider range of problems Much harder to implement efficiently prices of laptop with … sort store1store2storeN … movies in Seattle starring … join IMDBsidewalk rev2 … rev1 Join, sort aggregate revN

10. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration User must know which sites have relevant info User must go to each one in turn Slow: Sequential access takes time Confusing: Each site has a different interface User must manually integrate information Challenges

11. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 11 Practical Motivation Enterprise Business “dashboard’’; web-site construction. WWW Comparison shopping Portals integrating data from multiple sources B2B, electronic marketplaces Science and culture: Medical genetics: integrating genomic data Astrophysics: monitoring events in the sky. Environment: Puget Sound Regional Synthesis Model Culture: uniform access to all cultural databases produced by countries in Europe.

12. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 12 The Problem: Data Integration Uniform query capability across autonomous, heterogeneous data sources on LAN, WAN, or Internet

13. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 13 Current Solutions Mostly ad-hoc programming: create a special solution for every case; pay consultants a lot of money. Data warehousing: load all the data periodically into a warehouse. 6-18 months lead time Separates operational DBMS from decision support DBMS. (not only a solution to data integration). Performance is good; data may not be fresh. Need to clean, scrub you data.

14. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 14 Data Warehouse Architecture Data source Data source Data source Relational database (warehouse) User queries Data extraction programs Data extraction, cleaning/ scrubbing OLAP / Decision support/ Data cubes/ data mining

15. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 15 Warehouse Summary Pro Relatively simple Good performance (OLAP support) Mature technology (DB, ETL industries) Con Expensive Stale data Risky – most warehouse projects fail Rigid architecture Fixed schema Must know all queries ahead of time

16. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 16 Architecture for Virtual Integration Leave the data in the sources. When a query comes in: 1)Determine which sources are relevant to query. 2)Break query into sub-queries for each source. 3)Get answers from sources 4)Combine. Data is fresh. Challenge: performance.

17. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 17 Virtual Integration Architecture Data source wrapper Data source wrapper Data source wrapper Sources can be: relational, hierarchical (IMS), structured files, web sites. Mediator: User queries Mediated schema Data source catalog Reformulator Optimizer Execution engine

18. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 18 Research Projects Garlic (IBM), Information Manifold (AT&T) Tsimmis, InfoMaster (Stanford) Internet Softbot/Razor/Tukwila (U Wash.) Hermes (Maryland) Telegraph / Eddies (UC Berkeley) Niagara (Univ Wisconsin) DISCO, Agora (INRIA, France) SIMS/Ariadne (USC/ISI) Emerac/Havasu (ASU)

19. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 19 Industry Nimble Technology Enosys Markets IBM BEA

20. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 20 Dimensions to Consider How many sources are we accessing? How autonomous are they? Meta-data about sources? Is the data structured? Queries or also updates? Requirements: accuracy, completeness, performance, handling inconsistencies. Closed world assumption vs. open world?

21. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 21 Outline Motivation / introduction Wrappers / information extraction Database Review Integrating data sources Content, completeness, capabilities Reformulation algorithms: Bucket

22. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 22 Wrapper Programs Task to communicate with the data sources and do format translations. Built w.r.t. a specific source. Can sit either at the source or mediator. Often hard to build (very little science). Can be “intelligent” perform source-specific optimizations.

23. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 23 Example Introduction to DB Phil Bernstein Eric Newcomer Addison Wesley, 1999 Introduction to DB Phil Bernstein Eric Newcomer Addison Wesley 1999 Transform: into:

24. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 24 Wrapper Construction Use PERL, or Generate wrappers automatically Get training examples Human marks up selected pages with GUI tool Use shallow NLP to create features Favorite learning method HMMs, VS on prefix, postfix strings, ?? Boosting Co-training See research on information extraction

25. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 25 SemTag & Seeker WWW-03 Best Paper Prize Seeded with TAP ontology (72k concepts) And ~700 human judgments Crawled 264 million web pages Extracted 434 million semantic tags Automatically disambiguated

26. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 26 Outline Motivation / Introduction Wrappers / Information extraction Database Review Relational algebra, SQL, datalog Views Optimization (query planning) Integrating data sources Content, completeness, capabilities Reformulation algorithms: Bucket

27. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 27 Database (relational) Database Manager (DBMS) -Storage mgmt -Query processing -View management -(Transaction processing) Query (SQL) Answer (relation) Traditional Database Architecture

28. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 28 Relational Data: Terminology Name Price Category Manufacturer gizmo $19.99 gadgets GizmoWorks Power gizmo $29.99 gadgets GizmoWorks SingleTouch $149.99 photography Canon MultiTouch $203.99 household Hitachi tuple attribute Product Product(Name: string, Price: real, category: enum, Man: string ) schema relation (Arity=4)

29. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 29 Relational Algebra Operators tuple sets as input, new set as output Operations Union, Intersection, difference,.. Selection (  Projection (  ) Cartesian product (X) Join ( ) City Manufacturer GizmoWorks Canon Hitachi Tempe Kyoto Dayton

30. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 30 SQL: A query language for Relational Algebra Many standards out there: SQL92, SQL2, SQL3, SQL99 Select attributes From relations (possibly multiple, joined) Where conditions (selections) “Find companies that manufacture products bought by Joe Blow” SELECT Company.name FROM Company, Product WHERE Company.name=Product.maker AND Product.name IN (SELECT product FROM Purchase WHERE buyer = “Joe Blow”); Other features: aggregation, group-by etc.

31. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 31 Deductive Databases Tables viewed as predicates. Ops on tables expressed as “datalog” rules (Horn clauses, without function symbols) Enames(Name) :- Employe(Name, SSN) [Projection] Wealthy-Employee(Name) :- Employee(Name,SSN), Salary(SSN,Money),Money> 10 [Selection] Ed(Name, Dname) :- Employee(Name, SSN), E_Dependents(SSN, Dname) [Join] ERelated(Name,Dname) :- Ed(Name,Dname) ERelated(Name,Dname) :- Ed(Name,D1), ERelated(D1,D2) [Recursion]

32. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 32 More datalog terminology A datalog program is a set of datalog rules. A program with one rule is a conjunctive query. We distinguish EDB predicates and IDB predicates EDB’s are stored in the database, appear only in rule bodies IDB’s are intensionally defined, appear in both bodies and heads.

33. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 33 Views Views are relations, except that they are not physically stored. Uses: simplify complex queries, & define conceptually different views of DB for diff. users. Example: purchases of telephony products: CREATE VIEW telephony-purchases AS SELECT product, buyer, seller, store FROM Purchase, Product WHERE Purchase.product = Product.name AND Product.category = “telephony”

34. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 34 A Different View CREATE VIEW Seattle-view AS SELECT buyer, seller, product, store FROM Person, Purchase WHERE Person.city = “Seattle” AND Person.name = Purchase.buyer We can later use the view: SELECT name, store FROM Seattle-view, Product WHERE Seattle-view.product = Product.name AND Product.category = “shoes” What’s really happening when we query a view??

35. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 35 Materialized Views Views whose corresponding queries have been executed and the data is stored in a separate database Uses: Caching Issues Using views in answering queries Normally, views are available in addition to DB – (so, views are local caches) In information integration, views may be the only things we have access to. –An internet source specializing in tom hanks movies can be seen as a view on a database of all movies. –Except, there is no DB out there which contains all movies..

36. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 36 DB Inference Type 1 Evaluate query on instance of data Exponential in size of query Key is performance in size of data Type 2 Query containment Q1  Q2 No matter what data instances Logical entailment

37. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 37 Query Optimization Imperative execution plan: Declarative query Ideally: Want to find best plan. Practically: Avoid worst plans! Purchase Person Buyer=name City=‘seattle’ phone>’5430000’ buyer (Simple Nested Loops)  (Table scan)(Index scan) SELECT S.buyer FROM Purchase P, Person Q WHERE P.buyer=Q.name AND Q.city=‘seattle’ AND Q.phone > ‘5430000’ Inputs: the query available memory statistics about the data indexes, cardinalities, selectivity factors

38. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 38 Reserves Sailors sid=sid bid=100 rating > 5 sname (Simple Nested Loops) (On-the-fly) (Scan; write to temp T1) (Sort-Merge Join) Reserves Sailors sid=sid bid=100 sname (On-the-fly) rating > 5 (On-the-fly) SELECT S.sname FROM Reserves R, Sailors S WHERE R.sid=S.sid AND R.bid=100 AND S.rating>5 Goal of optimization : To find more efficient plans that compute the same answer. Reserves Sailors sid=sid bid=100 sname (On-the-fly) rating > 5 (Use hash index; do not write result to temp) Pipelined hash join

39. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration Relational Algebra Equivalences Allow us to choose different join orders and to ‘push’ selections and projections ahead of joins. ( Commute ) Projections: (Cascade) Joins: R (S T) (R S) T (Associative) (R S) (S R) (Commute) Selections:

40. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 40 Q(u,x) :- R(u,v), S(v,w), T(w,x) R S T Many ways of doing a single join R S Symmetric vs. asymmetric join operations Nested join, hash join, double pipe-lined hash join etc. Processing costs alone vs. proc. + transfer costs Get R and S together vs, get R, get just the tuples of S that will join with R (“semi-join”) Many orders in which to do the join (R join S) join T (S join R) join T (T join S) join R etc. All with different costs Optimizing Joins

41. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration Determining Join Order In principle, must consider all possible join orderings: As # of joins increases, # plans grows rapidly Must restrict search space. System-R: consider only left-deep join trees. This lets us generate all fully pipelined plans: Intermediate results not written to temporary files. (Not all left-deep trees are fully pipelined) B A C D B A C D C D B A

42. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration Enumeration of Left-Deep Plans Naïve approach: n! combinations. Principle of optimality: the best plan for the join of R 1,…R n-1 will be part of the best plan for the join of R 1,…,R n Enumerated using N passes (if N relations joined): Pass 1: Find best 1-relation plan for each relation. Pass 2: Find best way to join result of each 1-relation plan (as outer) to another relation. (All 2-relation plans.) Pass N: Find best way to join result of a (N-1)-relation plan (outer) to the N’th relation. (All N-relation plans.) For each subset of relations, retain only: Cheapest plan overall, plus Cheapest plan for each interesting order of the tuples.

43. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration Cost Estimation For each plan considered, estimate cost: Estimate cost of each operation in plan tree. Depends on input cardinalities. Estimate size of result for each op in tree! Use information about the input relations. Selectivity (Histograms) For selections and joins, assume independence of predicates. System R cost estimation approach. Very inexact, but works ok in practice. More sophisticated techniques known now.

44. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 44 Key Lessons in Optimization Classic planning / execution scenario Uncertainty / replanning key for data integration Main points Disk IO as cost metric Algebraic rules / use in query transformation.. Join ordering via dynamic programming Estimating cost of plans Size of intermediate results.

45. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 45 Integrator vs. DBMS No common schema Sources with heterogeneous schemas Semi-structured sources Legacy Sources Not relational-complete Access/process limitations Autonomous sources Uncontrolled source content overlap Lack of source statistics Tradeoffs: plan cost, coverage, quality, … Multi-objective cost models Unpredictable run-time behavior Makes query execution hard Reprise

46. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 46 Outline Motivation Wrappers / information extraction Database Review Integrating data sources Content, completeness, capabilities Reformulation algorithms: Bucket

47. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration Remnder: Info Integration Want agent such that User says what she wants Agent decides how & when to achieve it Example: Show me all reviews of movies starring Matt Damon that are currently playing in Seattle Ebert IMDB Fandango Sidewalk

48. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 48 Data Source Catalog Contains meta-information about sources: Logical source contents (books, new cars). Source capabilities (can answer SQL queries?) Source completeness (has all books). Physical properties of source and network. Statistics about the data (like in an RDBMS) Source reliability Mirror sources? Update frequency.

49. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 49 Content Descriptions User queries refer to the mediated schema. Source data is stored in a local schema. Content descriptions provide semantic mappings between different schemas. Data integration system uses the descriptions to translate user queries into queries on the sources.

50. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 50 Desiderata for Source Descriptions Expressive power: distinguish between sources with closely related data. Enable pruning of access to irrelevant sources. Easy addition: make it easy to add new data sources. Reformulation: be able to reformulate a user query into a query on the sources efficiently and effectively.

51. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 51 Reformulation Problem Given: A query Q posed over the mediated schema Descriptions of the data sources Find: A query Q’ over the data source relations, such that: Q’ provides only correct answers to Q, and Q’ provides all possible answers from to Q given the sources.

52. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 52 Approaches to Specifying Source Descriptions Global-as-view: express the mediated schema relations as a set of views over the data source relations Local-as-view: express the source relations as views over the mediated schema. Can be combined with no additional cost.

53. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 53 Global-as-View Mediated schema: Movie(title, dir, year, genre), Schedule(cinema, title, time). Create View Movie AS select * from S1 [S1(title,dir,year,genre)] union select * from S2 [S2(title, dir,year,genre)] union [S3(title,dir), S4(title,year,genre)] select S3.title, S3.dir, S4.year, S4.genre from S3, S4 where S3.title=S4.title

54. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 54 Global-as-View: Example 2 Mediated schema: Movie(title, dir, year, genre), Schedule(cinema, title, time). Create View Movie AS [S1(title,dir,year)] select title, dir, year, NULL from S1 union [S2(title, dir,genre)] select title, dir, NULL, genre from S2

55. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 55 Global-as-View: Example 3 Mediated schema: Movie(title, dir, year, genre), Schedule(cinema, title, time). Source S4: S4(cinema, genre) Create View Movie AS select NULL, NULL, NULL, genre from S4 Create View Schedule AS select cinema, NULL, NULL from S4. But what if we want to find which cinemas are playing comedies?

56. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 56 Global-as-View Summary  Very easy conceptually. Query reformulation  view unfolding.  Can build hierarchies of mediated schemas.  Sometimes loose information. Not always natural.  Adding sources is hard. Need to consider all other sources that are available. May need to modify every global view defn

57. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 57 Local-as-View: example 1 Mediated schema: Movie(title, dir, year, genre), Schedule(cinema, title, time). Create Source S1 AS [S1(title, dir, year, genre)] select * from Movie Create Source S3 AS [S3(title, dir)] select title, dir from Movie Create Source S5 AS [S5(title, dir, year)] select title, dir, year from Movie where year > 1960 AND genre=“Comedy”

58. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 58 Local-as-View: Example 2 Mediated schema: Movie(title, dir, year, genre), Schedule(cinema, title, time). Source S4: S4(cinema, genre) Create Source S4 select cinema, genre from Movie m, Schedule s where m.title=s.title. Now if we want to find which cinemas are playing comedies, there is hope!

59. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 59 Local-as-View Summary Very flexible. You have the power of the entire query language to define the contents of the source. Hence, can easily distinguish between contents of closely related sources. Adding sources is easy: They’re independent of each other. Query reformulation: Answering queries using views!

60. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 60 The General Problem Given a set of views V1,…,Vn, and a query Q, can we answer Q using only the answers to V1,…,Vn? Many, many papers on this problem. Great survey on the topic: (Halevy, 2001). The best performing algorithm: MiniCon (Pottinger & Levy, 2000).

61. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 61 Movie, Schedule Example Query: Q(x):- r 1 (x,y) & r 2 (y,x) Views: V 1 (a):-r 1 (a,b) V 2 (d):-r 2 (c,d) V 3 (f):- r 1 (f,g) & r 2 (g,f) Create Source V 1 as select a from r 1 Create Source V3 as select r1.arg1 from r1, r2 where r1.arg1 = r2.arg2

62. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 62 Bucket Algorithm (1) ¶1) For each subgoal in the query, place relevant views in the subgoal’s bucket Query: Q(x):- r 1 (x,y) & r 2 (y,x) Views: V 1 (a):-r 1 (a,b) V 2 (d):-r 2 (c,d) V 3 (f):- r 1 (f,g) & r 2 (g,f) r 1 (x,y) r 2 (y,x) V 1 (x),V 3 (x) V 2 (x), V 3 (x) Buckets:

63. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 63 Bucket Algorithm (2) 2) For every combo in the Cartesian product of the buckets, “check containment of the query” Candidate rewritings: Q’ 1 (x) :- V 1 (x) & V 2 (x) Q’ 2 (x) :- V 1 (x) & V 3 (x) Q’ 3 (x) :- V 3 (x) & V 2 (x) Q’ 4 (x) :- V 3 (x) & V 3 (x) r 1 (x,y) V 1 (x),V 3 (x) r 2 (y,x) V 2 (x), V 3 (x) Bucket Algorithm will check all possible combinations r 1 (x,y)r 2 (y,x) Q(x):-r 1 (x,y) & r 2 (y,x) Q’ 4 (x):-r 1 (x,a) & r 2 (a,x) & r 1 (x,b) & r 2 (b,x) 

64. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 64 Modeling Source Capabilities Negative capabilities: A web site may require certain inputs (in an HTML form). Positive capabilities: A source may be an ODBC compliant system. Need to decide placement of operations according to capabilities. Problem: how to describe and exploit source capabilities.

65. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 65 & cites(x, y) Example #1: Access Patterns Mediated schema relation: cites(paper1, paper2) Create Source S1 as S1($x,y) :- cites(x, y) select * from cites given paper1 Create Source S2 as S2(x) :- cites(x, y) select paper1 from cites Query: Q(x) :- cites(x, “W03”) Select paper1 from cites where paper2=“W03”

66. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 66 Example #2: Access Patterns Create Source S1 as select * from Cites given paper1 Create Source S2 as select paperID from UW-Papers Create Source S3 as select paperID from AwardPapers given paperID Query: select * from AwardPapers

67. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 67 Example #2: Solutions Can’t go directly to S3 (it requires a binding). Can go to S1, get UW papers, and check if they’re in S3. Can go to S1, get UW papers, feed them to S2, and then check if they’re in S3. Can go to S1, feed results into S2, feed results into S2 again, then check if they’re in S3. Note: we can’t a priori decide when to stop. Need recursive query processing.

68. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 68 Algorithms Inverse Rules [Duschka & Genesereth] Minicon [Pottinger & Levy]

69. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 69 Complexity of finding maximally- contained plans Complexity depends on sources, not query Sources as unions of conjunctive queries (NP-hard) Disjunctive descriptions Sources as recursive queries (Undecidable) True source contents Advertised description

70. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 70 Matching Objects Across Sources How do I know that D. Weld in source 1 is the same as Daniel S. Weld in source 2? If  uniform keys across sources, easy. If not: Domain specific solutions (e.g., maybe look at the address, …). Use IR techniques (Cohen, 98). Judge similarity as you would between documents. Use concordance tables. These are time-consuming to build, but you can then sell them for lots of money.

71. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 71 Schema Mapping Problem Types of structures: Database schemas, XML DTDs, ontologies, …, Input: Two (or more) structures, S 1 and S 2 (perhaps) Data instances for S 1 and S 2 Background knowledge Output: A mapping between S 1 and S 2 Should enable translating between data instances.

72. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 72 Semantic Mappings between Schemas Source schemas = XML DTDs house location contact house address name phone num-baths full-bathshalf-baths contact-info agent-name agent-phone 1-1 mapping non 1-1 mapping

73. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 73 Summary Data Integration Providing uniform access to disparate data srcs AI meets DB Modeling Data Sources GAV, LAV Query Reformulation Answering queries using views Bucket algorithm [Inverse rules, Minicon] Schema Matching

74. © Daniel S. Weld, PLANET 2003 Tutorial on Data Integration 74 Next Talk Executing data integration plans Variable latency, Poor info on size & speed of remote sources  Adaptive execution Service integration Invoking and composing web services Query and update Planning with incomplete information