CSE 590DB: Database Seminar Autumn 2002: Meta Data Management Phil Bernstein Microsoft Research

Course Administration Course credit Course credit 1 credit for reading the papers and showing up 1 credit for reading the papers and showing up 2 credits for leading a discussion 2 credits for leading a discussion I’m giving today’s lecture I’m giving today’s lecture So I get 2 credits So I get 2 credits Next week, we have a visitor Next week, we have a visitor Wang-Chiew Tan Wang-Chiew Tan Send me mail to sign up for later weeks Send me mail to sign up for later weeks

What’s Meta Data? DB defns, form defns, documents, interface defns, source code, help text, executables, icons, makefiles,... DB defns, form defns, documents, interface defns, source code, help text, executables, icons, makefiles,... Spec Table Defns Interface Defns Architecture View Defns ER Diagram Forms Bill Customer Update Marketing Inventory Authorize Credit Order Entry Schedule Delivery BusinessProcess Emp.Sal < Emp.Mgr.Sal Business Rules

The Schemas are the Data Plus relationships between the schemas Plus relationships between the schemas Depends-on, generated-from, authored-by, … Depends-on, generated-from, authored-by, … This isn’t like data-processing data that you store in relational databases This isn’t like data-processing data that you store in relational databases It’s more like the content of a SQL catalog It’s more like the content of a SQL catalog It’s inherently heterogeneous & distributed It’s inherently heterogeneous & distributed SQL doesn’t help a lot to manipulate it SQL doesn’t help a lot to manipulate it

What’s Meta Data Management? Generic mechanisms to help store, search, and manipulate meta data. Generic mechanisms to help store, search, and manipulate meta data. Main components Main components A database engine A database engine Information Models (i.e. meta models) Information Models (i.e. meta models) Schemas whose instances are meta data (i.e. models) Schemas whose instances are meta data (i.e. models) Tools Tools Generic – for any meta data application Generic – for any meta data application For Vertical Applications – usually design-time, but can be run-time For Vertical Applications – usually design-time, but can be run-time

Meta Data Manager Architecture select all cust emp dept dno dna Bill Customer Update Marketing Inventory Authorize Credit Order Entry Schedule Delivery Spec Tables Code Architecture Views ERD Forms Database Database System Database System Repository Mgr or OO DBMS Objects, properties Rich relationships Rich relationships Extensibility Extensibility Versioning Versioning Configurations Configurations Information Model Predefined types Generic Tools Browser Browser Scripting language Scripting language Data translators (import/export) Data translators (import/export) Model editor Model editor Model merge Model merge Component mgr Component mgr

Typical Usage Pattern Scanner (importer) Object-Oriented Structure Meta Data Source Database catalog, language introspection, modeling tool, prog. environment, … Meta Data Application Database design, impact analysis, data translation, data integration, view integration, message mapping, data whse loading, data migration, …. Generator (exporter) Code SQL DDL, interface defns, XSLT, Java, …

Meta Terminology meta-meta-meta data = meta-meta-model Definition of “object” meta-meta data = meta-model Definition of “Table” meta data = model Definition of (schema for) the Employee Table dataEmployee Table

Vertical Applications I Database design Database design Map ER model to SQL schema Map ER model to SQL schema Reverse engineer SQL schema to ER model Reverse engineer SQL schema to ER model DB Application development DB Application development Map SQL schema to default form Map SQL schema to default form Map business rule to SQL constraints and form validation code Map business rule to SQL constraints and form validation code Manage dependencies between code and schemas and forms Manage dependencies between code and schemas and forms

Vertical Applications II  Data translation Import source and target schemas Import source and target schemas Build a mapping between them Build a mapping between them Data translator interprets the mapping Data translator interprets the mapping  Schema integration Merge data sources into a global schema Merge data sources into a global schema View integration View integration Define use-case scenario Define use-case scenario Identify views for each use-case Identify views for each use-case Integrate views into a conceptual schema Integrate views into a conceptual schema

Vertical Applications III Message Mapping Message Mapping Map messages from one format to another Map messages from one format to another Data Warehousing Data Warehousing Import schemas of data sources Import schemas of data sources Identify overlapping attributes, etc. Identify overlapping attributes, etc. Build data cleaning scripts Build data cleaning scripts Build data transformation scripts Build data transformation scripts Enable data lineage tracing Enable data lineage tracing

Vertical Applications IV Data Migration Data Migration Import a schema and its modified version Import a schema and its modified version Build a mapping between them Build a mapping between them Data migration tool inteprets the mapping Data migration tool inteprets the mapping Scientific data management Scientific data management Merge schemas from related experiments Merge schemas from related experiments Manage transformations of experimental data Manage transformations of experimental data Track evolution of schemas and transformations Track evolution of schemas and transformations

Vertical Applications V Information Resource Management Information Resource Management Inventory control of schemas and apps for large enterprises Inventory control of schemas and apps for large enterprises Import & browse schemas and interfaces Import & browse schemas and interfaces Impact analysis Impact analysis Semantic query processing Semantic query processing Pose queries against conceptual model Pose queries against conceptual model Automatically map the query to DB schemas Automatically map the query to DB schemas

Vertical Applications VI Integrated CASE Integrated CASE Model the application in UML Model the application in UML Translate the UML to interfaces, schemas, etc. Translate the UML to interfaces, schemas, etc. Tools for consistency checking, test generation, impact analysis, …. Tools for consistency checking, test generation, impact analysis, …. Integrating customized applications Integrating customized applications Workflow design and management Workflow design and management Document management Document management Application configuration management Application configuration management

Course Topics Meta data applications Meta data applications Data translation [Popa et al] Data translation [Popa et al] Data integration [Spaccapietra & Parent] Data integration [Spaccapietra & Parent] Schema management Schema management Meta meta models [Noy et al] Meta meta models [Noy et al] Properties of mappings [Hull] Properties of mappings [Hull] Meta data mechanisms Meta data mechanisms Lineage tracing [Buneman et al.] (Wang-Chiew Tan) Lineage tracing [Buneman et al.] (Wang-Chiew Tan) Mapping generation [Atzeni & Torlone] Mapping generation [Atzeni & Torlone] Schema merging [Buneman, Davidson, Kosky] Schema merging [Buneman, Davidson, Kosky] Model Management [Bernstein] Model Management [Bernstein]

Model Management Model – a complex information structure, e.g., XML schema, SQL schema,.NET type def, UML model. Model – a complex information structure, e.g., XML schema, SQL schema,.NET type def, UML model. Mapping – a transformation from one model into another Mapping – a transformation from one model into another Map between two XML schemas Map between two XML schemas Map SQL schema to XML schema Map SQL schema to XML schema Map data source to data warehouse Map data source to data warehouse Map classes to data source defns Map classes to data source defns Model Management Algebra Model Management Algebra Match ( M 1, M 2, map ) Match ( M 1, M 2, map ) Merge ( M 1, M 2, map, M 3 ) Merge ( M 1, M 2, map, M 3 ) Compose ( map 1, map 2, map 3 ) Compose ( map 1, map 2, map 3 ) Copy, Delete, Update Copy, Delete, Update Diff, Apply, ModelGen Diff, Apply, ModelGen Select, Enumerate Select, Enumerate

Implementation Vision Implementation Vision Match Merge Compose Copy Apply … Model-Driven UI Generator Model Manager Object-Oriented Repository SQL DBMS Bill Customer Update Marketing Inventory Authorize Credit Order Entry Schedule Delivery select all cust emp dept dno dna Generic Tools Generic Tools Browser Browser Import/export Import/export Scripting Scripting Editors Editors Catalogs Catalogs Operation Speciali- zations Inferencing Engine            

Web pointers // // //research.microsoft.com/db/ModelMgt/ //research.microsoft.com/db/ModelMgt/