1. Data Warehouse success depends on metadata

2. Overview What is metadata? Why is it needed? Types of metadata
Metadata life cycle

3. Better end user data access and analysis tools can help users figure out how to get information they need out of the warehouse, but only good, easily accessible metadata can help them figure out what is available in the data warehouse
and how to ask for it.

4. Copyright © 1997, Enterprise Group, Ltd.
Data Warehouse Process
Data Characteristics
Raw Detail
No/Minimal History
Integrated
Scrubbed
History
Summaries
Targeted
Specialized (OLAP)
Source OLTP
Systems
Data Marts
Data
Warehouse
Design
Mapping
Extract
Scrub
Transform
Load
Index
Aggregation
Replication
Data Set Distribution
Access & Analysis
Resource Scheduling & Distribution
Meta Data
System Monitoring
Copyright © 1997, Enterprise Group, Ltd.

5. Meta Data Description Information about the data warehouse system
Content
Organizational
Structural
Management Information
Scheduling Information
Contact Information
Technical Information

6. Why Do You Need Meta Data?
Share resources
Users
Tools
Document system
Without metadata
Not Sustainable
Not able to fully utilize resource

7. Metadata Life Cycle
Collection - Identify metadata and capture into repository; automate
Maintenance - Put in place processes to synchronize metadata automatically with changing data architecture; automate
Deployment - Provide metadata to users in the right form and with the right tools; match metadata offered to specific needs of each audience

8. Metadata Collection Right metadata at the right time
Variety of collection strategies
Sources
potential sources of data for DW
external data
data structures
Data Models - enterprise data model start point
import from CASE tool
correlate enterprise and warehouse models

9. Metadata Collection Warehouse mappings Warehouse usage information
map operational data into warehouse data structure
Need record of logical connection used for mapping and transformation
Warehouse usage information
After roll out
What tables accessed, by whom and for what
What queries written
Capture nature of business problem or query

10. Maintaining Metadata Up to date with reality
Capture incremental changes

11. Metadata Deployment Warehouse developers need:
physical structure info for data sources
enterprise data model
warehouse data model
concerned with accuracy, completeness and flexibility of metadata
Need access to comprehensive impact analysis capabilities
Need to defend against accuracy & integrity questions

12. Meta Data
Types
Technical
Business / User
Levels
Core
Basic
Deluxe

13. Core Technical Meta Data
Source
Target
Algorithm

14. Basic Technical Meta Data
History of transformation changes
Business rules
Source program / system name
Source program author / owner
Extract program name & version
Extract program author / owner
Extract JCL / Script name
Extract JCL / Script author / owner
Load JCL / Script name

15. Basic Technical Meta Data (con’t)
Load JCL / Script author / owner
Load frequency
Extract dependencies
Transformation dependencies
Load dependencies
Load completion date / time stamp
Load completion record count
Load status

16. Deluxe Technical Meta Data
Source system platform
Source system network address
Source system support contact
Source system support phone / beeper
Target system platform
Target system network address
Target system support contact
Target system support phone / beeper
Etc.

17. Core Business Meta Data
Field / object description
Confidence level
Frequency of update

18. Basic Business Meta Data
Source system name
Valid entries (i.e. “There are three valid codes: A, B, C”)
Formats (i.e. Contract Date: 82/4/30)
Business rules used to calculate or derive the data
Changes in business rules over time

19. Deluxe Business Meta Data
Data owner
Data owner contact information
Typical uses
Level of summarization
Related fields / objects
Existing queries / reports using this field / object
Estimated size (tables / objects)

20. Amount of Meta Data How much Meta Data do I need?
As much as you can support!

21. The Meta Data Conundrum
Meta Data is absolutely required for success
Meta Data is 99% Manual
Cold, Hard Reality
5,000 data mart fields
7 manually populated and maintained meta data fields
35,000 total manual meta data fields
Are you ready for this, forever?
Copyright © 1997, Enterprise Group, Ltd.

22. The Meta Data Conundrum
Can you support 35,000 Meta Data fields?
Calculate available ongoing resources
Commit only to what you can maintain
You MUST deliver core, probably some basic to be viable

23. Meta Data Functions - Technical
Maintenance
Troubleshooting
Documentation
Logging / Metrics

24. Meta Data Location DB Resident Almost always relational
C/S predominantly
Normalized design
OODB is popular option for proprietary solutions

25. Repository
Specialized databases designed to maintain metadata, together with tools and interfaces that allow a company to collect and distribute its metadata
Repository Requirements
Logically Common
Open
Extensible

26. Multiple Repository Upside Downside Local instance, quick response
Local view
Users don’t have to wade through other’s material
Downside
More challenging implementation
Advanced replication
Requires maintenance resources
More susceptible to architecture modification to remote instances

27. Copyright © 1997, Enterprise Group, Ltd.
Multiple Repository
Where do I find all the information about sales?
Data Mart
Meta Data
Data
Marts
Requires multiple access points
Requires more system resources
Copyright © 1997, Enterprise Group, Ltd.

28. Common Repository Upside Downside Optimum solution
Avoids replication challenges
Allows central management/access
Downside
Requires remote access for remote DM’s
More network infrastructure
May require gateways

29. Copyright © 1997, Enterprise Group, Ltd.
Common Repository
Where do I find all the information about sales?
Data Mart
Meta Data
Data
Marts
Single access point for all information resources
Low system resources required
Copyright © 1997, Enterprise Group, Ltd.

30. Copyright © 1997, Enterprise Group, Ltd.
Meta Data Process
Integrated with entire process and data flow
Populated from beginning to end
Begin population at design phase of project
Dedicated resources throughout
Build
Maintain
Design
Mapping
Extract
Scrub
Transform
Load
Index
Aggregation
Replication
Data Set Distribution
Access & Analysis
Resource Scheduling & Distribution
Meta Data
System Monitoring
Copyright © 1997, Enterprise Group, Ltd.

31. Meta Data Vision vs. Reality
Standards
OMG standard (June 2000)
Common Warehouse Metadata Model
XML based
Supported by Oracle
Designed by Oracle, Unisys, IBM, NCR and Hyperion
Industry initiatives just taking hold
Proprietary solutions inadequate
Who is missing?

32. Meta Data Challenges The Meta Data conundrum
Thin tool support (pairing standards, MSFT coming)
Hidden resource trap
Absolute requirement for success

33. Web Sites
List of metadata tools
Universal Metadata
Metadata Project