1. Dr. Jerry Rosenbaum jrosenba@ix.netcom.com
Dimensional Modeling
Dr. Jerry Rosenbaum
The Rose Tree Group
Myriad Solutions

2. Agenda Dimensional Example The Bigger Picture
Steps to Build a Dimensional Model

3. An Understanding of the Problem is key to the solution
To rewire the Empire State Building you must
Understand the current wiring
Understand the goals for the new wiring
Design the new wiring system
Execute your design
Monitor the results
The “Fire, Ready, Aim” approach
can lead to a hole in your foot

4. Business User Perspective on Data
In the view of the business user, there are only two important things about data
Accessibility
Can I get the data that I need
Am I allowed to get the data I need (security)
Quality
When I get the data, can I trust it
Without quality, business intelligence devolves into business stupidity

5. Part I Dimensional Model Example

6. Up Front Points
Dimensional Models are generally a star schema, but may be a snow flake
Provides a slice of the total available data and is focused about the needs of a single department or user
Contains a fact table and multiple dimension tables
Attributes may be source data or derived data
A dimensional model is one type of data mart.

7. Sales Analysis Star Schema
From Len Silverston
Universal Data
Models

8. Dimensional Model One Fact table
Customer Sales
Fact is either one data item (or a group of tightly coupled data items of the same granularity)
Multiple Dimension Tables (each with one or more dimensions of a similar type)
Sales Rep
Product
Time by Day
Address
Customer
Customer Demographics
Internal Organization

9. Business Points
Data in Fact Table, and the Dimension Tables is based on
Business requirements
Data extracted from other databases
Internal or external data
Allows business users to slice and dice the data by any combination of dimensions to produce a business report

10. Business Points (2)
Design of the Dimensional Model is only one aspect, how about the data to be loaded
Are there any data quality issues with the data that will be loaded
Are there alternate sources of the data that differ from the data source we used
Are we violating any security or privacy issues

11. Questions Where did the data come from How often do we update the data
Is it an update or a complete refresh
Are the rules different for internal and external data sources
Why didn’t we just use the original databases
Was any of the data transformed
Why did we choose those 7 dimensions
What is the quality of the data

12. Truly Answering These Questions
We must look at
Business drivers
Where dimensional modeling fits into the bigger data picture
How do we do dimensional modeling
In other words, it helps if we understand the bigger picture so we can build the right solution the first time

13. Some Hidden Issues
This dimensional model can not reasonably help us answer
What was the typical total check out amount for all purchases by a customer
How many items did a typical customer purchase
Adding these items to every row of the fact table is not a good solution
This requires a second, but related dimensional model.

14. Hidden Issues (2)
How does one keep a set of dimensional models in synch
How does one ensure that for a given “fact” and set of dimensions that the sourcing of the data is consistent across multiple models
If a transformation (the T in ETL) is used, how can one ensure that everyone uses a consistent transformation
What about the data quality issues – were they resolved the same way every time.

15. Part II The Bigger Picture

16. Sagely Advice “Where should I begin your majesty”
“Begin at the beginning”, the king said gravely
- Louis Carroll
Alice’s Adventure in Wonderland

17. Business Intelligence
On of the earliest applications in the history of computing was a program to generate reports on operating system performance
FAST FORWARD TO TODAY
Today we call this process Business Intelligence (BI) – we gather and analyze data to increase business process efficiency.
Realize even the first application was written in the early 70s’ to provide intelligence to improve system performance.
So as we fast forward to today -----
We are still providing reports for the reason of improving a result, whether it is a business decisions or the performance of the system in a data center. So we must assume the reports we provide must have a positive end result with a high degree of confidence in the accuracy of the data being manipulated.
From J. O’Conner

18. Business Intelligence
C - Level – Considerations
Costly ERP and Major systems are implemented to provide information to management to make the best decisions relating to improving the “Bottom Line.”
IT Customer satisfaction does not seem to meet expectations – manage expectations
IT takes the blame for bad decisions due to the information available to the executives
UNWARRANTED ??? –
Corporations spend large amounts of money, time and numerous process changes with the expectation of saving money, improving productivity and gathering information for improved decision making and possessing the information for strategic direction and tactical improvements.
Information Technology has been the “whipping boy” for poor business decisions, increased cost of production, and loss of cost savings – even to being responsible for the systems failures to meet expectations. Many times IT does not even participate in the product selection, but are all of a sudden responsible for the results, not just the implementation and training.
Most times IT does not have the proper level of responsibility at the executive level who defend the fact IT cannot do any more than provide the best business intelligence possible and keep the system functioning – but we cannot control the business’s .
However, the information and reporting of Business Intelligence must meet the needs and expectations of the customer. So is the blame unwarranted?
This is our fault – we do not manage expectations or listen to the customer – which is the most common problem because we do not train our customers about what information is available and if it is not at this time how can we provide a solution to meet the expectation.
From J. O’Conner

19. Business Intelligence
C - Level Executive Considerations
ERP or Major System’s Value is dependent on how the information is disseminated to management (regions, functions, divisions, products, etc.) – but IT needs to understand what data is available and assure its accuracy.
IT must establish test and check points to assure data accuracy using standard data integrity methods.
IT must provide training and support in development of reports for accuracy – become detectives – look for anomalies, bringing them to the attention of the clients.
One thing to remember about the Executive management – they have already “selected” the best system (in their opinion).
They do not care about how query requirements work – joining, filtering, grouping or special calculations must be processed. Therefore, we need to train their staff to query and utilize the end user tools to produce the reports and information, but we also need to have a check and balance to assure the information is accurate, has been processed properly (review the report structure) and test all output for accuracy with specific test points and check points.
REMEMBER:
It must take a leadership role in investigating any anomalies because the end user or client cannot be wrong it is the system.
Story, For a considerable time the businesses and Finance would state that the information coming from the SAP system was incorrect. After some basic investigation it was found that Finance had a consolidating system that took the data from all of the SAP businesses and created the month end reports. They also made adjustments in the consolidating system that did NOT coincide with the business rules established by Finance when the system was implemented. To make it worse they made the changes in the system at the summary level and in their system so two things were wrong (1) The consolidating system did not “roll up” from the details and (2) they did not make the changes to the SAP system. So over time the error compounded itself, to a point that when SOX404 was being implemented the correcting and need to improve the check points and remediation was significant.
So create a data integrity team that is cross functional and review the business intelligence for obvious issues as well as those that are less obvious. SOX404 will help with a Control Self Assessment that should be adhered to by IT to assure the Business Intelligence is accurate and avoid the “finger pointing” as well as being the reason for business failures.
From J. O’Conner

20. Business Intelligence
C - Level Executive Considerations
Information Business Leaders need:
Customer Information – trends in product selection, billing management, order tracking, fulfillment visibility, marketing planning, campaign management , telemarketing, lead generation, lead generation, and custom segmentation.
One thing to remember about the Executive management – they have already “selected” the best system (in their opinion).
They do not care about how query requirements work – joining, filtering, grouping or special calculations must be processed. Therefore, we need to train their staff to query and utilize the end user tools to produce the reports and information, but we also need to have a check and balance to assure the information is accurate, has been processed properly (review the report structure) and test all output for accuracy with specific test points and check points.
REMEMBER:
Faced with the business demand for sustained profitability, companies of all sizes are looking for sourcing and procurement – and the bottom line advantage of managing supplier spends, reduce the cost of purchased goods and services all can boost profitability.
The Supplier trends – vendor to products, delivery trend (timely), - basically provide the business intelligence to control the complete supply cycle – from strategy to execution by optimizing supplier selection, compressing cycle times, and building sustainable workable supplier relations.
The customer trends – purchasing habits, product selection, delivery method, and payment trends.
From J. O’Conner

21. Business Intelligence
C - Level Executive Considerations
Information Business Leaders need:
Supplier Information – cost of purchased goods and services, optimization of supplier selection, compress cycle times, align the purchase of goods with the corporate strategy.
One thing to remember about the Executive management – they have already “selected” the best system (in their opinion).
They do not care about how query requirements work – joining, filtering, grouping or special calculations must be processed. Therefore, we need to train their staff to query and utilize the end user tools to produce the reports and information, but we also need to have a check and balance to assure the information is accurate, has been processed properly (review the report structure) and test all output for accuracy with specific test points and check points.
REMEMBER:
Faced with the business demand for sustained profitability, companies of all sizes are looking for sourcing and procurement – and the bottom line advantage of managing supplier spends, reduce the cost of purchased goods and services all can boost profitability.
The Supplier trends – vendor to products, delivery trend (timely), - basically provide the business intelligence to control the complete supply cycle – from strategy to execution by optimizing supplier selection, compressing cycle times, and building sustainable workable supplier relations.
The customer trends – purchasing habits, product selection, delivery method, and payment trends.
From J. O’Conner

22. Business Intelligence
C - Level Executive Considerations
Information Business Leaders need:
Product Lifecycle Management data
Supply Chain Management data
Financial Data that links Business controls to Finance and comply with Sarbanes Oxley (SOX404).
One thing to remember about the Executive management – they have already “selected” the best system (in their opinion).
They do not care about how query requirements work – joining, filtering, grouping or special calculations must be processed. Therefore, we need to train their staff to query and utilize the end user tools to produce the reports and information, but we also need to have a check and balance to assure the information is accurate, has been processed properly (review the report structure) and test all output for accuracy with specific test points and check points.
REMEMBER:
Faced with the business demand for sustained profitability, companies of all sizes are looking for sourcing and procurement – and the bottom line advantage of managing supplier spends, reduce the cost of purchased goods and services all can boost profitability.
The Supplier trends – vendor to products, delivery trend (timely), - basically provide the business intelligence to control the complete supply cycle – from strategy to execution by optimizing supplier selection, compressing cycle times, and building sustainable workable supplier relations.
The customer trends – purchasing habits, product selection, delivery method, and payment trends.
From J. O’Conner

23. Key Business Driver: The Need to Improve Business Intelligence
Nine out of 10 executives from the largest U.S. companies say they need stronger business intelligence capabilities that provide better analysis of, and insight into, their operations if they are to grow successfully in an uncertain economic and political environment
Accenture survey of 150 senior executives of Fortune 1000 firms.
From J. O’Conner

24. Top Needs 91% selected stronger analytical and business intelligence
84% selected an organizational culture that better accommodates change
74% selected a more robust information technology infrastructure

25. Building a Business and IT Foundation for BI
Organizations have never been so eager to adopt business intelligence (BI) technology. Unfortunately, lack of alignment between people, process, and technology has led to many misguided business intelligence deployments. Using a business-centric methodology and process improvement type of approach, organizations can leverage BI efficiently to enable Performance Management
M.A.Smith – Data Management Review

26. Zachman Framework

27. DoDAF

28. Row 1 - Planner Data – What
List of Subject Areas about which data is stored
Often presented as a taxonomy tree or a multilevel outline
Process - How
Business functional areas
Often presented as a set of functional decompositions
Also important to know
Relationship between data and process
Present, and proposed data and process
Transition plan / Road map

29. Row 2 – Business View/Owner
What - Data
Conceptual (or Business) Data Model. Has two components
Business data objects (forms, reports, etc) and their decomposition into data components and
A high level organization of the business data components and their relationships
How - System
Business work flow
End to end (incl. people)
Includes all aspects of the target system
The business workflow is the structure for organizing the business steps
There are rules for moving along the steps in the workflow
The business steps communicate to each other via the data

30. Row 3 – Logical View / Designer
What - Data
Logical Data Model
Fully normalized (through fifth normal form)
Determine which services use which data elements (entity level and attribute level)
How - System
For each business object
Develop a set of services to meet the business need
Search for potential common services
User interactions
Metadata for both data and systems should be collected,
Organized, and maintained

31. What is in a Logical Data Model
Graphic that depicts entities, attributes, primary keys, etc (Data items + structural rules + relationships) Plus
Metadata for Entities, Attributes and Relationships (CRUCIAL FOR USING DATA MARTS)
Definition
Data Domain values set (including possible representation)
Units of Measure
Cardinality (and optionality)
Management of synonyms and antonyms
Semantic rules
Status of an entity, or attribute

32. Logical Data Model (2)
Primary key, foreign key, uniqueness, use of nulls and default values
Data integrity and business rules (Data Quality Rules)
Originating Data Source
System of Record/Authoritative Source
Lead and steward business domain
Usage of data in an information exchange
Security
Notes for physical data model designer
Example
A data modeler may use abstract design templates (generalization and specialization) and bottom up design based current systems plus new requirements to build a complete model
All other models (especially the process models) are used as potential sources of data elements

33. Some Data Model Notes
The Conceptual Data Model tends to be a very wide scope, but limited detail (sets the context for data sharing).
Logical Data Model is developed in detail as needed
Physical Data Model is based on all or part of the Logical Data Model AND it may have a similar or very different data structure
Structure is based on planned usage

34. Row 4 – Physical View/Builder
What - Data
Physical Database Design
Transaction Path Analysis
Analysis of the need for indexes to improve performance
Determine Physical Data Structure
Determine if any services will be stored procedures
How - System
Transformation of the business flow to a physical flow
Determine groupings of services for implementation
Build physical flow based on business flow and services
SOA
Add physical details to the metadata and discovery services

35. Row 5 – Detail Representation / Subcontractor
What - Data
Determine the layout of the database tables across the disk farm
Develop the DCL for the physical database structure
Determine backup & recovery strategy
Determine SAN strategy
How - System
Determine detail specifications for each element of the physical flow
Write the programs for implementing the flow as well as each element of the flow

36. Row 6 – Information System (Actual physical system
What
Performance monitoring and adjustments
Business continuity
Archiving and retrieval
SPC & Audit
Data Stewardship
How
“Help Desk”
Change management (including data)

37. Corporate Information Factory
Metadata
Operational Systems
Reporting & BI Systems AR
Data
Mart for
Accounting
ETL
ETL AP
Data
Warehouse
Data
Mart for
Sales
Order
Entry
Etcetera
Etcetera
System Measurements
Based on work of C. Imhof

38. Data Quality Points You must measure your actual data quality
Quality must start in the production systems
Your ETL processes along with the data warehouse and data marts are not meant to be a sewerage treatment plant for bad data
If the quality is not very high in the operational systems, then
Your quest for business intelligence devolves into producing business stupidity
Solve your DQ problem in the operational systems (root cause) and then go forward.
Bad data can be created faster than you can correct it.

39. System Measures for Data and Systems
Frequency of use
Pattern of use (monthly, weekly, daily, hourly)
Resources consumed
CPU, Disk, Network
For Data both volume and rate of growth
Performance Metrics
Utilization Metrics

40. Metadata for Data and Systems
Need Metadata about Data and Systems
Written in Business Terms + Technical Terms
Metadata for data includes information about Entities, Attributes, and Relationships
Definition
Data Domain values set (including possible representation)
Units of Measure
Cardinality (and optionality)

41. Metadata (2) Management of synonyms and antonyms Semantic rules
Status of an Entity and Attribute
Primary key, foreign key, uniqueness, use of nulls and default values
Data integrity and business rules
Originating Data Source
System of Record / Authoritative Source
Lead and steward business domain
Usage of data in an information exchange
Example
AND Business Rules for Data Validity

42. Metadata (3) Without Metadata,
You may not be sure what you are looking at
For example what does a length of 6.2 mean (a Mars Lander crashed because of this problem)
You may not be sure what process you should use to execute a business process
Etc

43. Key Question Where do you get the data for the data marts.
Inmon: from the data warehouse
Kimball: directly from the operational systems
The problem is that source data may be available from several sources
What do you do if the sources do not have identical data
For derived data, does everyone use the exact same method (including data sources)
Are the semantics the same

44. A man with one watch knows what time it is A man with two watches is never sure
-Louis Carroll

45. Operational Systems
These are the systems that run the business on a daily basis
Most Customer interactions are with operational systems
Operational Systems AR AP
Order
Entry
Etcetera

46. Data Warehouse
Serves as the single, officially accepted and approved, valid source for all data needed for business analysis /intelligence
Helps insure that all Data Marts are reading from the same book and using the same rules.
Much easier change management
Fully normalized RDB with summary data added
Data
Warehouse
Archive

47. Data Mart Geared to meet the business users need Uses range from
Simple reports to
Data delivered in a manipulability form
Word
Excel
Small Data Warehouse
Star Schema
Delivery media depends upon planned usage
Reporting & BI Systems
Reporting & BI Systems
Data
Mart for
Accounting
Data
Mart for
Sales
Etcetera

48. Getting the Data In
Vendors promise us that the ETL process for moving data from operational systems to the Data Warehouse is “simple”
But we must deal with
Conflict resolution
Data Quality Issues
Duplicate data
etc

49. Getting the Data Out Generally use SQL queries
Care must still be taken to keep dimensions consistent
Marts sourced from a single Data Warehouse can be merged

50. Other Important Big Picture Data Issues

51. Net-Centricity Business Issue
Business Users may be located all over the place and each may move around
Home, Office, Customer site, etc
There are many computers in the network and each one
Can execute a specified set of processes
Maintains a specified set of data bases
Is not necessarily the same as any other computer.
The business user must be able to access any desired data and execute any desired process by connecting into any point on the network

52. Key Net Centric Services
Data services
Process services
Discovery services
Relies very heavily on metadata
Minimize SPOFs

53. Net-Centric Data Related Issues
If a piece of the network (or a processor) goes down, the overall system must still be functional
Data Discovery Services
Stress on Data Quality
Data bottlenecks must be mitigated
Data replication must be planned to
Guarantee latest version of data as soon as possible and reasonable
Inform the user of “old” data being presented

54. Governance What are the key tasks Who is responsible for the tasks
Management roles and responsibilities
Worker bee roles and responsibilities
What to do is things go wrong
Some key tasks include
Data Stewardship
Data Quality
Business Continuity
Performance Monitoring and Tuning
Testing

55. Part III Building a Dimensional Model

56. Why Build a Data Mart
OLTP systems are designed for rapid response and high transaction rates
Reporting from them is slow and degrades performance
Data Warehouse is big (hard for user to find things) and not designed for slicing and dicing. It is very well suited for general reporting

57. Design Options for Data Marts
Design chosen must consider the needs and abilities of the user
Excerpt from big Data Warehouse using the same or similar design
Dimensional Model (Star or Snow Flake)
Spreadsheet
Word Document
etcetera

58. Basic Steps for Building A Dimensional Data Mart
Determine the Business Needs
Determine Sources of Data in Data Warehouse
Identify the Granularity of the Data
Determine the data in the Fact Table(s)
Determine the Dimensions
Build the Dimensional Model

59. Sales Data Warehouse

60. 1. Determining Business Needs
Must be done in face to face meetings
Ask for samples or mock up reports
Probe for possible extensions
Determine which type of data mart is best suited to the set of users.

61. 2. Determine Source of Data in DW
Which table(s) in the data warehouse contain the data needed by the user for
Fact Tables
Dimension tables
Is any of the data from external sources
How do you plan to get that data
How reliable is the external data
Did you check it yourself
Fortunately you have already solved your internal DQ problems

62. Dimensional Definitions
A dimensional model requires
Fact table
Contains the measurements or metrics or facts of business processes PLUS foreign keys to the several dimension table
Dimension Table(s)
Context of the measurements (or facts) are represented in dimension tables. Typically the what, where, when, when, who and how of the measurement (or fact)

63. 3. Identify Granularity Needed
For the fact table
What is the lowest level of granularity (detail) needed
Is it atomic data or summarized data
How many fact tables are needed (each will have its own star schema)
If you wish to join data across star schemas, you must have consistent sourcing of data and consistent dimensions. and appropriately matching granularities.

64. 4. Determine Data in Fact Table
For each fact table,
What single fact (or tightly coupled group of facts) will be included.
Determine how the fact will be extracted from the source
What dimensions are available in the source data (Foreign keys)
Does your source database support the desired foreign keys.

65. 5. Determine the Dimensions
Dimensions generally follow the standard who, what, where, when, why, and how
Who are there people or organizations associated with the fact
What are the ways of categorizing the fact in the fact table
Where: what are the locations associated with the fact
May be physical locations or conceptual locations

66. 5. Dimensions continued When: What are the time factors involved
Day, week, month, quarter, day of week, etc
Why is the fact included (not very common)
How: Was any special process or method associated with the fact
E.g. store sale, phone sale, internet sale

67. 6. Build the Dimensional Model
For the fact table, extract the data and all the associated dimensions
Often requires data from several tables
For each dimension, determine the set of possible values (domain) and populate each dimension table
Note that all the needed data may not be in the source database (e.g. suppose there were no sales from Wyoming)

68. The Finale Test the dimensional database you built
Deliver it to the user and train the user
Questions
Jerry Rosenbaum

69. References Ralph Kimball and Margy Ross – The Data Warehouse Toolkit,
John Wiley & Sons
Len Silverston – The Data Model Resource Book, Vol 1 & 2
Graeme Simsion – Data Modeling Essentials, 3rd Edition
Morgan Kaufman