1. How (and Why) to Build a Data Warehouse 101
From Questions to
Eight Steps to DW Heaven!
@aupward & #meshU

2. Declaration of Data Independence
When in the Course of an increasingly competitive global economy it becomes necessary for one data set to dissolve its connections to a constraining environment, the separate but inherently unequal station to which the Laws of Whose budget is larger prevails…. <snip>
We hold these truths to be self-evident, that all data is created equal, that they are endowed by their Creator with metadata that holds important context and critical unalienable rights, that among these are compliance, security and the right to drive business value…

5. And finally for the database query writers…

6. Who is Antony Upward Funny after lunch?
Management Consultant – Business Systems – 20+ years
Business Analyst, Project Manager / Program Director, Business Architect
Apple – GUI to Data Warehouse over WAN (DECNet) in 1990!
Bell family of companies and elsewhere – including full SAP Data Warehouse Implementation
Academic… Teaching / Learning
Ryerson University School of Management
Business Technology Management Program
Business Process Analysis and Design
IT Governance and the Role of the CIO
Producing the people you need / want to hire
Understand Business AND Technology
New…Edward James Consulting… Sustainability Business Architect
Aligning People, Process and Technology to achieve Sustainable Results…
Sustainable for People, Planet and Profits
Returning to University… Masters in Environmental Studies with Graduate Diploma in Business and the York / Schulich

7. The Data Warehouse – a key tool
Why?
Your world is…
Complex
Dynamic
Data… coming at you like a fire hose
How do you:
Make sense of your world?
Make informed decisions to:
Lower costs, increase revenues, increase productivity?
Make plans… and know if you’re achieving them?
The Data Warehouse – a key tool

8. Customer – Supplier – Industry
You’ve Launched…
Your Communities
Customer – Supplier – Industry
Your Market
Your
Suppliers
Your
Customers
Your
Company
Your
Talent
Pool
Potential
Customers
Employees
Your
Regulator
Your
Bank
Your
Investors
Your
Advisors
(Accountant,
Lawyer, etc.)
..and You’re Starting to see Your
World is Complex…

9. …which you’ve started to interconnect…
…and you have systems…
Internet / Cloud
Customer Care
Order Entry
Billing
Assets
Projects
Retail
Web Site /
Portal
Finance
A/R GL
A/P
Inventory/
Warehouse
Web Site Content
Management
Purchasing
Community
Wholesale
& Supplier
Web Sites
Supplier
Bank
…which you’ve started to interconnect…

10. …and you have transactions…
Customer
Sales Orders
Financial
Journal
Entries
Payments to
Suppliers
(Cheques)
Proposals
From
Suppliers
Marketing
Campaigns
(Qualified
Prospects)
Community
(blog postings, tweets/buzz, wall postings…)
Payroll
Proposals &
Quotes for
Customer
Customer
Invoices
Purchase
Orders
Customer
Quotes
Supplier
Invoices
Website
Activity
(browsing, choosing,
sharing, using…)
Customer
Payments
Customer
Inquiries Praise
& Complaints
…more and more all the time…

11. … and you have started to realize your data has structure …
Transactional Data
Data related to specific business events
Master data
Data related to the objects involved in your business, which change over time – but remain constant over many transactions
Meta data
Data about your transactional and master data
Transactional Data
Data related to specific business events
Customer placing an order (Sales Orders), Buying something from a vendor (Purchase Orders), etc.
Master data
Data related to the objects involved in your business, which change over time – but remain constant over many transactions
Customers, Products, Employees, Suppliers, G/L Account Numbers, Warehouse Layout, Payment Methods, etc.
Meta data
Data about your transactional and master data
Usually not written down… but is known… e.g. how do people know what each field of a customer order means?
… perhaps realizing that all is not right…

12. …but most of all you have questions
How many customers do I have?
When did the last buy from me?
What did they buy?
Where are my customers?
What do my customers have in common with each other?
Which combinations of products do they buy…so I can make recommendations?
How did they behave on the website when deciding what to buy?
How are they using my application, product or service?
Who is paying me on time / late?
Who has upgraded and what led them to decide to upgrade…so I can encourage other customers to do the same?
Which suppliers ship to me reliably (on-time, right quantities, no DOAs)?
Which suppliers invoice me accurately?
What do my customers tell me they like?
What is being discussed about my company in the community?
What are the trends in the complaints I receive … how can I improve?
Am I easy to do business with?
Am I meeting my delivery promises to my customers? (no back-orders, on-time delivery, etc.)
Am I spending the right amount on my inventory? (too much, not enough)
How long does it take for me from getting an order to getting paid?
Is every order profitable?
Did I make money in the last month, week, day, hour?
How much did I spend on X?
…and despite all this data… …you don’t have ready answers!

13. … an aside… wondering why… if we have all the data why we don’t have information?
The way we build systems* to help run our transactional business processes … rarely has anything to do with the information we need to manage, plan, change those same processes†
Plus…
Data is very very slippery‡… it requires huge business discipline to keep it all consistent over time… and soon as data is inconsistent it is hard to turn it into information
* Historically this was because we had no choice – it simply wasn’t possible to cost effectively buy the hardware or build the systems with the complexity to meet both needs simultaneously
† New technologies like in-memory databases will change this… but it will take 5-10 years before it is normal for transactional systems to also be able to provide management information
‡ Despite the claims of the semantic web people we are no closer to solving this problem! Don’t believe the hype

14. Operational Process (OLTP)
The Big Picture…Where the data Warehouse Fits in Running Your Business…
Transaction
(e.g. Take Order)
(e.g. pack & ship)
(e.g. invoice)
Act
Operational Process (OLTP)
Delivery of Organizations Value Proposition to Stakeholders
Transactional Systems 1...n
OLTP = on line transaction processing
(aka ERP, CRM, etc.)
Management Process (OLAP)
OLAP = on line analytical processing (aka data warehouse)
Analyze
Measure
Gain Understanding…Answer Your Questions
OK we’ve just finished a deep dive on measurement – down at the detailed level. Before we wrap up our discussion of measurement let’s look at measurement from the big picture / macro perspective – more at the top-level process level.
Key:
OLTP (on-line transaction processing) systems are usually used to support the transactional or operational variants (aka transactional sub-processes) of the Core (Customer facing) and other types of top-level processes in an organization.
OLAP (on-line analytical processing) systems are usually used to support the management variants (aka the management sub-processes) of the Core and other types of top-level processes in an organization.
Review Slide
The management sub-processes measure, analyse and report on the transactional sub-processes. Then the management sub-processes planning acitivities will make decisions to alter the plans based on the analysis which drive the operational sub-processes.
For example in an Order to Cash top level process there will be (transactional) sub-processes which are focused on taking the customers order, ensuring it is manufactured, shipped, billing and the cash received. In addition there will (management) sub-processes which are focused on understanding how many orders are flowing through the process per minutes, hour, day, week, month, quarter and trying to forecast what will happen and identify problems / opportunities for improvement. The result of these analyese will result in new plans / targets being set and associated changes being made (increasing or reducing staff, increase or decreasing inventory levels, etc.) in order to optimize the delivery of the customers value proposition.
See more notes on next (Hidden) Slide
THE data
warehouse
Plan
Make Decisions… Decide How to Action Them

15. So…You Need a Data Warehouse (DW)… How Do You Get One
So…You Need a Data Warehouse (DW)… How Do You Get One? OR Eight Steps to DW Heaven
* Like data warehouse technology sales people!

16. First…Recognize That Building a DW is Both Like and Unlike Building Other Systems…
Same…you need:
People… who will be using the DW to be involved in designing, building and using it IF you want an ROI
Process… a great project manager who has the right plan – one customized to your needs
Technology… and great people who know the technology
Don’t let anyone* tell you they know the answer and can “magically” give it to you tomorrow
Different…
The users of a DW are YOU… Your leaders / managers!
Are you ready to get involved?
Are you ready to change – make decisions using the information from the DW?
You really don’t know what you don’t know about your data… get comfortable learning by iterating
You’ll spend less and get more
* e.g. DW technology vendor or consulting vendor sales people! There is a lot of hype out there. Be cautious.

17. Second… Do You Have the Skills You Need?
Enterprise Data Architect*
(Data Modeller, Entity Relationship Diagrams)
* Technological owner of the definition of all data in the transactional and DW systems – not to be confused with the business owner
DW Tool
Experts
..In which
ever tool you decide to
use†
Business Intelligence (BI)
Business Analyst
Understands the tech…
But can learn, understand and talk to you YOUR business, YOUR questions
Your Management Team
Whose questions is the reason the DW is being Built!
DataBase Knowledge
(Oracle, SQL Server,
etc.)
DBA
(Administrator)
Query Writer
(SQL etc.)
DW Infrastructure
Specialists
(Hardware, Operating
System, etc.)
Plan to get the skills you need at the right time…
1a. BI Business Analyst, 1b. Enterprise Data Architect 2. Tool Expert
† You’ll need more of this resource early on than later… so perhaps a good place to use contractors/consultants … at least initially and only hire later when you know the level of on-going need for this skill set

18. ...Third… Decide on What To Do First…
Make a list of the most pressing questions you can’t answer today
Get your management team involved
Brainstorm what the answers might be… and what you will change if that answer turns out to be correct
Get everyone used to the idea that the DW is a tool which will become a normal part of the process by which you will:
Make decisions , Make and prioritize plans for improving your business
Drive change, Measure management team member success
Determine the potential benefits if you were to decide to implement the changes… Your Business Case
Which answers would drive the biggest benefits?
Pick the n questions whose answers would enable the biggest benefits
2 < N < 10
Drop the rest for now… you need to focus… DON’T get side tracked…focus!
Make a list of the most pressing questions you can’t answer today
At all
Frequently enough
Accurately enough
Get your management team involved
Brainstorm what the answers might be… and what you will change if that answer turns out to be correct
Get your management team involved NOW… if they don’t buy in you are wasting your DW investment
Get everyone used to the idea that the DW is a tool which will become a normal part of the process by which you will:
Make decisions
Make and prioritize plans for improving y our business
Drive change
Measure management team member success
Determine the potential benefits if you were to decide to implement the changes
This is your business case
What are the benefits of each change?
Which answers would drive the biggest benefits?
Pick the n questions whose answers would enable the biggest benefits
2 < N < 10
Drop the rest for now… you need to focus… DON’T get side tracked…focus!
Yes some people will be unhappy that “their” questions won’t be tackled in first wave… get over it

19. Fourth…Which Data Do You Need ?
Mock-up the output from the DW which will answer each question
Show it to everyone… listen to the feedback
Remember why you are doing this… it is the ability to decide to change as a result of answering the question that matters not what the screen / report looks like!
How do you “calculate” each part of the answer?
Get everyone involved in figuring this out…
Decide which senior manager “owns” each calculation / measure
What data is required?
Identify both the master data (customer master, product master, etc.) and the transactional data (sales orders, payments, etc.)
You do have an enterprise data architect and model don’t you?
If not time to:
Hire that person
Build that model…and have your transactional systems people keep it up to date
Mock-up the output from the DW which will answer each question
Show it to everyone… listen to the feedback
Confirm that if the output of the DW looks like the mock-up people will be able to answer the questions and make the decisions they want to be able to make
Remember why you are doing this… it is the ability to decide to change as a result of answering the question that matters not what the screen / report looks like!
How do you “calculate” each part of the answer?
Get everyone involved in figuring this out… otherwise you’ll end up in arguments over whether the answer is valid rather than using the answer to make decisions and drive change
Decide which senior manager “owns” each calculation / measure
Who will defend the answer?
Who will decide when the calculation needs to be changed / improved?
What data is required?
Identify both the master data (customer master, product master, etc.) and the transactional data (sales orders, payments, etc.)
You do have an enterprise data architect and model don’t you?
If not time to:
Hire that person
Build that model…and have your transactional systems people keep it up to date

20. Fourth (part Deux)… Where Is Your Data?
Where is the data?
Does it exist… ask the enterprise data architect… if not can you substitute…
Start the feedback loop to future versions of your transactional systems
Is that data clean (consistent), what anomalies exist in that data?
Normal to find data isn’t clean… expect work arounds
Ensure everyone knows how these imperfections will impact the accuracy of the answer
Who in the business is responsible for cleansing the data and keeping clean

21. …Fourth (part Trois)… Get the Tech Ready
Set a budget for the tech
Use the benefits of answering the first group of questions to decide how much to spend
Decide on the tools
Ensure they can grow with you
Get help deciding
Hire the DW tools expert(s)
The people who know the tool you’ve chosen
Consider contractors or consultants
Set the technology standards and design principles
Will help ensure reliability, flexibility, agility
Response time, refresh frequency,
Data Integrity, Data Security, Disaster Recovery, Business Continuance
Change Control (Development, Test, Production, Training)

22. The (Tech) Parts of a Data Warehouse
The Data
Warehouse
DW Control
&
Operations
Data
Includes: scheduling, reporting schedules, extract, transformation, refresh, DR rules etc.
These need to be easy to use… Management will the users!
Manual Data Maintenance
Meta
Data†
† Data about the data in the ODS and Cubes – to allow users to understand, and “self document” their queries
Dimension
Tables
(Shared)
Your Transaction Systems
(OLTP)
Transactional
Databases
Operational
Data Store (ODS)
aka Persistent
Staging Area (PSA)
Clean Master
Data
Read the cubes
End User
Query & Reporting Tools
Clean
Trans-actional
Data
Cube 1*
Cube 2
Cube n
* Each cube (aka Data Mart) has the” fact” tables containg the transactional data transformed and the applicable dimension tables to answer groups of related questions
Extract, Transform and Load (ETL)‡
‡ Includes: matching, cleansing, versioning.
Can be “pull” or “push” depending on volumes
Read from & Write results to cubes
End User Analytic Tools
“Close Loop”
Analytic Tools
Make your decisions “real” by pushing process changes directly into operational / transactional systems. Can make changes in near real time!
Technical tools – to be used by DW Tool Experts and BI Biz Analysts

23. …Fifth… Build & Test a Prototype …Iteratively
Document the detailed design (BI Analyst and DW Tool Techs)
Take earlier mock-ups (step 4) and build a prototype
focus on getting to 80%
Expect the build to take 5-10x longer to answer first 2-3 questions than questions 3-10
There is a lot of one time set-up…
Make sure your tech team know this is a prototype…
Test with users
Set expectations… people should be starting to get excited that its becoming real… not upset because its not perfect!
Can your management now answer the questions they had and (more important) can they make (and execute) the decisions based on the answers
Expect this step to take 2-12 weeks depending on complexity
Time box it at 12 weeks… reduce the scope
Document the detailed design (BI Analyst and DW Tool Techs)
Take earlier mock-ups (step 4) and build a prototype
focus on getting to 80%
Expect the build to take 5-10x longer to answer first 2-3 questions than questions 3-10
There is a lot of one time set-up…
Make sure your tech team know this is a prototype… they will have time to productionize later… much will need to change based on what the users learn
Test with users
Set expectations… people should be starting to get excited that its becoming real… not upset because its not perfect!
Can your management now answer the questions they had and (more important) can they make (and execute) the decisions based on the answers
Expect this step to take 2-12 weeks depending on complexity
Time box it at 12 weeks… reduce the scope

24. …Sixth… Use it… Make Decisions (Mistakes)… Learn… Improve
Get the prototype so it can deliver answers
Expect regular manual effort
Cleansing data, running ETL or reports, hand holding users if UI isn’t perfect
Use the answers to make and execute decisions
Have a party… you have the start of a successful data warehouse!
Learn
After 2-12 weeks (depends on decision frequency) have a formal lessons learned
Are you getting the benefits you expected?
Involve everyone – leaders, managers, BI Analyst, Tech team… you all need to understand the challenges and what’s been learned
This is the difference between great high performing (BI/DW) teams and mediocre ones… their ability to learn together
Plan the changes needed to have the DW reliably deliver the answers

25. …Seventh… Productionize…
The real go live…
Formal training of users
Prepare documentation
Make them confident in the use of the tools
Empower them with great meta data
Mechanising manual steps
Scheduling of all the jobs
make it run smoothly
No wee hours of the morning handholding!
Implement disaster recovery
If your DW is enabling operational decisions your DW is mission critical!
Party some more
You’ve been working hard at this for 3-8 months at this stage and can now see real benefits happening!
Every day your decisions are being informed by your data via the DW

26. … Eighth… Now… Which Questions Didn’t You Answer?… Start Wave #2
Build on your success…
Go back to Third step and repeat…but go faster!
Remember incremental cost of answering more questions and adding more data to the warehouse is much lower than the first wave due to:
One time investments made
The experience you have

27. You Are On Your Way to a Sustainable Business
Having all management decisions and strategic planning informed by a single, consistent set of information which your management team all align around is the basis for the long term health of your business
The DW is just a tool, but one which when used with significant management / leadership discipline can make the difference between long term success and failure
The DW is not a one-time project… it is a tool to enable a better way of running your business… forever!
Expect to want to evolve it as you and the DW tools become more and more sophisticated

28. Thank-you… download slides from http://www.EdwardJames.biz/documents