Presentation is loading. Please wait.

Presentation is loading. Please wait.

Mission Critical BI in an EDW 2.0 world

Similar presentations

Presentation on theme: "Mission Critical BI in an EDW 2.0 world"— Presentation transcript:

1 Mission Critical BI in an EDW 2.0 world
By Jason Perkins & William O’Shea Summary:- The days of data warehouses being a back office system are over. Enterprise Data warehouse have to combine classic data warehousing with operational and system requirements to provide the mission critical support demanded by stakeholders. Mission Critical Business intelligence provides a framework for supporting service-level agreements and aligning business needs with system performance & availability. This presentation will cover best practice, practical techniques and real world examples:- • Application & data integration patterns. • Operational & serviceability • Data Migrations & transitions. • BI Exploitation models. • Information lifecycle management (ILM) • Maximum availability & resilience. • And more.

2 About the Presenters Jason B Perkins
Chief Architect on the Secondary Uses Service (SUS) Programme for BT Health Over 10 years working on some of the world’s largest and most complex Business intelligence and Data warehouses programmes. Highlights from Career Lead BI Architect for BT Retail ODM Consumer Reference Set (CRS) BT Mobile Data Strategy National Name & Address Database (NAD) Solution Architect for (Swift) BT Marketing Data warehouse. Qualifications TDWI Certified Intelligence Professional (CBIP) DAMA Certified Data Management Professional (CDMP) Subject matter expertise across Health, Retail and Telecoms Will O’Shea Data Warehouse Consultant at AMS Systems, currently assigned to the NHS Over 20 years of experience in consulting, focusing on Data Warehousing and Oracle RDBMS. Highlights from Career Worked with Gene Amdahl Development Lead at Oracle Oracle Consultant at Blue Cross DWH Consultant at Pfizer Data Warehouse consultant at Johnson & Johnson; awarded Innovation award for “Data warehouse in a box”. Technical Architect at the NHS, awarded Champagne award by Atos Origin for implementing RDM process Education MBA from University of Manchester (MBS) BSc from University of Waterloo, Canada. Oracle Certified Professional (10g DBA) Subject matter expertise Financial, Healthcare & Pharmaceutical. Summary Jason B Perkins – Has over 10 years experience working on the world’s largest and most complex Business intelligence and Data warehouses programmes. Jason is currently the Chief Architect on £20m per annum National Healthcare Data warehouse. He is both a TDWI Certified Intelligence Professional (CBIP) and DAMA Certified Data Management Professional (CDMP). Will O’Shea – An experienced consultant who has specialised in Very large Data Warehouses (VLDW) and Oracle RDBMS (VLDB). Will is a Technical Architect who has worked with blue chip organisations that include NHS, Pfizer, Johnson & Johnson, etc. He has a MBA from University of Manchester (MBS), BSC from University of Waterloo, Canada and is a Oracle Certified Professional (OCP). Amdahl’s law = 1/((1-P) + P/n)) where P = proportion of programme that can be parallelised and n=number of processors.

3 Agenda MCBI - The Business View Mission Critical Architecture
Mission Critical Method * BREAK * Mission Critical Principles & Operating Model Mission Critical Building Blocks Summary

4 Business Intelligence?
Expert Knowledge Facts Intuition JBP Birth of “Evidence based management” Jeffrey Pfeffer – Profiting from Evidence based management Bernard Marr – The Intelligent Company ~25-33% of medical decisions are based on science / facts. “Nearly two-thirds of managers believe poor information management is hurting productivity by 29 per cent, according to a recent survey by Capgemini.“ “In God we trust. Everyone else bring data?” W. Edwards Deming

5 TDWI “Three threes of Performance Dashboards”
Types of BI? Operational BI - Optimise & track core operational processes - Bottom up - Detailed - Monitoring Tactical BI - Project analysis and departmental activities - Departmental - Detailed / Summary - Analysis Strategic BI - Strategic Execution and analysis. - Top Down - Summary - Management JBP Examples of Operational BI – BAM – monitor and manage business processes that span multiple operational systems – Retail Order fulfilment. Operational MIS – call centre dashboard Decision Engines – retail next best offer Tactical BI - Departmental dashboard - retail marketing “slice and dice” analysis Analytical Sandpits – for ad hoc casual and analytical users Strategic BI – Corporate Scorecards; KPI (Telecoms RFT) Dashboards ; Budget forecasting For more information see TDWI Three Threes of Performance Dashboards. TDWI “Three threes of Performance Dashboards”

6 Mission Critical BI Mission Critical BI :-
“Systems that merit mission-critical status are those that affect a range of business processes, and warrant service-level agreements that align the business needs with system performance. Gartner ” Do not confuse the many other faces and names in BI:- Real Time / Right Time BI Real time integration / Data freshness On Demand BI High availability BI "What we've seen is that any time that we're talking to any of the online players -- say, some of the newer players in the marketplace, [such as] social media people or the community-based couponing [vendors] -- any time you talk to any of those organizations, it's all about real-time, any-time, all-the-time data warehousing," he says. "It's the same with financial services: they're all about fast, fast, fast and always-on as well. It's something you have to be able to [address] if you want to compete in this space." JBP

7 Mission Critical BI – Why?
Pervasive Business Intelligence Business 2.0 Always on Self service Joined up view of the customer. Available everywhere BI/DW no longer a back office function / system. Cost of entry in most industries. What you do with it remains a competitive differentiator. Operational Decision support Zero Latency Enterprise Globalisation JBP – Four key drivers which we are seeing in MCBI:- Business Agility Operational Secure Integrated Lots of terms – Real Time Enterprise “RTE compete by using up-to-date information to progressively remove delays to the management and execution of its critical business processes. Gartner” “Enterprises compete by using up-to-date information to progressively remove delays to the management and execution of its critical business processes. Gartner”

8 Mission Critical BI – Real World Examples
E-everything – 24x7 E-Government Health care monitoring – Commissioning / Payment for quality / results Referral to treatment times Payment for Quality Telecommunication Bandwidth management / Mobile Coverage Order to fulfilment MIS Retail – Just-in-time inventory JBP

9 Mission Critical – Challenges
Mission Critical BI is not new! So why is it so hard? “Pace of change” keeps increasing … Continued Pressure on IT Spend – estimated ~20-30% reduction in 2009/10. BI / DW keeps evolving – Many of the original mission statements of BI/DW remain elusive. Increased demand for integrated information – e.g. unstructured, social media, etc. Data Explosion – “Data volumes will grow exponentially while CPU capacity will increase only geometrically. Gartner”. Security of all the information is paramount BI/DW remains a predominately “build” activity. JBP Build as opposed to Buy. Pace of change : According to TDWI 2008 BI benchmark report – 6-8 weeks and for large enterprises far longer! to add a new data source. New Corporate report / dashboard 5 weeks for a new KPI. Security - Business Continuity; Compliance mandates; Privacy and confidentiality ; Data warehouse as corporate memory and Security threats BI – CIO’s No.1 priority for 3 years now falling … Gartner. BI is currently no.5 behind Cloud, virtualisation, web 2.0, etc.

10 Mission Critical – EDW Scale
Complexity Business Model Data Integration Mixed Workload Exploitation Number of Users Exploitation Maturity Size Data Loaded Data warehouse size Information output Number of different views need to be considered when quantifying the challenge ahead. Varies by industry, type of business and geography. JBP EWOC have specialised in the Enterprise DW space – top decile in terms of the 3 main measure above. Having said that we believe many of the MCBI principles discussed today apply across the whole vertical. Type of business – SoHo, SME, Corporation, Government, etc.

11 Mission Critical BI Architecture
Jason Perkins – 30m

12 Central Data Warehouse
EDW Architectures Independent Data Mart Virtual Data Warehouse Hub & Spoke Central Data Warehouse Easy to Build Organizationally Limit Scope Easy to Build Technically No need for ETL No need for separate platform Allows easier customization of user interfaces and reports Tailor spokes for business. Single Enterprise “Business” View Data reusability Consistency Lowest TCO Business Enterprise view unavailable Redundant data costs High ETL costs High App costs High DBA and operational costs Only viable for low volume access Meta data issues Network bandwidth and join complexity issues Workload typically placed on op systems Business Enterprise view challenging Medium ETL costs Data latency Requires corporate leadership and vision Requires fully performant and scalable technology JBP Many different EDW architectures – here are four of the most popular. Dimensional lifecycle similar to Hub & spoke – conformed dimension and facts act as the hub. Poll the audience to see what they have?

13 Mission Critical Maximum Availability Flexibility Maintenance Security
Lifecycle Method Infrastructure Adaptability Operations Migrations Technology JBP

14 Mission Critical DW Architecture
BI Applications OLTP & ODS Systems Business Applications Excel XML Process Staging Tier Operational Tier Integration Tier Performance Tier JBP

15 Mission Critical DW Architecture
BI Applications Operational Ad hoc Query Data Extracts Alerts Analytics Dashboards Reporting Web Services Auditing Security Performance Tier Metadata Services Resource Management Consolidation Marts Aggregates OLAP Sandpits Recovery / Restart Integration Tier Problem Resolution Data Quality SCD Manager Fact Loader Adoption Services Conforming Survivorship Error Management Workflow Monitor Staging Tier Job Scheduling JBP Adoption services (AKA late arriving data). Aligns with Oracle DW reference architecture and Kimball ETL sub systems. Auditing – Central data store for capturing and monitoring ETL and Exploitation audit. Audit dimension for managing against business facts. Security – responsible for ETL and Exploitation (RBAC) security. Metadata services – capture and exploit business and technical metadata. Resource management – responsible for monitoring and managing DW resources across ETL and exploitation. Recovery and restart – logical decomposition with regular checkpoint. Reusable ETL framework for recovery and restart. Problem resolution – capture, prioritise, manage and communicate all maintenance and operational issues. Error management – Central data store for capturing and monitoring business and technical errors. Error event fact for flagging issues. Workflow management – monitor job status include pending, running, completed and suspended jobs. Job scheduling – schedule jobs and nested jobs. Control relationship and dependencies. Auditing – Central data store for capturing and monitoring. Customer tracking – capturing and monitoring customer inbound (load) and outbound (exploitation) requests. Community Management Loader Services Change Data Capture Validation Services Customer Tracking Business Process Business Applications Unstructured Excel XML External MDM

16 Serviceability Architecture
Automation – lights out / zero touch Flexibility - meta data/reference data driven Robustness - error tracking, handling & reporting Maintenance - load/event tracking & reporting Resilience – Ability to stop individual parts of the system, restart Robustness - error tracking, handling & reporting JBP Far too often people neglect the operational side of a DWH when building; in a Mission critical system this becomes paramount. Operationally ready

17 Mission Critical Method

18 Nursery Method Raison d'être
BI/DW requires an Iterative approach. Mission critical is no different. New deliveries and changes must:- Protect core services. Facilitate “pace of change” Support re-use Allow experimentation Adapt to changing requirements Involve users Developed “Nursery” Method in response Supports front room and back room deliveries Reduce cycle time. “Nurseries” (AKA Sandboxes) – user initiated ETL process Production of Transformation and Load templates Will O’Shea … This new way is not really new, but is already known under a number of guises… Rapid Application Development (RAD) Agile Model Driven Development (AMDD) Agile Unified Process (AUP) Iterative and Incremental development Rational Unified Process (RUP) Scrum Development Joint Application Development (JAD) All involve multiple iterations along the path to implementation. All involve small teams, ideally co-located, at minimum meeting frequently and working collaboratively.

19 Nursery Method Growing a system
Requirements Analysis & Design Planning Implementation Nursery Planting the seed Initial Planning Transplant Implementation Evaluation Testing Delivery Must have involvement from all teams involved Must have buy in from all teams involved Everyone, business & developers, learns from both development and use of the system Introduces the ability to act on what has been learned Leaves Nursery when mature, and is transplanted into production – not re-grown.

20 Nursery Method The Growing Stages
Analysis & Design Integrated Small teams Design specification Implementation Did I mention Integrated Small teams Elaboration & Implementation specification Testing By both business and developers Delivery Delivery to users Evaluation User feed back Quality reports Transplant Final delivery should match 1.1 somewhat Initial Planning High level overall plan How long are iterations What deliverables are required High level requirements Planning Integrated Small teams Detail Iteration plan Higher level plan for 2 & 3 iteration Requirements Requirements for iteration Should fit within iteration or get broken into small bits Start with lowest level Will O’Shea …

21 Nursery Method Creating a Nurturing Environment
First Steps Initial Plan Overall objective? By when? Define Roles Assign Roles Business roles? User roles? Supplier roles? Commitment from those in the roles!! Define communication Meetings? Frequency Types Periodic weeding - Scrum Watering sessions – Stand-ups others Roles involved in each Tight Integration of roles Documentation from each role – small Frequency of documentation Type of documentation Define outputs from each iteration/phase Plan for cycle Roles involved at what stage Requirement documentation – small Initial Schedule Length of iterations Potential number of iterations Building the Nursery Will O’Shea …

22 Nursery Method Creating a Nurturing Environment
Next Steps Define system requirements Number of data suppliers ? Amount of data? Number of users? Size of infrastructure required Define First few iterations Cycle 1 Get data ? Load data ? Extract data ? Distribute data ? Cycle 2 Build some validation? Extract validation outcome? Cycle 3 Build in some robustness? Size of Plot Growth cycles Will O’Shea …

23 Nursery Method Principals
Focuses on: Users – Not Processes and tools Working systems – Not exhaustive documentation Working together – Not adhering to the contract Delivering what is wanted – Not following a plan Adapting to Change – Not Issuing Change Requests Both the Left side and the Right side must exist, but the emphasis is On the Left – Not the Right. Benefits Cycle time from months to weeks, even days! Improve quality – leverage “Lessons Learned”, as they happen Reduce: Cost Delivery time Happy Users !!! Our Real world examples Large International pharmaceutical company (delivered in Months not years) Healthcare Provider (implemented new functionality in days) Will O’Shea …

24 Nursery Method Greenhouses - Sandboxes
What Constitutes a Sandbox What are the characteristics How do they need to act & interact Users’ play areas Using the “Build Once – Use Many” principal users can Load new data sets Create new tables Create new reports Play with existing data Needs Work Flow Management – Key in a Mission Critical system Isolates the effects of users’ play areas from production Does Not isolate the data. User can access production data Other users can access their data Mechanism should exist to release into Production – if required Sandboxes are not Production; but rather a pathway to production Sandboxes are used as design, not as code Let’s users develop Though users are NOT developers You must supply a set to tools and templates for them to use. Tool across the spectrum, not just reporting Must have a way of getting the data to move throughout the system, this is where workflow management comes in Users can set up a series of connected jobs that load data from source though to utilization

25 Nursery Method Planning
Will O’Shea …

26 Nursery Method Exploitation – Managing “live” changes
Differentiate between types of changes – one size does not fit all. Determines how many Cycles it should stay in the Nursery. Minor Changes to Reports and Semantic Layer Category 1 – Changes to pre-canned reports / extracts Do not require changes to Semantic layer Category 2 – Deployment to live of new reports created by information analysts. Category 3 – Simple changes to the Semantic layer. New Reports Category 4 – Creation of new reports / extracts. Changes Impacting semantic layer Category 5 – Other changes to the semantic layer. creation of new derived fields (not to be performed in the universe). Category 6 – Changes to pre-canned reports / extracts that require changes to semantic layer. Category 7 – Creation of new semantic later. Different approach adopted based on business priority (of change and the existing business asset), size and complexity of changes required.

27 Nursery Method Exploitation elaboration workflow
Will O’Shea … Further complicated if already providing a live service. Example work flow we have used for delivering exploitation requirements into a live service. Talk around how you use a offshore / outsource partners.

28 Mission Critical Principles & Operating Model
Will O’Shea – 90m The Detail ..

29 Mission Critical Adaptability
“Pace of change” – keeps increasing … Its all about speed Speed of change Speed of information access “Design for change” – as opposed to “built to last” Design to: Build Once – Use Many Enter “Business Rule Management” (BRM) Process – Business Process Management (BPM). Rules – Decision logic Data – Decision variables Process Rules Jason Perkins Gain more flexibility by separating decision logic from implementation. Rules – how an organisation want to act Data

30 Mission Critical Adaptability
Design for change Process – Business Process Management for operational decision support Process flow or workflow for tactical / strategic decision support Rules – Rules Drive the Process Declarative approach Business user managed Descriptive Data – Meta/Reference data Enforces the Rules Thus data Drives the Process Contextual Volatile Flexible Process Rules Data Jason Perkins

31 Mission Critical Adaptability
Examples of rules management … Jason Perkins Examples of Business Rules Management:- Fair Issacs Blaze - Oracle Data Integrator (ODI) - Expressor -

32 Operational Principles Flexibility
Users require “flexibility” without the need to re-develop. Need to be able to Add and/or Modify Load Process Application processing Error processing Validations Recipients of Load statistics (DQ, Errors, etc) Encryption Process Load and use new data (joined to existing data) As and when they want to Without new code !!! Will O’Shea …

33 Operational Principles Maintenance
Operational team require the ability to configure and monitor processes. View ETL progress (real time) Loads Load steps Load Statistics Reporting and tracking by: Load Business Unit Time Status Performance and statistic reporting. Error tracking & maintenance against Load Control Loads if needed Start (automatically & manually) Hold/Pause all or part of a load(s) Stop Loads Restartable (from where needed) Will O’Shea … Reference data populated error messages, each with a severity code Message ids unique throughout application (for code maintenance) Error maintenance, reporting & tracking against loads Operations and user maintainable error cause and solution(s) repository for each id.

34 Operational Principles Administration
Business require Knowledge System should output meaningful & understood Error messages. Specific Messages throughout application, so business know the area. Visibility of Operations Error maintenance. Ability to feed into process Statistical Real-time reporting & tracking of loads. Know what data has been loaded Know how much data has been loaded Know what stage each load is at. Know what business units have loaded data. Will O’Shea … Overall goal is to automate and minimise / reduce administration. System should be self managing – with administration by exception.

35 Operational Principles Resilience
Business & Operations require A robust & resilient system Loads may be automatically restarted from where they were stopped/failed (as required) Each load job, step and statistic has start/end times and status ETL checks status of job to determine if it needs to/can be run. Fatal errors need manual intervention before they may be rerun. Performance and statistic reporting Self initiating Loads Will O’Shea … Resilience Loads may be automatically restarted from where they were stopped/failed Each load job, load job step and load job step statistic has start and end times as well as status ETL checks status of job to determine if it needs to/can be run. Fatal errors need manual intervention before they may be rerun. Performance and statistic reporting

36 Operational Principles Summary
Data Warehouses require “Metadata Driven Processing” (MDP) How? Where can MDP help your DWH? What Metadata does MDP need? Feed MDP into Development stream? Educate developers to use it Educate user to request it. Educate the business to use it. What can be MDP and what can’t? Loading Data – Types of loads, Source to target Load Control – Starting, stopping, branching, etc Errors & Messages – effects of & reporting on, Validation (DQ) – how, what, when & reports Encryption – how, what & when Reverence Data Processing Will O’Shea …

37 Integration & Quality Team?
Metadata Driven Processing Enterprise Warehouse Operational Components (EWOC) The Concept Business Unit Instance of Job Project Severity Validation Outcome Work-Flow Load Step Message Will O’Shea … Validation Rules Data Integration & Quality Team? Application Users Admin Load Statistics

38 Metadata Driven Processing The Metadata Driven ETL
Infrastructure Storage Allocation CPU Allocation Memory Allocation Sand Pit Business Unit Severity Fatal Error Warning Information Job Collection of Steps Has a start and an End Schemas Project Validation Lookups Static values Data Quality Patterns Linkage Man/Ops Etc Message Validation Load Processing Job Step Get data Load staging Load Atomic Human Interaction Etc. BU Job Cause & Solution Source SUS Cancer Registry Internal Target Internal BO / OBI BU Job Step BU Validation Additional Less the non-mandatory Will O’Shea … Meta driven code and processing Type CSV File XML Table Report/Extract Schema Source Schema Target

39 Metadata Driven Processing The Jobs - ETL
Metadata Driven Processing (MDP) Definition of Jobs Loads are specific instances of a Job Build re-usable modules Metadata driven code, promote MDP Quicker time to delivery, develop and test once Add/Change source and target by changing MDP data Add/Change ETL by changing MDP data Job Validation Lookups Static values Data Quality Patterns Linkage Man/Ops Etc Job Step Source Target Pick Lists Defined by Reference data Examples: Date range validation Foreign Key Lookups Mandatory / Optional dd-mm-yyyy vs. yyyy/mm/dd Y/N vs. 1/0 Will O’Shea … Type CSV File XML Table Report/Extract

40 Metadata Driven Processing The Messages – Driving force
Fatal – Fails the load Invalid file format Error – Load keeps going Max number of errors? % of load rather than # Warning – not following rules Date format etc. Information – no affect on load Dates out of range Visit after treatment Supports MDP Feeds Metadata Driven ETL Should be used throughout ETL Failure Checks/Traps Exceptions Reporting (DQ & Validation) Each error/trap/exception has a unique Message ID Headings/Titles/Text Severity can be changed Changes processing when changed Severity Fatal Error Warning Information Message Grouping Message Validation Load Processing Usage Error reporting Textual objects Information Messages Load Reporting Load Control Will O’Shea … Cause & Solution Helps with future occurrences Updated & Maintained

41 Metadata Driven Processing Data Quality & Linkage
Supports MDP Key in any system, but more so in a MC one. Use Metadata to Drive process Important right people get right data Quickly Rules Based Validation Data Quality Validation Linkage Validation New rules can be added/removed When needed(no code required) Businesses users decide to add rules From pick list Defined using building blocks Severity of failure of rule can be changed Businesses users decide severity Business Unit Canadian Office Finish Office UK Office Audit Data Reports Metadata Driven ETL Validation Outcome Validation Rules Will O’Shea … Lookups Static values Range Conversions Patterns Linkage Man/Ops Etc Reports Data Integration & Quality Team?

42 Metadata Driven Processing Encryption
Supports MDP Encryption is simply a specific Instance of a Job Built to perform Encryption New Encryption Types can be added but do require code New columns to be encrypted can be added by simply adding metadata, no code. Keys can be stored or added at run-time AES128 Triple DES Look-up Home-Grown? Name DoB ID # Encryption Type Audit Data Source Data Column Type Parameters (keys) Will O’Shea … Metadata Driven ETL Target Data Source & Target Definition

43 Metadata Driven Processing Reference Data Management
Supports MDP New reference data can be added without new code Different BUs can have different data but though same RDMT Different Import types are catered for Different Table Types are catered for e.g. K-Type 1, 2 & 3, Home grown, etc. e.g. CSV, XML, Excel Table Types Business Unit Import Types Reference Table Definitions BU Sources Metadata Driven ETL Column Definitions Source Attribute Definitions Source Definitions Will O’Shea … Target Data Audit Data Source Data

44 Metadata Driven Processing The Metadata Model
Can be as large as you want to take it. It is as good as you use it. If you don’t use it, no matter how good it is, it won’t be working Should be easy to set up and run, too much complication make it difficult to use. Try to keep application areas seperate though with common bonds

45 Metadata Driven Processing Extensibility
Extending the Mission Critical Data Warehouse. Most BI/DW requirements are not green field. Extending existing is a key design objective. Build Once – Use Many Adding new data sources Change existing data sources Data linage - Metadata Where data has come from Where it has gone What has happened to it along the way Impact Analysis New exploitation (analysis and reporting) of existing DW Adding new exploitation capabilities to DW Audit Data Majority of DW are over 2 years old. However few DW are ever finished …

46 More building blocks Jason Perkins – 150m

47 Technology Drivers Examples of technology features supporting Mission Critical BI. Analytics outside Data warehouse BI Web Services High Availability Data Warehousing Real-Time Data Warehousing Master Data Management (MDM) JBP – Question who in audience is pursing one of these DW features at the moment? From “TDWI Best Practice Report, Next Generation Data Warehouse Platforms, By Philip Russom”.

48 Mission Critical Performance
Leaving the Nursery (or Sandbox) Productionise the code Performance!! Balance Brute force – MPP (medium to high volumes / complexity / users) SMP (low volume / complexity / users) Performance Layer BI tool and RDBMS calibration Speed of ETL vs. Need of Retrieval - when to do something and when to not. 80 – 20 rule Selective Denormalisation Selective Pre-Joins Aggregates and Summaries – are they always needed DWA no?, SMP yes? OLAP Performance metadata Row counts Elapsed time JBP Goal shows be to move administrators up the value chain – i.e. how can they help us exploit the data and gain new / additional insights. Do not want them spending all time on tuning and keeping system running.

49 Mission Critical Administration
Resources Management JBP Resource Management - Goal Oriented, Workload-centric, Automated Enable management of mixed workloads – ETL and Exploitation; Operational, Tactical & Strategic; Based on business priorities / SLA’s. Not all BI is mission critical – phew! Prioritise resources for Mission Critical BI Applications Back office workload

50 Information Lifecycle Management
Not all information is mission critical – phew! Many benefits to segmenting information by its usefulness to the business. Performance / Throughput Cost effective Prioritisation of resources ILM - Number of levels Separate active and non active data. Compression non volatile data Read only for historic ILM - Intelligent storage based on usage of information. Automation is a key (emerging) requirement for supporting MCBI. JBP A key enabler for dealing with the explosion in data volumes. ~ 30% compressing non volatile data – Forrester.

51 Mission Critical Security
Security includes … Business Continuity Confidentiality Information Classification Non Repudiation Privacy Apply principle of “defence in depth” with multiple layers relating to security of information. Protecting customer identifying information. Pseudonymisation (P14n) Anonymisation Linkage across datasets and over time but NOT customer identifying. Usable Audit Services: provision of audit trail for Transactions applied to the database. Access to data in the database. Will O'Shea For more information please see ISO Information Security Standard Confidentiality (patient consent) Access control Audit Information Security Business Continuity Data integrity

52 Mission Critical Security
Pseudonymisation (P14n) Encryption Reversible Non Reversible Substitution Surrogates Anonymisation Other considerations Harvesting / Sharing Usability of output Key destruction Will O'Shea

53 Mission Critical Infrastructure
Mission critical infrastructure requirements Availability & Resilience Capacity on demand Ease of management Linear Scalability Data warehouse infrastructure “Roll your own” data warehouses Declining … Data warehouse appliance (DWA) The “new” kid on the block Cloud Services Way of the future? Jason Perkins

54 Mission Critical – Maximum Availability
Data warehouse now have to meet following with NO downtime. Planned Outages System Changes Application Changes Migrations / Transitions Unplanned Outages Infrastructure Failures Issues Data Error Human Degraded Service Insufficient Capacity Workload Management JBP

55 Mission Critical – Maximum Availability
Requirements Measured in 9’s No single point of failure. Tolerates many outages transparently Straightforward administration Availability and Resilience Active / Standby Active / Passive Dual Active Fallback Backup and recovery Automation Hot vs. Cold Incremental vs. Full Second site Software Operational Network Hardware JBP A&R Active / stand by – common “stand by” will be re-used as a NFT environment. Active / Passive – redundant version maintained. Idle resources. Dual active – minutes to hours unplanned outage protection – Second site for disaster recovery and data centre failure. Fallback – protection against “no data loss”. Possible reduced service in case of failure. B&R Automation – do you have to design or will system detect changes? MCBI is “Hot” Challenge for the new wave of Data warehouse Appliances. Low cost commodity components … SAN backup and recovery options. Fault isolation – failure in any component does not cause cascade failure to whole system.

56 Mission Critical Service Availability
Data Migrations New requirements – No downtime for on boarding data or exploitation. No impact to data freshness. Minimise impact on existing system. Differentiate between Migrations of new data source Migrations for existing subject areas (more common) Phased data migrations. Emerging Integration patterns Green field data migration Parallel Trickle data migrations. Mini batch data migrations JBP Most companies have data warehouse in 21st. Average DW is over 5 years old so migrations against existing subject areas are the most common these days.

57 Mission Critical Data Migrations
Independent data migration of (new) data source. Partition data migration in order to batch / trickle. Impact volumes against pattern to understand impact of additional throughput. Resource management a key requirement to protect existing system. No downtime or data freshness impact on business. Original structures New structures ETL Data Migration Green field Mini batch Or Trickle 1 2 3 JBP

58 Mission Critical Data Migrations
Concurrent maintenance of new and old structures. Cut over on completion of data migration to new structures. Impact volumes against pattern to understand impact of additional throughput. Failure to either new or original structures must result in rollback of both. No downtime or data freshness impact on business. Original structures New structures ETL Data Migration Parallel Trickle Pattern Trickle 1 2 JBP

59 Mission Critical Data Migrations
ETL Maintenance at single data structure at any point in time. Logically segment the source data into discrete partitions. Execute mini batch migrations, focusing on each partition in turn. Partition on volatility with early phases based on least volatile data. Catch-up mini batches required for changes during transition before final cut over. No downtime or data freshness impact on business. Original structures New structures ETL Data Migration Mini Batch Pattern 1 2 3 Mini batches JBP

60 Mission Critical Data Migrations
Pre-requisites Data profiling and analysis of new / changes in data migration Up front planning for Pipe cleaning and Rehearsal Practically Selective Only select entities you know you will need in that phase. If your hitting an entity consider taking it all. Transition – Fail to plan is plan to fail! rehearsal is key. Rolling Data quality monitors Audit and Reconciliation JBP 20% up front time on data analysis, pipe cleaning and rehearsal.

61 Summary Mission Critical is here …
What we need is an “Intelligent Data warehouse” Metadata driven Build once – use many Why do we need it? Business Agility through Nursery Method – Facilitates “pace of change” of business. Protects existing Mission Critical BI Services. Operational patterns Empower the business Support the Mission Critical BI Services. Integrated – exploitation of the customer “360 view” Secure – ensuring the right information to the right person Will O’Shea

62 References Massive But Agile: Best Practices For Scaling - The Next-Generation Enterprise Data Warehouse, Forrester. TDWI Best Practice Report, Next Generation Data Warehouse Platforms, Philip Russom. The ETL Toolkit, Ralph Kimball. Smart (Enough) Systems, James Taylor. Best Practices Mitigate Data Migration Risks and Challenges, Gartner.

63 Questions Thank you Further queries contact us at:-
Jason & Will can be contacted at:- For further information at

Download ppt "Mission Critical BI in an EDW 2.0 world"

Similar presentations

Ads by Google