Presentation on theme: "Mission Critical BI in an EDW 2.0 world"— Presentation transcript:
1 Mission Critical BI in an EDW 2.0 world By Jason Perkins & William O’SheaSummary:-The days of data warehouses being a back office system are over. Enterprise Data warehouse have to combine classic data warehousing with operational and system requirements to provide the mission critical support demanded by stakeholders.Mission Critical Business intelligence provides a framework for supporting service-level agreements and aligning business needs with system performance & availability. This presentation will cover best practice, practical techniques and real world examples:-• Application & data integration patterns.• Operational & serviceability• Data Migrations & transitions.• BI Exploitation models.• Information lifecycle management (ILM)• Maximum availability & resilience.• And more.
2 About the Presenters Jason B Perkins Chief Architect on the Secondary Uses Service (SUS) Programme for BT HealthOver 10 years working on some of the world’s largest and most complex Business intelligence and Data warehouses programmes.Highlights from CareerLead BI Architect for BT RetailODM Consumer Reference Set (CRS)BT Mobile Data StrategyNational Name & Address Database (NAD)Solution Architect for (Swift) BT Marketing Data warehouse.QualificationsTDWI Certified Intelligence Professional (CBIP)DAMA Certified Data Management Professional (CDMP)Subject matter expertise across Health, Retail and TelecomsWill O’SheaData Warehouse Consultant at AMS Systems, currently assigned to the NHSOver 20 years of experience in consulting, focusing on Data Warehousing and Oracle RDBMS.Highlights from CareerWorked with Gene AmdahlDevelopment Lead at OracleOracle Consultant at Blue CrossDWH Consultant at PfizerData Warehouse consultant at Johnson & Johnson; awarded Innovation award for “Data warehouse in a box”.Technical Architect at the NHS, awarded Champagne award by Atos Origin for implementing RDM processEducationMBA from University of Manchester (MBS)BSc from University of Waterloo, Canada.Oracle Certified Professional (10g DBA)Subject matter expertise Financial, Healthcare & Pharmaceutical.SummaryJason B Perkins – Has over 10 years experience working on the world’s largest and most complex Business intelligence and Data warehouses programmes. Jason is currently the Chief Architect on £20m per annum National Healthcare Data warehouse. He is both a TDWI Certified Intelligence Professional (CBIP) and DAMA Certified Data Management Professional (CDMP).Will O’Shea – An experienced consultant who has specialised in Very large Data Warehouses (VLDW) and Oracle RDBMS (VLDB). Will is a Technical Architect who has worked with blue chip organisations that include NHS, Pfizer, Johnson & Johnson, etc. He has a MBA from University of Manchester (MBS), BSC from University of Waterloo, Canada and is a Oracle Certified Professional (OCP).Amdahl’s law = 1/((1-P) + P/n)) where P = proportion of programme that can be parallelised and n=number of processors.
3 Agenda MCBI - The Business View Mission Critical Architecture Mission Critical Method* BREAK *Mission Critical Principles& Operating ModelMission Critical Building BlocksSummary
4 Business Intelligence? ExpertKnowledgeFactsIntuitionJBPBirth of “Evidence based management”Jeffrey Pfeffer – Profiting from Evidence based managementBernard Marr – The Intelligent Company~25-33% of medical decisions are based on science / facts.“Nearly two-thirds of managers believe poor information management is hurting productivity by 29 per cent, according to a recent survey by Capgemini.““In God we trust. Everyone else bring data?”W. Edwards Deming
5 TDWI “Three threes of Performance Dashboards” Types of BI?OperationalBI- Optimise & track core operational processes- Bottom up- Detailed- MonitoringTactical BI- Project analysis and departmental activities- Departmental- Detailed / Summary- AnalysisStrategic BI- Strategic Execution and analysis.- Top Down- Summary- ManagementJBPExamples ofOperational BI –BAM – monitor and manage business processes that span multiple operational systems – Retail Order fulfilment.Operational MIS – call centre dashboardDecision Engines – retail next best offerTactical BI -Departmental dashboard - retail marketing “slice and dice” analysisAnalytical Sandpits – for ad hoc casual and analytical usersStrategic BI – Corporate Scorecards; KPI (Telecoms RFT) Dashboards ; Budget forecastingFor more information see TDWI Three Threes of Performance Dashboards.TDWI “Three threes of Performance Dashboards”
6 Mission Critical BI Mission Critical BI :- “Systems that merit mission-critical status are those that affect a range of business processes, and warrant service-level agreements that align the business needs with system performance. Gartner ”Do not confuse the many other faces and names in BI:-Real Time / Right Time BIReal time integration / Data freshnessOn Demand BIHigh availability BI"What we've seen is that any time that we're talking to any of the online players -- say, some of the newer players in the marketplace, [such as] social media people or the community-based couponing [vendors] -- any time you talk to any of those organizations, it's all about real-time, any-time, all-the-time data warehousing," he says. "It's the same with financial services: they're all about fast, fast, fast and always-on as well. It's something you have to be able to [address] if you want to compete in this space."JBP
7 Mission Critical BI – Why? PervasiveBusinessIntelligenceBusiness 2.0Always onSelf serviceJoined up view of the customer.Available everywhereBI/DWno longer a back office function / system.Cost of entry in most industries. What you do with it remains a competitive differentiator.OperationalDecisionsupportZero LatencyEnterpriseGlobalisationJBP –Four key drivers which we are seeing in MCBI:-Business AgilityOperationalSecureIntegratedLots of terms – Real Time Enterprise“RTE compete by using up-to-date information to progressively remove delays to the management and execution of its critical business processes. Gartner”“Enterprises compete by using up-to-date information to progressively remove delays to the management and execution of its critical business processes. Gartner”
8 Mission Critical BI – Real World Examples E-everything – 24x7E-GovernmentHealth care monitoring –Commissioning / Payment for quality / resultsReferral to treatment timesPayment for QualityTelecommunicationBandwidth management / Mobile CoverageOrder to fulfilment MISRetail – Just-in-time inventoryJBP
9 Mission Critical – Challenges Mission Critical BI is not new! So why is it so hard?“Pace of change” keeps increasing …Continued Pressure on IT Spend – estimated ~20-30% reduction in 2009/10.BI / DW keeps evolving –Many of the original mission statements of BI/DW remain elusive.Increased demand for integrated information – e.g. unstructured, social media, etc.Data Explosion – “Data volumes will grow exponentially while CPU capacity will increase only geometrically. Gartner”.Security of all the information is paramountBI/DW remains a predominately “build” activity.JBPBuild as opposed to Buy.Pace of change : According to TDWI 2008 BI benchmark report –6-8 weeks and for large enterprises far longer!to add a new data source.New Corporate report / dashboard5 weeks for a new KPI.Security - Business Continuity; Compliance mandates; Privacy and confidentiality ; Data warehouse as corporate memory and Security threatsBI – CIO’s No.1 priority for 3 years now falling … Gartner. BI is currently no.5 behind Cloud, virtualisation, web 2.0, etc.
10 Mission Critical – EDW Scale ComplexityBusiness ModelData IntegrationMixed WorkloadExploitationNumber of UsersExploitation MaturitySizeData LoadedData warehouse sizeInformation outputNumber of different views need to be considered when quantifying the challenge ahead.Varies by industry, type of business and geography.JBPEWOC have specialised in the Enterprise DW space – top decile in terms of the 3 main measure above. Having said that we believe many of the MCBI principles discussed today apply across the whole vertical.Type of business – SoHo, SME, Corporation, Government, etc.
11 Mission Critical BI Architecture Jason Perkins – 30m
12 Central Data Warehouse EDW ArchitecturesIndependent Data MartVirtual Data WarehouseHub & SpokeCentral Data WarehouseEasy to Build OrganizationallyLimit ScopeEasy to Build TechnicallyNo need for ETLNo need for separate platformAllows easier customization of user interfaces and reportsTailor spokes for business.Single Enterprise “Business” ViewData reusabilityConsistencyLowest TCOBusiness Enterprise view unavailableRedundant data costsHigh ETL costsHigh App costsHigh DBA and operational costsOnly viable for low volume accessMeta data issuesNetwork bandwidth and join complexity issuesWorkload typically placed on op systemsBusiness Enterprise view challengingMedium ETL costsData latencyRequires corporate leadership and visionRequires fully performant and scalable technologyJBPMany different EDW architectures – here are four of the most popular.Dimensional lifecycle similar to Hub & spoke – conformed dimension and facts act as the hub.Poll the audience to see what they have?
13 Mission Critical Maximum Availability Flexibility Maintenance Security LifecycleMethodInfrastructureAdaptabilityOperationsMigrationsTechnologyJBP
15 Mission Critical DW Architecture BI ApplicationsOperationalAd hocQueryDataExtractsAlertsAnalyticsDashboardsReportingWeb ServicesAuditingSecurityPerformance TierMetadata ServicesResourceManagementConsolidationMartsAggregatesOLAPSandpitsRecovery / RestartIntegration TierProblem ResolutionDataQualitySCDManagerFactLoaderAdoptionServicesConformingSurvivorshipError ManagementWorkflow MonitorStaging TierJob SchedulingJBPAdoption services (AKA late arriving data).Aligns with Oracle DW reference architecture and Kimball ETL sub systems.Auditing – Central data store for capturing and monitoring ETL and Exploitation audit. Audit dimension for managing against business facts.Security – responsible for ETL and Exploitation (RBAC) security.Metadata services – capture and exploit business and technical metadata.Resource management – responsible for monitoring and managing DW resources across ETL and exploitation.Recovery and restart – logical decomposition with regular checkpoint. Reusable ETL framework for recovery and restart.Problem resolution – capture, prioritise, manage and communicate all maintenance and operational issues.Error management – Central data store for capturing and monitoring business and technical errors. Error event fact for flagging issues.Workflow management – monitor job status include pending, running, completed and suspended jobs.Job scheduling – schedule jobs and nested jobs. Control relationship and dependencies. Auditing – Central data store for capturing and monitoring.Customer tracking – capturing and monitoring customer inbound (load) and outbound (exploitation) requests.CommunityManagementLoaderServicesChange DataCaptureValidationServicesCustomer TrackingBusinessProcessBusinessApplicationsUnstructuredExcelXMLExternalMDM
16 Serviceability Architecture Automation – lights out / zero touchFlexibility - meta data/reference data drivenRobustness - error tracking, handling & reportingMaintenance - load/event tracking & reportingResilience –Ability to stop individual parts of the system, restartRobustness - error tracking, handling & reportingJBPFar too often people neglect the operational side of a DWH when building; in a Mission critical system this becomes paramount.Operationally ready
18 Nursery Method Raison d'être BI/DW requires an Iterative approach. Mission critical is no different.New deliveries and changes must:-Protect core services.Facilitate “pace of change”Support re-useAllow experimentationAdapt to changing requirementsInvolve usersDeveloped “Nursery” Method in responseSupports front room and back room deliveriesReduce cycle time.“Nurseries” (AKA Sandboxes) – user initiated ETL processProduction of Transformation and Load templatesWill O’Shea …This new way is not really new, but is already known under a number of guises…Rapid Application Development (RAD)Agile Model Driven Development (AMDD)Agile Unified Process (AUP)Iterative and Incremental developmentRational Unified Process (RUP)Scrum DevelopmentJoint Application Development (JAD)All involve multiple iterations along the path to implementation.All involve small teams, ideally co-located, at minimum meeting frequently and working collaboratively.
19 Nursery Method Growing a system RequirementsAnalysis & DesignPlanningImplementationNurseryPlanting the seedInitial PlanningTransplantImplementationEvaluationTestingDeliveryMust have involvement from all teams involvedMust have buy in from all teams involvedEveryone, business & developers, learns from both development and use of the systemIntroduces the ability to act on what has been learnedLeaves Nursery when mature, and is transplanted into production – not re-grown.
20 Nursery Method The Growing Stages Analysis & DesignIntegrated Small teamsDesign specificationImplementationDid I mention Integrated Small teamsElaboration & Implementation specificationTestingBy both business and developersDeliveryDelivery to usersEvaluationUser feed backQuality reportsTransplantFinal delivery should match 1.1 somewhatInitial PlanningHigh level overall planHow long are iterationsWhat deliverables are requiredHigh level requirementsPlanningIntegrated Small teamsDetail Iteration planHigher level plan for 2 & 3 iterationRequirementsRequirements for iterationShould fit within iterationor get broken into small bitsStart with lowest levelWill O’Shea …
21 Nursery Method Creating a Nurturing Environment First StepsInitial PlanOverall objective?By when?Define RolesAssign RolesBusiness roles?User roles?Supplier roles?Commitment from those in the roles!!Define communicationMeetings?FrequencyTypesPeriodic weeding - ScrumWatering sessions – Stand-upsothersRoles involved in eachTight Integration of rolesDocumentation from each role – smallFrequency of documentationType of documentationDefine outputs from each iteration/phasePlan for cycleRoles involved at what stageRequirement documentation – smallInitial ScheduleLength of iterationsPotential number of iterationsBuilding the NurseryWill O’Shea …
22 Nursery Method Creating a Nurturing Environment Next StepsDefine system requirementsNumber of data suppliers ?Amount of data?Number of users?Size of infrastructure requiredDefine First few iterationsCycle 1Get data ?Load data ?Extract data ?Distribute data ?Cycle 2Build some validation?Extract validation outcome?Cycle 3Build in some robustness?Size of PlotGrowth cyclesWill O’Shea …
23 Nursery Method Principals Focuses on:Users – Not Processes and toolsWorking systems – Not exhaustive documentationWorking together – Not adhering to the contractDelivering what is wanted – Not following a planAdapting to Change – Not Issuing Change RequestsBoth the Left side and the Right side must exist, but the emphasis isOn the Left – Not the Right.BenefitsCycle time from months to weeks, even days!Improve quality – leverage “Lessons Learned”, as they happenReduce:CostDelivery timeHappy Users !!!Our Real world examplesLarge International pharmaceutical company (delivered in Months not years)Healthcare Provider (implemented new functionality in days)Will O’Shea …
24 Nursery Method Greenhouses - Sandboxes What Constitutes a SandboxWhat are the characteristicsHow do they need to act & interactUsers’ play areasUsing the “Build Once – Use Many” principal users canLoad new data setsCreate new tablesCreate new reportsPlay with existing dataNeeds Work Flow Management – Key in a Mission Critical systemIsolates the effects of users’ play areas from productionDoes Not isolate the data.User can access production dataOther users can access their dataMechanism should exist to release into Production – if requiredSandboxes are not Production; but rather a pathway to productionSandboxes are used as design, not as codeLet’s users developThough users are NOT developersYou must supply a set to tools and templates for them to use.Tool across the spectrum, not just reportingMust have a way of getting the data to move throughout the system, this is where workflow management comes inUsers can set up a series of connected jobs that load data from source though to utilization
26 Nursery Method Exploitation – Managing “live” changes Differentiate between types of changes – one size does not fit all.Determines how many Cycles it should stay in the Nursery.Minor Changes to Reports and Semantic LayerCategory 1 –Changes to pre-canned reports / extractsDo not require changes to Semantic layerCategory 2 – Deployment to live of new reports created by information analysts.Category 3 – Simple changes to the Semantic layer.New ReportsCategory 4 – Creation of new reports / extracts.Changes Impacting semantic layerCategory 5 –Other changes to the semantic layer.creation of new derived fields (not to be performed in the universe).Category 6 – Changes to pre-canned reports / extracts that require changes to semantic layer.Category 7 – Creation of new semantic later.Different approach adopted based on business priority (of change and the existing business asset), size and complexity of changes required.
27 Nursery Method Exploitation elaboration workflow Will O’Shea …Further complicated if already providing a live service.Example work flow we have used for delivering exploitation requirements into a live service.Talk around how you use a offshore / outsource partners.
28 Mission Critical Principles & Operating Model Will O’Shea – 90mThe Detail ..
29 Mission Critical Adaptability “Pace of change” – keeps increasing …Its all about speedSpeed of changeSpeed of information access“Design for change” – as opposed to “built to last”Design to: Build Once – Use ManyEnter “Business Rule Management” (BRM)Process – Business Process Management (BPM).Rules – Decision logicData – Decision variablesProcessRulesJason PerkinsGain more flexibility by separating decision logic from implementation.Rules – how an organisation want to actData
30 Mission Critical Adaptability Design for changeProcess –Business Process Management for operational decision supportProcess flow or workflow for tactical / strategic decision supportRules –Rules Drive the ProcessDeclarative approachBusiness user managedDescriptiveData –Meta/Reference data Enforces the RulesThus data Drives the ProcessContextualVolatileFlexibleProcessRulesDataJason Perkins
31 Mission Critical Adaptability Examples of rules management …Jason PerkinsExamples of Business Rules Management:-Fair Issacs Blaze -Oracle Data Integrator (ODI) -Expressor -
32 Operational Principles Flexibility Users require “flexibility” without the need to re-develop.Need to be able to Add and/or ModifyLoad ProcessApplication processingError processingValidationsRecipients of Load statistics (DQ, Errors, etc)Encryption ProcessLoad and use new data (joined to existing data)As and when they want toWithout new code !!!Will O’Shea …
33 Operational Principles Maintenance Operational team require the ability to configure and monitor processes.View ETL progress (real time)LoadsLoad stepsLoad StatisticsReporting and tracking by:LoadBusiness UnitTimeStatusPerformance and statistic reporting.Error tracking & maintenance against LoadControl Loads if neededStart (automatically & manually)Hold/Pause all or part of a load(s)Stop LoadsRestartable (from where needed)Will O’Shea …Reference data populated error messages, each with a severity codeMessage ids unique throughout application (for code maintenance)Error maintenance, reporting & tracking against loadsOperations and user maintainable error cause and solution(s) repository for each id.
34 Operational Principles Administration Business require KnowledgeSystem should output meaningful & understood Error messages.Specific Messages throughout application, so business know the area.Visibility of Operations Error maintenance.Ability to feed into processStatistical Real-time reporting & tracking of loads.Know what data has been loadedKnow how much data has been loadedKnow what stage each load is at.Know what business units have loaded data.Will O’Shea …Overall goal is to automate and minimise / reduce administration.System should be self managing – with administration by exception.
35 Operational Principles Resilience Business & Operations require A robust & resilient systemLoads may be automatically restarted from where they were stopped/failed (as required)Each load job, step and statistic has start/end times and statusETL checks status of job to determine if it needs to/can be run.Fatal errors need manual intervention before they may be rerun.Performance and statistic reportingSelf initiating LoadsWill O’Shea …ResilienceLoads may be automatically restarted from where they were stopped/failedEach load job, load job step and load job step statistic has start and end times as well as statusETL checks status of job to determine if it needs to/can be run.Fatal errors need manual intervention before they may be rerun.Performance and statistic reporting
36 Operational Principles Summary Data Warehouses require “Metadata Driven Processing” (MDP)How?Where can MDP help your DWH?What Metadata does MDP need?Feed MDP into Development stream?Educate developers to use itEducate user to request it.Educate the business to use it.What can be MDP and what can’t?Loading Data – Types of loads, Source to targetLoad Control – Starting, stopping, branching, etcErrors & Messages – effects of & reporting on,Validation (DQ) – how, what, when & reportsEncryption – how, what & whenReverence Data ProcessingWill O’Shea …
38 Metadata Driven Processing The Metadata Driven ETL InfrastructureStorage AllocationCPU AllocationMemory AllocationSand PitBusinessUnitSeverityFatalErrorWarningInformationJobCollection of StepsHas a start and an End…SchemasProjectValidationLookupsStatic valuesData QualityPatternsLinkageMan/OpsEtcMessageValidationLoadProcessingJob StepGet dataLoad stagingLoad AtomicHuman InteractionEtc.BUJobCause &SolutionSourceSUSCancer RegistryInternalTargetInternalBO / OBIBUJob StepBUValidationAdditionalLess the non-mandatoryWill O’Shea …Meta driven code and processingTypeCSV FileXMLTableReport/ExtractSchema SourceSchemaTarget
39 Metadata Driven Processing The Jobs - ETL Metadata Driven Processing (MDP)Definition of JobsLoads are specific instances of a JobBuild re-usable modulesMetadata driven code, promote MDPQuicker time to delivery, develop and test onceAdd/Change source and target by changing MDP dataAdd/Change ETL by changing MDP dataJobValidationLookupsStatic valuesData QualityPatternsLinkageMan/OpsEtcJob StepSourceTargetPick ListsDefined by Reference dataExamples:Date range validationForeign Key LookupsMandatory / Optionaldd-mm-yyyy vs. yyyy/mm/ddY/N vs. 1/0Will O’Shea …TypeCSV FileXMLTableReport/Extract
40 Metadata Driven Processing The Messages – Driving force Fatal – Fails the loadInvalid file formatError – Load keeps goingMax number of errors?% of load rather than #Warning – not following rulesDate format etc.Information – no affect on loadDates out of rangeVisit after treatmentSupports MDPFeeds Metadata Driven ETLShould be used throughout ETLFailure Checks/TrapsExceptionsReporting (DQ & Validation)Each error/trap/exception has a unique Message IDHeadings/Titles/TextSeverity can be changedChanges processing when changedSeverityFatalErrorWarningInformationMessageGroupingMessageValidationLoadProcessingUsageError reportingTextual objectsInformation MessagesLoad ReportingLoad ControlWill O’Shea …Cause &SolutionHelps with future occurrencesUpdated & Maintained
41 Metadata Driven Processing Data Quality & Linkage Supports MDPKey in any system, but more so in a MC one.Use Metadata to Drive processImportant right people get right dataQuicklyRules Based ValidationData Quality ValidationLinkage ValidationNew rules can be added/removedWhen needed(no code required)Businesses users decide to add rulesFrom pick listDefined using building blocksSeverity of failure of rule can be changedBusinesses users decide severityBusinessUnitCanadian OfficeFinish OfficeUK OfficeAudit DataReportsMetadataDrivenETLValidationOutcomeValidationRulesWill O’Shea …LookupsStatic valuesRangeConversionsPatternsLinkageMan/OpsEtcReportsDataIntegration & Quality Team?
42 Metadata Driven Processing Encryption Supports MDPEncryption is simply a specific Instance of a JobBuilt to perform EncryptionNew Encryption Types can be added but do require codeNew columns to be encrypted can be added by simply adding metadata, no code.Keys can be stored or added at run-timeAES128Triple DESLook-upHome-Grown?NameDoBID #EncryptionTypeAudit DataSourceDataColumnTypeParameters(keys)Will O’Shea …MetadataDrivenETLTargetDataSource & TargetDefinition
43 Metadata Driven Processing Reference Data Management Supports MDPNew reference data can be added without new codeDifferent BUs can have different data but though same RDMTDifferent Import types are catered forDifferent Table Types are catered fore.g. K-Type 1, 2 & 3, Home grown, etc.e.g. CSV, XML, ExcelTableTypesBusinessUnitImportTypesReferenceTableDefinitionsBU SourcesMetadataDrivenETLColumnDefinitionsSourceAttributeDefinitionsSourceDefinitionsWill O’Shea …TargetDataAudit DataSourceData
44 Metadata Driven Processing The Metadata Model Can be as large as you want to take it.It is as good as you use it. If you don’t use it, no matter how good it is, it won’t be workingShould be easy to set up and run, too much complication make it difficult to use.Try to keep application areas seperate though with common bonds
45 Metadata Driven Processing Extensibility Extending the Mission Critical Data Warehouse.Most BI/DW requirements are not green field.Extending existing is a key design objective.Build Once – Use ManyAdding new data sourcesChange existing data sourcesData linage - MetadataWhere data has come fromWhere it has goneWhat has happened to it along the wayImpact AnalysisNew exploitation (analysis and reporting) of existing DWAdding new exploitation capabilities to DWAudit DataMajority of DW are over 2 years old. However few DW are ever finished …
47 Technology DriversExamples of technology features supporting Mission Critical BI.Analytics outside Data warehouseBI Web ServicesHigh Availability Data WarehousingReal-Time Data WarehousingMaster Data Management (MDM)JBP – Question who in audience is pursing one of these DW features at the moment?From “TDWI Best Practice Report, Next Generation Data Warehouse Platforms, By Philip Russom”.
48 Mission Critical Performance Leaving the Nursery (or Sandbox)Productionise the codePerformance!!BalanceBrute force –MPP (medium to high volumes / complexity / users)SMP (low volume / complexity / users)Performance LayerBI tool and RDBMS calibrationSpeed of ETL vs. Need of Retrieval - when to do something and when to not.80 – 20 ruleSelective DenormalisationSelective Pre-JoinsAggregates and Summaries – are they always needed DWA no?, SMP yes?OLAPPerformance metadataRow countsElapsed timeJBPGoal shows be to move administrators up the value chain – i.e. how can they help us exploit the data and gain new / additional insights.Do not want them spending all time on tuning and keeping system running.
49 Mission Critical Administration Resources ManagementJBPResource Management - Goal Oriented, Workload-centric, AutomatedEnable management of mixed workloads – ETL and Exploitation; Operational, Tactical & Strategic;Based on business priorities / SLA’s.Not all BI is mission critical – phew!Prioritise resources for Mission CriticalBI ApplicationsBack office workload
50 Information Lifecycle Management Not all information is mission critical – phew!Many benefits to segmenting information by its usefulness to the business.Performance / ThroughputCost effectivePrioritisation of resourcesILM - Number of levelsSeparate active and non active data.Compression non volatile dataRead only for historicILM - Intelligent storage based on usage of information.Automation is a key (emerging) requirement for supporting MCBI.JBPA key enabler for dealing with the explosion in data volumes.~ 30% compressing non volatile data – Forrester.
51 Mission Critical Security Security includes …Business ContinuityConfidentialityInformation ClassificationNon RepudiationPrivacyApply principle of “defence in depth” with multiple layers relating to security of information.Protecting customer identifying information.Pseudonymisation (P14n)AnonymisationLinkage across datasets and over time but NOT customer identifying.UsableAudit Services: provision of audit trail forTransactions applied to the database.Access to data in the database.Will O'SheaFor more information please see ISO Information Security StandardConfidentiality (patient consent)Access controlAuditInformation SecurityBusiness ContinuityData integrity
53 Mission Critical Infrastructure Mission critical infrastructure requirementsAvailability & ResilienceCapacity on demandEase of managementLinear ScalabilityData warehouse infrastructure“Roll your own” data warehousesDeclining …Data warehouse appliance (DWA)The “new” kid on the blockCloud ServicesWay of the future?Jason Perkins
54 Mission Critical – Maximum Availability Data warehouse now have to meet following with NO downtime.Planned OutagesSystem ChangesApplicationChangesMigrations / TransitionsUnplanned OutagesInfrastructureFailuresIssuesDataErrorHumanDegradedServiceInsufficientCapacityWorkload ManagementJBP
55 Mission Critical – Maximum Availability RequirementsMeasured in 9’sNo single point of failure.Tolerates many outages transparentlyStraightforward administrationAvailability and ResilienceActive / StandbyActive / PassiveDual ActiveFallbackBackup and recoveryAutomationHot vs. ColdIncremental vs. FullSecond siteSoftwareOperationalNetworkHardwareJBPA&RActive / stand by – common “stand by” will be re-used as a NFT environment.Active / Passive – redundant version maintained. Idle resources.Dual active – minutes to hours unplanned outage protection – Second site for disaster recovery and data centre failure.Fallback – protection against “no data loss”. Possible reduced service in case of failure.B&RAutomation – do you have to design or will system detect changes?MCBI is “Hot”Challenge for the new wave of Data warehouse Appliances.Low cost commodity components …SAN backup and recovery options.Fault isolation – failure in any component does not cause cascade failure to whole system.
56 Mission Critical Service Availability Data MigrationsNew requirements –No downtime for on boarding data or exploitation.No impact to data freshness.Minimise impact on existing system.Differentiate betweenMigrations of new data sourceMigrations for existing subject areas (more common)Phased data migrations.Emerging Integration patternsGreen field data migrationParallel Trickle data migrations.Mini batch data migrationsJBPMost companies have data warehouse in 21st. Average DW is over 5 years old so migrations against existing subject areas are the most common these days.
57 Mission Critical Data Migrations Independent data migration of (new) data source.Partition data migration in order to batch / trickle.Impact volumes against pattern to understand impact of additional throughput.Resource management a key requirement to protect existing system.No downtime or data freshness impact on business.Original structuresNewstructuresETLData MigrationGreen fieldMini batchOr Trickle123JBP
58 Mission Critical Data Migrations Concurrent maintenance of new and old structures.Cut over on completion of data migration to new structures.Impact volumes against pattern to understand impact of additional throughput.Failure to either new or original structures must result in rollback of both.No downtime or data freshness impact on business.Original structuresNewstructuresETLData MigrationParallel Trickle PatternTrickle12JBP
59 Mission Critical Data Migrations ETL Maintenance at single data structure at any point in time.Logically segment the source data into discrete partitions.Execute mini batch migrations, focusing on each partition in turn.Partition on volatility with early phases based on least volatile data.Catch-up mini batches required for changes during transition before final cut over.No downtime or data freshness impact on business.Original structuresNewstructuresETLData MigrationMini Batch Pattern123Mini batchesJBP
60 Mission Critical Data Migrations Pre-requisitesData profiling and analysis of new / changes in data migrationUp front planning for Pipe cleaning and RehearsalPractically SelectiveOnly select entities you know you will need in that phase.If your hitting an entity consider taking it all.Transition –Fail to plan is plan to fail!rehearsal is key.RollingData quality monitorsAudit and ReconciliationJBP20% up front time on data analysis, pipe cleaning and rehearsal.
61 Summary Mission Critical is here … What we need is an “Intelligent Data warehouse”Metadata drivenBuild once – use manyWhy do we need it?Business Agility through Nursery Method –Facilitates “pace of change” of business.Protects existing Mission Critical BI Services.Operational patternsEmpower the businessSupport the Mission Critical BI Services.Integrated – exploitation of the customer “360 view”Secure – ensuring the right information to the right personWill O’Shea
62 ReferencesMassive But Agile: Best Practices For Scaling - The Next-Generation Enterprise Data Warehouse, Forrester.TDWI Best Practice Report, Next Generation Data Warehouse Platforms, Philip Russom.The ETL Toolkit, Ralph Kimball.Smart (Enough) Systems, James Taylor.Best Practices Mitigate Data Migration Risks and Challenges, Gartner.
63 Questions Thank you Further queries contact us at:- Jason & Will can be contacted at:-For further information at