Presentation on theme: "Iterative Waterfall Case Study: Network Information Data On-line Analysis Alessandro Zorer"— Presentation transcript:
Iterative Waterfall Case Study: Network Information Data On-line Analysis Alessandro Zorer
Agenda Iterative Waterfall methodology (based on Sodalia SIMEP) General approach DWH ‘tailoring’ Case Study: Network Information Data On-line Analysis Needs Approach Focus on System Architecture Functional View Process View Development View Physical View Fifth View Summary Q&A
Iterative Waterfall Methodology & Process Iteration # 2 Iteration # 1 System Development Strategy Concept Exploration and Context Analysis Iteration # n System Requirements Iterative approach Multilayer Multiperspective System Architecture. Component Design. Component Development. Component Design. Component Test. System Test.
Adaptation to DWH Requirement Manager Use Case Prioritization Context Analyst Use Case and Actor identification Architect DW Designer Data Mart Designer Use Case model Structuring Integration Analysis Data Warehouse design DW Analysis Interface Prototyping DW may be tailored to each specific project and domain Product impact Analysis Expressed as a UML activity diagram
Context Analysis Business Model Develop Business Model Stakeholder Identification Manage Dependencies Elicit Stakeholder Needs Activity Diagrams Glossary Find Actors and Use Cases Structure the Use-Case Model Requirement Attributes Use-Case Model Responsible for Capture a Common Vocabulary Context Analyst
Concept Exploration UML Requirement Definition and System Architecture DESMET Analysis Requirements Analysis Business Analysis Use case identification Data Sourcces identification Data Consumers identification System Architecture System Architecture Design Performance Scalability Capacity Architectural Qualities Tools Assessment & Evaluation
Architectural Views Logical View A logical abstract view of system elements and of the services to be provided to the end user Process View Analysis the dynamic aspect of the system through scenarios and other diagrams (e.g., sequence, collaboration and activity diagrams). The elements focused are: tasks with their workflow, processes with their dependency, synchronization and concurrence aspects. Development View Organization of the actual software modules in the software-development environment. The modules may be packaged in components or subsystems (component diagram) which may be organized in a hierarchy of layers, each layer providing a narrow well-defined interface to other layers. Physical View Provide the deployment configurations in terms of Hardware and Software Components. This view shows the System Topology, a network of processing nodes with the software running on them. Capacity issues are addressed. Fifth View Orthogonal view. Issues addressed are: Potential Software Reuse Analysis, Requirements allocation on Components, Performance Analysis, Functionality Categorization and Ranking.
DESMET Methodology for evaluating COTS based on: Functional Qualities Architectural qualities (i.e. adaptability, Scalability) Performance analysis Business aspects Time to market System lifecycle Contractual constraints Support organization
DW Design System Architecture Design UML E-R Physical Architecture Design Data Modeling MD Schema Data Flow Design Metadata Management Design Data MartsInput Design Output Design
DM Design Data Mart Construction TestingTraining Deployment Iteration CustomizationUnit Testing Data FlowDatabaseHardware
Support tools infrastructure Business Process Modeling Object / Component Modeling Logical / Physical Database Modeling Enterprise Integrated DW Modeling
Case Study: Network Information Data On-line Analysis Business needs: Definition and development of a DataWarehouse Framework for Multidimensional Analysis based on: Call Data (Network Management) Fault Data (Problem Management) Performance Data (End-to-End Analysis) Optimization of network performances through gathering and analysis High integrability of new data sources Optimization and extension of on-line analysis functionalities Quick creation of reports and flexibility for the end user (through custom Data Marts) Extension of capabilities in term of historical data management.
Solution A specialized and adaptable Data Warehouse solution to support Network Traffic Management and Call Behavior Analysis through a smart data correlation among CDR and configuration, performance and trouble tickets Highly scalable to adapt from small to large business needs Based an a mix of COTS and developed components Flexible to accomadate a variety of different sources and Call Data Record formats Detailed data analysis capabilities to support different DSS customer organizations Predefined “good example” analysis library to quikly develop and deliver QoS monitors and Service Level Management functions
System Framework Approach Simplify the design, implementation, and management of data warehousing solutions An open architecture that allows easy integration with and extended by third party vendors Heterogeneous data import, export, validation and cleansing services with optional data lineage Integrated metadata for warehouse design, data extraction/transformation, server management, and end-user analysis tools Core management services for scheduling, storage management, performance monitoring, alerts/events, and notification
Logical View UML Domain and System Modeling describes system concepts in a formal way drives data modeling drives components design drives dynamic modeling Standard-based Object Information Model (OIM) from Microsoft and Metadata Coalition
Layered Modeling Organization Data Transformation Layer Data Warehouse Layer Data Analysis Layer MetaData ManagementWorkFlow Management
Generic Record-oriented Model Element DeployedRecordLogicalRecord 88Level Field Record Column Attribute Classifier ModelElement Group RecordFormat 0..* +Type +DeployedCatalogs SummaryInformationTransformableObject GroupDef DeployedGroupLogicalGroupDeployedFieldLogicalField RecordItem
Generic Call Data Record Model NetworkElementServiceType CallDataRecord DeployedRecord SourceIDDestinationIDElapsedMeasure NEType
OLAP and DSS Fast five seconds or less. Analysis Performs basic numerical and statistical analysis of the data, predefined or ad hoc Shared Implements the security requirements across a large user population Multidimensional Is the essential characteristic of OLAP Information Accesses all the data and information wherever it may reside and not limited by volume.
Metadata Management 1. The link between the DSS system and the business analysts. 2. Critical for maintaining, controlling, and expanding the DSS system. Reduces the cost and cycle time of problem resolution. Technical Users (Developers & Analysts) Business Users (Executives & Business Analysts) Data Administrator METADATA
Metadata Consumer Business Users Less technical Use predefined queries & reports DSS navigation and definition Power Business Users More technical Ad-hoc Technical Users Acquisition & access developers, analysts, data modelers, architects Need users access patterns & frequency Transformation rules
Metadata Management Business Meta Data Technical Meta Data Technical Users (Developers & Analysts ) Business Users (Executives & Business Analysts) Transformation Rules Attribute Names Domain Values Access Patterns Entity Relationships Attribute Business Definitions Entity Business Definitions Aggregation Rules Report Business Descriptions List of Available Reports Power Business Users Data Administrator
Data transformation u Finding the right data to satisfy end users needs u Moving the right data to the target u Scheduling and monitoring u Providing visual access u Linking transformations and movement metadata with all other metadata activity
Workflow Management Process integration DataintegrationDataintegration Metadata Operational Data DW
Functional Architecture Operational Systems Data Trasformation Data Transformation Tools Load Validation Tools Warehouse Management Tools Data Warehouse Operational DB Applications Meta Data Sources Data Cleansing Tools Metadata Repository Enterprise Reference Data CASE & Modeling Tools Source Data Extract Tools Database Utilities Data Quality Assessment Tools DWData Marts Data Mining & Simulation Tools OLAP Data Query, Reporting and Visualization Tools Query Meta Data Movement & Replication Tools Meta Data Access Tools Meta Data Administration Utilities Project Deliverables Generator Meta Data Management Change Management Tools
Layered Architecture Data Transformation Enterprise Warehouse Replication & Propagation Dependent Data Mart Knowledge Discovery / Data Mining Information Access / Applications Business Users IT Users Data Warehouse Middleware Network ManagementDatabase ManagementSystems Management Network Admin Operations Manager Database Admin Data Analyst Applic. Developer Power Analyst Knowledge Worker Executive/ Manager Customer Contact Application Server Source Data (Internal and/or External) Metadata Logical Data Model Physical Data base Design Data Dictionaries Workflow Management Data Capture Data Management Data Analysis Support Infrastructure Staging Area
Components Integration Data Analysis Integrated Support Infrastructure Data Management Data Capture
Components Integration Data Analysis & DSS Data capture Integrated Support Infrastructure Summarization Data Cleaning & Maintenance Communication Service Infrastructure Query Data Management Schedule-driven acquisition Transformation chain Asynchronous acquisition Data Browser Workflow Change Management Report Sched WEB Services System Management
Physical View UnixMVS Server Platform Client Platform Windows NT Directory Services DW Intranet Windows 95/98/NT Unix WS OLAP Corporate Data
System Scalability System Sizing Small Size ( <= 10 M CDR/day ) Medium Size ( >= 10 M <= 50 M CDR /day ) Large Size ( >= 50 M <= 200 M CDR / day ) Solutions : Process distribution (divide et impera) Different COTS choice (performance and TCO) Hardware platform
Architectural Qualities Performance (Canned queries, MD Analysis, Ad hoc, Min. Impact on Operational System) Flexibility (MD Flex, Ad hoc, Change data structure) Scalability (No. of Users, Volume of Data) Ease of Use (Location, Formulation, Navigation, Manipulation) Data Quality (Consistent, Correct, Timely, Integrated) Connection to the Detail Business Transactions
Summary Iterative waterfall approach for large projects … Architecture as a CENTRAL activity for the success of projects Scalability as a driving factor in this case Standard adoption (Metadata Coalition OIM Model) COTS + developed components to meets Time to market and Best-in-class solution Flexibility in data capturing and high modularity to improve the level of integration with already in place systems Q&A