Reference Architecture Subgroup NIST Big Data Public Working Group Reference Architecture Subgroup September 30, 2013 Co-chairs: Orit LevinMicrosoft James KetnerAT&T Don KrapohlAugmented Intel

Reference Architecture Subgroup Agenda Deliverable #1: White Paper: Survey of Existing Big Data RAs Deliverable #2: NIST Big Data Reference Architecture Next Steps 2

Reference Architecture Subgroup NIST Survey Big Data Architecture Models Input Document M0151

Reference Architecture Subgroup List Of Surveyed Architectures Vendor-neutral and technology-agnostic proposals – Bob MarcusET-Strategies – Orit LevinMicrosoft – Gary MazzaferroAlloyCloud – Yuri DemchenkoUniversity of Amsterdam Vendors’ Architectures – IBM – Oracle – Booz Allen Hamilton – EMC – SAP – 9sight – LexusNexis 4

Reference Architecture Subgroup Vendor-neutral and Technology-agnostic Proposals 5 Data Processing Flow M0039 Data Transformation Flow M0017 IT Stack M0047

Reference Architecture Subgroup Vendor-neutral and Technology-agnostic Proposals 6 Data Processing Flow M0039 Data Transformation Flow M0017 IT Stack M0047

Reference Architecture Subgroup Vendor-neutral and Technology-agnostic Proposals 7 Data Processing Flow M0039 IT Stack M0047 Data Transformation Flow M0017

Reference Architecture Subgroup Vendor-neutral and Technology-agnostic Proposals 8 Data Transformation Flow M0017 IT Stack M0047 Data Processing Flow M0039

Reference Architecture Subgroup Draft Agreement / Rough Consensus Transformation includes – Processing functions – Analytic functions – Visualization functions Data Infrastructure includes – Data stores – In-memory DBs – Analytic DBs 9 Sources Transformation Usage Data Infrastructure Security Management Cloud Computing Network

Reference Architecture Subgroup NIST BIG DATA Reference Architecture Input Document M0226

Reference Architecture Subgroup 11 A superset of a “traditional data” system A representation of a vendor- neutral and technology- agnostic system A functional architecture comprised of logical roles Applicable to a variety of business models –Tightly-integrated enterprise systems –Loosely-coupled vertical industries A business architecture representing internal vs. external functional boundaries A deployment architecture A detailed IT RA of a specific system implementation All of the above will be developed in the next stage in the context of specific use cases. What the Baseline Big Data RA IsIs Not

Reference Architecture Subgroup Main Functional Blocks 12 Big Data Application Provider System Orchestrator Data Consumer Data Provider Big Data Framework Provider Application Specific Identity Management & Authorization Etc. Discovery of data Description of data Access to data Code execution on data Etc. Discovery of services Description of data Visualization of data Rendering of data Reporting of data Code execution on data Etc. Analytic processing of data Machine learning Code execution on data et situ Storage, retrieval, search, etc. of data Providing computing infrastructure Providing networking infrastructure Etc.

Reference Architecture Subgroup Big Data Lifecycle 13 Big Data Application Provider System Orchestrator DATA Visualization Access Analytics Curation Collection Data Consumer Data Provider Big Data Framework Provider DATA

Reference Architecture Subgroup Big Data Frameworks 14 Big Data Application Provider Visualization Access Analytics Curation Collection System Orchestrator DATA Data Consumer Data Provider Horizontally Scalable (VM clusters) Vertically Scalable Horizontally Scalable Vertically Scalable Horizontally Scalable Vertically Scalable Processing Frameworks (analytic tools, etc.) Platforms (databases, etc.) Infrastructures Physical and Virtual Resources (networking, computing, etc.) DATA Big Data Framework Provider

Reference Architecture Subgroup Bringing Tools to the Data 15 Big Data Application Provider Visualization Access Analytics Curation Collection System Orchestrator DATA SW DATA SW Data Consumer Data Provider Horizontally Scalable (VM clusters) Vertically Scalable Horizontally Scalable Vertically Scalable Horizontally Scalable Vertically Scalable Big Data Framework Provider Processing Frameworks (analytic tools, etc.) Platforms (databases, etc.) Infrastructures Physical and Virtual Resources (networking, computing, etc.) DATA SW

Reference Architecture Subgroup Management Security & Privacy 16 Big Data Application Provider Visualization Access Analytics Curation Collection System Orchestrator DATA SW DATA SW INFORMATION VALUE CHAIN IT VALUE CHAIN Data Consumer Data Provider Horizontally Scalable (VM clusters) Vertically Scalable Horizontally Scalable Vertically Scalable Horizontally Scalable Vertically Scalable Big Data Framework Provider Processing Frameworks (analytic tools, etc.) Platforms (databases, etc.) Infrastructures Physical and Virtual Resources (networking, computing, etc.) DATA SW

Reference Architecture Subgroup Outline 17 Executive Summary 1 Introduction 2 Big Data System Requirements 3 Conceptual Model 4 Main Components 4.1 Data Provider 4.2 Big Data Application Provider 4.3 Big Data Framework Provider 4.4 Data Consumer 4.5 System Orchestrator 5 Management 5.1 System Management 5.2 Lifecycle Management 6 Security and Privacy 7 Big Data Taxonomy Appendix A: Terms and Definitions Appendix B: Acronyms Appendix C: References Appendix D: Deployment Considerations 1 Big Data Framework Provider 1.1 Traditional On-Premise Frameworks 1.2 Cloud Service Providers

Reference Architecture Subgroup Summary – The NIST Big Data functional reference architecture (RA v.1.0) is available for review as input document M0226. Next Steps – Continue the editorial and alignment effort – Map generic Big Data use cases to RA – Map specific collected Big Data cases to RA Let’s exchange additional ideas this afternoon at the breakout session! 18