NIST BIG DATA WG Reference Architecture Subgroup Agenda for the Subgroup Call Co-chairs: Orit Levin (Microsoft) James Ketner (AT&T) Don Krapohl (Augmented Intelligence) September 5, 2013

Agenda Synchronization with other subgroups Deliverable #1 Deliverable #2 Open Issues to be resolved TODAY 09/04/2013NIST Big Data WG / Ref Arch Subgroup2

Deliverable #1: The White Paper (outline posted as M0151) Status: no real progress so far The situation will be discussed tomorrow, Thu, during the weekly subgroup call Call for volunteers to describe different approaches Especially for the authors of proposed architectures 09/04/2013NIST Big Data WG / Ref Arch Subgroup3

Deliverable #2: The Common RA Great progress has been made The two proposals under discussion are very close Both are based on previous contributions and the work of all subgroups The main open issue is around the role of the “Data Sources” We need to resolve this issue before we can move forward Today Clarify the decision / reaching consensus process Make the decision The table of additional open issues is provided Less critical, but needs to be resolved ASAP Close as many as possible open issues today 09/04/2013NIST Big Data WG / Ref Arch Subgroup4

Open Issue: The role of Data Sources in the NIST BD Reference Architecture Proposal I: “Data Manager” uses the “Data Service Abstraction” provided by the Transformation to drive the system Data Sources are passive and therefore are included in the Data Manager Proposal II: “Data Producer” abstracts and introduces the data sources into the system by exposing the “Data Service Abstraction” to be used by other components “System Manager” is a new fifth main component that drives the system (i.e., assumes part of the Data Manager functionality per above) 09/04/2013NIST Big Data WG / Ref Arch Subgroup5

M0189v.1 (+ most recent feedback) M0126v.4 09/04/2013NIST Big Data WG / Ref Arch Subgroup6

Data Manager Is responsible for processing given data using certain methodologies and placing the processed results into certain formats. Main responsibility is to provide data source description (location of data which can be centralized or distributed, data at rest or data in motion, file types and attributes, etc.) to the TF and request one or more data services for the given data. Data services can include (a) collecting data from one or more data sources, (b) deciding what curation process should be performed, (c) determining how data should be analyzed, (d) choosing what methods and formats the processed results should be stored as, and (e) deciding how and where processed results should be pushed to (end users or systems, if any) 09/04/2013NIST Big Data WG / Ref Arch Subgroup7 Proposal I

System Manager (or Vertical Application) Responsible for the supply chain from a variety of Data Producers Data Capabilities Providers Defines and integrates the required data transformations components into the vertical system Uses both internal and external to the system service abstractions Scope depends on the Vertical’s business environment: Tightly-coupled Enterprise: System Manager is a central functional entity Loosely-coupled Vertical: every independent stakeholder implements its System Manager 09/04/2013NIST Big Data WG / Ref Arch Subgroup8 Data Producer Introduces new information feeds into the big data system for discovery, access and use by the BD system: Creates the metadata describing the information source(s), usage policies, and other relevant attributes Using the metadata, publishes the availability of the information and the means to access it through a well-defined interface. Using various technology means, makes the information accessible by the BD system components. These technology means allow subscription to events, listening to data feeds, querying for specific data properties or content, and the ability to upload and host software tools to process the data in situ. Note: New feeds are distinct from the data already being in use by the system and residing the various system repositories (including memory, databases, etc.) although similar technologies can be used to access both. Proposal II

AbstractAlternative ApproachesResolutionNotes Data Sources role1.Data Sources are included in “Data Manager”, which uses the “services” provided by the “transformation” to drive the system. 2.Data Sources are a stand-alone Level 1 component, which provides “data services” to other RA components. TBDBlocking the progress Terminology and Taxonomy used for the Level 1 and Level 2 RA blocks 1. Collector, Manager, Producer, Consumer, etc. 2.Collection, Management, Sources, Use, etc. TBDRelates to the “actors” vs. “roles” vs. “activities” open issue in Def&Tax subgroup Granularity of interfaces shown in terms of data flow and tools flows 1.Shown together by a single arrow 2. Shown separately TBD Placement of Management and “Security & Privacy” blocks 1.Inside the “Management Capabilities” block 2.Each block (Capabilities, Security&Privacy, Management) is stand alone TBD “Management” block resolution 1.Management only, no additional details on the figure 2.Management block contains System Management and Lifecycle Management sub-blocks 3.System Management and Lifecycle Management blocks are stand alone TBD “Capabilities’” sub-block’ vocabulary TBD: Needs targeted input and explicit discussion from the Roadmap subgroup TBDPending sync with the Roadmap subgroup Open Issues 09/04/2013NIST Big Data WG / Ref Arch Subgroup9