1. Time Series Data Repository (TSDR)
Project Proposal

2. TSDR Functional Objectives
To capture ODL data into a persistent time series data repository
This includes:
Statistics counters
Performance data
Health status information
Operational configuration data
To facilitate various applications built on top of TSDR
Applications include:
Operational configuration optimization
Traffic engineering
Network analytics with automated intelligence
Security risk detection
Performance analysis
Major functions
Data Collection
Data Storage
Data Queries
Data Aggregation
Data Purge
Lithium Focus
TSDR functionalities on OpenFlow Statistics data

3. TSDR Design Objectives
Generic and Extensible architectural framework
Generic and extensible TSDR Data Model.
Abstract and generic TSDR Persistence Layer
with TSDR Persistence APIs
Allow implementation of various data store plugins under TSDR Persistence Layer
with HBase Plugin as an example TSDR Data Store implementation.
Scalable with high performance
Providing both integrated and distributed architectures
to handle different scales of time series data
Fully utilizing MD-SAL’s clustering capability
to handle performance and scalability in large scale deployment scenarios

4. TSDR Integrated Architecture
TSDR Data Services including Data Collection, Data Storage, Data Query, Data Purging, and Data Aggregation are MD-SAL services.
Data Collection service receives time series data published on MD-SAL messaging bus from MD-SAL southbound plugins.
Data Collection service communicates with Data Storage service to store the data into TSDR.
TSDR data services access TSDR Data Store such as HBase through generic TSDR Data Persistence Layer.
Needs MD-SAL notification subsystem support.

5. TSDR Distributed Architecture
In large data center deployment scenarios, TSDR Distributed Architecture would be needed to handle the performance and scalability.
In distributed architecture, TSDR data services are deployed in a separate MD-SAL instance.
The data pushed onto MD-SAL messaging bus by ODL southbound plugin are propagated to the other MD-SAL instance for TSDR data services to process into TSDR data repository.
Needs ODL clustering support.

6. TSDR Data Flow with multiple data models
TSDR Data Flow involves multiple data models including source data model ( OpenFlow statistics), TSDR data model, and TSDR plugin ( HBase) data model.
Data Collection Service subscribes to receive OpenFlow Statistics data from MD-SAL Notification Subsystem and passes the data to Data Storage Service.
Data Storage Service converts OpenFlow Statistics data model to TSDR data model.
HBase TSDR Plugin converts TSDR data model to HBase specific data model based on HBase TSDR schema design.

7. Unstructured or Semi-Structured data consideration – for future release
For unstructured or semi-structured data such as syslog data, MD-SAL receives the data in the format of syslog specifica data model.
Data Filtering and Preprocessing can be added to filter out the data noise and optionally extract structured information from the semi-structured data.
Third party specific TSDR plugin such as Splunk Plugin could be added under TSDR Data Persistence Layer to work with proprietary data stores.
Data Aggregation Service is not needed when handling unstructured data.
Third party tools such as Splunk could leverage Data Query Service to obtain the unstructured data from TSDR and add application specific processing on top of it.

8. TSDR Data Model The goal of the TSDR data model design:
Generic
Extensible
Scalable
Performance Optimized
The data model captures:
Statistics data
Log type of data
Note: To add a new group, extend TSDRBaseRecord
DataCategory contains:
Flow Stats
Interface Stats
Queue Stats
Flow Group Stats
Flow Meter Stats
Log Records
Note: More categories can be added to the above list.
RecordKeys contains:
A list of composite keys
Different categories contain different set of keys
Key set validation is needed based on different data categories

9. TSDR Persistence APIs Interface Name Description/comments
Extends from ODL Common APIs?
Specific to TSDR Persistence API?
Will be implemented in HBase plugin in Lithium?
save()
Including saving one or a list of objects
Yes No
find()
Including query based on a list of IDs, with specified criteria, and paging support
count()
delete()
Including delete with one or a list of IDs, and delete the entire table
exists()
Including query based on one or a list of IDs
min(), max(), avg()
For Data Aggregation purpose

10. HBase TSDR Schema – Raw Data
TableName
RowKey
Column Family: Column Qualifier = Cell Value
FlowMetrics
MetricID_NodeID_TableID(_FlowID)_timestamp
‘raw’ = metric_value
InterfaceMetrics
MetricID_NodeID_TableID(_PortID)_timestamp
QueueMetrics
MetricID_NodeID_TableID_PortID_QueueID_timestamp
GroupMetrics
MetricID_NodeID_GroupID(_GroupBucketID)_timestamp
MeterMetrics
MetricID_NodeID_GroupID(_MeterID)_timestamp
Schema Design considerations:
General HBase Schema Design Rules applied:
Keep RowKey, Column Family Key, Column Qualifier as short as possible.
Design the RowKey properly so as to keep rows evenly distributed in multiple data nodes.
Keep the number of column family low
Other performance considerations:
Multiple tables are created based on the data categories in the TSDR data model.
Data storage and query operations run much faster on smaller data sets stored in HBase tables with structured keys.

11. HBase TSDR Schema – Aggregated Data
TableName
RowKey
Column Family: Column Qualifier = Cell Value
HourlyFlowMetrics
MetricID_NodeID_TableID(_FlowID)_timestamp
‘min = metric_value
‘max’ = metric_value
‘avg’ = metric_value
HourlyInterfaceMetrics
MetricID_NodeID_TableID(_InterfaceID)_timestamp
HourlyQueueMetrics
MetricID_NodeID_TableID_PortID_QueueID_timestamp
HourlyGroupMetrics
MetricID_NodeID_GroupID(_GroupBucketID)_timestamp
HourlyMeterMetrics
MetricID_NodeID_GroupID(_MeterID)_timestamp
For performance consideration, we design multiple aggregation tables with different granularity.
Aggregation tables with different granularity will have similar schema as displayed above

12. HBase TSDR Data Model
TSDR HBase Plugin converts the generic TSDR data model into HBase specific data model based on HBase schema design.
TSDR HBase Plugin leverages this HBase specific data model to implement the generic
TSDR Persistence APIs including storage, query, purging, and aggregation to complete the TSDR data services in HBase.

13. TSDR Scope in Lithium
In the Lithium release, we will focus on the following deliverables:
Architectural framework
Data Type Support
as specified in the architectural design
OpenFlow Statistics
Deployment scenarios support
Data Collection mechanisms
TSDR Integrated Architecture
HBase on Hadoop single node deployment scenario
Implement Pub/Sub collection mechanism
Data Persistence Layer
Functionality implementation
Complete TSDR Persistence APIs with interface definition
Data Collection
Data Storage
TSDR Plugin
Data Model implementation
HBase plugin as an example implementation
Focus on the storage API implementation in HBase plugin to support Data Storage Service in Lithium
TSDR Data Model to support OpenFlow Statistics
HBase Data Model for HBase Plugin implementation