1. An overview of roadmap plans for the APSD
Big Data-Fueled
Feedback Loops in
SDN/NFV
Matt Olson
CenturyLink
An overview of roadmap plans for the APSD
Big Data Analytics Platform (“BDAP”).
B=Big Data
R= Repository
A = Artificial Intelligence
I = Integrated
N = Network
DANN - Data & Analytics Neural Network
Cognitive Computing
March 31, 2016
Strata+Hadoop World

2. Who am I?
Part of a small team deploying focused on transformation of networks with Software Defined Networking and Network Function Virtualization
Background in Network Performance Engineering and Analytics
Accidental foray into Data Science motivated by the desire to solve problems in the network space
Relative newcomer to the field, but with years of experience data wrangling and building tools in Splunk, and a variety of other platforms

3. What’s my problem?
CenturyLink is deploying a cloud based SDN/NFV platform capable of transforming network service delivery.
The SDN/NFV platform is awesome for rapid and automated provisioning, but . . .
To be really awesome, it needs a nervous system and brain
Also, it’s crazy complicated - we need systems to help people manage the complexity until the robots are ready
The challenge is to build a brain for the SDN/NFV fabric. At its core, this is a Big Data and an algorithmic challenge.

4. Why does it matter? SDN is poised to transform Telecom completely
“Within 5 years SDN/NFV will be the mainstream option for service providers deploying cloud and network architectures “ Tech Target, February 2015
CenturyLink is leading that charge with a huge commitment
“(CenturyLink) plans to have full global virtualization of its IP core network and data centers by ” FierceTelecom, October 22, 2015
Telecom is changing, and to be successful, we need to jump ahead of the curve and drive that change.
Under the hood, SDN/NFV is highly complex – Success depends inherently on automation, and intelligent use of the data generated from the platform.

5. A little background - SDN
Software Defined Networking (SDN) is a natural corollary to service virtualization
The lines between Network and Services are blurring – Services are networked, and network provisioning should be as agile as the service provisioning.
This need has prompted the development of SDN
API driven routing and switching
Automated configuration and management of network connectivity
Separation of the control plane and forwarding plane, allowing for use of commodity hardware
Provisioning in minutes, rather than days!
Scaling and cost advantages (OPEX and CAPEX)

6. SDN Framework Software! Programmable via APIs
Opportunity to use commodity hardware
Centralized interfaces

7. A little background - NFV
Network Function Virtualization allows us to leverage cloud services to provision network functions in a dynamic and flexible manner
Network services run on VM’s.
Control and data plane elements are fully abstracted, leveraging virtualized compute resources for scaling, resiliency, and agile provisioning across the VM platform.
Introduces sophisticated Layer functionality in areas such as firewalls, load balancers, and video and IMS  services.
API driven provisioning of complex ‘service chains’ across the Layer 2 and 3 fabric.
Single source of state and transactional data related to services end-to-end and state of the network and elements delivering those services.  

8. NFV Framework Heterogeneity & Complexity Homogeneity & Simplicity
Non-NFV
Fragmented non-commodity hardware
Physical install per appliance per site
Hardware development large barrier to entry for new vendors, constraining innovation & competition
Heterogeneity
& Complexity
Virtualize the Network…
NFV – Network Functions Virtualization.
SDN – Software Defined Network.
Open source
API driven
At Cloud Scale…
Scale-out, not up
Commodity hardware + Acceleration
Consumable REST API’s.
(Automation / Reduce or No-touch)
Value the data…
Collect network data into a Data Lake
Easy tools to access & mine the data.
Homogeneity
& Simplicity

9. Programmable Services Backbone
7 Countries
44 Total Global Locations
Data Center & Telecom Facilities
1 University & 4 DoD Labs
SDN
NFV
TeraPoP/Core
PSB
SDN/NFV
MIA
ORL
XXX
Orchestration & Automation
API’s
CTL Cloud
MPLS
Central Offices
& Data Centers
Donation of 22 years of computing time in less than 2 years (Cancer, Ebola, AIDS, Genome Markers)
Distributed-NFV CPE w/X86 Virtualization
& IoT
Premise Based

10. PSB Stack for SDN/NFV Network as a Service Platform
API
Orchestration NFV)
Programmable
Services
Backbone
Networking (SDN)
Virtualization (OpenStack / VMWare)
Hardware

11. Orchestration & Controllers
BRAIN and Adaptive Service Chains
SVC2
SVC1
SVC3
Services Layer
SVC4
SVCn
API
Portal
OSS/BSS
SDN/NFV
Orchestration & Controllers
TOSCA
Engine
Network
Forwarding Graph
BRAIN
Prescribe
Predict
Inform
Simple on the outside (service chain, and API wrapper, but this hides a tremendous amount of complexity)
BRAIN - Big data Repository and Artificial intelligence Integrated Networking
SDN/NFV Platform Generates massive amounts of metadata related to the platform, and the services provisioned on the platform
The SDN/NFV platform is also highly complex, with multiple layers of abstraction, complex end-to-end service chains, and dynamic topology
This presents a huge operational challenge!
But . . .
The platform provides an integrated source of rich Real Time data
The platform provides API interfaces and inherent agility
=> A huge opportunity for automation and Intelligent services!
Data Warehouses
& Data Lakes

12. Big Data? Diverse metadata sources:
Platform health and state data (bare metal, hypervisors, VMs, Applications)
Network interface statistics and network traffic data (packet stats, NetFlow, SFlow etc.)
Service health and usage data (transactional data, active test and probe data, QoS)
Reference and topology data – Needed to enrich and make sense of platform and service data

13. Machine Learning and Analytics
The availability of rich platform data and service level data allows the application of Machine Learning techniques, and the training of models to drive optimized service chain architecture.
Supervised learning focused on measured service quality and availability
Tuning for specific services offered (e.g. Voice, Video, etc.)
Incorporation of service and node interaction and usage patterns / node interactions
Incorporation of platform element and link failure modes
Train models on accumulated historical data, and apply model derived analytics to real time streaming data for pattern detection.

14. Feedback Loops
With trained models feeding on streaming platform data . . .
Automated . . .
Trigger detection (e.g. a surge in traffic between nodes, service degradation observed on a network segment, etc.)
Determination of remediation / optimization action
Execution of orchestration action via API calls
Evaluation of results with ‘commit’ or roll-back
Through the use of Neural Network modeling and multi-dimensional feedback loops, network and service rebalancing can be supported in a broad manner enabling network and service resiliency, even under conditions of highly dynamic load shifts and/or multiple failures.
Predictive, as well as responsive – Anticipate emerging needs
Add contextual awareness and service intelligence

15. Use Cases - Adaptation

16. Target Architecture Streaming and Batch Ingest
Connectivity to various reference data stores (RDBMS and API interfaces)
Persistence in HDFS with batch analytics using MapReduce
Spark stack for analytics
MLlib – Machine Learning
TensorFlow - Deep Learning

17. Reference and Topology
BRAIN
PSB Streaming Data
Apex / Kafka
Hadoop
People
Data
Science
Reference and Topology
Spark
R / SQL
MLlib
Robots

18. Practical Considerations - Scaling and Leverage
We are a very small team, ‘the tip of the spear’
So We need tools which are solid, resilient and scalable. Ideally, we need pre-existing solutions for scaling, resiliency, governance, etc. so we can focus our energies on new development further up the stack
Ingest: DataTorrent (Apache Apex)
Persistence: MapR (Hadoop and Mapr-FS)
Presentation and analytics: Splunk / Hunk

19. More Practical Considerations – Legacy Integration
This is an interesting technical problem, but the real issue is that we’re brownfield . . .
Interconnected legacy networks
Legacy organizations
Legacy systems
Scarcity of Big Data skills
Incremental incorporation of automation. Human interaction and exploration in a familiar environment is key
Integration with existing legacy OSS platforms is a must
=> Splunk (with Hunk)

20. Splunk/Hunk Architecture
Analyze
Explore
Visualize
Dashboards
Share
Hadoop Client Libraries
Hunk offers Full-featured Analytics in an Integrated Platform
Explore, analyze and visualize data, create dashboards and share reports from one integrated platform.
Hunk enables everyone in your organization to unlock the business value of data locked in Hadoop
Hunk integrates the processes of data exploration, analysis and visualization into a single, fluid user experience designed to drive rapid insights from your big data in Hadoop. Enable powerful analytics for everyone with Splunk’s Data Models and the Pivot interface, first released in Splunk Enterprise 6.
And Hunk works with what you have today
Hunk works on Apache Hadoop and most major distributions, including those from Cloudera, Hortonworks, IBM, MapR and Pivotal, with support for both first-generation MapReduce and YARN (Yet Another Resource Negotiator, the technical acronym for 2nd generation MapReduce) and Amazon EMR and S#. Preview results and interactively search across one or more Hadoop clusters, including from different distribution vendors. Customers can also use the ODBC driver to feed data from Hunk to third-party data visualization tools or business intelligence software.
Because it leverages our experience across thousands of organizations, we’ve naturally made Hunk easy to deploy.
Hadoop Clusters

21. Splunk / Hunk Unified Search
Intelligently Search Across Real-Time and Historical Data Using the Same Splunk Interface
Intelligently search across real-time and historical data with unified search. With one simple search you can access data from Splunk Enterprise and Hadoop.
Optimize Splunk Enterprise search head performance for real-time monitoring, alerting and dashboarding with short-term historical context
Hunk search, analyze and visualize months or years of historical data in Hadoop
Run unified queries and dashboards across Splunk Enterprise and Hunk
Real-Time Data
Historical Data in Hadoop

22. Splunk / Hunk – Legacy Integration
Immediate support for OSS / Operational support functions
Existing REST API interface to Northbound systems
Support for streaming ingest as well as Hadoop data store and batch processes
Leveraging existing resources and expertise
Ease of use – Coping with scarcity of Specialized Big Data related skills
Splunkers and Splunk community
Existing infrastructure and resources
Spanning all relevant data sets
Legacy Network data (already streaming into Splunk) – Across all relevant networks, bridging the legacy network and Virtualized
RDBMS (via DBConnect) – reference, Topology, and customer data
Native Splunk Indexed data
Hadoop via HUNK

23. Splunk / Hunk – User Interface
Opportunity for exploration of the data
For developers – Rapid development with low barrier to entry
For Operations and Engineering
Search language, Data Models, and Pivot
Interactive / Real Time
Data Drill Down
Flexible Schema on Search (with Hunk support for HIVE Metastore)
Visualization and analytics
Operational support
Data Governance
RBAC
Support for LDAP Single Sign on
Resource bundling into Apps
Environment for development and sharing of views and analytics

24. Hunk Deployment Overview
Hunk shares all architectural aspects in common with Splunk Enterprise – it runs splunkweb and splunkd. It only differs operationally by its license key.
Hunk
NameNode (CLDB)
SplunkWeb
ResourceManager
Hadoop Cluster
Splunkd
ERP
DataNode 1
NodeManager
Linux Local
HDFS
The Hunk license will enable Hunk ERPs to interface with the Hadoop framework. Splunkd on Hunk determines if a search addressed is data external to Splunk index and dispatches a job accordingly.
Your data
DataNode n
Linux Local
NodeManager

25. Mixed-mode Search
Previews data and allows users to search interactively by pausing and refining queries
Pause or stop MapReduce jobs
Preview results
Switch over
time
preview
Splunk Stream
Hunk starts the streaming and reporting modes concurrently. Streaming results show until the reporting results come in. Allows users to search interactively by pausing and refining queries.
This is a major, unique advantage of Hunk compared to alternative approaches such as Hive or SQL on Hadoop which require fixed schema in an effort to speed up searches, while Hunk retains the combination of schema on the fly with results preview.
- Streaming mode (scanning) – this low-latency ERP process does the rudimentary search by retrieving a superset of the results from external stores.
- Reporting mode – this ERP process runs the full Splunk remote search pipeline by pushing down the computation to DataNodes. This has the inherent advantage of using distributed computation such as Hadoop. A knowledge bundle is also pushed to HDFS and gets used for data enrichment.
preview
Hadoop MR / Splunk Index
Time

26. BRAIN Integrated Analytics Platform
PSB Streaming Data
Apex / Kafka / Splunk Fwd
Splunk
Index
Splunk (SH)
OSS
Legacy Network Streaming Data
People
- Ops
Dev BI
Data
Science
Hadoop
Hunk
Reference and Topology
Spark
R / SQL
MLlib
Robots

27. Summary SDN/NFV offers tremendous transformative potential
The convergence of Big Data Technologies, ML, and SDN/NFV presents a huge opportunity to develop next generation service delivery
But, introducing this transformation in an existing large network environment requires incremental change, and iterative improvement
Need to Crawl -> Walk -> Run and work with tools that are resilient, scalable, open, and extensible
Though the ‘Singularity’ is the ultimate goal, need to engage existing teams and systems with the data
And, ultimately, the biggest challenges are organizational, and cultural, not technical . . .