Software-defined network(SDN)

What is SDN? It is an approach to networking that focuses on centralizing control functionality and providing programmatic interfaces into a wide range of network equipment It is a new approach to networking that aims to make data networks more flexible, easier to operate and manage, and better able to respond to the changing demands of applications and network conditions

SDN Architecture

OpenFlow

A network of OpenFlow-enabled commercial switches and routers

NOX The network operating system presents programs with a centralized programming model; programs are written as if the entire network were present on a single machine Programs are written in terms of high-level abstractions, not low-level configuration parameters

Related work 4D project – The goal of 4D project is to control forwarding(their network view only includes the network infrastructure(link, switch, router)) SANE and Ethane – Provide a broader class of functionality by including a namespace for users and nodes and mapping NOX – Extends SANE/Ethane in two dimensions. First, it attempts to scale this centralized paradigm to very large systems. Second, it allow general programmatic control of the network.

NOX vs Maestro NOX focuses on providing applications with a higher level of abstraction so they need not deal with low-level details Maestro focuses on controlling the interactions between applications

OpenFlow vs NOX OpenFlow -> an abstraction for a particular network component -> a device driver NOX provides network-wide abstractions -> operating systems provide system-wide abstractions

Programming Your Network at Run- time for Big Data Applications Explore the tight integration of application and network control Study the run-time network configuration for big data applications to jointly optimize application performance and network utilization Discuss the integrated network control architecture, job scheduling, topology and routing configuration mechanisms for Hadoop jobs

System Architecture

Mesos

Traffic pattern of big data applications The shuffle phase of MapReduce In parallel database systems, most operations require merging and splitting data from different tables

Disadvantages of other current approaches It is difficult to estimate real application traffic demand based only on readings of network level statistics Blindly optimizing circuit throughput without considering application structure could cause blocking among interdependent applications and poor application performace

Job scheduling in Hadoop Traffic demand estimation of hadoop jobs Network-aware job scheduling

Topology and Routing for Aggregation Patterns

Several issues Fairness, priority and fault tolerance Traffic engineering for big data applications