1. The Design and Implementation of Declarative Networks Boon Thau Loo University of Pennsylvania, University of California-Berkeley * *This dissertation was completed as a graduate student at the University of California- Berkeley

2. Declarative Networking A declarative framework for networks: Network protocols are declaratively specified using a database query language Distributed query engine executes specifications to implement network protocols Success of database research: 70’s – today: Database research has revolutionized data management Today: Similar opportunity to revolutionize the Internet architecture

3. Motivation Internet faces many challenges today: Unwanted, harmful traffic Complexity/fragility in Internet routing Proliferation of new applications and services Efforts at improving the Internet: Evolutionary: App-level “Overlay” networks Revolutionary: Clean-slate designs  NSF GENI initiative, FIND program Opportunity: Software tools that can significantly accelerate network innovation

4. A Declarative Network Distributed recursive query Traditional Networks Declarative Networks Network State Distributed database Network protocol Recursive Query Execution Network messages Distributed Dataflow Dataflow messages Dataflow messages

5. The Case for Declarative Networking Ease of programming: Compact and high-level representation of protocols Orders of magnitude reduction in code size  Declarative Chord DHT is 48 lines instead of 10,000. Easy customization Safety: Queries are “sandboxed” within query processor Potential for static analysis of safety What about efficiency? No fundamental overhead when executing standard routing protocols Application of well-studied query optimizations Note: Same question was asked of relational databases in the 70’s.

6. Main Contributions of Dissertation Declarative Routing [SIGCOMM ’05]: Extensible Routers: balance of flexibility, efficiency and safety Declarative Overlays [SOSP ’05]: Rapid prototyping of new overlay networks Database Fundamentals [SIGMOD ‘06] : Network specific query language and semantics Distributed recursive query execution strategies Query optimizations, classical and new

7. A Breadth of Use Cases Example implementations to date: Textbook routing protocols Chord DHT Narada mesh for end-system multicast Distributed Gnutella/Web crawlers Pastry DHT Replication protocols Lamport/Chandy snapshots Paxos distributed consensus Overlays for host mobility Sensor network protocols P2 declarative networking system http://p2.cs.berkeley.edu

8. Outline Background The Connection: Routing as a Query Execution Model Path-Vector Protocol Example  Query specification  protocol implementation Query Processing Beyond routing: Declarative Overlays Ongoing work @ Penn

9. Traditional Router Packets Traditional Router Control Plane Forwarding Plane Routing Protocol Neighbor Table updates Forwarding Table updates

10. Declarative RouterTraditional Router Declarative Router Declarative Queries Control Plane Forwarding Plane Packets Query Engine Routing Protocol Neighbor Table updates Forwarding Table updates

11. All-Pairs Reachability R2: reachable(S,D)  link(S,Z), reachable(Z,D) R1: reachable(S,D)  link(S,D) Input: link(source, destination) Output: reachable(source, destination) “For all nodes S,D, If there is a link from S to D, then S can reach D”. link(a,b) – “there is a link from node a to node b” reachable(a,b) – “node a can reach node b”

12. All-Pairs Reachability R2: reachable(S,D)  link(S,Z), reachable(Z,D) R1: reachable(S,D)  link(S,D) Input: link(source, destination) Output: reachable(source, destination) “For all nodes S,D and Z, If there is a link from S to Z, AND Z can reach D, then S can reach D”.

13. Towards Network Datalog Specify tuple placement Value-based partitioning of tables Tuples to be combined are co-located Rule rewrite ensures body is always single-site All communication is among neighbors No multihop routing during basic rule execution Link-restricted rules: Enforced via simple syntactic restrictions

14. All-Pairs Reachability R1: reachable(@S,D)  link(@S,D) R2: reachable(@S,D)  link(@S,Z), reachable(@Z,D) Network Datalog Query: reachable(@M,N) @SD @ab c d reachable Output table: Input table: Query: reachable(@a,N) @SD @cb d link @SD @bc a link @SD @ab link @SD @dc link bdc a @SD @ba c d reachable @SD @ca b d reachable @SD @da b c reachable Location Specifier “@S” Query: reachable(@a,N)

15. Path Vector in Network Datalog Input: link(@source, destination) Query output: path(@source, destination, pathVector) R1: path(@S,D,P)  link(@S,D), P=(S,D). R2: link(@Z,S), path(@S,D,P) P=S  P 2. path(@Z,D,P 2 ),  Query: path(@S,D,P) Add S to front of P 2

16. R1: path(@S,D,P)  link(@S,D), P=(S,D). R2: path(@S,D,P)  link(@Z,S), path(@Z,D,P 2 ), P=S  P 2. @SDP DP @cd[c,d] Query Execution @SDP DP Query: path(@a,d,P) Neighbor table: @SD @cb d link @SD @bc a link @SD @ab link @SD @dc link bdca path Forwarding table:

17. @SDP DP DP @cd[c,d] Query Execution Forwarding table: @SDP @bd[b,c,d] bdca path(@b,d,[b,c,d]) R1: path(@S,D,P)  link(@S,D), P=(S,D). R2: path(@S,D,P)  link(@Z,S), path(@Z,D,P 2 ), P=S  P 2. Query: path(@a,d,P) Neighbor table: @SD @cb d link @SD @bc a link @SD @ab link @SD @dc link path @SDP @ad[a,b,c,d] path(@a,d,[a,b,c,d]) Communication patterns are identical to those in the actual path vector protocol Matching variable Z = “Join”

18. Other Routing Examples Best-Path Routing Distance Vector Dynamic Source Routing (Wireless) Policy Decisions QoS-based Routing Link-state Multicast Overlays (Single-Source & CBT)

19. Outline Background The Connection: Routing as a Query Query Processing Beyond routing: Declarative Overlays Sampling of ongoing work

20. Recursive Query Evaluation Semi-naïve evaluation: Iterations (rounds) of synchronous computation Results from iteration i th used in (i+1) th Path Table 8 7 3-hop 10 92 1 1-hop 36 5 2-hop 4 Link TableNetwork 5 10 0 2 1 3 4 6 8 7 Problem: Unpredictable delays and failures 9

21. Pipelined Semi-naïve (PSN) Fully-asynchronous evaluation: Computed tuples in any iteration pipelined to next iteration Natural for network protocols Path Table 4 1 7 Link TableNetwork 2 5 8 3 6 9 10 5 0 2 1 3 4 6 8 7 9 Relaxation of semi-naïve

22. Pipelined Evaluation Challenges: Does PSN produce the correct answer? Is PSN bandwidth efficient?  I.e. does it make the minimum number of inferences? Proofs for Basic technique: local timestamps p(x,z) :- p 1 (x,y), p 2 (y,z), …, p n (y,z), q(z,w) recursive w.r.t. p

23. Execution Plan Strands Messages Network In Messages Network Out Single Node Nodes in execution plan (“operators”): Network operators (send/recv, cc, retry, rate limitation) Relational operators (selects, projects, joins, aggregates) Flow operators (mux, demux, queues)

24. Localization Rewrite Rules may have body predicates at different locations: R2: path(@S,D,P)  link(@S,Z), path(@Z,D,P 2 ), P=S  P 2. R2b: path(@S,D,P)  linkD(S,@Z), path(@Z,D,P 2 ), P=S  P 2. R2a: linkD(S,@D)  link(@S,D) Matching variable Z = “Join” Rewritten rules: Matching variable Z = “Join”

25. Localized Rule Compilation Execution Plan  path Join path.Z = linkD.Z linkD Project path(S,D,P) Send to path.S R2b: path(@S,D,P)  linkD(S,@Z), path(@Z,D,P 2 ), P=S  P 2. Network In Network Out  linkD Join linkD.Z = path.Z path Project path(S,D,P) Send to path.S

26. Optimizations Traditional: evaluate in the NW context Aggregate Selections Magic Sets rewrite Predicate Reordering New: motivated by NW context Multi-query optimizations:  Query Results caching  Opportunistic message sharing Cost-based optimizations  Neighborhood density function  Hybrid rewrites PV/DV (Wired)  DSR (Wireless) Hybridized protocols: Zone Routing Protocol

27. Beyond Routing: Declarative Overlays Language extensions to support events and soft-state predicates Chord Routing, including: Multiple successors Stabilization Optimized finger maintenance Failure detection 48 rules 11 table definitions MIT-Chord: x100 more code Another example: Narada mesh in 22 rules 10 pt font

28. Outline Background The Connection: Routing as a Query Query Processing Beyond routing: Declarative Overlays Ongoing work @ Penn

29. Ongoing Work @ Penn Declarative secure networking Difficult to design/implement/reason about secure networks Network Datalog + logic-based security languages [NetDB ’07] Authenticated path vector protocol, DNSSEC, secure DHTs,… Moving forward:  Exploit fine-grained control over networks and security policies Data-centric querying and routing in heterogeneous networks Internet: Wired infrastructure with wireless clouds at the edges Flexible network support for mobility [ACM MobiArch ’07]  Declarative queries for addressing and naming mobile hosts  Session-aware customizable QoS routing Moving forward:  Declarative wireless ad-hoc networks  Cost-based query optimizations to adapt protocols

30. Summary Declarative networking: Declarative Routing:  Extensible routing infrastructure Declarative Overlays  Rapid prototyping overlay networks Database fundamentals  Query language  New distributed query execution strategies and optimizations  Semantics in dynamic networks P2 declarative networking system (http://p2.cs.berkeley.edu)http://p2.cs.berkeley.edu

31. Many Thanks… Advisors: Joseph M. Hellerstein, Ion Stoica Collaborators: UC Berkeley: Tyson Condie, Ryan Huebsch Intel Research: David Gay, Petros Maniatis, Timothy Roscoe Yahoo! Research: Minos Garofalakis, Raghu Ramakrishnan Rice University: Atul Singh Many others…