1. 1 Quality of Service vs. Any Service at All IWQoS 2005 Passau Germany Randy H. Katz Computer Science Division Electrical Engineering and Computer Science Department University of California, Berkeley Berkeley, CA 94720-1776

2. 2 Presentation Outline The Problem System and Network Trends Checking-Observing-Protecting Services Inspection-and-Action Boxes Annotation Layer Scenario Call to Action

3. 3 Presentation Outline The Problem System and Network Trends Checking-Observing-Protecting Services Inspection-and-Action Boxes Annotation Layer Scenario Call to Action

4. 4 Some Observations Internet reasonably robust to point problems like link and router failures (“fail stop”) Successfully operates under a wide range of loading conditions and over diverse technologies During 9/11/01, Internet worked reasonable well, under heavy traffic conditions and with some major facilities failures in Lower Manhattan

5. 5 The Problem Networks awash in illegitimate traffic: port scans, propagating worms, p2p file swapping –Legitimate traffic starved for bandwidth –Essential network services (e.g., DNS, NFS) compromised Needed: better network management of services/applications to achieve good performance and resilience even in the face of network stress –Self-aware network environment –Observing and responding to traffic changes –While sustaining the ability to control the network

6. 6 From the Frontlines Berkeley Campus Network –Unanticipated traffic surges render the network unmanageable (and may cause routers to fail) –Denial of service attacks, latest worm, or the newest file sharing protocol largely indistinguishable –In-band control channel is starved, making it difficult to manage and recover the network Berkeley EECS Department Network (12/04) –Suspected denial-of-service attack against DNS –Poorly implemented/configured spam appliance adds to DNS overload –Traffic surges render it impossible to access Web or mount file systems Network problems contribute to brittleness of distributed systems

7. 7 Why and How Networks Fail Complex phenomenology of failure Recent Berkeley experience suggests that traffic surges render enterprise networks unusable Indirect effects of DoS traffic on network infrastructure: role of unexpected traffic patterns –Cisco Express Forwarding: random IP addresses flood route cache forcing all traffic to go through router slow path—high CPU utilization yields inability to manage router table updates –Route Summarization: powerful misconfigured peer overwhelms weaker peer with too many router table entries –SNMP DoS attack: overwhelm SNMP ports on routers –DNS attack: response-response loops in DNS queries generate traffic overload

8. 8 Possible Approach New technology: packet flow manipulations at L4-L7 made possible by new PNEs and stateful routers Enables identification/segregation of traffic –Good: protect it –Bad: block it –Suspicious: slow it Check/Observe/Protect Services (COPS) –Inspection-and-Action Boxes (iBoxes) –Annotation layer between routing and transport Yielding new service building blocks –Beyond packet marking and annotation … –To flow extraction and path-oriented statistics collection … –Based on traffic analysis, model extraction, statistical correlation & causality testing

9. 9 Presentation Outline The Problem System and Network Trends Checking-Observing-Protecting Services Inspection-and-Action Boxes Annotation Layer Scenario Call to Action

10. 10 Edge Network Wide Area Network Server Managing Edge Network Services and Applications Not shrink wrap software—but cascaded “appliances” Data Center in-a-box blade servers, network storage Brittle to traffic surges and shifts, yielding network disruption FirewallIDS Traffic Shaper Egress Checker Load Balancer Blades Edge Network Middleboxes

11. 11 Appliances Proliferate: Management Nightmare! F5 Networks BIG-IP LoadBalancer Web server load balancer Packeteer PacketShaper Traffic monitor and shaper Ingrian i225 SSL offload appliance Network Appliance NetCache Localized content delivery platform Nortel Alteon Switched Firewall CheckPoint firewall and L7 switch Cisco IDS 4250-XL Intrusion detection system Cisco SN 5420 IP-SAN storage gateway Extreme Networks SummitPx1 L2-L7 application switch NetScreen 500 Firewall and VPN

12. 12 Network Support for Tiered Applications LAN Wide Area Network Server App Tier Egress Checker Server Web Tier Server Database Tier Firewall Load Balancer Datacenter Network(s) Unified LAN Wide Area Network Server Servers on Demand Load Balancer + Firewall + Egress Checker Blades Server Servers on Demand Configure servers, storage, connectivity net functionality as needed

13. 13 “The Computer is the Network” Emergence of Programmable Network Elements –Network components where net services/applications execute –Virtualization (hosts, storage, nets) and flow filtering (blocking, delaying) Computation-in-the-Network is NOT Unlimited –Packet handling complexity limited by latency/processing overhead –NOT arbitrary per packet programming (aka active networking) –Allocate general computation like proxies to network blades Beyond Per Packet Processing: Network Flows –Managing/configuring network for performance and resilience –Emergence of stateful routers for flow-based management –Adaptation based on Observe (Monitor), Analyze (Detect, Diagnose), Act (Redirect, Reallocate, Balance, Throttle)

14. 14 Presentation Outline The Problem System and Network Trends Checking-Observing-Protecting Services Inspection-and-Action Boxes Annotation Layer Scenario Call to Action

15. 15 Check Checkable Protocols: Maintain invariants and techniques for checking and enforcing protocols –Listen & Whisper: well-formed BGP behavior –Traffic Rate Control: Self-Verifiable Core Stateless Fair Queuing (SV-CSFQ) Existing work requires changes to protocol end points or routers on the path –Difficult to retrofit checkability to existing protocols without embedded processing in PNEs –Develop building blocks for new protocols »Observable protocol behavior »Cryptographic techniques »Statistical methods

16. 16 Observe Observation and Action Points –Network points where control is exercised, traffic classified, resources allocated –In the datapath statistical collection + annotating, prioritizing, shaping, blocking, … –Inspection-and-Action Boxes (iBoxes) »Prototyped on commercial PNEs »Placed at Internet and Server edges of enterprise net »Cascaded with existing routers to extend their functionality »Migration into (some current and) future router architectures

17. 17 Protect Protect Crucial Services –Minimize and mitigate effects of attacks and traffic surges –Classify traffic into good, bad, and ugly (suspicious) »Good: standing patterns and operator-tunable policies »Bad: evolves faster, harder to characterize »Ugly: cannot immediately be determined as good or bad –Filter the bad, slow the suspicious, maintain resources for the good (e.g., control traffic) »Sufficient to reduce false positives »Some suspicious-looking good traffic may be slowed down, but won’t be blocked

18. 18 Presentation Outline The Problem System and Network Trends Checking-Observing-Protecting Services Inspection-and-Action Boxes Annotation Layer Scenario Call to Action

19. 19 iBoxes: Observe, Analyze, Act Enterprise Network Architecture Inspection-and-Action Boxes: Deep multiprotocol packet inspection No routing; observation & marking Policing points: drop, fence, block

20. 20 Generic Network Element Architecture Interconnection Fabric Input Ports Output Ports Buffers “Tag” Mem CP AP Action Processor CP Classification Processor Rules & Programs

21. 21 RouterVM High-level specification environment for describing packet processing Virtualized: abstracted view of underlying hardware resources of target PNEs –Portable across diverse architectures –Simulate before deployment Services, policies, and standard routing functions managed thru composed packet filters –Generalized packet filter: trigger + action bundle, cascaded, allows new services and policies to be implemented / configured thru GUI –New services can be implemented without new code through library building blocks Mel Tsai

22. 22 Extended Router Architecture Virtualized components representative of a “common” router implementation Structure independent of particular hardware Virtual line card instantiated for every port required by application Virtual backplane shuttles packets between line cards CPU handles routing protocols & mgmt tasks Compute engines perform complex, high-latency processing on flows Blue “standard” components Yellow components added & configured per-application Filters are key to flexibility Mel Tsai

23. 23 GPF “Fill-in” Specification “Packet filter” as high-level, programmable building-block for network appliance apps FILTER 19 SETUP NAME - SIP - SMASK - DIP - DMASK - PROTO - SRC PORT - DST PORT - VLAN - ACTION - example any 255.255.255.255 10.0.0.0 255.255.255.0 tcp,udp any 80 default drop Classification Parameters Action Traditional Filter RouterVM Generalized Packet Filter (type L7)

24. 24 GPF Action Language Basic set of assignments, evaluations, expressions, control-flow ops, “physical” actions on packets/flows –Control-flow: If-then-else, if-not –Evaluation: ==, =, != –Packet flow control: Allow, unallow, drop, skip filter, jump filter –Packet manipulation: Field rewriting (ip_src == blah, tcp_src = blah), truncation, header additions –Actions: NAT, loadbalance, ratelimit, (perhaps others) –Meta actions: packet generation, logging, statistics gathering Basic Filter –Simple L2-L4 header classifications –Any RouterVM actions L7 Filter –REs, TCP termination, ADU recon NAT Filter –Capabilities beyond simple NAT action available to all GPFs Content Caching –Builds on L7 filter functionality WAN Link Compression –Simple to specify, but requires lots of computation IP-to-FC Gateway –Requires own table format & processing XML Preprocessing –Not very well documented, and difficulty is unknown…

25. 25 Presentation Outline The Problem System and Network Trends Checking-Observing-Protecting Services Inspection-and-Action Boxes Annotation Layer Scenario Call to Action

26. 26 Network-Level Observe-Analyze-Act Observe –Packet, path, protocol, service invocation statistical collection and sampling: frequencies, latencies, completion rates –Construct the collection infrastructure Analyze –Determine correlations among observations –“Normal” model discovery + anomaly detection –Exploit SLT Act –Experiment to test correlations –Prioritize and throttle –Mark and annotate –Control theory? Distributed analyses and actions

27. 27 Network Layer Mechanism: Annotations Enhance network visibility: disseminate observations, communicate actions, provide in- band network management actions, iBox-to-iBox communications iBoxes label packets at annotation layer but do not rewrite packet contents Annotations stack, must be removed from packets before delivery to A-layer unaware end nodes Expose annotations to application layer? Phy Link Network Annotation Transport Session Presentation Application

28. 28 Annotation Layer: Simple Marking Example Marking vs. rewriting approach –E.g., mark packets as internally vs. externally sourced using IP header options Prioritize internal vs. external access to services solves some but not all traffic surge problems

29. 29 Annotation Layer: iBox Piggybacked Control Plane Problem: Control plane starvation Use A-layer for iBox-to-iBox communication –Passively piggyback on existing flows –“Busy” parts of network have lots of control plane b/w –Actively inject control packets to less active parts –Embedded control info authenticated and sent redundantly –Priority given to packets w/control when net under stress Network monitoring and statistics collection dissemination subsystem

30. 30 Presentation Outline The Problem System and Network Trends Checking-Observing-Protecting Services Inspection-and-Action Boxes Annotation Layer Scenario Call to Action

31. 31 R R Distribution Tier E E E S S I R IAIA E Internet Edge Access Edge Server Edge Spam Appliance Primary & Secondary DNS Servers ISIS S Mail Server S Scenario: Traffic Surge Inhibiting Network Services DNS Server swamped by excessive request traffic –Observe: DNS time outs, Web access traffic slowed, but also higher than normal mail delivery latency implying busy server edge (correlation between Mail Server and DNS Server utilization?) –Root Cause: High DNS request rates generated by Spam Appliance triggered by mail surge

32. 32 Scenario Continued How Diagnosed? –I-S detects high link utilization but abnormally high DNS traffic –Stats from I-I: high mail traffic, low outgoing web traffic, in traffic high but link utilization not high –Stats from I-A: lower web traffic, no unusual mail origination –Problem localized to Server edge, but visibility limited R R Distribution Tier E E E S S I R IAIA E Internet Edge Access Edge Server Edge Spam Appliance Primary & Secondary DNS Servers ISIS S Mail Server S

33. 33 Scenario Continued Possible Action Responses –Experiment: Redirect local DNS requests to Secondary DNS server: if these complete, can infer the server is the problem, not the network –Throttle: Due to MS-DNS correlation, block/slow email traffic at Server Edge: should expect reduced DNS server utilization R R Distribution Tier E E E S S I R IAIA E Internet Edge Access Edge Server Edge Spam Appliance Primary & Secondary DNS Servers ISIS S Mail Server S

34. 34 Presentation Outline Problem and Approach System and Network Trends Checking-Observing-Protecting Services Inspection-and-Action Boxes Annotation Layer Scenario Call to Action

35. 35 Edge Network System Perspective Needed! Distributed Middleware Client SLT Services Distributed Middleware Server Internet IP Network Router Edge Network iBox Prototype Applications Programming Abstractions For Roll-back and wide-area distributed computations Crash-only services + Observation Infrastructure for System SLT Checkable Protocols Fast Detection & Route Recovery Observation Infrastructure for network SLT Commodity Internet Operator User Application- Specific Overlay Network

36. 36 iBoxes implemented on commercial PNEs –Don’t: route or implement (full) protocol stacks –Do: protect routers and shield network services »Classify packets »Extract flows »Redirect traffic »Log, count, collect stats »Filter/shape traffic Hope for Emerging Platforms

37. 37 Summary and Conclusions Processing-in-the-Network is real –Networking plus processing in switched and routed infrastructures –Configuration and management of packet processing cast onto PNEs (network appliances, blades, stateful routers) Needed: Unifying Framework –Methods to specify functionality and processing »RouterVM: Filtering, Redirecting, Transformation »Map from policy intentions to network actions? »Local observations/correct global behavior? Application-specific network processing based on session extraction

38. 38 Summary and Conclusions PNEs: foundation of a pervasive infrastructure for observation and action at the network level –iBoxes Observation and Action points –Annotation Layer for marking and control Check-Observe-Protect paradigm for protecting critical resources when network is under stress Functionality eventually migrates into future generations of routers –E.g., Blades embedded in routers

39. 39 Quality of Service vs. Any Service at All Randy H. Katz Thank You!

40. 40