1. Data Streaming in Computer Networking
Cristian Estan, George Varghese
University of California, San Diego

2. Data streaming in computer networking - MPDS 2003
Talk structure
Traditional streaming in networking
Rules of the game
Iteration paradigm: packet scheduling example
New streaming problems
Detecting malicious traffic
Understanding network workloads
June 8, 2003
Data streaming in computer networking - MPDS 2003

3. Internet service model
Source port
Destination port
Source IP address
Destination IP address
Data
Header
Conversations (flows) broken up into packets handled independently by the network
Packets contain detailed information
Destination IP address
Source IP address
“Application”: protocol field + source and destination port
At the core of the network high speed routers
Decide what to do with each packet
Flow
Internet
June 8, 2003
Data streaming in computer networking - MPDS 2003

4. Traditional router functions
IP Lookup ?
Incoming 1
Outgoing 1
Incoming 2
Outgoing 2
Decide which interface to send the packet on (route lookup)
Incoming 3
Outgoing 3
June 8, 2003
Data streaming in computer networking - MPDS 2003

5. Traditional router functions
IP Lookup
Out2
Incoming 1
Outgoing 1
Incoming 2
Outgoing 2
Incoming 3
Outgoing 3
June 8, 2003
Data streaming in computer networking - MPDS 2003

6. Traditional router functions
Switching
Out2
Out3
Incoming 1
Outgoing 1
Out3
Incoming 2
Outgoing 2
Move packets from between interfaces (switching)
Out1
Out2
Incoming 3
Outgoing 3
June 8, 2003
Data streaming in computer networking - MPDS 2003

7. Traditional router functions
Scheduling
Incoming 1
Outgoing 1
Flow 1
Flow 2
Incoming 2
Outgoing 2
Flow 3
Decide which packets to send and which to delay or drop (scheduling)
Incoming 3
Outgoing 3
June 8, 2003
Data streaming in computer networking - MPDS 2003

8. Traditional router functions
Scheduling
Incoming 1
Outgoing 1
Flow 1
Flow 3
Flow 2
Incoming 2
Outgoing 2
Incoming 3
Outgoing 3
June 8, 2003
Data streaming in computer networking - MPDS 2003

9. Data streaming in computer networking - MPDS 2003
Rules of the game
Wire speed processing
At 40 gigabits/s 8 nanoseconds per packet - need fast SRAM
Limited SRAM (say 32 megabits) but millions of flows
What does this mean for algorithms?
Low worst case complexity bounds
Low bounds on the amount of memory used
Differences from databases
One pass vs. multiple passes
Worst case vs. average case
Small constants vs. asymptotic complexity
June 8, 2003
Data streaming in computer networking - MPDS 2003

10. Data streaming in computer networking - MPDS 2003
Talk structure
Traditional streaming in networking
Rules of the game
Iteration paradigm: packet scheduling example
New streaming problems
Detecting malicious traffic
Understanding network workloads
June 8, 2003
Data streaming in computer networking - MPDS 2003

11. Data streaming in computer networking - MPDS 2003
Iteration paradigm
Many networking algorithms use iteration in time
Way to allow multi-pass algorithms without storing input by assuming inputs do not change quickly
Many examples (MULTOPS for DoS detection [Gil01], CSFQ for scheduling [Stoica98])
Would be nice to formalize tradeoff between quality of results and drift rate of input
Perhaps exponential averaging is not enough
June 8, 2003
Data streaming in computer networking - MPDS 2003

12. Example: Core Stateless FQ
If R>F drop with probability 1-F/R
Iteratively compute fair share F R
Fair queuing: if traffic is larger than link capacity, limit the large flows to the “fair share”
The size of the fair share depends on the rates of all flows
Per flow state is impractical
Core Stateless:
Uses labels in packets to determine rates of flows
Computes the fair share using iterative approach
Minimal state
Exploits stationarity in the traffic mix
Mark rate R
June 8, 2003
Data streaming in computer networking - MPDS 2003

13. Data streaming in computer networking - MPDS 2003
Talk structure
Traditional streaming in networking
Rules of the game
Iteration paradigm: packet scheduling example
New streaming problems
Detecting malicious traffic
Understanding network workloads
June 8, 2003
Data streaming in computer networking - MPDS 2003

14. New streaming problems
Detecting malicious activity
Flooding (denial of service attacks)
Worms
Scans looking for vulnerable servers
Understanding workloads
Billing
Planning network growth
Application mix
June 8, 2003
Data streaming in computer networking - MPDS 2003

15. Detecting malicious traffic
Well defined building blocks
Detecting large aggregates
Similar to iceberg queries
Counting active flows in an aggregate
Similar to counting distinct values
Many open problems: e.g. detect worms and DoS attacks (not clear what is right formal problem statement)
June 8, 2003
Data streaming in computer networking - MPDS 2003

16. Data streaming in computer networking - MPDS 2003
Talk structure
Traditional streaming in networking
Rules of the game
Iteration paradigm: packet scheduling example
New streaming problems
Detecting malicious traffic
Understanding network workloads
June 8, 2003
Data streaming in computer networking - MPDS 2003

17. Informal problem definition
Analysis
Traffic reports
Applications:
50% of traffic is Kazaa
Sources:
20% of traffic comes from Steve’s PC
Terabytes of measurement data
June 8, 2003
Data streaming in computer networking - MPDS 2003

18. Informal problem definition
Analysis
Traffic reports
20% is Kazaa from Steve’s PC
50% is Kazaa from the dorms
Terabytes of measurement data
June 8, 2003
Data streaming in computer networking - MPDS 2003

19. Formal problem definition
Define clusters:
Atoms: fields 1 to n with hierarchies in each field including *
Cluster: intersection of one set from each field hierarchy
Example: Source=*, Destination=CS Net, App=
Threshold clusters:
Report traffic clusters above threshold T (e.g. 1% of traffic)
Omit redundant clusters:
Compression rule: remove general clusters from report when its traffic can be inferred (up to error T) from on non-overlapping more specific clusters
June 8, 2003
Data streaming in computer networking - MPDS 2003

20. Data streaming in computer networking - MPDS 2003
Solution status
The good:
Offline tool AutoFocus; SIGCOMM 2003 paper
Detected worm, busy servers, squid cache, etc.
Network managers like it
The bad:
Takes long: 3 hours at T=0.5% for one day trace
Needs much memory 300 Mbytes
The wanted:
Streaming algorithm - we invite improvements
June 8, 2003
Data streaming in computer networking - MPDS 2003

21. Data streaming in computer networking - MPDS 2003
Conclusions
New rules: strict constraints on algorithms running in routers
Iteration in time: can give simple algorithms, but needs more formalization as to quality of results
General open problems: many challenges in detecting malicious traffic such as worms and DoS attacks
Specific open problem: computing traffic cluster reports in streaming fashion
June 8, 2003
Data streaming in computer networking - MPDS 2003

22. Data streaming in computer networking - MPDS 2003
Thank you!
Algorithms ?
Databases
Networking
June 8, 2003
Data streaming in computer networking - MPDS 2003

23. Unidimensional clusters15 35 30 40
160
110 35 75
June 8, 2003
Data streaming in computer networking - MPDS 2003

24. Unidimensional clusters
/28
500
/29
120
/29
380
/30 50
/30 70
/30
305 75
/30
/31
/31 50
/31 70
/31
270 35 75
/31 15 35 30 40
160
110 35 75
June 8, 2003
Data streaming in computer networking - MPDS 2003

25. Unidimensional clusters
/28
500
/29
120
/29
380
/30 50
/30 70
/30
305 75
/30
/31
/31 50
/31 70
/31
270 35 75
/31 15 35 30 40
160
110 35 75
June 8, 2003
Data streaming in computer networking - MPDS 2003

26. Unidimensional clusters
/28
500
/29
120
/29
380
/30
305
/31
270
160
110
June 8, 2003
Data streaming in computer networking - MPDS 2003

27. Unidimensional clusters
/28
500
/29
120
/29
380
/30
305
/31
270
160
110
June 8, 2003
Data streaming in computer networking - MPDS 2003

28. Multidimensional clusters
Two dimensions
Source network
Protocol (traffic type)
Trees turn into lattice
Multiple parents
Nodes overlap
June 8, 2003
Data streaming in computer networking - MPDS 2003

29. Data streaming in computer networking - MPDS 2003
Offline solution
June 8, 2003
Data streaming in computer networking - MPDS 2003

30. Data streaming in computer networking - MPDS 2003
Sample report
June 8, 2003
Data streaming in computer networking - MPDS 2003