1. Fast, Memory-Efficient Traffic Estimation by Coincidence Counting Fang Hao 1, Murali Kodialam 1, T. V. Lakshman 1, Hui Zhang 2, 1 Bell Labs, Lucent Technologies 2 University of Southern California

2. 2 Traffic flow measurement  Related work: Sample and hold [Estan et al. 2002], Smart Sampling [Duffield et al. 2004], RATE [Kodialam et al. 2004], ACCEL-RATE[Hao et al. 2004], etc.  Flow rate p f : the proportion of packets belonging to flow f during a certain time period. Arrival rate r f = p f  : - total arrival rate (packets / second) router packets flows Flow rate statistics network link

3. 3 Problem definition What’s the traffic flow composition through a network link during a certain time period, given an estimation accuracy requirement ( ,  )?   - confidence percentile;   - estimation error. For any given flow f with its rate be p f, determine an estimate p’ f for p f such that p’ f  (p f -  /2, p f +  /2) with probability greater than .  e.g.,  = 0.9975,  = 0.0001.

4. 4 One solution – naive counting Directly counting the number of packets for each flow.  It’s simple;  It estimates the rate of the flows rapidly;  But it requires frequent access of a large amount of high-speed memory. There can be millions of active flows on backbone links [Duffield et al. 2001]

5. 5 Arrival model 1.Flow rates are stationary during the estimation period.  An arriving packet belongs to flow f with probability p f. 2.Packet arrivals are independent.  An arriving packet belongs to flow f independent of other arrivals.

6. 6 Performance metrics Sample size (estimation time): the number of arrivals needed to achieve the specified estimation accuracy. Memory size: the number of flows that are tracked during the estimation.

7. 7 Can we … Can we design a scheme which runs as fast as naïve counting but only catches “interesting” flows? By counting the exact number of arrivals for each flow, naïve accounting requires minimally arrivals to meet the accuracy requirement ( ,  ).  This will capture at least one packet in expectation for any flow f with p f  (Z  =3,  = 0.0001, ).

8. 8 Problem re-definition What’s the traffic flow composition through a network link during a certain time period given an estimation accuracy requirement ( , , ,  )?   - confidence percentile;  - estimation error;   - threshold rate;  - error relaxation factor. For any given flow f with its rate be p f, determine an estimate p’ f for p f such that p’ f  (p f -  /2, p f +  /2) if p f   p’ f  (p f -  /2, p f +  /2) if p f   with probability greater than .  e.g.,  = 0.9975 (Z  =3),  = 0.0001,  = 0.01,  = 10.

9. 9 Our solution – CATE Coincidence bAsed Traffic Estimation  It’s simple;  It estimates the size of the flows rapidly;  It requires a small amount of memory. A generalization of RATE scheme [ Kodialam et al. 2004 ]

10. 10 CATE – scheme description (I) k-length

11. 11 CATE – scheme description (II) Estimation procedure 1.Specify the estimation accuracy requirement ( , , ,  ); 2.Calculate sample size N, memory size M, and k (the length of PT); 3.Run and count the coincidences CC(f) for each flow f; 4.At the end of the estimation, output the estimated flow rates as a)For each flow f in CCT, p’(f) = ; b)For the rest of the flows, report 0 as the estimated rates.

12. 12 Intuition behind CATE Counting coincidence dramatically amplifies (squares) the ratio of catching probability between a large flow and a small flow.  Good news: CATE sample size is still for the estimation accuracy requirement ( , , ,  ). Multiple (i.e., k) comparisons increase the number of membership testing to kN with N arrivals.  Good news: the reduction on estimation variance due to increase in testing number is no less than the increase on estimation variance due to comparison correlation.

13. 13 Given the accuracy requirement ( ,  ) and k-length predecessor table for CATE,  The minimal sample size CATE – sample size  Specifically, for a flow f with p f , the minimal sample size

14. 14 Given an estimation accuracy requirement ( , , ,  ), if then setting gives the sample size = Theorem 1 – CATE sample size  = 99.9 %  = 0.0001  = 0.01   = 10  k = 50  = 0.001   = 20  k = 500 …

15. 15 Given the estimation setting in Theorem 1, the maximum expected memory Theorem 2 – CATE memory size Totally 100,000 flows, rate range [10 -6, 1] Z  = 3  = 0.002 CATE will catch no more than 1650 flows. Naïve counting will record all flows w.h.p.

16. 16 CATE – experiment 1 Real IP traces from NLANR; A size-1000 buffer between new arrival and PT. (The memory size = 547)(The memory size = 1464)

17. 17 CATE – experiment 2 (I) methodology  Totally 1 million flows, synthetic traces;  5 “large” flows, rates uniformly distributed between 0.1 and 0.2 of the entire traffic;  1000 “medium-sized” flows, rates uniformly distributed between 0.0001 and 0.0002 of the entire traffic;  All the rest are “small” flows, each with rate roughly 10 -7 ;  Deliberately chose a short sample time (1 million packet time) to illustrate the impact of k, the predecessor table length.

18. 18 CATE – experiment 2 (II)

19. 19 CATE – experiment 2 (III)

20. 20 Conclusion & future work CATE, a memory efficient traffic estimation scheme as fast as naïve counting. Future work:  Extending CATE for byte rate estimation.  Extending CATE to minimize the impact of arrival dependence without (excessive) additional overhead.