1. Neuro-Fuzzy Processing of Packet Dispersion Traces for Highly Variable Cross-Traffic Estimation Marco A. Alzate 1,2,3, Néstor M. Peña 1, Miguel A. Labrador 2 1 Universidad de los Andes, Bogotá, Colombia 2 University of South Florida, Tampa, FL 3 Universidad Distrital, Bogotá, Colombia In the context of constant-rate fluid-flow traffic, it has been shown that there is a minimum probing traffic rate at which the dispersion exhibits correlation with cross traffic, so packet length and input gap must be adjusted to reach that minimum. However, here we show evidence that, with highly variable traffic, it is possible to have very short probing packets at a very low rate and still get an important correlation between dispersion and traffic, over a long range of measurement time scales, even when the utilization factor is low. As an example of how the previous finding can be exploited for estimation purposes, we design a heuristically-modified neuro-fuzzy estimator (HNFE) for real time high resolution traffic estimation that, instead of looking for a long run average, tracks the cross traffic rate signal. This HNFE exploits the high variability property of modern traffic by using an extremely low rate probing traffic and multiple time scale analysis, achieving high accuracy, low computational cost, very low transmission overhead and high robustness against varying network conditions. Main Ideas Packet Dispersion and High Variability As we increase the variability through the coefficient of variation (C = 1,2,4), the correlation becomes less dependent on the utilization factor, ro. Similarly, as we increase the variability through the Hurst parameter (H = 0.5, 0.65, 0.8, 0.95), the correlation becomes less dependent on T as well. Correspondingly, as traffic variability increases, we can test the link over a wide range of time scales obtaining a high correlation even with a low utilizations factor. Cross traffic Probing traffic C t XtXt t PtPt L/C T t PrPr D A link carries a cross-traffic, characterized by a given coefficient of variation and a given Hurst parameter, along with a probing traffic consisting L-bits long packets sent every T seconds. How much correlation is there between the average cross- traffic arrival rate at the n th measurement period, X n, and the corresponding packet dispersion measure, D n ? Neuro-Fuzzy Estimator A Simple Estimator For a FIFO link of capacity C bps that does not become idle between the n th and n+1 st probing packets, the n th dispersion measurement D n, will give an exact value of the cross-traffic arrival rate during the n th measurement period, X n. Simulation experiment with the Bellcore traffic trace BCpAug89 on a T1 link, with 24-byte probing packets sent every second. Normalized variables Considering the mean and variance of the previous 12 dispersion measures,  D and  D 2, we plot the relative error of SE (left) to establish the following intuitively plausible rules: These rules call for a fuzzy approach to our estimation problem Based on the experimental data (d n, x n ), we estimated that information about x n conveyed by these four selected variables, I(x n ; {  i (n), i=1,2,3,4}), is comparable to that conveyed by the 12 previous dispersion measurements together, I(x n ;{d n-k, k=0,…,11}). Fitting the histograms of each input variable conditioned on an “exact” (f E (. )) or “poor” (f P (. )) performance of SE (shown left), we define fuzzy sets “Far from zero” and “Close to zero” through the relationships If  D is far from 0, the simple estimation is exact If  D is close to 0 and  D 2 is small, the simple estimation is poor If  D is close to 0 and  D 2 is large, the simple estimation is fair. Normalized simple estimator (SE) Selected input variables: We have two inputs at the scale of the measurement period T,  1 and  2, and two inputs at the scale of 12T,  3 and  4, so we can classify SE according to each scale and then combine their results according to the following fuzzy classifiers: IF both variables are close to 0 THEN SE is poor ELSEIF both variables are far from 0 THEN SE is good ELSE SE is fair END IF one scale says SE is poor AND the other does not say it is good THEN SE is poor ELSEIF one scale says SE is good AND the other does not say it is poor THEN SE is good ELSE SE is fair END At each scale, (  1,  2 ) and (  3,  4 ) Combining the result of each scale Which leads to the following fuzzy inference system for classifying SE, The final system, a Heuristically-modified Neuro-Fuzzy Estimator (HNFE), adds two additional elements: A different affine estimator, based on (  1,  2 ), for each class above, A queue simulator [q n = max(0, q n-1 + x ^ n + L/(CT))], to recover SE whenever the queue length has exceed a given threshold, thr. Low computational cost, Good generalization (low overfiting), good interpretability (rules were chosen, not learned) fast learning speed. 1.Initialize 1, 3, and 4 by fitting the conditional histograms. 2.Compute the nine linear parameters of the affine estimators by a least square procedure. 3.Compute the optimal exponents { i, i=1,3,4} through a quasi-Newton line search algorithm. 4.Iterate steps 2 and 3 until convergence. 5.Look for the optimal queue threshold through bracketing. Leading closely to the following closed form expressions T-norm = product S-norm = maximum Training: Through its reduced structural complexity, HNFE achieves Conclusions References Results 1. R.S. Prasad, M. Murray, C. Dovrolis, and K.C. Claffy, "Bandwidth Estimation: Metrics, Measurement Techniques, and Tools," IEEE Network Magazine, Vol. 17, No. 6, pp. 27--35, Nov/Dec. 2003. 2. N. Hu and P. Steenkiste, “Evaluation and Characterization of Available Bandwidth Probing Techniques,” IEEE JSAC, Vol. 21, No. 6, pp. 879-894, Aug., 2003. 3. M. Jain and C. Dovrolis, “End-to-End Available Bandwidth Measure Methodology, Dynamics and Relation with TCP throughput,” IEEE/ACM Transactions on Networking, Vol. 11, No. 4, August 2003, pp. 537-549. 4. V. Ribeiro, R. Riedi, R. Baraniuk, J. Navratil and L. Cottrell, “PathChirp: Efficient Available Bandwidth Estimation for Network Paths,” Proceedings of Passive and Active Measurements (PAM) Workshop, La Jolla, CA, USA, Apr. 2003. 5. J. Strauss, D. Katabi, F. Kaashoek, and B. Prabhakar, “Spruce: A Lightweight End-to-End Tool for Measuring Available Bandwidth,” Proceedings of Internet Measurement Conference (IMC) 2003, Miami, Florida, October 2003. 6. R. Ribeiro, M. Coates, R. Riedi, S. Sarvotham, B. Hendricks, and R. Baraniuk, “MultiFractal Cross-Traffic Estimation,” Proceedings of ITC Specialist Seminar on IP Traffic Measurement, Monterey California, September 18-20 2000 7. Lawrence Berkeley National Laboratory. The Internet Traffic Archives, BC – Ethernet traces of LAN and WAN traffic, http://ita.ee.lbl.gov/html/contrib/BC.html 8. Video Traces Research Group, Arizona State University, http://trace.eas.asu.edu/TRACE/pics/FrameTrace/mp4/Verbose_Jurassic.dat. For the training trace and with the training conditions, the proposed system reduced the estimation error, even when the probing packets do not belong to the same occupation period. Passive and Active Measurement Conference, PAM 2007, Louvain-la-neuve, Belgium, April 5-6 2007 The performance for the training trace, in terms of the SNR (where the traffic trace is the signal and the estimation error is the noise) is high, even under low utilization factors, on a wide range of measurement time scales, for different link utilizations and different measurement periods. The SNR is still high for another traffic trace, a 768 kbps MPEG4 version of “Jurassic Park”, a very different cross traffic trace than the one used for training, revealing the good generalization properties of the system. The dynamics of packet pair probing techniques are influenced by traffic variability, beyond what constant rate fluid flow models have suggested. In particular, dispersion measurements can still be highly correlated with the cross traffic on a wide range of measurement time scales, even for lowly used links. We do not need to exhaust available bandwidth with probing traffic in order to obtain good cross-traffic estimates, as long as the measurement time scale falls within the high variability range. By exploiting the properties above, our HNFE demonstrates that it is possible to obtain a good cross-traffic tracking signal (not just a long run average) with a negligible probing load.