Integrating heterogeneous network monitoring data
In this paper, we investigate the integration of heterogeneous network monitoring data. Specifically, we will synchronize and integrate flow-level records, exemplified by Cisco NetFlow, and packet-level traces, exemplified by NLANR PMA. The integration can facilitate cross-validation and complementary utility. However, finding the correspondences of timestamps/flows/packets between the PMA and Netflow is non-trivial, because they have different levels of granularity, different sampling strategy, different time sources, and different IP address masking. To integrate heterogeneous monitoring data, we first synchronize their timestamps, and then match their masked IP addresses. Our key observation is that although the IP addresses are masked, some other header fields can be exploited to match different types of monitoring data. In order to reduce the search space and the processing overhead, we have adopted a top-down approach to limit the search scope, and iterative algorithms to reduce the matching errors step by step.
KeywordsHeterogeneous network monitoring data NetFlow PMA
Unable to display preview. Download preview PDF.
- 1.Duffield, N., & Lund, C. (2003). Predicting resource and estimation accuracy in an IP flow measurement collection intrastate. In ACM internet measurement conference, October 2003. Google Scholar
- 2.Duffield, N. G., Lund, C., & Thorup, M. (2002). Properties and prediction of flow statistics from sampled packet streams. In ACM internet measurement workshop, November 2002. Google Scholar
- 3.Duffield, N., Lund, C., & Thorup, M. (2003). Estimating flow distributions from sampled flow statistics. In ACM SIGCOMM, August 2003. Google Scholar
- 4.Estan, C., Keys, K., Moore, D., & Varghese, G. (2002). Building a better NetFlow. In ACM SIGCOMM, August 2002. Google Scholar
- 5.Estan, C., Keys, K., Moore, D., & Varghese, G. (2002). New directions in traffic measurement and accounting. In ACM SIGCOMM, August 2002. Google Scholar
- 6.Estan, C., Savage, S., & Varghese, G. (2003). Automatically inferring patterns of resource consumption in network traffic. In ACM SIGCOMM, August 2003. Google Scholar
- 7.Kumar, A., Sung, M., Xu, J., & Wang, J. (2004). Data streaming algorithms for efficient and accurate estimation of flow distribution. In ACM SIGMETRICS, June 2004. Google Scholar
- 8.Micheel, J., Donnelly, S., & Graham, I. (2001). Precision timestamping of network packets. In ACM internet measurement workshop, November 2001. Google Scholar
- 9.McGregor, A., Hall, M., Lorier, P., & Brunskill, J. (2004). Flow clustering using machine learning techniques. In Passive and active measurement workshop, April 2004. Google Scholar
- 10.Moon, S. B., Skelly, P., & Towsley, D. (1999). Estimation and removal of clock skew from network delay measurement. In IEEE INFOCOM, March 1999. Google Scholar
- 11.Mori, T., Uchida, M., & Kasahara, R., et al. (2004). Identifying elephant flows through periodically sampled packets. In ACM internet measurement conference, October 2004. Google Scholar
- 12.Cisco NetFlow. http://www.cisco.com/warp/public/732/Tech/nmp/NetFlow/.
- 13.Paxson, V. (1998). On calibrating measurements of packet transit times. In ACM SIGMETRICS, June 1998. Google Scholar
- 14.Rupp, A., Dreger, H., Fedlmann, A., & Sommer, R. (2004). Packet trace manipulation framework for test labs. In ACM internet measurement conference, October 2004. Google Scholar
- 15.Sommer, R., & Feldmann, A. (2002). NetFlow: information loss or win. In Internet measurement workshop, November 2002. Google Scholar
- 16.Veitch, D., Babu, S., & Pasztor, A. (2004). Robust synchronization of software clocks across the internet. In ACM internet measurement conference, October 2004. Google Scholar
- 17.Zhang, Y., Singh, S., Sen, S., Duffield, N., & Lund, C. (2004). Online identification of hierarchical heavy hitters: algorithms, evaluation, and applications. In Internet measurement conference, October 2004. Google Scholar