Telecommunication Systems

, Volume 37, Issue 1–3, pp 71–84 | Cite as

Integrating heterogeneous network monitoring data

  • Chi Zhang
  • Bin Liu
  • Xun Su
  • Heidi Alvarez
  • Julio Ibarra
Article

Abstract

In this paper, we investigate the integration of heterogeneous network monitoring data. Specifically, we will synchronize and integrate flow-level records, exemplified by Cisco NetFlow, and packet-level traces, exemplified by NLANR PMA. The integration can facilitate cross-validation and complementary utility. However, finding the correspondences of timestamps/flows/packets between the PMA and Netflow is non-trivial, because they have different levels of granularity, different sampling strategy, different time sources, and different IP address masking. To integrate heterogeneous monitoring data, we first synchronize their timestamps, and then match their masked IP addresses. Our key observation is that although the IP addresses are masked, some other header fields can be exploited to match different types of monitoring data. In order to reduce the search space and the processing overhead, we have adopted a top-down approach to limit the search scope, and iterative algorithms to reduce the matching errors step by step.

Keywords

Heterogeneous network monitoring data NetFlow PMA 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Duffield, N., & Lund, C. (2003). Predicting resource and estimation accuracy in an IP flow measurement collection intrastate. In ACM internet measurement conference, October 2003. Google Scholar
  2. 2.
    Duffield, N. G., Lund, C., & Thorup, M. (2002). Properties and prediction of flow statistics from sampled packet streams. In ACM internet measurement workshop, November 2002. Google Scholar
  3. 3.
    Duffield, N., Lund, C., & Thorup, M. (2003). Estimating flow distributions from sampled flow statistics. In ACM SIGCOMM, August 2003. Google Scholar
  4. 4.
    Estan, C., Keys, K., Moore, D., & Varghese, G. (2002). Building a better NetFlow. In ACM SIGCOMM, August 2002. Google Scholar
  5. 5.
    Estan, C., Keys, K., Moore, D., & Varghese, G. (2002). New directions in traffic measurement and accounting. In ACM SIGCOMM, August 2002. Google Scholar
  6. 6.
    Estan, C., Savage, S., & Varghese, G. (2003). Automatically inferring patterns of resource consumption in network traffic. In ACM SIGCOMM, August 2003. Google Scholar
  7. 7.
    Kumar, A., Sung, M., Xu, J., & Wang, J. (2004). Data streaming algorithms for efficient and accurate estimation of flow distribution. In ACM SIGMETRICS, June 2004. Google Scholar
  8. 8.
    Micheel, J., Donnelly, S., & Graham, I. (2001). Precision timestamping of network packets. In ACM internet measurement workshop, November 2001. Google Scholar
  9. 9.
    McGregor, A., Hall, M., Lorier, P., & Brunskill, J. (2004). Flow clustering using machine learning techniques. In Passive and active measurement workshop, April 2004. Google Scholar
  10. 10.
    Moon, S. B., Skelly, P., & Towsley, D. (1999). Estimation and removal of clock skew from network delay measurement. In IEEE INFOCOM, March 1999. Google Scholar
  11. 11.
    Mori, T., Uchida, M., & Kasahara, R., et al. (2004). Identifying elephant flows through periodically sampled packets. In ACM internet measurement conference, October 2004. Google Scholar
  12. 12.
  13. 13.
    Paxson, V. (1998). On calibrating measurements of packet transit times. In ACM SIGMETRICS, June 1998. Google Scholar
  14. 14.
    Rupp, A., Dreger, H., Fedlmann, A., & Sommer, R. (2004). Packet trace manipulation framework for test labs. In ACM internet measurement conference, October 2004. Google Scholar
  15. 15.
    Sommer, R., & Feldmann, A. (2002). NetFlow: information loss or win. In Internet measurement workshop, November 2002. Google Scholar
  16. 16.
    Veitch, D., Babu, S., & Pasztor, A. (2004). Robust synchronization of software clocks across the internet. In ACM internet measurement conference, October 2004. Google Scholar
  17. 17.
    Zhang, Y., Singh, S., Sen, S., Duffield, N., & Lund, C. (2004). Online identification of hierarchical heavy hitters: algorithms, evaluation, and applications. In Internet measurement conference, October 2004. Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  • Chi Zhang
    • 1
  • Bin Liu
    • 2
  • Xun Su
    • 3
  • Heidi Alvarez
    • 4
  • Julio Ibarra
    • 4
  1. 1.Juniper NetworksSunnyvaleUSA
  2. 2.MicrosoftRedmondUSA
  3. 3.Fulcrum MicrosystemsCalabasasUSA
  4. 4.Center for Internet Augmented Research and AssessmentFlorida International UniversityMiamiUSA

Personalised recommendations