Assessing the Quality of Packet-Level Traces Collected on Internet Backbone Links

  • Behrooz Sangchoolie
  • Mazdak Rajabi Nasab
  • Tomas Olovsson
  • Wolfgang John
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7617)


The quality of captured traffic plays an important role for decisions made by systems like intrusion detection/prevention systems (IDS/IPS) and firewalls. As these systems monitor network traffic to find malicious activities, a missing packet might lead to an incorrect decision. In this paper, we analyze the quality of packet-level traces collected on Internet backbone links using different generations of DAG cards. This is accomplished by inferring dropped packets introduced by the data collection system with help of the intrinsic structural properties inherently provided by TCP traffic flows. We employ two metrics which we believe can detect all kinds of missing packets: i) packets with ACK numbers greater than the expected ACK, indicating that the communicating parties acknowledge a packet not present in the trace; and ii) packets with data beyond the receiver’s window size, which with a high probability, indicates that the packet advertising the correct window size was not recorded. These heuristics have been applied to three large datasets collected with different hardware and in different environments.

We also introduce flowstat, a tool developed for this purpose which is capable of analyzing both captured traces and real-time traffic. After assessing more than 400 traces (75M bidirectional flows), we conclude that at least 0.08% of the flows have missing packets, a surprisingly large number that can affect the quality of analysis performed by firewalls and intrusion detection/prevention systems. The paper concludes with an investigation and discussion of the spatial and temporal aspects of the experienced packet losses and possible reasons behind missing data in traces.


Traffic measurement measurement errors packet drop intrusion detection/prevention system firewall 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    John, W.: Characterization and Classification of Internet Backbone Traffic, Chalmers University of Technology, gothenburg, Sweden. PhD Thesis 0346-718X (2010)Google Scholar
  2. 2.
    Brauckhoff, D., Dimitropoulos, X., Wagner, A., Salamatian, K.: Anomaly extraction in backbone networks using association rules. In: Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement Conference, IMC 2009, New York, NY, USA, pp. 28–34 (2009)Google Scholar
  3. 3.
    Fraleigh, C., et al.: Packet-Level Traffic Measurements from the Sprint IP Backbone. IEEE Network 17(6), 6–16 (2003)CrossRefGoogle Scholar
  4. 4.
    Shang, F.: Research on the Link Traffic Measurement System Based on Edge Measurement. In: International Conference on Communications, Circuits and Systems, Guilin, Guangzi, China, pp. 1791–1795 (2006)Google Scholar
  5. 5.
    Mellia, M., Cigno, R.L., Neri, F.: Measuring IP and TCP behavior on edge nodes with Tstat. Computer Networks 47(1), 1–21 (2005)Google Scholar
  6. 6.
    Endace. Enterprise Network Monitoring Tools, Network Security System, Application Performance Monitoring (September 2011),
  7. 7.
    NSS lab, Network Intrusion Detection System individual product test result, NSS Labs, Auckland, New Zealand (2010)Google Scholar
  8. 8.
    Paxson, V.: Automated packet trace analysis of TCP implementations. In: Proceedings of the ACM SIGCOMM Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication, Cannes, France, pp. 167–179 (1997)Google Scholar
  9. 9.
    SUNET, History of the Swedish University Computer Network (Online) (September 2011),
  10. 10.
    SUNET. The Swedish University Computer Network OptoSUNET (Online) (September 2011),
  11. 11.
    John, W., Tafvelin, S., Olovsson, T.: Passive Internet Measurement Overview and Guidelines Based on Experiences. Computer Communications 33(5) (March 2010)Google Scholar
  12. 12.
    John, W., Tafvelin, S.: Analysis of Internet Backbone Traffic and Header Anomalies Observed. In: IMC 2007: Proceedings of the 7th ACM SIGCOMM conference on Internet measurement, San Diego, pp. 111–116 (2007)Google Scholar
  13. 13.
    Keys, K., et al.: The Architecture of CoralReef:An Internet Traffic Monitoring Software Suite. In: A Workshop in Passive and Active Measurement, Amsterdam, The Netherlands (2001)Google Scholar
  14. 14.
    Fan, J., Xu, J.J., Ammar, M.H., Moon, S.B.: Prefix-preserving IP address anonymization: measurement-based security evaluation and a new cryptography-based scheme. In: ICNP: Proceedings of the 10th IEEE International Conference on Network Protocols, Washington, DC, USA, pp. 280–289 (2002)Google Scholar
  15. 15.
    Qian, F., et al.: TCP revisited: a fresh look at TCP in the wild. In: Proceeding of the 9th ACM SIGCOMM Conference on Internet Measurement Conference, Chicago, Illinois, pp. 76–89 (2009)Google Scholar
  16. 16.
    Olovsson, T., John, W.: Detection of malicious traffic on backbone links via packet header analysis. Campus-Wide Information Systems 25(5), 342–358 (2008)CrossRefGoogle Scholar
  17. 17.
    Claffy, K.C., Braun, H.W., Polyzos, G.C.: A parameterizable methodology for Internet traffic flow profiling. Selected Areas in Communications 13(8), 1481–1494 (1995)CrossRefGoogle Scholar
  18. 18.
    John, W., Dusi, M., Claffy, K.C.: Estimating routing symmetry on single links by passive flow measurements. In: IWCMC 2010 Proceedings of the 6th International Wireless Communications and Mobile Computing Conference, Caen, France, pp. 473–478 (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Behrooz Sangchoolie
    • 1
  • Mazdak Rajabi Nasab
    • 1
  • Tomas Olovsson
    • 1
  • Wolfgang John
    • 2
  1. 1.Department of Computer Science and EngineeringChalmers University of TechnologyGothenburgSweden
  2. 2.Ericsson ResearchKistaSweden

Personalised recommendations