Journal of Network and Systems Management

, Volume 25, Issue 3, pp 643–668

A Modular Traffic Sampling Architecture: Bringing Versatility and Efficiency to Massive Traffic Analysis

  • João Marco C. Silva
  • Paulo Carvalho
  • Solange Rito Lima
Article
  • 184 Downloads

Abstract

The massive traffic volumes and heterogeneity of services in today’s networks urge for flexible, yet simple measurement solutions to assist network management tasks, without impairing network performance. To turn treatable tasks requiring traffic analysis, sampling the traffic has become mandatory, triggering substantial research in the area. Despite that, there is still a lack of an encompassing solution able to support the flexible deployment of sampling techniques in production networks, adequate to diverse traffic scenarios and measurement activities. In this context, this article proposes a modular traffic sampling architecture able to foster the flexible design and deployment of efficient measurement strategies. The architecture is composed of three layers—management plane, control plane and data plane—covering key components to achieve versatile and lightweight measurements in diverse traffic scenarios and measurement activities. Each component of the architecture is described considering the different strategies, technologies and protocols that compose the several stages of a measurement process. Following the proposed architecture, a sampling framework prototype has been developed, providing a fair environment to assess and compare sampling techniques under distinct measurement scenarios, evaluating their performance in balancing computational burden and accuracy. The results have demonstrated the relevance and applicability of the proposed architecture, revealing that a modular and configurable approach to sampling is a step forward for improving sampling scope and efficiency.

Keywords

Traffic sampling Sampling techniques Traffic measurement architecture Traffic sampling taxonomy Traffic monitoring and analysis 

References

  1. 1.
    Zseby, T., Molina, M., Duffield, N.: Sampling and Filtering Techniques for IP Packet Selection RFC 5475. Technical report, IETF. http://datatracker.ietf.org/doc/rfc5475/ (2009)
  2. 2.
    Silva, J.M.C., Carvalho, P., Rito Lima, S.: Analysing traffic flows through sampling: a comparative study. In: 20th IEEE Symposium on Computers and Communication (ISCC), Cyprus (2015)Google Scholar
  3. 3.
    Jadwab, J., Phall, P., Pinna, B.: Traffic estimation for the largest sources on a network using packet sampling with limited storage. Technical report, Hewllet-Packard Laboratories, Bristol (1992)Google Scholar
  4. 4.
    Claffy, K.C., Polyzos, G.C., Braun, H.W.: Application of sampling methodologies to network traffic characterization, SIGCOMM. Comput. Commun. Rev. 23(4), 194–203 (1993). doi:10.1145/167954.166256 CrossRefGoogle Scholar
  5. 5.
    Cozzani, I., Giordano, S.: Traffic sampling methods for end-to-end QoS evaluation in large heterogeneous networks. Comput. Netw. ISDN Syst. 30(16–18), 1697–1706. http://www.sciencedirect.com/science/article/pii/S0169755298001986 (1998)
  6. 6.
    Amer, P., Cassel, L.: Management of sampled real-time network measurements. In: Proceedings of 14th Conference on Local Computer Networks. IEEE Comput. Soc. Press, pp. 62–68. http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=65244 (1989)
  7. 7.
    Tammaro, D., Valenti, S., Rossi, D., Pescapé, A.: Exploiting packet-sampling measurements for traffic characterization and classification. Int. J. Netw. Manag. 22(6), 451–476 (2012). doi:10.1002/nem.1802 CrossRefGoogle Scholar
  8. 8.
    Duffield, N.: Fair sampling across network flow measurements. ACM SIGMETRICS Perform. Eval. Rev. 40(1), 367 (2012). doi:http://dl.acm.org/citation.cfm?id=2318857.2254800
  9. 9.
    Hernandez, E.A., Chidester, M.C., George, A.D.: Adaptive sampling for network management. J. Netw. Syst. Manag. 9(4), 409–434 (2001). doi:10.1023/A:1012980307500 CrossRefGoogle Scholar
  10. 10.
    Silva, J.M.C., Carvalho, P., Rito Lima, S.: A multiadaptive sampling technique for cost-effective network measurements. Comput. Netw. 57(17), 3357–3369 (2013). doi:10.1016/j.comnet.2013.07.023
  11. 11.
    Duffield, N.G., Grossglauser, M.: Trajectory sampling for direct traffic observation. ACM SIGCOMM Comput. Commun. Rev. 30(4), 271–282 (2000). doi:10.1145/347057.347555 CrossRefGoogle Scholar
  12. 12.
    Estan, C., Varghese, G.: New directions in traffic measurement and accounting. SIGCOMM Comput. Commun. Rev. 32(4), 323–336 (2002). doi:10.1145/964725.633056 CrossRefGoogle Scholar
  13. 13.
    Singh, R., Kumar, H., Singla, R.K.: Analyzing statistical effect of sampling on network traffic dataset. In: Satapathy, S.C., Avadhani, P.S., Udgata, S.K., Lakshminarayana, S. (eds.). ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India. Springer International Publishing, pp. 401–408. http://link.springer.com/chapter/10.1007/978-3-319-03107-1_43 (2014)
  14. 14.
    Yang, L., Michailidis, G.: Sampled based estimation of network traffic flow characteristics. In: IEEE INFOCOM 2007---26th IEEE International Conference on Computer Communications, (IEEE) pp. 1775–1783. doi:10.1109/INFCOM.2007.207. http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4215789 (2007)
  15. 15.
    Carela-Español, V., Barlet-Ros, P., Cabellos-Aparicio, A., Solé-Pareta, J.: Analysis of the impact of sampling on NetFlow traffic classification. Comput. Netw. 55(5), 1083–1089 (2011). doi:10.1016/j.comnet.2010.11.002 CrossRefGoogle Scholar
  16. 16.
    Lin, R., Li, O., Li, Q., Dai, K.: Exploiting adaptive packet-sampling measurements for multimedia traffic classification. J. Commun. 9(12) (2014). http://www.jocm.us/uploadfile/2014/1231/20141231030404520
  17. 17.
    Kandula, S., Mahajan, R.: Sampling biases in network path measurements and what to do about it. In: Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement Conference IMC ’09 (ACM, New York, NY, USA) , pp. 156–169. doi:10.1145/1644893.1644912 (2009)
  18. 18.
    Lee, M., Duffield, N., Kompella, R.: Two samples are enough: opportunistic flow-level latency estimation using NetFlow. In: 2010 Proceedings IEEE INFOCOM, pp. 1–9. doi:10.1109/INFCOM.2010.5462044. http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5462044 (2010)
  19. 19.
    Mahmood, A.N., Hu, J., Tari, Z., Leckie, C.: Critical infrastructure protection: resource efficient sampling to improve detection of less frequent patterns in network traffic. J. Netw. Comput. Appl. 33(4), 491–502 (2010). http://www.sciencedirect.com/science/article/B6WKB-4YBMFB6-1/2/9b91d8daa2364e0d025aed6088160da7
  20. 20.
    Zhang, J., Luo, X., Perdisci, R., Gu, G., Lee, W., Feamster, N.: Boosting the scalability of botnet detection using adaptive traffic sampling. In: Proceedings of the 6th ACM Symposium on Information, Computer and Communications Security, Ser. (ACM, New York, NY, USA), ASIACCS ’11, pp. 124–134. doi:10.1145/1966913.1966930 (2011)
  21. 21.
    Huang, Y., Pullen, J.: Countering denial-of-service attacks using congestion triggered packet sampling and filtering. In: Proceedings Tenth International Conference on Computer Communications and Networks (Cat. No.01EX495), (IEEE), pp. 490–494 (2001). http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=956309
  22. 22.
    Brauckhoff, D., Tellenbach, B., Wagner, A., May, M., Lakhina, A.: Impact of packet sampling on anomaly detection metrics. In: Proceedings of the 6th ACM SIGCOMM Conference on Internet Measurement, Ser. (ACM, New York, NY, USA) IMC ’06, pp. 159–164. doi:10.1145/1177080.1177101 (2006)
  23. 23.
    Paredes-Oliva, I., Barlet-Ros, P., Solé-Pareta, J.: Portscan detection with sampled Netflow. In: Traffic Monitoring and Analysis (Springer), pp. 26–33. http://link.springer.com/chapter/10.1007/978-3-642-01645-5_4 (2009)
  24. 24.
    Mai, J., Chuah, C.N., Sridharan, A., Ye, T., Zang, H.: Is sampled data sufficient for anomaly detection? In: Proceedings of the 6th ACM SIGCOMM on Internet measurement—IMC’06, Ser. (ACM Press, New York, NY, USA) p. 165 (2006). http://portal.acm.org/citation.cfm?doid=1177080.1177102
  25. 25.
    Jae-Hyun, J., Cheol-Woong, A., Dongjoon, L., Sung-Ho, K.: DDoS attack detection using flow entropy and packet sampling on huge networks. In: ICN 2014 : The Thirteenth International Conference on Networks (IARIA), pp. 183–190 (2014)Google Scholar
  26. 26.
    Zseby, T.: Deployment of sampling methods for SLA validation with non-intrusive measurements. In: Proceedings of Passive and Active Measurements Conference (Fort Collins) (2002)Google Scholar
  27. 27.
    Zseby, T.: Comparison of sampling methods for non-intrusive SLA validation. In: Proceedings of the Second Workshop on End-to-End Monitoring Techniques and Services (E2EMon) (2004)Google Scholar
  28. 28.
    Serral-Gracia, R., Cabellos-Aparicio, A., Domingo-Pascual, J.: Packet loss estimation using distributed adaptive sampling. In: Network Operations and Management Symposium Workshops, 2008. NOMS Workshops 2008. IEEE (IEEE), pp. 124–131 (2008). http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4509938
  29. 29.
    Sommers, J., Barford, P., Duffield, N., Ron, A.: Improving accuracy in end-to-end packet loss measurement. In: Proceedings of the 2005 conference on Applications, Technologies, Architectures, and Protocols for Computer Communications—SIGCOMM ’05, (ACM Press, New York, New York, USA), vol. 35, p. 157 (2005). http://dl.acm.org/citation.cfm?id=1080091.1080111
  30. 30.
    Dogman, A., Saatchi, R., Al-Khayatt, S.: An adaptive statistical sampling technique for computer network traffic. In: 7th International Symposium on Communication Systems Networks and Digital Signal Processing (CSNDSP, 2010), pp. 479–483 (2010)Google Scholar
  31. 31.
    Gu, Y., Breslau, L., Duffield, N., Sen, S.: On passive one-way loss measurements using sampled flow statistics. In: IEEE INFOCOM 2009—The 28th Conference on Computer Communications (IEEE), pp. 2946–2950 (2009). http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5062264
  32. 32.
    Androulidakis, G., Chatzigiannakis, V., Papavassiliou, S.: Network anomaly detection and classification via opportunistic sampling. IEEE Netw. 23(1), 6–12 (2009). http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4804318
  33. 33.
    Choi, B.Y., Bhattacharyya, S.: Observations on cisco sampled netflow. ACM SIGMETRICS Perform. Eval. Rev. 33(3), p. 18 (2005). http://portal.acm.org/citation.cfm?doid=1111572.1111579
  34. 34.
    Zseby, T., Hirsch, T., Claise, B.: Packet sampling for flow accounting: challenges and limitations. In: Claypool, M., Uhlig, S. (eds.) Passive and Active Network Measurement, Ser. Lecture Notes in Computer Science, vol. 4979, (Springer Berlin / Heidelberg), pp. 61–71 (2008). doi:10.1007/978-3-540-79232-1_7
  35. 35.
    Pescape, A., Rossi, D., Tammaro, D., Valenti, S.: On the impact of sampling on traffic monitoring and analysis. In: 2010 22nd International Teletraffic Congress (lTC 22) (IEEE), pp. 1–8. (2010). http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5608718
  36. 36.
    Chabchoub, Y., Fricker, C., Guillemin, F., Robert, P.: Deterministic versus probabilistic packet sampling in the internet. In: Mason, L., Drwiega, T., Yan, J. (eds.) Managing Traffic Performance in Converged Networks, Lecture Notes in Computer Science, vol. 4516, Springer, Berlin, Heidelberg, pp. 678–689 (2007). http://link.springer.com/chapter/10.1007/978-3-540-72990-7_60
  37. 37.
    Castro, V., Carvalho, P., Lima, S.R., In: A cooperative network monitoring overlay. Smart Spaces and Next Generation Wired/Wireless Networking, Springer, pp. 475–486 (2011). http://link.springer.com/chapter/10.1007/978-3-642-22875-9_43
  38. 38.
    Schad, J., Dittrich, J., Quiané-Ruiz, J.A.: Runtime measurements in the cloud: observing, analyzing, and reducing variance. Proc. VLDB Endow. 3(1–2), 460–471 (2010). doi:10.14778/1920841.1920902 CrossRefGoogle Scholar
  39. 39.
    Pras, A., Schoenwaelder, J.: On the difference between information models and data models—RFC 3444. Technical Report, IETF (2003). https://datatracker.ietf.org/doc/rfc3444/
  40. 40.
    Claise, B., Trammell, B.: Specification of the IP Flow Information eXport (IPFIX) Protocol for the Exchange of Flow Information. RFC 7011 (2013). http://datatracker.ietf.org/doc/draft-ietf-ipfix-protocol-rfc5101bis/
  41. 41.
    Claise, B., Trammel, B.: Information Model for IP Flow Information Export (IPFIX)—RFC 7012. Technical Report IETF (2013). https://datatracker.ietf.org/doc/rfc7012/
  42. 42.
    Dietz, T., Claise, B., Aitken, P., Dressler, F., Carle, G.: Information Model for Packet Sampling Exports. Technical Report, IETF RFC 5477 (2009). https://datatracker.ietf.org/doc/rfc5477/
  43. 43.
    IP Flow Information Export (IPFIX).: Entities (2015).http://www.iana.org/assignments/ipfix/ipfix.xhtml
  44. 44.
    Dietz, T., Claise, B., Quittek. J.: Definitions of Managed Objects for Packet Sampling. RFC 6727 (2012). http://datatracker.ietf.org/doc/rfc6727/
  45. 45.
    Case, J., Mundy, R., Partain, D., Stewart, B.: Introduction and Applicability Statements for Internet-Standard Management Framework—RFC 3410. Technical Report, IETF (2002). https://datatracker.ietf.org/doc/rfc3410/
  46. 46.
    Aitken, P., Claise, B., McDowall, C., Schoenwaelder, J.: Exporting MIB Variables using the IPFIX Protocol draft-ietf-ipfix-mib-variable-export-09. Technical Report, IETF (2015). https://datatracker.ietf.org/doc/draft-ietf-ipfix-mib-variable-export/
  47. 47.
    McCloghrie, K., Seligson, J., Reichmeyer, F., Smith, A., Sahita, R.: Structure of policy provisioning information (SPPI)—RFC 3159. Technical Report, IETF (2001). https://datatracker.ietf.org/doc/rfc3159/
  48. 48.
    Uslar, M., Specht, M., Rohjans, S., Trefke, J., González, J.M.: The Common Information Model CIM: IEC 61968/61970 and 62325—A Practical Introduction to the CIM, vol. 66. Springer, New York (2012)Google Scholar
  49. 49.
    Silva, J.M.C., Carvalho, P., Rito Lima, S.: Enhancing traffic sampling scope and efficiency. In: 2013 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS) (IEEE), pp. 71–72 (2013). http://ieeexplore.ieee.org/articleDetails.jsp?arnumber=6562848
  50. 50.
    Hofstede, R., Celeda, P.,  Trammell, B., Drago, I., Sadre, R., Sperotto, A., Pras, A.: Flow monitoring explained: from packet capture to data analysis With NetFlow and IPFIX. IEEE Commun. Surv. Tutor. 16(4), 2037–206 (2014). http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6814316
  51. 51.
    Claise, B., Johnson, A., Quittek. J.: Packet Sampling (PSAMP) Protocol Specifications. RFC 5476 (2009). http://datatracker.ietf.org/doc/rfc5476/
  52. 52.
    Orebaugh, A., Ramirez, G., Beale, J.: Wireshark and Ethereal Network Protocol Analyzer Toolkit. Syngress, Rockland (2006)Google Scholar
  53. 53.
    Jacobson, V., McCanne, S.: Lawrence Berkeley Laboratory, Berkeley, CA (2009)Google Scholar
  54. 54.
    Alcock, S., Lorier, P., Nelson, R.: ACM SIGCOMM Comput. Commun. Rev. 42(2), 42 (2012). http://dl.acm.org/citation.cfm?doid=2185376.2185382
  55. 55.
    Silva, J.M.C., Carvalho, P., Lima, S.R.: Computational weight of network traffic sampling techniques. In: 2014 IEEE Symposium on Computers and Communications (ISCC) (IEEE, Madeira, Portugal), pp. 1–6 (2014). http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6912467
  56. 56.
    Shannon, C., Aben, E., Claffy, K., Andersen, D., Brownlee, N.: The CAIDA UCSD Anonymized Internet Traces 2008—equinix-chicago.dirA.20080430-170200.UTC.anon. Downloaded from http://www.caida.org/data/passive/passive_2008_dataset.xml (2008)
  57. 57.
    Shannon, C., Aben, E., Claffy, K., Andersen, D., Brownlee, N.: The CAIDA UCSD Anonymized Internet Traces 2014—-equinix-chicago.dirA.20140619-131100.UTC.anon. Downloaded from http://www.caida.org/data/passive/passive_2014_dataset.xml (2014)
  58. 58.
    Krishnan, R., Yong, L., Ghanwani, A., So, N., Khasnabish, B.: Mechanisms for Optimizing Link Aggregation Group (LAG) and Equal-Cost Multipath (ECMP) Component Link Utilization in Networks—RFC 7424. Technical Report, IETF (2015). https://datatracker.ietf.org/doc/rfc7424/
  59. 59.
    Silverman, B.W.: Density Estimation for Statistics and Data Analysis, vol. 26. CRC Press, Boca Raton (1986)CrossRefMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.Departamento de Informática, Centro AlgoritmiUniversidade do MinhoBragaPortugal

Personalised recommendations