Silent Consensus: Probabilistic Packet Sampling for Lightweight Network Monitoring

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11619)


Artificial intelligence based methods for operations of IT-systems (AIOps) support the process of maintaining and operating large IT infrastructures on different levels, e.g. anomaly detection, root cause analysis, or initiation of self-stabilizing activities. The foundation for the deployment of such methods are extensive and reliable metric data on the current state of the overall system. In particular, network information expressing the core parameters latency, throughput, and bandwidth have crucial impact on modern IoT and edge computing environments. Collecting the data is a challenging problem, as the communication is limited to existent network protocols, and adding new features requires a major infrastructure adaptation. The usage of additional monitoring protocols increases the CPU/network overhead and should be avoided as well. Therefore, we propose a two step approach for measuring latency between adjacent hops without manipulating or generating any network traffic. Inspired by audio and image compression algorithms, we developed a probabilistic method named silent consensus, where we keep the precision within a desired interval while reducing the overhead significantly. This method identifies the same packets on a sequence of network hops solely by observing the regular traffic. A linear regression helps to predict packets that are likely to appear after a fixed temporal offset based on a constrained set of historic observations. A correction of the predicted entity increases the probability for consensus between the involved hops. An extensive experimental evaluation proves that the approach delivers the expected foundation for further analysis of the network streams and the overall system.


Network monitoring Packet sampling Measurement Traffic engineering 


  1. 1.
    Alexandrov, A., et al.: The stratosphere platform for big data analytics. VLDB J. Int. J. Very Large Data Bases 23(6), 939–964 (2014)CrossRefGoogle Scholar
  2. 2.
    Arista Networks, Inc.: arista.comwhite paperarista 7170 multi-function programmable networking. Technical report, Arista (2018).
  3. 3.
    Bosshart, P., et al.: P4: Programming protocol-independent packet processors. ACM SIGCOMM Comput. Commun. Rev. 44(3), 87–95 (2014)CrossRefGoogle Scholar
  4. 4.
    Bosshart, P., et al.: Forwarding metamorphosis: fast programmable match-action processing in hardware for sdn. In: ACM SIGCOMM Computer Communication Review, vol. 43, pp. 99–110. ACM (2013)Google Scholar
  5. 5.
    Brebner, G.: P4 for an FPGA target. In: P4 Workshop (2015)Google Scholar
  6. 6.
    Cisco: Cisco visual networking index: forecast and trends, 2017–2022. Technical report, Cisco (2018).
  7. 7.
    Curtis, A.R., Mogul, J.C., Tourrilhes, J., Yalagandula, P., Sharma, P., Banerjee, S.: DevoFlow: scaling flow management for high-performance networks. In: ACM SIGCOMM Computer Communication Review, vol. 41, pp. 254–265. ACM (2011)CrossRefGoogle Scholar
  8. 8.
    Datta, R., Choi, S., Chowdhary, A., Park, Y.: P4Guard: designing P4 based firewall. In: MILCOM 2018–2018 IEEE Military Communications Conference (MILCOM), pp. 1–6. IEEE (2018)Google Scholar
  9. 9.
    Duffield, N.G., Grossglauser, M.: Trajectory sampling for direct traffic observation. IEEE/ACM Trans. Netw. (ToN) 9(3), 280–292 (2001)CrossRefGoogle Scholar
  10. 10.
    Feamster, N., Rexford, J.: Why (and how) networks should run themselves. CoRR abs/1710.11583 (2017).
  11. 11.
    Feng, Y., Wu, X., Hu, Y.: Forecasting research on the wireless mesh network throughput based on the support vector machine. Wirel. Pers. Commun. 99(1), 581–593 (2018)CrossRefGoogle Scholar
  12. 12.
    Goyal, M., Guerin, R., Rajan, R.: Predicting TCP throughput from non-invasive network sampling. In: INFOCOM 2002. Proceedings of the IEEE Twenty-First Annual Joint Conference of the IEEE Computer and Communications Societies, vol. 1, pp. 180–189. IEEE (2002)Google Scholar
  13. 13.
    Gulenko, A., Wallschläger, M., Kao, O.: A practical implementation of in-band network telemetry in Open vSwitch. In: 2018 IEEE 7th International Conference on Cloud Networking (CloudNet), pp. 1–4. IEEE (2018)Google Scholar
  14. 14.
    Gupta, A., Birkner, R., Canini, M., Feamster, N., Mac-Stoker, C., Willinger, W.: Network monitoring as a streaming analytics problem. In: Proceedings of the 15th ACM Workshop on Hot Topics in Networks, pp. 106–112. ACM (2016)Google Scholar
  15. 15.
    Gupta, A., et al.: Sonata: query-driven network telemetry. arXiv preprint arXiv:1705.01049 (2017)
  16. 16.
    Kim, C., et al.: In-band network telemetry (INT). Technical report, The P4 Language Consortium (2016). Accessed 01 June 2018
  17. 17.
    Kim, C., Sivaraman, A., Katta, N., Bas, A., Dixit, A., Wobker, L.J.: In-band network telemetry via programmable dataplanes. In: ACM SIGCOMM Symposium on SDN Research (SOSR) (2015)Google Scholar
  18. 18.
    Liu, Z., Vorsanger, G., Braverman, V., Sekar, V.: Enabling a RISC approach for software-defined monitoring using universal streaming. In: Proceedings of the 14th ACM Workshop on Hot Topics in Networks, p. 21. ACM (2015)Google Scholar
  19. 19.
    Liu, Z., Bi, J., Zhou, Y., Wang, Y., Lin, Y.: NetVision: towards network telemetry as a service. In: 2018 IEEE 26th International Conference on Network Protocols (ICNP), pp. 247–248. IEEE (2018)Google Scholar
  20. 20.
    Lu, D., Qiao, Y., Dinda, P.A., Bustamante, F.E.: Characterizing and predicting TCP throughput on the wide area network. In: Proceedings of the 25th IEEE International Conference on Distributed Computing Systems, ICDCS 2005, pp. 414–424. IEEE (2005)Google Scholar
  21. 21.
    Mahalingam, M., Dutt, D.G., Duda, K., Agarwal, P.: Virtual eXtensible Local Area Network (VXLAN): a framework for overlaying virtualized layer 2 networks over layer 3 networks. RFC 7348, RFC Editor, August 2014.
  22. 22.
  23. 23.
    Mirza, M., Sommers, J., Barford, P., Zhu, X.: A machine learning approach to TCP throughput prediction. SIGMETRICS Perform. Eval. Rev. 35(1), 97–108 (2007). Scholar
  24. 24.
    Shahbaz, M., et al.: PISCES: a programmable, protocol-independent software switch. In: SIGCOMM Conference, pp. 525–538. ACM (2016)Google Scholar
  25. 25.
    Suh, J., Kwon, T.T., Dixon, C., Felter, W., Carter, J.: OpenSample: a low-latency, sampling-based measurement platform for commodity SDN. In: 2014 IEEE 34th International Conference on Distributed Computing Systems (ICDCS), pp. 228–237. IEEE (2014)Google Scholar
  26. 26.
    Tammana, P., Agarwal, R., Lee, M.: Simplifying datacenter network debugging with pathdump. In: OSDI, pp. 233–248 (2016)Google Scholar
  27. 27.
    Yoon, S., Ha, T., Kim, S., Lim, H.: Scalable traffic sampling using centrality measure on software-defined networks. IEEE Commun. Mag. 55(7), 43–49 (2017)CrossRefGoogle Scholar
  28. 28.
    Yu, M., Jose, L., Miao, R.: Software defined traffic measurement with opensketch. In: NSDI, vol. 13, pp. 29–42 (2013)Google Scholar
  29. 29.
    Zhu, Y., et al.: Packet-level telemetry in large datacenter networks. In: ACM SIGCOMM Computer Communication Review, vol. 45, pp. 479–491. ACM (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Complex and Distributed SystemsTechnische Universität BerlinBerlinGermany

Personalised recommendations