Journal of Network and Systems Management

, Volume 23, Issue 1, pp 58–88 | Cite as

Measuring Cloud Service Health Using NetFlow/IPFIX: The WikiLeaks Case

  • Idilio DragoEmail author
  • Rick Hofstede
  • Ramin Sadre
  • Anna Sperotto
  • Aiko Pras


The increasing trend of outsourcing services to cloud providers is changing the way computing power is delivered to enterprises and end users. Although cloud services offer several advantages, they also make cloud consumers strongly dependent on providers. Hence, consumers have a vital interest to be immediately informed about any problems in their services. This paper aims at a first step toward a network-based approach to monitor cloud services. We focus on severe problems that affect most services, such as outages or extreme server overload, and propose a method to monitor these problems that relies solely on the traffic exchanged between users and cloud providers. Our proposal is entirely based on NetFlow/IPFIX data and, therefore, explicitly targets high-speed networks. By combining a methodology to reassemble and classify flow records with stochastic estimations, our proposal has the distinct characteristic of being applicable to both sampled and non-sampled data. We validate our proposal and show its applicability using data collected at both the University of Twente and an international backbone during the WikiLeaks Cablegate. Our results show that, in contrast to Anonymous’ claims, the users of the targeted services have been only marginally affected by the attacks.


Cloud computing Performance Measurements 



This work has been carried out in the context of the FP7 FLAMINGO Network of Excellence Project (CNECT-ICT-318488), the EU FP7-257513 UniverSelf Collaborative Project and the IOP GenCom project Service Optimization and Quality (SeQual). SeQual is supported by the Dutch Ministry of Economic Affairs, Agriculture and Innovation via its agency Agentschap NL


  1. 1.
    Hajjat, M., Sun, X., Sung, Y.W.E., Maltz, D., Rao, S., Sripanidkulchai, K., Tawarmalani, M.: Cloudward bound: planning for beneficial migration of enterprise applications to the cloud. SIGCOMM Comput. Commun. Rev. 40(4), 243–254 (2010)CrossRefGoogle Scholar
  2. 2.
    Rish, I., Brodie, M., Odintsova, N., Ma, S., Grabarnik, G.: Real-Time Problem Determination in Distributed Systems Using Active Probing. In: Proceedings of the IEEE/IFIP Network Operations and Management Symposium, NOMS’04, pp. 133–146 (2004)Google Scholar
  3. 3.
    Xu, K., Wang, F., Wang, H.: Lightweight and informative traffic metrics for data center monitoring. J. Netw. Syst. Manag. 20, 226–243 (2012)CrossRefGoogle Scholar
  4. 4.
    Clarke, R.: How reliable is cloudsourcing? A review of articles in the technical media 2005–11. Comput. Law Secur. Rev. 28(1), 90–95 (2012)CrossRefGoogle Scholar
  5. 5.
    Gehlen, V., Finamore, A., Mellia, M., Munafò, M.M.: Uncovering the Big Players of the Web. In: Proceedings of the 4th International Conference on Traffic Monitoring and Analysis, TMA’12, pp. 15–28 (2012)Google Scholar
  6. 6.
    Claise, B.: Cisco Systems NetFlow Services Export Version 9. RFC 3954 (Informational) (2004)Google Scholar
  7. 7.
    Claise, B.: Specification of the IP flow information export (IPFIX) protocol for the exchange of IP traffic flow information. RFC 5101 (Standards Track) (2008)Google Scholar
  8. 8.
    Garcia-Dorado, J., Finamore, A., Mellia, M., Meo, M., Munafò, M.M.: Characterization of ISP traffic: trends, user habits, and access technology impact. IEEE Trans. Netw. Serv. Manag. 9(2), 142–155 (2012)CrossRefGoogle Scholar
  9. 9.
    Labovitz, C., Iekel-Johnson, S., McPherson, D., Oberheide, J., Jahanian, F.: Internet Inter-domain Traffic. In: Proceedings of the ACM SIGCOMM 2010 Conference, SIGCOMM’10, pp. 75–86 (2010)Google Scholar
  10. 10.
    Mansfield-Devine, S.: Anonymous: serious threat or mere annoyance? Netw. Secur. 2011(1), 4–10 (2011)CrossRefGoogle Scholar
  11. 11.
    Sommer, R., Feldmann, A.: NetFlow: Information Loss or Win? In: Proceedings of the 2nd ACM SIGCOMM Workshop on Internet Measurement, IMW’02, pp. 173–174 (2002)Google Scholar
  12. 12.
    Paxson, V.: Bro: a system for detecting network intruders in real-time. Comput. Netw. 31(23–24), 2435–2463 (1999)CrossRefGoogle Scholar
  13. 13.
    Duffield, N., Lund, C., Thorup, M.: Estimating flow distributions from sampled flow statistics. IEEE/ACM Trans. Netw. 13(5), 933–946 (2005)CrossRefMathSciNetGoogle Scholar
  14. 14.
    Draper, N., Guttman, I.: Bayesian estimation of the binomial parameter. Technometrics 13(3), 667–673 (1971)CrossRefzbMATHGoogle Scholar
  15. 15.
    Tang, V.K.T., Sindler, R.B., Shirven, R.M.: Bayesian estimation of n in a binomial distribution. Tech. Rep. CRM 87–185, Center for Naval Analyses (1987)Google Scholar
  16. 16.
    Tang, V.K.T., Sindler, R.B.: Confidence interval for parameter n in a binomial distribution. Tech. Rep. CRM 86–265, Center for Naval Analyses (1987)Google Scholar
  17. 17.
    Finamore, A., Mellia, M., Meo, M., Munafò, M.M., Rossi, D.: Experiences of Internet traffic monitoring with Tstat. IEEE Netw. 25(3), 8–14 (2011)CrossRefGoogle Scholar
  18. 18.
    Haag, P.: Watch your flows with NfSen and NFDUMP. 50th RIPE Meeting. (2005). Accessed June 2013
  19. 19.
    Fullmer, M., Romig, S.: The OSU Flow-tools Package and CISCO NetFlow Logs. In: Proceedings of the 14th USENIX conference on System administration, LISA’00, pp. 291–304 (2000)Google Scholar
  20. 20.
    Inacio, C.M., Trammell, B.: YAF: Yet Another Flowmeter. In: Proceedings of the 24th International Conference on Large Installation System Administration, LISA’10, pp. 1–16 (2010)Google Scholar
  21. 21.
    MaxMind: GeoIP Organization. (2013). Accessed June 2013
  22. 22.
    Limmer, T., Dressler, F.: Flow-Based TCP Connection Analysis. In: Proceedings of the 2nd IEEE International Workshop on Information and Data Assurance, WIDA’09, pp. 376–383 (2009)Google Scholar
  23. 23.
    Arlitt, M., Williamson, C.: An analysis of TCP reset behaviour on the Internet. SIGCOMM Comput. Commun. Rev. 35(1), 37–44 (2005)CrossRefGoogle Scholar
  24. 24.
    Fawcett, T.: An introduction to ROC analysis. Pattern Recogn. Lett. 27(8), 861–874 (2006)CrossRefMathSciNetGoogle Scholar
  25. 25.
    van Rijsbergen, C.: Information Retrieval, 2 edn. Butterworth, London (1979)Google Scholar
  26. 26.
    Lampert, R.T., Sommer, C., Munz, G., Dressler, F.: Vermont—A Versatile Monitoring Toolkit for IPFIX and PSAMP. In: Proceedings of the IEEE/IST Workshop on Monitoring, Attack Detection and Mitigation, MonAM’06 (2006)Google Scholar
  27. 27.
    Čeleda, P., Kováčik, M., Koníř, T., Krmíček, V., Špringl, P., Žádník, M.: FlowMon Probe. Tech. Rep., CESNET (2007)Google Scholar
  28. 28.
    Deri, L.: nProbe: An Open Source NetFlow Probe for Gigabit Networks. In: Proceedings of the Terena, TNC’03 (2003)Google Scholar
  29. 29.
    Zseby, T., Molina, M., Duffield, N., Niccolini, S., Raspall, F.: Sampling and Filtering Techniques for IP Packet Selection. RFC 5475 (Standards Track) (2009)Google Scholar
  30. 30.
    Estan, C., Varghese, G.: New directions in traffic measurement and accounting. SIGCOMM Comput. Commun. Rev. 32(4), 323–336 (2002)CrossRefGoogle Scholar
  31. 31.
    Estan, C., Keys, K., Moore, D., Varghese, G.: Building a Better NetFlow. In: Proceedings of the ACM SIGCOMM 2004 Conference, SIGCOMM’04, pp. 245–256 (2004)Google Scholar
  32. 32.
    Dropbox: DropboxOps. (2013). Accessed June 2013
  33. 33.
    Twitter: Status. (2013). Accessed June 2013
  34. 34.
    Google: Apps Status Dashboard. (2013). Accessed June 2013
  35. 35.
    Drago, I., Mellia, M., Munafò, M.M., Sperotto, A., Sadre, R., Pras, A.: Inside Dropbox: Understanding Personal Cloud Storage Services. In: Proceedings of the 12th ACM Internet Measurement Conference, IMC’12, pp. 481–494 (2012)Google Scholar
  36. 36.
    Pras, A., Sperotto, A., Moura, G.C.M., Drago, I., Barbosa, R.R.R., Sadre, R., de Oliveira Schmidt, R., Hofstede, R.: Attacks by “Anonymous” WikiLeaks Proponents not Anonymous. Tech. Rep. TR-CTIT-10-41, CTIT, University of Twente, Enschede (2010)Google Scholar
  37. 37.
    Kossmann, D., Kraska, T., Loesing, S.: An Evaluation of Alternative Architectures for Transaction Processing in the Cloud. In: Proceedings of the ACM SIGMOD International Conference on Management of data, SIGMOD’10, pp. 579–590 (2010)Google Scholar
  38. 38.
    Lenk, A., Menzel, M., Lipsky, J., Tai, S., Offermann, P.: What Are You Paying for? Performance Benchmarking for Infrastructure-as-a-Service Offerings. In: Proceedings of the 4th IEEE International Conference on Cloud Computing, CLOUD’11, pp. 484–491 (2011)Google Scholar
  39. 39.
    Li, A., Yang, X., Kandula, S., Zhang, M.: CloudCmp: Comparing Public Cloud Providers. In: Proceedings of the 10th ACM SIGCOMM Conference on Internet Measurement, IMC’10, pp. 1–14 (2010)Google Scholar
  40. 40.
    Meng, S., Iyengar, A.K., Rouvellou, I.M., Liu, L., Lee, K., Palanisamy, B., Tang, Y.: Reliable State Monitoring in Cloud Datacenters. In: Proceedings of the 5th IEEE International Conference on Cloud Computing, CLOUD’12, pp. 951–958 (2012)Google Scholar
  41. 41.
    Meng, S., Liu, L.: Enhanced monitoring-as-a-service for effective cloud management. IEEE Trans. Comput. (2012)
  42. 42.
    Hu, W., Yang, T., Matthews, J.N.: The good, the bad and the ugly of consumer cloud storage. SIGOPS Oper. Syst. Rev. 44(3), 110–115 (2010)CrossRefGoogle Scholar
  43. 43.
    Wang, G., Ng, T.E.: The Impact of Virtualization on Network Performance of Amazon EC2 Data Center. In: Proceedings of the 29th Conference on Information Communications, INFOCOM’10, pp. 1–9 (2010)Google Scholar
  44. 44.
    Zhang, Q., Cheng, L., Boutaba, R.: Cloud computing: state-of-the-art and research challenges. J. Internet Serv. Appl. 1, 7–18 (2010)CrossRefGoogle Scholar
  45. 45.
    Glatz, E., Dimitropoulos, X.: Classifying Internet One-Way Traffic. In: Proceedings of the 12th ACM Internet Measurement Conference, IMC’12, pp. 37–50 (2012)Google Scholar
  46. 46.
    Schatzmann, D., Leinen, S., Kögel, J., Mühlbauer, W.: FACT: Flow-Based Approach for Connectivity Tracking. In: Proceedings of the 12th International Conference on Passive and Active Network Measurement, PAM’11, pp. 214–223 (2011)Google Scholar
  47. 47.
    Caracas, A., Kind, A., Gantenbein, D., Fussenegger, S., Dechouniotis, D.: Mining Semantic Relations using NetFlow. In: Proceedings of the 3rd IEEE/IFIP International Workshop on Business-driven IT Management, BDIM’08, pp. 110–111 (2008)Google Scholar
  48. 48.
    Bermudez, I., Mellia, M., Munafò, M.M., Keralapura, R., Nucci, A.: DNS to the Rescue: Discerning Content and Services in a Tangled Web. In: Proceedings of the 12th ACM Internet Measurement Conference, IMC’12, pp. 413–426 (2012)Google Scholar
  49. 49.
    Quittek, J., Bryant, S., Claise, B., Aitken, P., Meyer, J.: Information Model for IP Flow Information Export. RFC 5102 (Standards Track) (2008)Google Scholar
  50. 50.
    Trammell, B., Boschi, E.: An introduction to IP flow information export (IPFIX). Commun. Mag. 49(4), 89–95 (2011)CrossRefGoogle Scholar
  51. 51.
    Lewis, D.D., Gale, W.A.: A Sequential Algorithm for Training Text Classifiers. In: Proceedings of the 17th annual international ACM SIGIR conference on research and development in information retrieval, SIGIR’94, pp. 3–12 (1994)Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Idilio Drago
    • 1
    Email author
  • Rick Hofstede
    • 1
  • Ramin Sadre
    • 2
  • Anna Sperotto
    • 1
  • Aiko Pras
    • 1
  1. 1.University of TwenteEnschedeThe Netherlands
  2. 2.Aalborg UniversityAalborgDenmark

Personalised recommendations