Abstract
Visualizing communication logs, like NetFlow records, is extremely useful for numerous tasks that need to analyze network traffic traces, like network planning, performance monitoring, and troubleshooting. Communication logs, however, can be massive, which necessitates designing effective visualization techniques for large data sets. To address this problem, we introduce a novel network traffic visualization scheme based on the key ideas of (1) exploiting frequent itemset mining (FIM) to visualize a succinct set of interesting traffic patterns extracted from large traces of communication logs; and (2) visualizing extracted patterns as hypergraphs that clearly display multi-attribute associations. We demonstrate case studies that support the utility of our visualization scheme and show that it enables the visualization of substantially larger data sets than existing network traffic visualization schemes based on parallel-coordinate plots or graphs. For example, we show that our scheme can easily visualize the patterns of more than 41 million NetFlow records. Previous research has explored using parallel-coordinate plots for visualizing network traffic flows. However, such plots do not scale to data sets with thousands of even millions of flows.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Choi H, Lee H, Kim H (2009) Fast detection and visualization of network attacks on parallel coordinates. Comput Secur 28(5):276–288
Berthier R, Cukier M, Hiltunen M, Kormann D, Vesonder G, Sheleheda D (2010) Nfsight: netflow-based network awareness tool. In: Proceedings of LISA
Borgelt C, Wang X (2009) Sam: A split and merge algorithm for fuzzy frequent item set mining. In: Proceedings of IFSA/EUSFLAT
Boschetti A, Salgarelli L, Muelder C, Ma K.-L (2011) Tvi: a visual querying system for network monitoring and anomaly detection. In: Proceedings of the 8th International Symposium on Visualization for Cyber, Security
Brauckhoff D, Dimitropoulos X, Wagner A, Salamatian K (2009) Anomaly extraction in backbone networks using association rules. In: Proceedings of the 9th ACM SIGCOMM conference on Internet measurement conference, ACM pp 28–34
Cirneci A, Boboc S, Leordeanu C, Cristea V, Estan C (2009) Netpy: Advanced network traffic monitoring. In: Proceedings of the 2009 International Conference on Intelligent Networking and Collaborative Systems. INCOS’ 09
D’Amico AD, Goodall JR, Tesone DR, Kopylec JK (2007) Visual discovery in computer network defense. IEEE Comput Graph Appl 27(5):20–27
Ellson J, Gansner ER, Koutsofios E, North SC, Woodhull G (2003) Graphviz and dynagraph static and dynamic graph drawing tools. In: GRAPH DRAWING SOFTWARE, Springer, Berlin, pp 127–148
Ertek G, Demiriz A (2006) A framework for visualizing association mining results. In: Proceedings of the 21st International Conference on computer and information sciences
Estan C, Savage S, Varghese G (2003) Automatically inferring patterns of resource consumption in network traffic. Comp Commun Rev 33(4):137–150
Fischer F, Mansmann F, Keim DA, Pietzko S, Waldvogel M (2008) Large-scale network monitoring for visual analysis of attacks. In: Proceedings of the 5th International Workshop on visualization for computer, security
Fruchterman TMJ, Reingold EM (1991) Graph drawing by force-directed placement. Softw Pract Exp 21(11):1129–1164
Glanfield J, Brooks S, Taylor T, Paterson D, Smith C, Gates C, Mchugh J (2009) OverFlow: An overview visualization for network analysis. In: Proceedings of workshop on visualization for cyber security (VizSec)
Glatz E (2010) Visualizing host traffic through graphs. In: Proceedings of the Seventh International Symposium on visualization for cyber, security
Glatz E, Dimitropoulos X (2012) Classifying internet one-way traffic. In: Proceedings of ACM SIGCOMM Internet Measurement Conference
Haag P (2005) Watch your flows with nfsen and nfdump. In: In 50th RIPE Meeting
Hahsler M, Chelluboina S (2011) Visualizing association rules: Introduction to the R-extension package arulesViz. R project module
Iliofotou M, Pappu P, Faloutsos M, Mitzenmacher M, Singh S, Varghese G (2007) Network monitoring using traffic dispersion graphs (TDGs). In: Proceedings of ACM SIGCOMM Internet Measurement Conference
Jin Y, Sharafuddin E, Zhang Z.-L (2009) Unveiling core network-wide communication patterns through application traffic activity graph decomposition. In: Proceedings of SIGMETRICS
Kamada T, Kawai S (1989) An algorithm for drawing general undirected graphs. Inf Process Lett 31(1):7–15
Karagiannis T, Papagiannaki K, Faloutsos M (2005) Blinc: multilevel traffic classification in the dark. In: Proceedings of the 2005 Conference on applications, technologies, architectures, and protocols for computer communications
Lakkaraju K, Yurcik W, Lee AJ (2004) Nvisionip: netflow visualizations of system state for security situational awareness. In: Proceedings of the 2004 ACM workshop on visualization and data mining for computer, security
Luca Deri. ntop. http://www.ntop.org
Plonka D (2000) Flowscan: a network traffic flow reporting and visualization tool. In: Proceedings of the 14th USENIX Conference on system administration
Leinen S (2001) Fluxoscope —a system for flow-based accounting. In: Proceedings of the 2001 passive and active measurement workshop (poster). http://www.switch.ch/network/operation/statistics/fluxoscope/
Srikant R, Agrawal R (1997) Mining generalized association rules. Future Gener Comput Syst 13(2–3): 161–180
Taylor T, Paterson D, Glanfield J, Gates C, Brooks S, McHugh J (2009) Flovis: flow visualization system. In: Conference for homeland security, 2009. CATCH ’09. Cybersecurity applications technology
Wang J, Han J, Lu Y, Tzvetkov P (2005) Tfp: an efficient algorithm for mining top-k frequent closed itemsets. Knowl Data Eng IEEE Trans 17(5):652–663
Yin X, Yurcik W, Treaster M, Li Y, Lakkaraju K (2004) Visflowconnect: netflow visualizations of link relationships for security situational awareness. In: Proceedings of the 2004 ACM workshop on visualization and data mining for computer, security
Author information
Authors and Affiliations
Corresponding author
Additional information
First IMC Workshop on Internet Visualization (WIV 2012), November 13, 2012, Boston, MA, USA.
Rights and permissions
About this article
Cite this article
Glatz, E., Mavromatidis, S., Ager, B. et al. Visualizing big network traffic data using frequent pattern mining and hypergraphs. Computing 96, 27–38 (2014). https://doi.org/10.1007/s00607-013-0282-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00607-013-0282-8