Metrics Matter in Community Detection

Conference paper
Part of the Studies in Computational Intelligence book series (SCI, volume 881)


We critically evaluate normalized mutual information (NMI) as an evaluation metric for community detection. NMI exaggerates the leximin method’s performance on weak communities: Does leximin, in finding the trivial singletons clustering, truly outperform eight other community detection methods? Three NMI improvements from the literature are AMI, rrNMI, and cNMI. We show equivalences under relevant random models, and for evaluating community detection, we advise one-sided AMI under the \(\mathbb {M}_\mathbf{all }\) model (all partitions of \(n\) nodes). This work seeks (1) to drive a conversation on robust measurements (2) to advocate evaluations which do not give “free lunch”.


  1. 1.
    Blondel, V.D., Guillaume, J.L., Lambiotte, R., Lefebvre, E.: Fast unfolding of communities in large networks. J. Stat. Mech. 2008(10), P10008 (2008)CrossRefGoogle Scholar
  2. 2.
    Clauset, A., Newman, M.E.J., Moore, C.: Finding community structure in very large networks. Phys. Rev. E 70, 066111 (2004)CrossRefGoogle Scholar
  3. 3.
    Danon, L., Díaz-Guilera, A., Duch, J., Arenas, A.: Comparing community structure identification. J. Stat. Mech. 2005(09), P09008 (2005)CrossRefGoogle Scholar
  4. 4.
    Decelle, A., Krzakala, F., Moore, C., Zdeborová, L.: Asymptotic analysis of the stochastic block model for modular networks and its algorithmic applications. Phys. Rev. E 84, 066106 (2011)CrossRefGoogle Scholar
  5. 5.
    Dhillon, I.S., Mallela, S., Kumar, R.: A divisive information-theoretic feature clustering algorithm for text classification. JMLR 3, 1265–1287 (2003)MathSciNetzbMATHGoogle Scholar
  6. 6.
    Gates, A.J., Ahn, Y.Y.: The impact of random models on clustering similarity. JMLR 18(87), 1–28 (2017)MathSciNetzbMATHGoogle Scholar
  7. 7.
    Horta, D., Campello, R.J.G.B.: Comparing hard and overlapping clusterings. JMLR 16(1), 2949–2997 (2015)MathSciNetzbMATHGoogle Scholar
  8. 8.
    Hubert, L., Arabie, P.: Comparing partitions. J. Classif. 2, 1 (1985)CrossRefGoogle Scholar
  9. 9.
    Kingman, J.F.C.: The representation of partition structures. J. London Math. Soc. s2–18(2), 374–380 (1978)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Lai, D., Nardini, C.: A corrected normalized mutual information for performance evaluation of community detection. J. Stat. Mech. (9) (2016)Google Scholar
  11. 11.
    Lancichinetti, A., Fortunato, S., Radicchi, F.: Benchmark graphs for testing community detection algorithms. Phys. Rev. E 78, 046110 (2008) CrossRefGoogle Scholar
  12. 12.
    Liu, X., Cheng, H.M., Zhang, Z.Y.: Evaluation of community structures using kappa index and F-score instead of NMI. arXiv (2018)Google Scholar
  13. 13.
    Matula, D.W., Shahrokhi, F.: Sparsest cuts and bottlenecks in graphs. Discrete Appl. Math. 27(1), 113–123 (1990)MathSciNetCrossRefGoogle Scholar
  14. 14.
    McCarthy, A.D.: Gridlock in networks: the Leximin method for hierarchical community detection. Master’s thesis, Southern Methodist University (2017)Google Scholar
  15. 15.
    McCarthy, A.D., Chen, T., Ebner, S.: An exact no free lunch theorem for community detection. In: Proceedings of the 8th International Conference on Complex Networks and Their Applications: Complex Networks 2019, Lisbon, Portugal (2019)Google Scholar
  16. 16.
    Meilă, M.: Comparing clusterings by the variation of information. In: Learning Theory and Kernel Machines (2003)CrossRefGoogle Scholar
  17. 17.
    Newman, M.E.J.: Finding community structure in networks using the eigenvectors of matrices. Phys. Rev. E 74, 036104 (2006)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Peel, L., Larremore, D.B., Clauset, A.: The ground truth about metadata and community detection in networks. Sci. Adv. 3(5), e1602548 (2017)CrossRefGoogle Scholar
  19. 19.
    Pons, P., Latapy, M.: Computing communities in large networks using random walks. In: Computer and Information Sciences - ISCIS (2005)CrossRefGoogle Scholar
  20. 20.
    Radicchi, F., Castellano, C., Cecconi, F., Loreto, V., Parisi, D.: Defining and identifying communities in networks. PNAS 101(9), 2658–2663 (2004)CrossRefGoogle Scholar
  21. 21.
    Raghavan, U.N., Albert, R., Kumara, S.: Near linear time algorithm to detect community structures in large-scale networks. Phys. Rev. E 76, 036106 (2007)CrossRefGoogle Scholar
  22. 22.
    Reichardt, J., Bornholdt, S.: Statistical mechanics of community detection. Phys. Rev. E 74, 016110 (2006)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Romano, S., Bailey, J., Nguyen, V., Verspoor, K.: Standardized mutual information for clustering comparisons: one step further in adjustment for chance. In: Proceedings of 31st International Conference on Machine Learning, vol. 32 (2014)Google Scholar
  24. 24.
    Romano, S., Vinh, N.X., Bailey, J., Verspoor, K.: Adjusting for chance clustering comparison measures. JMLR 17(1), 4635–4666 (2016)MathSciNetzbMATHGoogle Scholar
  25. 25.
    Rosenberg, A., Hirschberg, J.: V-measure: a conditional entropy-based external cluster evaluation measure. In: Proceedings EMNLP-CoNLL (2007)Google Scholar
  26. 26.
    Rosvall, M., Axelsson, D., Bergstrom, C.T.: The map equation. Eur. Phys. J. Spec. Top. 178(1), 13–23 (2009)CrossRefGoogle Scholar
  27. 27.
    Rosvall, M., Bergstrom, C.T.: An information-theoretic framework for resolvingcommunity structure in complex networks. PNAS 104(18), 7327–7331 (2007)CrossRefGoogle Scholar
  28. 28.
    Shahrokhi, F., Matula, D.W.: The maximum concurrent flow problem. J. ACM 37(2), 318–334 (1990)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Veldt, N., Gleich, D.F., Wirth, A.: A correlation clustering framework for community detection. In: Proceedings of 2018 WWW Conference (2018)Google Scholar
  30. 30.
    Vinh, N.X., Epps, J., Bailey, J.: Information theoretic measures for clusterings comparison: is a correction for chance necessary? In: Proceedings of ICML (2009)Google Scholar
  31. 31.
    Wolpert, D.H.: The lack of a priori distinctions between learning algorithms. Neural Comput. 8(7), 1341–1390 (1996)CrossRefGoogle Scholar
  32. 32.
    Yang, Z., Algesheimer, R., Tessone, C.J.: A comparative analysis of community detection algorithms on artificial networks. Sci. Rep. 6, 30750 (2016)CrossRefGoogle Scholar
  33. 33.
    Young, J.G., St-Onge, G., Desrosiers, P., Dubé, L.J.: Universality of the stochastic block model. Phys. Rev. E 98, 032309 (2018)CrossRefGoogle Scholar
  34. 34.
    Zhang, J., Chen, T., Hu, J.: On the relationship between Gaussian stochastic blockmodels and label propagation algorithms. J. Stat. Mech. 2015(3), P03009 (2015)MathSciNetCrossRefGoogle Scholar
  35. 35.
    Zhang, P.: Evaluating accuracy of community detection using the relative normalized mutual information. J. Stat. Mech. 2015(11), P11006 (2015)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Johns Hopkins UniversityBaltimoreUSA
  2. 2.Southern Methodist UniversityUniversity ParkUSA

Personalised recommendations