Abstract
With SINr, we introduced a way to design graph and word embeddings based on community detection. Contrary to deep learning approaches, this approach does not require much compute and was proven to be at the state-of-the-art for interpretability in the context of word embeddings. In this paper, we investigate how filtering communities detected on word co-occurrence networks can improve performances of the approach. Community detection algorithms tend to uncover communities whose size follows a power-law distribution. Naturally, the number of activations per dimensions in SINr follows a power-law: a few dimensions are activated by many words, and many dimensions are activated by a few words. By filtering this distribution, removing part of its head and tail, we show improvement on intrinsic evaluation of the embedding while dividing their dimensionality by five. In addition, we show that these results are stable through several runs, thus defining a subset of distinctive features to describe a given corpus.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Adamic, L.: Unzipping Zipf’s law. Nature 474(7350), 164–165 (2011)
Agirre,E., Alfonseca,E., Hall,K., Kravalova, J., Paşca, M.: A study on similarity and relatedness using distributional and WordNet-based approaches. In: NAACL, pp. 19–27, (2009)
Baroni, M.: 39 distributions in text. In: Corpus Linguistics: An international handbook, vol. 2, pp. 803–822. Mouton de Gruyter (2005)
Baroni, M., Bernardini, S., Ferraresi, A., Zanchetta, E.: The wacky wide web: a collection of very large linguistically processed web-crawled corpora. Lang. Resour. Eval. 43(3), 209–226 (2009)
Blondel, V.D., Guillaume,J.-L., Lambiotte,R., Lefebvre, E.: Fast unfolding of communities in large networks. J. Stat. Mech.: Theory Exp., p. P10008 (2008)
Bruni, E., Tran, N.K., Baroni, M.: Multimodal distributional semantics. J. Artif. Int. Res. 49(1), 1–47 (2014)
BNC Consortium. British national corpus, XML edition (2007). Oxford Text Archive
Dao, V.L., Bothorel, C., Lenca, P.: Community structure: a comparative evaluation of community detection methods. Netw. Sci. 8(1), 1–41 (2020)
Dugué, Nicolas, Lamirel, Jean-Charles., Perez, Anthony: Bringing a feature selection metric from machine learning to complex networks. In: Aiello, Luca Maria, Cherifi, Chantal, Cherifi, Hocine, Lambiotte, Renaud, Lió, Pietro, Rocha, Luis M.. (eds.) COMPLEX NETWORKS 2018. SCI, vol. 813, pp. 107–118. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-05414-4_9
Guillot, S., Prouteau, T., Dugué, N.: Sparser is better: one step closer to word embedding interpretability. In: International Workshop on Computational Semantics (2023)
Hill, F., Reichart, R., Korhonen, A.: SimLex-999: evaluating semantic models with (genuine) similarity estimation. CoRR, abs/1408.3456 (2014)
Huang, E., Socher, R., Manning, C., Ng, A.: Improving word representations via global context and multiple word prototypes. In: ACL, pp. 873–882 (2012)
Lambiotte, R.: Multi-scale modularity and dynamics in complex networks. In: Mukherjee, A., Choudhury, M., Peruani, F., Ganguly, N., Mitra, B. (eds.) Dynamics On and Of Complex Networks, vol. 2, pp. 125–141. Springer, New York (2013). https://doi.org/10.1007/978-1-4614-6729-8_7
Launay, J., et al.: PAGnol: an extra-large french generative model. arXiv preprint arXiv:2110.08554 (2021)
Levy, O., Goldberg, Y.: Neural Word Embedding as Implicit Matrix Factorization. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., Weinberger, K.Q., (eds.) Advances in Neural Information Processing Systems, vol. 27. Curran Associates, Inc. (2014)
Levy, O., Goldberg, Y., Dagan, I.: Improving distributional similarity with lessons learned from word embeddings. ACL 3, 211–225 (2015)
Louis Martin, et al.: CamemBERT: a tasty French language model. arXiv:1911.03894 (2019)
Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: ICLR (2013)
Murphy, B., Talukdar, P., Mitchell, T.: Learning effective and interpretable semantic models using non-negative sparse embedding. In: COLING, pp. 1933–1950 (2012)
Pennington, J., Socher, R., Manning, C.D.: GloVe: global vectors for word representation. In: EMNLP, pp. 1532–1543 (2014)
Prouteau, T., et al.: SINr: fast computing of sparse interpretable node representations is not a sin! In: Abreu, P.H., Rodrigues, P.P., Fernández, A., Gama, J. (eds.) IDA 2021. LNCS, vol. 12695, pp. 325–337. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-74251-5_26
Prouteau, T., Dugué, N., Camelin, N., Meignier, S.: Are embedding spaces interpretable? Results of an intrusion detection evaluation on a large French corpus. In: LREC (2022)
Subramanian, A., Pruthi, D., Jhamtani, H., Berg-Kirkpatrick, T., Hovy, E.: Spine: SParse interpretable neural embeddings. In: AAAI (2018)
Acknowledgements
The work has been funded by the ANR project DIGING (ANR-21-CE23-0010).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Béranger, A., Dugué, N., Guillot, S., Prouteau, T. (2024). Filtering Communities in Word Co-Occurrence Networks to Foster the Emergence of Meaning. In: Cherifi, H., Rocha, L.M., Cherifi, C., Donduran, M. (eds) Complex Networks & Their Applications XII. COMPLEX NETWORKS 2023. Studies in Computational Intelligence, vol 1141. Springer, Cham. https://doi.org/10.1007/978-3-031-53468-3_32
Download citation
DOI: https://doi.org/10.1007/978-3-031-53468-3_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-53467-6
Online ISBN: 978-3-031-53468-3
eBook Packages: EngineeringEngineering (R0)