Abstract
Large-scale production recognition systems are crucial for building efficient E-commerce platforms. However, various traditional product recognition approaches are based on single-modal data input (e.g., image or text), which limits recognition performance. To tackle this issue, in this paper, we propose a Multimodal Deep Fusion Network (MDF-Net) for accurate large-scale product recognition. The MDF-Net has a two-stream late fusion architecture, with a CNN model and a bi-directional language model that respectively extract semantic latent features from multimodal inputs. Image and text features are fused via Hadamard product, then jointly generate results. Further, we investigated the integration of attention mechanism and residual connection to respectively improve the text and image representations. We conduct experiments on a large-scale multimodal E-commerce product dataset MEP-3M, which consists of three million image-text product data. MDF-Net achieves a 93.72% classification accuracy over 599 fine-grained classes. Empirical results demonstrated that the MDF-Net yields better performance than traditional approaches.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Zhong, C., Jiang, L., Liang, Y., Sun, H., Ma, C.: Temporal multiple-convolutional network for commodity classification of online retail platform data. In: Proceedings of the 2020 12th International Conference on Machine Learning and Computing, pp. 236–241 (2020)
Zahera, H.M., Sherif, M.: ProBERT: product data classification with fine-tuning BERT model. In: MWPD@ ISWC (2020)
Yang, Y., Wang, X., Zhao, Q., Sui, T.: Two-level attentions and grouping attention convolutional network for fine-grained image classification. Appl. Sci. 9(9), 1939 (2019)
Morency, L.P., Liang, P.P., Zadeh, A.: Tutorial on multimodal machine learning. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Tutorial Abstracts, Seattle, United States, pp. 33–38. Association for Computational Linguistics, July 2022
Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng, A.Y.: Multimodal deep learning. In: ICML (2011)
Conneau, A., Schwenk, H., Barrault, L., Lecun, Y.: Very deep convolutional networks for text classification. arXiv preprint arXiv:1606.01781 (2016)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Zhang, Z., Bizer, C., Peeters, R., Primpeli, A.: MWPD 2020: semantic web challenge on mining the web of html-embedded product data. In: MWPD@ ISWC (2020)
Wazarkar, S., Keshavamurthy, B.N.: Fashion image classification using matching points with linear convolution. Multimedia Tools Appl. 77(19), 25941–25958 (2018). https://doi.org/10.1007/s11042-018-5829-4
Zhang, W., Yu, J., Hu, H., Hu, H., Qin, Z.: Multimodal feature fusion by relational reasoning and attention for visual question answering. Inf. Fusion 55, 116–126 (2020)
Misikir Tashu, T., Fattouh, S., Kiss, P., Horvath, T.: Multimodal e-commerce product classification using hierarchical fusion. arXiv e-prints (2022) arXiv-2207
Li, L., Nie, Y., Han, W., Huang, J.: A multi-attention-based bidirectional long short-term memory network for relation extraction. In: Liu, D., Xie, S., Li, Y., Zhao, D., El-Alfy, E.-S.M. (eds.) ICONIP 2017. LNCS, vol. 10638, pp. 216–227. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-70139-4_22
Chen, D., Liu, F., Du, X., Gao, R., Xu, F.: MEP-3M: a large-scale multi-modal e-commerce products dataset
Liu, Z., Shen, Y., Lakshminarasimhan, V.B., Liang, P.P., Zadeh, A., Morency, L.P.: Efficient low-rank multimodal fusion with modality-specific factors. arXiv preprint arXiv:1806.00064 (2018)
Zadeh, A., Chen, M., Poria, S., Cambria, E., Morency, L.P.: Tensor fusion network for multimodal sentiment analysis. arXiv preprint arXiv:1707.07250 (2017)
Acknowledgment
This work was partially funded by Research Fund from Science and Technology on Underwater Vehicle Technology Laboratory(2021JCJQ-SYSJJ-LB06905), Water Science and Technology Project of Jiangsu Province under grant No. 2021072, 2021063.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Pan, Y., Zhou, R., Zhao, G., Zhang, W., Chen, D., Liu, F. (2022). MDF-Net: Multimodal Deep Fusion for Large-Scale Product Recognition. In: Deng, W., et al. Biometric Recognition. CCBR 2022. Lecture Notes in Computer Science, vol 13628. Springer, Cham. https://doi.org/10.1007/978-3-031-20233-9_51
Download citation
DOI: https://doi.org/10.1007/978-3-031-20233-9_51
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20232-2
Online ISBN: 978-3-031-20233-9
eBook Packages: Computer ScienceComputer Science (R0)