Skip to main content
SpringerLink
Log in

A framework for low cost, ubiquitous and interactive smart refrigerator

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Internet of Things (IoT) and Artificial Intelligence (AI)-enabled technologies are essential in developing innovative environments and intelligent applications. IoT and AI-enabled appliances are entering our kitchens, adding more comfort and usability. However, these appliances are not economical and are beyond the reach of a commoner with a moderate income. An intelligent fridge is one such appliance. This paper proposes a design for developing a cost-effective, ubiquitous, and intelligent refrigerator. Unlike existing approaches, the proposed method identifies and predicts the fridge items based on Night Vision images and provides minimal natural language interaction with the fridge. The proposed design aims to convert any standard refrigerator into its more intelligent counterpart with minimal hardware and software requirements. The design allows users to view fridge contents on the go using a mobile application and interact with it using natural language. The transfer learning technique enables us to use a YOLOv5n model for object detection. As there are no publicly available Night Vision image datasets of fridge items, we created a custom dataset of Night Vision images to train and validate the object recognition model. Our model for object detection achieved a mAP of 97.1% compared to the YOLOv3-tiny and YOLOv4-tiny models, whose mAP are 94.8% and 96.3%, respectively. The overall cost of the refrigerator after deployment of the module is less than $300, making it an affordable option. The proposed framework meets most of the requirements of a low-cost, ubiquitous, interactive smart refrigerator.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Algorithm 2
Fig. 10
Fig. 11
Algorithm 3
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

Data Availability

The dataset generated and analysed in the current work is obtainable on request from the corresponding author.

References

  1. Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Mané D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Viégas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X (2015) TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Software available from tensorflow.org. https://www.tensorflow.org/

  2. Adarsh P, Rathi P, Kumar M (2020) Yolo v3-tiny: Object detection and recognition using one stage improved model. In: 2020 6th International Conference on Advanced Computing and Communication Systems (ICACCS), pp 687–694 IEEE

  3. Al-Sarawi S, Anbar M, Abdullah R, Al Hawari AB (2020) Internet of things market analysis forecasts, 2020–2030. In: 2020 4th World conference on smart trends in systems, security and sustainability (WorldS4), pp 449–453. IEEE

  4. Anand G, Prakash L (2018) Iot based novel smart refrigerator to curb food wastage. In: 2018 3rd International Conference on Contemporary Computing and Informatics (IC3I), pp 268–272. IEEE

  5. Bayya M (2019) Low cost smart refrigerator. In: TENCON 2019-2019 IEEE Region 10 Conference (TENCON), pp 1702–1705. IEEE

  6. Bochkovskiy A, Wang C, Liao HM (2020) YOLOV4: Optimal speed and accuracy of object detection preprint at arXiv:2004.10934v1

  7. Bansal T, Agrawal SS, Kumar D, Shambu M, Inbarajan P (2021) Ai based diagnostic service for iot enabled smart refrigerators. In: 2021 8th International Conference on Future Internet of Things and Cloud (FiCloud), pp 163–168. IEEE

  8. Cappelletti F, Papetti A, Rossi M, Germani M (2022) Smart strategies for household food waste management. Procedia Comput Sci 200:887–895

    Article  Google Scholar 

  9. Dong Z, Abdulghani AM, Imran MA, Abbasi QH (2020) Artificial intelligence enabled smart refrigeration management system using internet of things framework. In: Proceedings of the 2020 International Conference on Computing, Networks and Internet of Things, pp 65–70

  10. Ferrero R, Vakili MG, Giusto E, Guerrera M, Randazzo V (2019) Ubiquitous fridge with natural language interaction. In: 2019 IEEE International Conference on RFID Technology and Applications (RFID-TA), pp 404–409. IEEE

  11. Floarea A-D, Sgârciu V (2016) Smart refrigerator: A next generation refrigerator connected to the iot. In: 2016 8th International Conference on Electronics, Computers and Artificial Intelligence (ECAI), pp. 1–6. IEEE

  12. Gao X, Ding X, Hou R, Tao Y (2019) Research on food recognition of smart refrigerator based on ssd target detection algorithm. In: Proceedings of the 2019 International conference on artificial intelligence and computer science, pp 303–308

  13. Gupta S, Giri S, Srivastava T, Agarwal P, Sharma R, Agrawal A (2021) Smart refrigerator based on ’internet of things’. In: 2021 International Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE), pp. 436–439. IEEE

  14. Hossain S, Abdelgawad A (2018) Smart refrigerator based on internet of things (iot) an approach to efficient food management. In: Proceedings of the 2nd International conference on smart digital environment, pp 15–18

  15. InstaView door-in-door™Wi-Fi refrigerators (2022) https://www.lg.com/in/instaview-door-in-door Accessed 1 Jul 2022

  16. Jocher G, Changyu L, Hogan A, Changyu98 LY, Rai P, Sullivan T (2020) Ultralytics/yolov5: Initial Release. https://doi.org/10.5281/zenodo.3908560

  17. Jiang Z, Zhao L, Li S, Jia Y (2020) Real-time object detection method based on improved yolov4-tiny. arXiv:2011.04244

  18. Jain P, Chawla P (2021) Smart module design for refrigerators based on inception-v3 cnn architecture. In: 2021 2nd International Conference on Electronics and Sustainable Communication Systems (ICESC), pp 1852–1859. IEEE

  19. Jocher G, Stoken A, Chaurasia A, Borovec J, NanoCode012, TaoXie, Kwon Y, Michael K, Changyu L, Fang J, Laughing VA, tkianai, yxNONG, Skalski P, Hogan A, Nadar J, imyhxy, Mammana L, AlexWang1900, Fati C, Montes D, Hajek J, Diaconu L, Minh MT, Marc, albinxavi, fatih, oleg, wanghaoyang0106 (2021) ultralytics/yolov5: V6.0 - YOLOv5n ’Nano’ Models, Roboflow Integration, TensorFlow Export, OpenCV DNN Support. https://doi.org/10.5281/zenodo.5563715

  20. Jocher G, Stoken A, Borovec J, NanoCode012, ChristopherSTAN, Changyu L, Laughing, tkianai, yxNONG, Hogan A, lorenzomammana, AlexWang1900, Chaurasia A, Diaconu L, Marc, wanghaoyang0106, ml5ah, Doug, Durgesh, Ingham F, Frederik, Guilhen, Colmagro A, Ye H, Jacobsolawetz, Poznanski J, Fang J, Kim J, Doan K , L.Y. (2021) ultralytics/yolov5: V4.0 - nn.SiLU() Activations, Weights & Biases Logging, PyTorch Hub Integration. https://doi.org/10.5281/zenodo.4418161

  21. Kang J, Gwak J (2022) Ensemble of multi-task deep convolutional neural networks using transfer learning for fruit freshness classification. Multimed Tools Appl 81(16):22355–22377

    Article  Google Scholar 

  22. Khan MA, Shahid MHB, Mansoor H, Shafique U, Khan MB et al (2019) Iot based grocery management system: Smart refrigerator and smart cabinet. In: 2019 International Conference on Systems of Collaboration Big Data, Internet of Things & Security (SysCoBIoTS), pp. 1–5. IEEE

  23. Kim I (2016) The framework for implementation of smart refrigerators using iot. Transportation 1(2):3

    MathSciNet  Google Scholar 

  24. Krishnamoorthy R, Krishnan K, Bharatiraja C (2021) Deployment of iot for smart home application and embedded real-time control system. Mater Today Proc 45:2777–2783

    Article  Google Scholar 

  25. Lin T-Y, Maire M, Belongie S, Bourdev L, Girshick R, Hays J, Perona P, Ramanan D, Zitnick CL, Dollár P (2014) Microsoft COCO: Common Objects in Context. arXiv. https://doi.org/10.48550/ARXIV.1405.0312

  26. Laker B, Patel C, Budhwar P, Malik A (2021) Six steps to innovate remotely. MIT Sloan Management Review

  27. Lee T-H, Kang S-W, Kim T, Kim J-S, Lee H-J (2021) Smart refrigerator inventory management using convolutional neural networks. In: 2021 IEEE 3rd International Conference on Artificial Intelligence Circuits and Systems (AICAS), pp 1–4. IEEE

  28. Lakhan A, Mohammed MA, Ibrahim DA, Abdulkareem KH (2021) Bio-inspired robotics enabled schemes in blockchain-fog-cloud assisted iomt environment. Journal of King Saud University - Computer and Information Sciences. https://doi.org/10.1016/j.jksuci.2021.11.009

  29. Li Y, Kumar R, Lasecki WS, Hilliges O (2020) Artificial intelligence for hci: A modern approach. CHI EA ’20, pp 1–8. Association for computing machinery. https://doi.org/10.1145/3334480.3375147

  30. Lakhan A, Mohammed MA, Abdulkareem KH, Jaber MM, Nedoma J, Martinek R, Zmij P (2022) Delay optimal schemes for internet of things applications in heterogeneous edge cloud computing networks. Sensors 22(16). https://doi.org/10.3390/s22165937

  31. Lakhan A, Mohammed MA, Rashid AN, Kadry S, Abdulkareem KH, Nedoma J, Martinek R, Razzak I (2022) Restricted boltzmann machine assisted secure serverless edge system for internet of medical things. IEEE Journal of Biomedical and Health Informatics

  32. Lakhan A, Mohammed MA, Kadry S, AlQahtani SA, Maashi MS, Abdulkareem KH (2022) Federated learning-aware multi-objective modeling and blockchain-enable system for iiot applications. Comput Electr Eng 100:107839

    Article  Google Scholar 

  33. Lakhan A, Mohammed MA, Nedoma J, Martinek R, Tiwari P, Vidyarthi A, Alkhayyat A, Wang W (2022) Federated-learning based privacy preservation and fraud-enabled blockchain iomt system for healthcare. IEEE Journal of Biomedical and Health Informatics

  34. Mallikarjun B, Harshitha S, Harshita B, Bhavani S, Tarwey S (2020) Smart refrigerator: An iot and machine learning based approach. In: 2020 International Conference for Emerging Technology (INCET), pp 1–4. IEEE

  35. Mohammad I, Mazumder MSI, Saha EK, Razzaque ST, Chowdhury S (2020) A deep learning approach to smart refrigerator system with the assistance of iot. In: Proceedings of the international conference on computing advancements, pp 1–7

  36. Nasir H, Aziz WBW, Ali F, Kadir K, Khan S (2018) The implementation of iot based smart refrigerator system. In: 2018 2nd International Conference on Smart Sensors and Application (ICSSA), pp 48–52. IEEE

  37. Olivas ES, Guerrero JDM, Sober MM, Benedito JRM, Lopez AJS (2009) Handbook of research on machine learning applications and trends: Algorithms, Methods and Techniques-2 Volumes. Information Science Reference-Imprint of: IGI Publishing

  38. Prapulla S, Shobha G, Thanuja T (2015) Smart refrigerator using internet of things. J Multidiscip Eng Sci Technol 2(1):1795–801

    Google Scholar 

  39. Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L et al (2019) Pytorch: an imperative style, high-performance deep learning library. Advances in neural information processing systems 32

  40. Programme UE (2021) UNEP Food Waste Index Report. https://www.unep.org/resources/report/unep-food-waste-index-report-2021 Accessed 1 Jul 2022

  41. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: Unified, real-time object detection. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp 779–788

  42. Redmon J, Farhadi A (2018) Yolov3: An incremental improvement. arXiv:1804.02767

  43. Saha D, Yadav R, Rachha S et al (2020) Using machine learning in refrigerator to keep inventory. In: Proceedings of the 3rd International Conference on Advances in Science & Technology (ICAST)

  44. Song L, Fei Y (2022) New lite YOLOv4-tiny algorithm and application on crack intelligent detection. J. Shanghai Jiaotong Univ. (Sci.) https://doi.org/10.1007/s12204-022-2504-8

  45. Samsung: Family Hub (2022) https://www.samsung.com/us/explore/family-hub-refrigerator/overview/ Accessed 1 Jul 2022

  46. Stojkoska BLR, Trivodaliev KV (2017) A review of internet of things for smart home: Challenges and solutions. J Clean Prod 140:1454–1464

    Article  Google Scholar 

  47. Tzutalin (2015) LabelImg. Free software: MIT license. https://github.com/tzutalin/labelImg

  48. Tajbakhsh N, Shin JY, Gurudu SR, Hurst RT, Kendall CB, Gotway MB, Liang J (2016) Convolutional neural networks for medical image analysis: Full training or fine tuning? IEEE Trans Med Imaging 35(5):1299–1312

    Article  Google Scholar 

  49. Tusor B, Gubo Š, KmeŤ T, Tóth JT (2020) Augmented smart refrigerator—an intelligent space application. https://doi.org/10.1007/978-3-030-36841-8_17

  50. The Android Profiler (2021) https://developer.android.com/studio/profile/android-profiler Accessed 1 Jul 2022

  51. Tensorflow Lite (2022) https://www.tensorflow.org/lite Accessed 1 Jul 2022

  52. Weiss K, Khoshgoftaar TM, Wang D (2016) A survey of transfer learning. J Big data 3(1):1–40

    Article  Google Scholar 

  53. Wang X, Lv F, Li L, Yi Z, Jiang Q (2022) A novel optimized tiny yolov3 algorithm for the identification of objects in the lawn environment. Sci Rep 12(1):1–9

    Google Scholar 

  54. Wang A, Dadmun CH, Hand RM, O’Keefe SF, J”Nai BP, Anders KA, Duncan SE (2018) Efficacy of light-protective additive packaging in protecting milk freshness in a retail dairy case with led lighting at different light intensities. Food Res Int 114:1–9

    Article  Google Scholar 

  55. Wang K, Ti Y, Liu D, Chen S (2019) A smart refrigerator architecture that reduces food ingredients waste materials and energy consumption. Ekoloji 28(107):4873–4878

    Google Scholar 

  56. Zhang W, Zhang Y, Zhai J, Zhao D, Xu L, Zhou J, Li Z, Yang S (2018) Multi-source data fusion using deep learning for smart refrigerators. Comput Ind 95:15–21

    Article  Google Scholar 

  57. Zhuang F, Qi Z, Duan K, Xi D, Zhu Y, Zhu H, Xiong H, He Q (2020) A comprehensive survey on transfer learning. Proc IEEE 109 (1):43–76

    Article  Google Scholar 

  58. Zhou F, Zhao H, Nie Z (2021) Safety helmet detection based on yolov5. In: 2021 IEEE International Conference on Power Electronics, Computer Applications (ICPECA), pp 6–11. IEEE

  59. Zhou Y, Shi L, Yuan B (2021) A generative adversarial network-based framework for fruit and vegetable occlusion detection in smart refrigerators. In: 2021 International Conference on Signal Processing and Machine Learning (CONF-SPML), pp 290–295. IEEE

Download references

Author information

Authors and Affiliations

  1. Department of Information Technology, National Institute of Technology Karnataka, Surathkal, Mangalore, India

    Sona Mundody & Ram Mohana Reddy Guddeti

Authors
  1. Sona Mundody
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ram Mohana Reddy Guddeti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sona Mundody.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The authors contributed equally to this work

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mundody, S., Guddeti, R.M.R. A framework for low cost, ubiquitous and interactive smart refrigerator. Multimed Tools Appl 83, 13337–13368 (2024). https://doi.org/10.1007/s11042-023-15544-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15544-1

Keywords

Search

Navigation