Computer Vision and the Internet of Things Ecosystem in the Connected Home

Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 800)


An automatic food replenishment system for fridges may help people with cognitive and motor impairments to have a constant food supply at home. More even, sane people may benefit from this system because it is difficult to know accurately and precisely which goods are present in the fridge every day. This system has been a wish and a major challenge for both white good companies and food distributors for decades. It is known that this system requires two things: a sensing module for food stock tracking and another actuating module for food replenishment. The last module can be easily addressed since nowadays there exist many smartphone applications for food delivering, in fact, many food distributors allow their end-users to schedule food replenishment. On the contrary, food stock tracking is not that easy since this requires artificial intelligence to determine not only the different type of goods present in the fridge but also their quantity and quality. In this work, we address the problem of food detection in the fridge by a supervised computer vision algorithm based on Fast Region-based Convolutional Network and an internet of things ecosystem architecture in the connected home for getting high performance on training and deployment of the proposed method. We have tested our method on a data set of images containing sixteen types of goods in the fridge, built with the aid of a fridge-cam. Preliminary results suggest that it is possible to detect different goods in the fridge with good accuracy and that our method may rapidly scale.


Computer vision Connected home Ecosystem Internet of things Machine learning 



This is a working progress being part of a research project (Computational Prototype for IoT Environments) among Colombia Institutions: Universidad Nacional, Universidad de Caldas and MABE, with code 36715.


  1. 1.
    S. A. LTD: FridgeCAM (2016). Accessed 30 Dec 2018
  2. 2.
    Färnström, F., Johansson, B., Åström, K.: Computer vision for determination of fridge contents. In: Proceedings SSAB 2002, Symposium on Image Analysis, pp. 45–48 (2002)Google Scholar
  3. 3.
    Floarea, A.D., Sgârciu, V.: Smart refrigerator : a next generation refrigerator connected to the IoT. In: Proceedings of the 2016 8th International Conference Electronics Computers and Artificial Intelligence (ECAI) (2017)Google Scholar
  4. 4.
    Hachani, A., Barouni, I., Ben Said, Z., Amamou, L.: RFID based smart fridge. In: 2016 8th IFIP International Conference on New Technologies, Mobility and Security (NTMS), pp. 1–4 (2016)Google Scholar
  5. 5.
    Hazen, T.J.: Microsoft and Liebherr Collaborating on New Generation of Smart Refrigerators (2016). Accessed 15 Dec 2017
  6. 6.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)CrossRefGoogle Scholar
  7. 7.
    Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2017)zbMATHGoogle Scholar
  8. 8.
    Girshick, R.: Fast R-CNN. In: IEEE International Conference on Computer Vision (ICCV) 2015, pp. 1440–1448 (2015)Google Scholar
  9. 9.
    Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition 2014, pp. 580–587 (2014)Google Scholar
  10. 10.
    Everingham, M., Eslami, S.M.A., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The Pascal Visual Object Classes Challenge: A Retrospective. Int. J. Comput. Vis. 111(1), 98–136 (2015)CrossRefGoogle Scholar
  11. 11.
    Hosang, J., Benenson, R., Schiele, B.: How good are detection proposals, really? In: Proceedings of the British Machine Vision Conference 2014, pp. 24.1–24.12 (2014)Google Scholar
  12. 12.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks.In: Advances in Neural Information Processing System, pp. 1–9 (2012)Google Scholar
  13. 13.
    He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, Sun, Jian: Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition. In: Fleet, David, Pajdla, Tomas, Schiele, Bernt, Tuytelaars, Tinne (eds.) ECCV 2014. LNCS, vol. 8691, pp. 346–361. Springer, Cham (2014). Scholar
  14. 14.
    Hosang, J., Benenson, R., Schiele, B.: Learning non-maximum suppression. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017, pp. 6469–6477 (2017)Google Scholar
  15. 15.
    Microsoft: Microsoft Cognitive Toolkit. Microsoft (2016)Google Scholar
  16. 16.
    Uijlings, J.R.R., van de Sande, K.E.A., Gevers, T., Smeulders, A.W.M.: Selective search for object recognition. Int. J. Comput. Vis. 104(2), 154–171 (2013)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Facultad de Ingeniería Y ArquitecturaUniversidad Nacional de ColombiaManizalesColombia
  2. 2.Universidad del ValleCaliColombia
  3. 3.GITIR Grupo Investigación Tecnologías Información y RedesUniversidad de CaldasManizalesColombia

Personalised recommendations