Skip to main content
SpringerLink
Log in

GarmNet: Improving Global with Local Perception for Robotic Laundry Folding

  • Conference paper
  • First Online:
Towards Autonomous Robotic Systems (TAROS 2019)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 11650))

Included in the following conference series:

Abstract

Developing autonomous assistants to help with domestic tasks is a vital topic in robotics research. Among these tasks, garment folding is one of them that is still far from being achieved mainly due to the large number of possible configurations that a crumpled piece of clothing may exhibit. Research has been done on either estimating the pose of the garment as a whole or detecting the landmarks for grasping separately. However, such works constrain the capability of the robots to perceive the states of the garment by limiting the representations for one single task. In this paper, we propose a novel end-to-end deep learning model named GarmNet that is able to simultaneously localize the garment and detect landmarks for grasping. The localization of the garment represents the global information for recognising the category of the garment, whereas the detection of landmarks can facilitate subsequent grasping actions. We train and evaluate our proposed GarmNet model using the CloPeMa Garment dataset that contains 3,330 images of different garment types in different poses. The experiments show that the inclusion of landmark detection (GarmNet-B) can largely improve the garment localization, with an error rate of 24.7% lower. Solutions as ours are important for robotics applications, as these offer scalable to many classes, memory and processing efficient solutions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://keras.io/.

  2. 2.

    https://www.tensorflow.org/.

  3. 3.

    https://github.com/danfergo/garment.

References

  1. Corona, E., Alenyà, G., Gabas, A., Torras, C.: Active garment recognition and target grasping point detection using deep learning. Pattern Recogn. 74, 629–641 (2018). https://doi.org/10.1016/j.patcog.2017.09.042. http://www.sciencedirect.com/science/article/pii/S0031320317303941

    Article  Google Scholar 

  2. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR 2009 (2009)

    Google Scholar 

  3. Engels, G., Heckel, R., Sauer, S.: UML—a universal modeling language? In: Nielsen, M., Simpson, D. (eds.) ICATPN 2000. LNCS, vol. 1825, pp. 24–38. Springer, Heidelberg (2000). https://doi.org/10.1007/3-540-44988-4_3

    Chapter  Google Scholar 

  4. Everingham, M., Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. Int. J. Comput. Vis. 88(2), 303–338 (2010). https://doi.org/10.1007/s11263-009-0275-4

    Article  Google Scholar 

  5. Girshick, R.B.: Fast R-CNN. CoRR abs/1504.08083 (2015). http://arxiv.org/abs/1504.08083

  6. Girshick, R.B., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. CoRR abs/1311.2524 (2013). http://arxiv.org/abs/1311.2524

  7. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015). http://arxiv.org/abs/1512.03385

  8. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems (2012)

    Google Scholar 

  9. Lecun, Y., Bengio, Y.: Convolutional networks for images, speech, and time-series. In: The Handbook of Brain Theory and Neural Networks, January 1995

    Google Scholar 

  10. Li, Y., Chen, C.F., Allen, P.K.: Recognition of deformable object category and pose. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) (2014)

    Google Scholar 

  11. Liu, Z., Luo, P., Qiu, S., Wang, X., Tang, X.: Deepfashion: powering robust clothes recognition and retrieval with rich annotations. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016

    Google Scholar 

  12. Maitin-Shepard, J., Cusumano-Towner, M., Lei, J., Abbeel, P.: Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding. In: 2010 IEEE International Conference on Robotics and Automation, pp. 2308–2315, May 2010. https://doi.org/10.1109/ROBOT.2010.5509439

  13. Mariolis, I., Peleka, G., Kargakos, A., Malassiotis, S.: Pose and category recognition of highly deformable objects using deep learning. In: 2015 International Conference on Advanced Robotics (ICAR), pp. 655–662. IEEE, July 2015. https://doi.org/10.1109/ICAR.2015.7251526. http://ieeexplore.ieee.org/document/7251526/

  14. Redmon, J., Divvala, S.K., Girshick, R.B., Farhadi, A.: You only look once: Unified, real-time object detection. CoRR abs/1506.02640 (2015). http://arxiv.org/abs/1506.02640

  15. Redmon, J., Farhadi, A.: YOLO9000: better, faster, stronger. CoRR abs/1612.08242 (2016). http://arxiv.org/abs/1612.08242

  16. Ren, S., He, K., Girshick, R.B., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. CoRR abs/1506.01497 (2015). http://arxiv.org/abs/1506.01497

  17. Seo, Y., Shik Shin, K.: Hierarchical convolutional neural networks for fashion image classification. Expert Syst. Appl. 116, 328–339 (2019). https://doi.org/10.1016/j.eswa.2018.09.022. http://www.sciencedirect.com/science/article/pii/S0957417418305992

    Article  Google Scholar 

  18. Wagner, L., K.D., Smutný, V.: CTU color and depth image dataset of spread garments. Technical Report CTU-CMP-2013-25, Center for Machine Perception, K13133 FEE Czech Technical University, Prague, Czech Republic, September 2013

    Google Scholar 

  19. Yamazaki, K.: Instance recognition of clumped clothing using image features focusing on clothing fabrics and wrinkles. In: 2015 IEEE International Conference on Robotics and Biomimetics, IEEE-ROBIO 2015, pp. 1102–1108 (2016). https://doi.org/10.1109/ROBIO.2015.7418919, http://dx.doi.org/10.1007/s10514-016-9559-z

  20. Yang, M., Yu, K.: Real-time clothing recognition in surveillance videos. In: Macq, B., Schelkens, P. (eds.) ICIP, pp. 2937–2940. IEEE (2011). http://dblp.uni-trier.de/db/conf/icip/icip2011.html#YangY11

Download references

Acknowledgment

This work was supported by the EPSRC project “Robotics and Artificial Intelligence for Nuclear (RAIN)” (EP/R026084/1).

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Liverpool, Liverpool, UK

    Daniel Fernandes Gomes & Shan Luo

  2. Faculdade de Engenharia, Universidade do Porto, Porto, Portugal

    Luis F. Teixeira

  3. INESC TEC, Porto, Portugal

    Luis F. Teixeira

Authors
  1. Daniel Fernandes Gomes
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shan Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Luis F. Teixeira
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniel Fernandes Gomes .

Editor information

Editors and Affiliations

  1. Queen Mary University of London, London, UK

    Kaspar Althoefer

  2. Queen Mary University of London, London, UK

    Jelizaveta Konstantinova

  3. Queen Mary University of London, London, UK

    Ketao Zhang

A Appendix

A Appendix

See Table 2.

Table 2. Summary of landmark Classification+Localization, as follows: GarmNet, the results obtained using the base model; GarmNet (S.C), the base model optimized with the spacial constraint (Sect. 4.4); GarmNet-B, the modified model with the bridge connection (4.5); and, GarmNet-B (A.D.), the latter model optimized using augmented dataset.
Full size table

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fernandes Gomes, D., Luo, S., Teixeira, L.F. (2019). GarmNet: Improving Global with Local Perception for Robotic Laundry Folding. In: Althoefer, K., Konstantinova, J., Zhang, K. (eds) Towards Autonomous Robotic Systems. TAROS 2019. Lecture Notes in Computer Science(), vol 11650. Springer, Cham. https://doi.org/10.1007/978-3-030-25332-5_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-25332-5_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-25331-8

  • Online ISBN: 978-3-030-25332-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation