Skip to main content

PressureVision: Estimating Hand Pressure from a Single RGB Image

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Abstract

People often interact with their surroundings by applying pressure with their hands. While hand pressure can be measured by placing pressure sensors between the hand and the environment, doing so can alter contact mechanics, interfere with human tactile perception, require costly sensors, and scale poorly to large environments. We explore the possibility of using a conventional RGB camera to infer hand pressure, enabling machine perception of hand pressure from uninstrumented hands and surfaces. The central insight is that the application of pressure by a hand results in informative appearance changes. Hands share biomechanical properties that result in similar observable phenomena, such as soft-tissue deformation, blood distribution, hand pose, and cast shadows. We collected videos of 36 participants with diverse skin tone applying pressure to an instrumented planar surface. We then trained a deep model (PressureVisionNet) to infer a pressure image from a single RGB image. Our model infers pressure for participants outside of the training data and outperforms baselines. We also show that the output of our model depends on the appearance of the hand and cast shadows near contact regions. Overall, our results suggest the appearance of a previously unobserved human hand can be used to accurately infer applied pressure.

Data, code, and models are available online (https://github.com/facebookresearch/pressurevision).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Asadi, H., Zhou, G., Lee, J.J., Aggarwal, V., Yu, D.: A computer vision approach for classifying isometric grip force exertion levels. Ergonomics 63(8), 1010–1026 (2020)

    Article  Google Scholar 

  2. Bhirangi, R., Hellebrekers, T., Majidi, C., Gupta, A.: ReSkin: versatile, replaceable, lasting tactile skins. In: Conference on Robot Learning (CoRL) (2021)

    Google Scholar 

  3. Brahmbhatt, S., Ham, C., Kemp, C.C., Hays, J.: ContactDB: analyzing and predicting grasp contact via thermal imaging. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  4. Brahmbhatt, S., Handa, A., Hays, J., Fox, D.: ContactGrasp: functional multi-finger grasp synthesis from contact. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2386–2393. IEEE (2019)

    Google Scholar 

  5. Brahmbhatt, S., Tang, C., Twigg, C.D., Kemp, C.C., Hays, J.: ContactPose: a dataset of grasps with object contact and hand pose. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12358, pp. 361–378. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58601-0_22

    Chapter  Google Scholar 

  6. Brahmbhatt, S.M.: Grasp contact between hand and object: capture, analysis, and applications. Ph.D. thesis, Georgia Institute of Technology (2020)

    Google Scholar 

  7. Büscher, G.H., Kõiva, R., Schürmann, C., Haschke, R., Ritter, H.J.: Flexible and stretchable fabric-based tactile sensor. Robot. Auton. Syst. 63, 244–252 (2015)

    Article  Google Scholar 

  8. Cavazos, J.G., Phillips, P.J., Castillo, C.D., O’Toole, A.J.: Accuracy comparison across face recognition algorithms: where are we on measuring race bias? IEEE Trans. Biometrics Behav. Identity Sci. 3(1), 101–111 (2020)

    Article  Google Scholar 

  9. Chen, N., Westling, G., Edin, B.B., van der Smagt, P.: Estimating fingertip forces, torques, and local curvatures from fingernail images. Robotica 38(7), 1242–1262 (2020)

    Article  Google Scholar 

  10. Chu, F.J., Xu, R., Vela, P.A.: Real-world multiobject, multigrasp detection. IEEE Robot. Autom. Lett. 3(4), 3355–3362 (2018)

    Article  Google Scholar 

  11. Clever, H.M., Erickson, Z., Kapusta, A., Turk, G., Liu, C.K., Kemp, C.C.: Bodies at rest: 3D human pose and shape estimation from a pressure image using synthetic data. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2020)

    Google Scholar 

  12. Clever, H.M., Grady, P., Turk, G., Kemp, C.C.: BodyPressure - inferring body pose and contact pressure from a depth image. IEEE Trans. Pattern Anal. Mach. Intell. (2021). https://ieeexplore.ieee.org/document/9743547

  13. Cook, C.M., Howard, J.J., Sirotin, Y.B., Tipton, J.L., Vemury, A.R.: Demographic effects in facial recognition and their dependence on image acquisition: an evaluation of eleven commercial systems. IEEE Trans. Biometrics Behav. Identity Sci. 1(1), 32–41 (2019)

    Article  Google Scholar 

  14. Davis, A., Rubinstein, M., Wadhwa, N., Mysore, G.J., Durand, F., Freeman, W.T.: The visual microphone: passive recovery of sound from video. ACM Trans. Graph. 33(4), 1–10 (2014)

    Article  Google Scholar 

  15. Dehaene, S.: The neural basis of the weber-fechner law: a logarithmic mental number line. Trends Cogn. Sci. 7(4), 145–147 (2003)

    Article  MathSciNet  Google Scholar 

  16. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248–255. IEEE (2009)

    Google Scholar 

  17. Ehsani, K., Tulsiani, S., Gupta, S., Farhadi, A., Gupta, A.: Use the force, Luke! Learning to predict physical forces by simulating effects. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 224–233 (2020)

    Google Scholar 

  18. Fitzpatrick, P.M., Torres-Jara, E.R.: The power of the dark side: using cast shadows for visually-guided touching. In: 2004 4th IEEE/RAS International Conference on Humanoid Robots, vol. 1, pp. 437–449. IEEE (2004)

    Google Scholar 

  19. Garcia-Hernando, G., Yuan, S., Baek, S., Kim, T.K.: First-person hand action benchmark with RGB-D videos and 3D hand pose annotations. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 409–419 (2018)

    Google Scholar 

  20. Grady, P., Tang, C., Twigg, C.D., Vo, M., Brahmbhatt, S., Kemp, C.C.: ContactOpt: Optimizing contact to improve grasps. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1471–1481 (2021)

    Google Scholar 

  21. Hampali, S., Rad, M., Oberweger, M., Lepetit, V.: HOnnotate: a method for 3D annotation of hand and object poses. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3196–3206 (2020)

    Google Scholar 

  22. Hasson, Y., Tekin, B., Bogo, F., Laptev, I., Pollefeys, M., Schmid, C.: Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 571–580 (2020)

    Google Scholar 

  23. Hasson, Y., et al.: Learning joint reconstruction of hands and manipulated objects. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11807–11816 (2019)

    Google Scholar 

  24. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)

    Google Scholar 

  25. Holden, D., Kanoun, O., Perepichka, M., Popa, T.: Learned motion matching. ACM Trans. Graph. 39(4), 1–12 (2020)

    Article  Google Scholar 

  26. Hu, H.H., Gooch, A.A., Creem-Regehr, S.H., Thompson, W.B.: Visual cues for perceiving distances from objects to surfaces. Presence: Teleoper. Virtual Environ. 11(6), 652–664 (2002)

    Article  Google Scholar 

  27. Hu, H.H., Gooch, A.A., Thompson, W.B., Smits, B.E., Rieser, J.J., Shirley, P.: Visual cues for imminent object contact in realistic virtual environments. In: 2000 Proceedings Visualization, pp. 179–185. IEEE (2000)

    Google Scholar 

  28. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7132–7141 (2018)

    Google Scholar 

  29. Hubona, G.S., Wheeler, P.N., Shirah, G.W., Brandt, M.: The relative contributions of stereo, lighting, and background scenes in promoting 3D depth visualization. ACM Trans. Comput. Human Interact. 6(3), 214–242 (1999)

    Article  Google Scholar 

  30. Hwang, W., Lim, S.: Inferring interaction force from visual information without using physical force sensors. Sensors 17(11), 2455 (2017)

    Article  Google Scholar 

  31. Johnson, M.K., Adelson, E.H.: Retrographic sensing for the measurement of surface texture and shape. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1070–1077. IEEE (2009)

    Google Scholar 

  32. Karunratanakul, K., Yang, J., Zhang, Y., Black, M.J., Muandet, K., Tang, S.: Grasping field: learning implicit representations for human grasps. In: 8th International Conference on 3D Vision, pp. 333–344. IEEE (2020)

    Google Scholar 

  33. Krishnapriya, K., Albiero, V., Vangara, K., King, M.C., Bowyer, K.W.: Issues related to face recognition accuracy varying based on race and skin tone. IEEE Transa. Technol. Soc. 1(1), 8–20 (2020)

    Article  Google Scholar 

  34. Li, Z., Sedlár, J., Carpentier, J., Laptev, I., Mansard, N., Sivic, J.: Estimating 3D motion and forces of person-object interactions from monocular video. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8640–8649 (2019)

    Google Scholar 

  35. Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2117–2125 (2017)

    Google Scholar 

  36. Mascaro, S.A., Asada, H.H.: Photoplethysmograph fingernail sensors for measuring finger forces without haptic obstruction. IEEE Trans. Robot. Autom. 17(5), 698–708 (2001)

    Article  Google Scholar 

  37. Mascaro, S.A., Asada, H.H.: Measurement of finger posture and three-axis fingertip touch force using fingernail sensors. IEEE Trans. Robot. Autom. 20(1), 26–35 (2004)

    Article  Google Scholar 

  38. Meier, M., Streli, P., Fender, A., Holz, C.: TapID: rapid touch interaction in virtual reality using wearable sensing. In: 2021 IEEE Virtual Reality and 3D User Interfaces (VR), pp. 519–528. IEEE (2021)

    Google Scholar 

  39. Morph: sensel morph haptic sensing tablet, www.sensel.com/pages/the-sensel-morph. Accessed 25 Feb 2020

  40. Narasimhaswamy, S., Nguyen, T., Nguyen, M.H.: Detecting hands and recognizing physical contact in the wild. In: Advances in Neural Information Processing Systems, vol. 33 (2020)

    Google Scholar 

  41. Niikura, T., Watanabe, Y., Ishikawa, M.: Anywhere surface touch: utilizing any surface as an input area. In: Proceedings of the 5th Augmented Human International Conference, pp. 1–8 (2014)

    Google Scholar 

  42. Pantone LLC: pantone skintone guide (2012)

    Google Scholar 

  43. Pérez-González, A., Vergara, M., Sancho-Bru, J.L.: Stiffness map of the grasping contact areas of the human hand. J. Biomech. 46(15), 2644–2650 (2013)

    Article  Google Scholar 

  44. Pham, T.H., Kheddar, A., Qammaz, A., Argyros, A.A.: Towards force sensing from vision: observing hand-object interactions to infer manipulation forces. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2810–2819 (2015)

    Google Scholar 

  45. Pham, T.H., Kyriazis, N., Argyros, A.A., Kheddar, A.: Hand-object contact force estimation from markerless visual tracking. IEEE Trans. Pattern Anal. Mach. Intell. 40(12), 2883–2896 (2017)

    Article  Google Scholar 

  46. Pressure Profile Systems: PPS TactileGlove. www.pressureprofile.com/body-pressure-mapping/tactile-glove

  47. Rogez, G., Supancic, J.S., Ramanan, D.: Understanding everyday hands in action from RGB-D images. In: IEEE International Conference on Computer Vision (ICCV), pp. 3889–3897 (2015)

    Google Scholar 

  48. Romero, J., Kjellström, H., Kragic, D.: Hands in action: real-time 3D reconstruction of hands in interaction with objects. In: 2010 IEEE International Conference on Robotics and Automation, pp. 458–463. IEEE (2010)

    Google Scholar 

  49. Rong, Y., Shiratori, T., Joo, H.: FrankmoCap: a monocular 3D whole-body pose estimation system via regression and integration. In: IEEE International Conference on Computer Vision Workshops (2021)

    Google Scholar 

  50. Rosales, C., Porta, J.M., Ros, L.: Global optimization of robotic grasps. In: Proceedings of Robotics: Science and Systems VII (2011)

    Google Scholar 

  51. Saxena, A., Driemeyer, J., Ng, A.Y.: Robotic grasping of novel objects using vision. The Int. J. Robot. Res. 27(2), 157–173 (2008)

    Article  Google Scholar 

  52. Scott, J., Ravichandran, B., Funk, C., Collins, R.T., Liu, Y.: From image to stability: learning dynamics from human pose. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12368, pp. 536–554. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58592-1_32

    Chapter  Google Scholar 

  53. Starke, S., Zhao, Y., Komura, T., Zaman, K.: Local motion phases for learning multi-contact character movements. ACM Trans. Graph. 39(4), 1–13 (2020). https://doi.org/10.1145/3386569.3392450

    Article  Google Scholar 

  54. Sundaram, S., Kellnhofer, P., Li, Y., Zhu, J.Y., Torralba, A., Matusik, W.: Learning the signatures of the human grasp using a scalable tactile glove. Nature 569(7758), 698–702 (2019)

    Article  Google Scholar 

  55. Taheri, O., Ghorbani, N., Black, M.J., Tzionas, D.: GRAB: a dataset of whole-body human grasping of objects. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12349, pp. 581–600. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58548-8_34

    Chapter  Google Scholar 

  56. TekScan: TekScan Grip System. www.tekscan.com

  57. Tzionas, D., Ballan, L., Srikantha, A., Aponte, P., Pollefeys, M., Gall, J.: Capturing hands in action using discriminative salient points and physics simulation. Int. J. Comput. Vis. 118(2), 172–193 (2016)

    Article  MathSciNet  Google Scholar 

  58. Ward-Cherrier, B., et al.: The TacTip family: soft optical tactile sensors with 3D-printed biomimetic morphologies. Soft Rob. 5(2), 216–227 (2018)

    Article  Google Scholar 

  59. Wettels, N., Santos, V.J., Johansson, R.S., Loeb, G.E.: Biomimetic tactile sensor array. Adv. Robot. 22(8), 829–849 (2008)

    Article  Google Scholar 

  60. Wu, H., Rubinstein, M., Shih, E., Guttag, J.V., Durand, F., Freeman, W.T.: Eulerian video magnification for revealing subtle changes in the world. ACM Trans. Graph. 31(4), 1–8 (2012)

    Article  Google Scholar 

  61. Xiao, R., Schwarz, J., Throm, N., Wilson, A.D., Benko, H.: MRTouch: adding touch input to head-mounted mixed reality. IEEE Trans. Visual Comput. Graphics 24(4), 1653–1660 (2018)

    Article  Google Scholar 

  62. Xie, S., Girshick, R.B., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5987–5995 (2017)

    Google Scholar 

  63. Yakubovskiy, P.: Segmentation models pytorch (2020)

    Google Scholar 

  64. Ye, Y., Liu, C.K.: Synthesis of detailed hand manipulations using contact sampling. ACM Trans. Graph. (TOG) 31(4), 41 (2012)

    Article  Google Scholar 

  65. Yuan, W., Dong, S., Adelson, E.H.: GelSight: high-resolution robot tactile sensors for estimating geometry and force. Sensors 17(12), 2762 (2017)

    Article  Google Scholar 

  66. Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_53

    Chapter  Google Scholar 

  67. Zhang, H., Ye, Y., Shiratori, T., Komura, T.: ManipNet: neural manipulation synthesis with a hand-object spatial representation. ACM Trans. Graph. (TOG) 40(4), 1–14 (2021)

    Google Scholar 

  68. Zhang, Y., Hassan, M., Neumann, H., Black, M.J., Tang, S.: Generating 3D people in scenes without people. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6193–6203 (2020)

    Google Scholar 

  69. Zhu, Y., Jiang, C., Zhao, Y., Terzopoulos, D., Zhu, S.C.: Inferring forces and learning human utilities from videos. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3823–3833 (2016)

    Google Scholar 

  70. Zonios, G., Bykowski, J., Kollias, N.: Skin melanin, hemoglobin, and light scattering properties can be quantitatively assessed in vivo using diffuse reflectance spectroscopy. J. Investig. Dermatol. 117(6), 1452–1457 (2001)

    Article  Google Scholar 

Download references

Acknowledgements

We thank Kevin Harris, Steve Miller, and Steve Olsen for their help in data collection, and Robert Wang, Minh Vo, Tomas Hodan, Amy Zhao, Kenrick Kin, Mark Richardson, and Cem Keskin for their advice.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Patrick Grady .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 7734 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Grady, P. et al. (2022). PressureVision: Estimating Hand Pressure from a Single RGB Image. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13666. Springer, Cham. https://doi.org/10.1007/978-3-031-20068-7_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20068-7_19

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20067-0

  • Online ISBN: 978-3-031-20068-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics