Skip to main content
SpringerLink
Log in

Deep Touch: Sensing Press Gestures from Touch Image Sequences

  • Chapter
  • First Online:
Artificial Intelligence for Human Computer Interaction: A Modern Approach

Part of the book series: Human–Computer Interaction Series ((HCIS))

Abstract

Capacitive touch sensors capture a sequence of images of a finger’s interaction with a surface that contain information about its contact shape, posture, and biomechanical structure. These images are typically reduced to two-dimensional points, with the remaining data discarded—restricting the expressivity that can be captured to discriminate a user’s touch intent. We develop a deep touch hypothesis that (1) the human finger performs richer expressions on a touch surface than simple pointing; (2) such expressions are manifested in touch sensor image sequences due to finger-surface biomechanics; and (3) modern neural networks are capable of discriminating touch gestures using these sequences. In particular, a press gesture based on an increase in a finger’s force can be sensed without additional hardware, and reliably discriminated from other common expressions. This work demonstrates that combining capacitive touch sensing with modern neural network algorithms is a practical direction to improve the usability and expressivity of touch-based user interfaces.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Sensing \(\Delta C\) is known as sensing the mutual capacitance between electrodes. In a related (but more limited) technique, the self capacitance C of each electrode is measured individually [7, 55].

  2. 2.

    If the electrodes are allowed the ‘float’ with respect to each other, then changes in the distance between them from external forces can be detected by Eq. 1.

  3. 3.

    All convolutional filters have a depth of 16, with ReLU activation between each operation [17].

  4. 4.

    https://developer.android.com/reference/android/view/GestureDetector.

References

  1. Albinsson PA, Zhai S (2003) High precision touch screen interaction. In: Proceedings of the SIGCHI conference on human factors in computing systems, association for computing machinery, New York, NY, CHI ’03, pp 105–112. https://doi.org/10.1145/642611.642631

  2. Aliakbarian MS, Saleh FS, Salzmann M, Fernando B, Petersson L, Andersson L (2017) Encouraging LSTMs to anticipate actions very early. In: 2017 IEEE international conference on computer vision (ICCV), pp 280–289. https://doi.org/10.1109/ICCV.2017.39

  3. Antoine A, Malacria S, Casiez G (2017) Forceedge: Controlling autoscroll on both desktop and mobile computers using the force. In: Proceedings of the 2017 CHI conference on human factors in computing systems, ACM, New York, NY, CHI ’17, pp 3281–3292. https://doi.org/10.1145/3025453.3025605

  4. Arif AS, Stuerzlinger W (2013) Pseudo-pressure detection and its use in predictive text entry on touchscreens. In: Proceedings of the 25th australian computer-human interaction conference: augmentation, application, innovation, collaboration, ACM, New York, NY, OzCHI ’13, pp 383–392. https://doi.org/10.1145/2541016.2541024

  5. Azenkot S, Zhai S (2012) Touch behavior with different postures on soft smartphone keyboards. In: Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services, ACM, New York, NY, MobileHCI ’12, pp 251–260. https://doi.org/10.1145/2371574.2371612

  6. Baglioni M, Malacria S, Lecolinet E, Guiard Y (2011) Flick-and-Brake: Finger control over inertial/sustained scroll motion. In: CHI ’11 Extended abstracts on human factors in computing systems, ACM, New York, NY, CHI EA ’11, pp 2281–2286. https://doi.org/10.1145/1979742.1979853

  7. Barrett G, Omote R (2010) Projected-capacitive touch technology. Inf Display 26(3):16–21

    Article  Google Scholar 

  8. Benko H, Wilson AD, Baudisch P (2006) Precise selection techniques for multi-touch screens. In: CHI ’06, ACM, New York, NY, pp 1263–1272. https://doi.org/10.1145/1124772.1124963

  9. Bi X, Li Y, Zhai S (2013) FFitts law: Modeling finger touch with Fitts’ law. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, NY, CHI ’13, pp 1363–1372. https://doi.org/10.1145/2470654.2466180

  10. Birznieks I, Jenmalm P, Goodwin AW, Johansson RS (2001) Encoding of direction of fingertip forces by human tactile afferents. J Neurosci 21(20):8222–8237. https://doi.org/10.1523/jneurosci.21-20-08222.2001

    Article  Google Scholar 

  11. Boceck T, Le HV, Sprott S, Mayer S (2019) Force touch detection on capacitive sensors using deep neural networks. In: Proceedings of the 21st international conference on human-computer interaction with mobile devices and services. https://doi.org/10.1145/3338286.3344389

  12. Boring S, Ledo D, Chen XA, Marquardt N, Tang A, Greenberg S (2012) The fat thumb: using the thumb’s contact size for single-handed mobile interaction. In: Proceedings of the 14th international conference on human-computer interaction with mobile devices and services, ACM, New York, NY, MobileHCI ’12, pp 39–48. https://doi.org/10.1145/2371574.2371582

  13. Brewster SA, Hughes M (2009) Pressure-based text entry for mobile devices. In: Proceedings of the 11th international conference on human-computer interaction with mobile devices and services, ACM, New York, NY, MobileHCI ’09, pp 9:1–9:4. https://doi.org/10.1145/1613858.1613870

  14. Buxton W (1995) Touch, gesture, and marking. In: Baecker RM, Grudin J, Buxton W, Greenberg S (eds) Human-computer interaction: toward the year 2000, Morgan Kaufmann Publishers, San Francisco, CA, chap 7, pp 469–482

    Google Scholar 

  15. Buxton W, Hill R, Rowley P (1985) Issues and techniques in touch-sensitive tablet input. In: Proceedings of the 12th annual conference on Computer graphics and interactive techniques, ACM, New York, NY, SIGGRAPH ’85, pp 215–224. https://doi.org/10.1145/325334.325239, http://doi.acm.org/10.1145/325334.325239

  16. Forlines C, Wigdor D, Shen C, Balakrishnan R (2007) Direct-touch vs. mouse input for tabletop displays. In: Proceedings of the SIGCHI conference on human factors in computing systems, association for computing machinery, New York, NY, CHI ’07, pp 647–656. https://doi.org/10.1145/1240624.1240726

  17. Glorot X, Bordes A, Bengio Y (2011) Deep sparse rectifier neural networks. In: Gordon G, Dunson D, Dudík M (eds) Proceedings of the fourteenth international conference on artificial intelligence and statistics, PMLR, Fort Lauderdale, FL, Proceedings of machine learning research, vol 15, pp 315–323

    Google Scholar 

  18. Goel M, Jansen A, Mandel T, Patel SN, Wobbrock JO (2013) ContextType: using hand posture information to improve mobile touch screen text entr. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, NY, CHI ’13, pp 2795–2798. https://doi.org/10.1145/2470654.2481386

  19. Goguey A, Casiez G, Vogel D, Gutwin C (2018) Characterizing finger pitch and roll orientation during atomic touch actions. In: Proceedings of the 2018 CHI conference on human factors in computing systems, ACM, New York, NY, CHI ’18, pp 589:1–589:12. https://doi.org/10.1145/3173574.3174163

  20. Goguey A, Malacria S, Gutwin C (2018) Improving discoverability and expert performance in force-sensitive text selection for touch devices with mode gauges. In: Proceedings of the 2018 CHI conference on human factors in computing systems, ACM, New York, NY, CHI ’18. https://doi.org/10.1145/3173574.3174051

  21. Graves A (2012) Supervised sequence labelling with recurrent neural networks. Springer, Berlin. https://doi.org/10.1007/978-3-642-24797-2

  22. Grosse-Puppendahl T, Holz C, Cohn G, Wimmer R, Bechtold O, Hodges S, Reynolds MS, Smith JR (2017) Finding common ground: A survey of capacitive sensing in human-computer interaction. In: Proceedings of the 2017 CHI conference on human factors in computing systems, ACM, New York, NY, CHI ’17, pp 3293–3315. https://doi.org/10.1145/3025453.3025808

  23. Harrison C, Hudson S (2012) Using shear as a supplemental two-dimensional input channel for rich touchscreen interaction. In: Proceedings of the sigchi conference on human factors in computing systems, ACM, New York, NY, CHI ’12, pp 3149–3152. https://doi.org/10.1145/2207676.2208730

  24. Heo S, Lee G (2011) Force gestures: augmenting touch screen gestures with normal and tangential forces. In: Proceedings of the 24th annual ACM symposium on User interface software and technology, ACM, New York, NY, UIST ’11, pp 621–626. https://doi.org/10.1145/2047196.2047278

  25. Heo S, Lee G (2011) ForceTap: extending the input vocabulary of mobile touch screens by adding tap gestures. In: Proceedings of the 13th international conference on human computer interaction with mobile devices and services, ACM, New York, NY, MobileHCI ’11, pp 113–122. https://doi.org/10.1145/2037373.2037393

  26. Heo S, Lee G (2013) Indirect shear force estimation for multi-point shear force operations. In: Proceedings of the SIGCHI Conference on human factors in computing systems, ACM, New York, NY, CHI ’13, pp 281–284. https://doi.org/10.1145/2470654.2470693

  27. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  28. Holz C, Baudisch P (2010) The generalized perceived input point model and how to double touch accuracy by extracting fingerprints. In: Proceedings of the 28th international conference on Human factors in computing systems, ACM, New York, NY, CHI ’10, pp 581–590. https://doi.org/10.1145/1753326.1753413

  29. Holz C, Baudisch P (2011) Understanding touch. In: Proceedings of the 2011 annual conference on Human factors in computing systems, ACM, New York, NY, CHI ’11, pp 2501–2510. https://doi.org/10.1145/1978942.1979308

  30. Hu Y, Huang L, Rieutort-Louis W, Sanz-Robinson J, Wagner S, Sturm JC, Verma N (2014) 3D gesture-sensing system for interactive displays based on extended-range capacitive sensing. In: 2014 IEEE international solid-state circuits conference digest of technical papers, ISSCC, pp 212–213. https://doi.org/10.1109/ISSCC.2014.6757404

  31. Kaaresoja T, Brewster S, Lantz V (2014) Towards the temporally perfect virtual button: Touch-feedback simultaneity and perceived quality in mobile touchscreen press interactions. ACM Trans Appl Percep 11(2):9:1–9:25, https://doi.org/10.1145/2611387

  32. Lee B, Lee H, Lim SC, Lee H, Han S, Park J (2012) Evaluation of human tangential force input performance. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, NY, CHI ’12, pp 3121–3130. https://doi.org/10.1145/2207676.2208727

  33. Lee J, Cole MT, Lai JCS, Nathan A (2014) An analysis of electrode patterns in capacitive touch screen panels. J Display Technol 10(5):362–366. https://doi.org/10.1109/JDT.2014.2303980

    Article  Google Scholar 

  34. Lee S, Buxton W, Smith KC (1985) A multi-touch three dimensional touch-sensitive tablet. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, NY, CHI ’85, pp 21–25. https://doi.org/10.1145/317456.317461

  35. Ma S, Sigal L, Sclaroff S (2016) Learning activity progression in LSTMs for activity detection and early detection. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 1942–1950. https://doi.org/10.1109/CVPR.2016.214

  36. Miyaki T, Rekimoto J (2009) GraspZoom: Zooming and scrolling control model for single-handed mobile interaction. In: Proceedings of the 11th international conference on human-computer interaction with mobile devices and services, ACM, New York, NY, MobileHCI ’09, pp 11:1–11:4. https://doi.org/10.1145/1613858.1613872

  37. Miyata N, Yamaguchi K, Maeda Y (2007) Measuring and modeling active maximum fingertip forces of a human index finger. In: 2007 IEEE/RSJ international conference on intelligent robots and systems, pp 2156–2161. https://doi.org/10.1109/IROS.2007.4399243

  38. O’Connor T (2010) mTouch projected capacitive touch screen sensing theory of operation. Technical Report, TB3064, Microchip Technology Inc

    Google Scholar 

  39. Pawluk DTV, Howe RD (1999) Dynamic contact of the human fingerpad against a flat surface. J Biomech Eng 121(6):605–611. https://doi.org/10.1115/1.2800860

    Article  Google Scholar 

  40. Potter RL, Weldon LJ, Shneiderman B (1988) Improving the accuracy of touch screens: an experimental evaluation of three strategies. In: CHI ’88, ACM, New York, NY, pp 27–32. https://doi.org/10.1145/57167.57171

  41. Quinn P, Feng W (2020) Sensing force-based gestures on the Pixel 4. Google AI Blog, https://ai.googleblog.com/2020/06/sensing-force-based-gestures-on-pixel-4.html

  42. Quinn P, Malacria S, Cockburn A (2013) Touch scrolling transfer functions. In: Proceedings of the 26th annual ACM symposium on user interface software and technology, ACM, New York, NY, UIST ’13, pp 61–70, https://doi.org/10.1145/2501988.2501995

  43. Rekimoto J, Schwesig C (2006) PreSenseII: Bi-directional touch and pressure sensing interactions with tactile feedback. In: CHI ’06 extended abstracts on human factors in computing systems, ACM, New York, NY, CHI EA ’06, pp 1253–1258. https://doi.org/10.1145/1125451.1125685

  44. Rendl C, Greindl P, Probst K, Behrens M, Haller M (2014) Presstures: exploring pressure-sensitive multi-touch gestures on trackpads. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, NY, CHI ’14, pp 431–434. https://doi.org/10.1145/2556288.2557146

  45. Rosenberg I, Perlin K (2009) The unmousepad: an interpolating multi-touch force-sensing input pad. ACM Trans Graph 28(3):65:1–65:9. https://doi.org/10.1145/1531326.1531371

  46. Roudaut A, Lecolinet E, Guiard Y (2009) Microrolls: expanding touch-screen input vocabulary by distinguishing rolls vs. slides of the thumb. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, NY, CHI ’09, pp 927–936. https://doi.org/10.1145/1518701.1518843

  47. Sakai N, Shimawaki S (2006) Mechanical responses and physical factors of the fingertip pulp. Appl Bionics Biomech 3(4):273–278. https://doi.org/10.1533/abbi.2006.0046

    Article  Google Scholar 

  48. Serina ER, Mote CD Jr, Rempel D (1997) Force response of the fingertip pulp to repeated compression–effects of loading rate, loading angle and anthropometry. J Biomech 30(10):1035–1040. https://doi.org/10.1016/S0021-9290(97)00065-1

    Article  Google Scholar 

  49. Serina ER, Mockensturm E, Mote CD Jr, Rempel D (1998) A structural model of the forced compression of the fingertip pulp. J Biomech 31(7):639–646. https://doi.org/10.1016/S0021-9290(98)00067-0

    Article  Google Scholar 

  50. Srinivasan MA, LaMotte RH (1995) Tactual discrimination of softness. J Neurophysiol 73(1):88–101. https://doi.org/10.1152/jn.1995.73.1.88

    Article  Google Scholar 

  51. Srinivasan MA, Gulati RJ, Dandekar K (1992) In vivo compressibility of the human fingertip. Adv Bioeng 22:573–576

    Google Scholar 

  52. Suzuki K, Sakamoto R, Sakamoto D, Ono T (2018) Pressure-sensitive zooming-out interfaces for one-handed mobile interaction. In: Proceedings of the 20th international conference on human-computer interaction with mobile devices and services, ACM, New York, NY, MobileHCI ’18, pp 30:1–30:8. https://doi.org/10.1145/3229434.3229446

  53. Taher F, Alexander J, Hardy J, Velloso E (2014) An empirical characterization of touch-gesture input-force on mobile devices. In: Proceedings of the ninth ACM international conference on interactive tabletops and surfaces, ACM, New York, NY, ITS ’14, pp 195–204. https://doi.org/10.1145/2669485.2669515

  54. Vogel D, Baudisch P (2007) Shift: a technique for operating pen-based interfaces using touchs. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, NY, CHI ’07, pp 657–666. https://doi.org/10.1145/1240624.1240727

  55. Walker G (2012) A review of technologies for sensing contact location on the surface of a display. J Soc Inf Display 20(8):413–440. https://doi.org/10.1002/jsid.100

    Article  Google Scholar 

  56. Wang F, Ren X (2009) Empirical evaluation for finger input properties in multi-touch interaction. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, NY, CHI ’09, pp 1063–1072. https://doi.org/10.1145/1518701.1518864

  57. Wang F, Cao X, Ren X, Irani P (2009) Detecting and leveraging finger orientation for interaction with direct-touch surfaces. In: Proceedings of the 22nd annual ACM symposium on user interface software and technology, ACM, New York, NY, UIST ’09, pp 23–32. https://doi.org/10.1145/1622176.1622182

  58. Wang T, Blankenship T (2011) Projected-capacitive touch systems from the controller point of view. Inf Display 27(3):8–11

    Article  Google Scholar 

  59. Westerman W (1999) Hand tracking, finger identification, and chordic manipulation on a multi-touch surface. PhD thesis, University of Delaware

    Google Scholar 

  60. Wilson G, Stewart C, Brewster SA (2010) Pressure-based menu selection for mobile devices. In: Proceedings of the 12th international conference on human computer interaction with mobile devices and services, ACM, New York, NY, MobileHCI ’10, pp 181–190. https://doi.org/10.1145/1851600.1851631

  61. Yaniger SI (1991) Force sensing resistors: a review of the technology. In: Electro international, pp 666–668. https://doi.org/10.1109/ELECTR.1991.718294

  62. Yin Y, Ouyang TY, Partridge K, Zhai S (2013) Making touchscreen keyboards adaptive to keys, hand postures, and individuals: a hierarchical spatial backoff model approach. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, NY, CHI ’13, pp 2775–2784. https://doi.org/10.1145/2470654.2481384

  63. Zimmerman TG, Smith JR, Paradiso JA, Allport D, Gershenfeld N (1995) Applying electric field sensing to human-computer interfaces. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM Press/Addison-Wesley Publishing Co., New York, NY, CHI ’95, pp 280–287. https://doi.org/10.1145/223904.223940

Download references

Acknowledgements

We thank many Google and Android colleagues in engineering, design, and product management for their direct and indirect contributions to the project.

Author information

Authors and Affiliations

  1. Google Research, Mountain View, CA, USA

    Philip Quinn, Wenxin Feng & Shumin Zhai

Authors
  1. Philip Quinn
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wenxin Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shumin Zhai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shumin Zhai .

Editor information

Editors and Affiliations

  1. Google Research (United States), Mountain View, CA, USA

    Yang Li

  2. Advanced Interactive Technologies Lab, ETH Zurich, Zurich, Switzerland

    Otmar Hilliges

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Quinn, P., Feng, W., Zhai, S. (2021). Deep Touch: Sensing Press Gestures from Touch Image Sequences. In: Li, Y., Hilliges, O. (eds) Artificial Intelligence for Human Computer Interaction: A Modern Approach. Human–Computer Interaction Series. Springer, Cham. https://doi.org/10.1007/978-3-030-82681-9_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-82681-9_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-82680-2

  • Online ISBN: 978-3-030-82681-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation