Abstract
We present the See-Through-your-Skin Display (STS-d), a device that integrates visual and tactile sensing with a surface display to provide an interactive user experience. The STS-d expands the application of visuo-tactile optical sensors to Human-Robot Interaction (HRI) tasks and Human-Computer Interaction (HCI) tasks more generally. A key finding of this paper is that it is possible to display graphics on the reflective membrane of semi-transparent optical tactile sensors without interfering with their sensing capabilities, thus permitting simultaneous sensing and visual display. A proof of concept demonstration of the technology is presented where the STS Visual Display (STS-d) is used to provide an animated countertop that responds to visual and tactile events. We show that the integrated sensor can monitor interactions with the countertop, such as predicting the timing and location of contact with an object, or the amount of liquid in a container being placed on it, while displaying visual cues to the user.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Hogan, F.R., Tremblay, J.F., Baghi, B.H., Jenkin, M., Siddiqi, K., Dudek, G.: Finger-STS: combined proximity and tactile sensing for robotic manipulation. IEEE Robot. Autom. Lett. 7, 10865–10872 (2022)
Yuan, W., Dong, S., Adelson, E.H.: GelSight: high-resolution robot tactile sensors for estimating geometry and force. Sensors 17(12), 2762 (2017)
Donlon, E., Dong, S., Liu, M., Li, J., Adelson, E., Rodriguez, A.: GelSlim: a high-resolution, compact, robust, and calibrated tactile-sensing finger. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1927–1934 Madrid, Spain (2018)
Alspach, A., Hashimoto, K., Kuppuswarny, N., Tedrake, R.: Soft-bubble: a highly compliant dense geometry tactile sensor for robot manipulation. In: IEEE International Conference on Soft Robotics (RoboSoft), pp. 597–604 Seoul, Korea (2019)
Padmanabha, A., Ebert, F., Tian, S., Calandra, R., Finn, C., Levine, S.: OmniTact: a multi-directional high-resolution touch sensor. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 618–624 Held Online (2020)
Gomes, D.F., Lin, Z., Luo, S.: GelTip: a finger-shaped optical tac- tile sensor for robotic manipulation. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 9903–9909. Las Vegas, NV, (2020)
Hogan, F.R., Jenkin, M., Rezaei-Shoshtari, S. Girdhar, Y., Meger, D., Dudek, G.: Seeing through your skin: recognizing objects with a novel visuotactile sensor. In: WACV, Held Online (2021)
Zhang, Y., Chen, X., Wang, M.Y., Yu, H.: Multidimensional tactile sensor with a thin compound eye-inspired imaging system. Soft Robot. 9(5), 861–870 (2022)
Cao, G., Jiang, J., Lu, C., Gomes, D.F., Luo, S.: TouchRoller: a rolling optical tactile sensor for rapid assessment of textures for large surface areas. Sensors 23, 2661 (2023)
Abad, A., Ranasinghe, A.: Visuotactile sensors with emphasis on GelSight sensor: a review. IEEE Sens. 20, 7628–7638 (2020)
Hogan, F., Jenkin, M., Dudek, G.L., Girdhar, Y., Rezaei-Shoshtari, S., Meger, D.: Semitransparent tactile surface sensor and a method of sensing an interaction with an object using the semitransparent tactile surface sensor (2023). US Patent 11656759
Denk, W.E.: Electronic pointer for television images (1946). US Patent 2,487,641
Brandl, P., et al.: An adapatable rear-projection screen using digital pens and hand gestures. In: 17th International Conference on Artificial Reality and Telexistence (ICAT), Esbjerg, Denmark (2007)
Jenkin, M., Tsotsos, J., Andreopoulos, A., Rotenstein, A., Robinson, M., Laurence, J.: A large-scale touch sensitive display. In: 6th International Conference on Informatics and Systems, Cairo, Egypt (2008)
Bachmann, D., Weichert, F., Rinkenauer, G.: Evaluation of the leap motion controller as a pointing device. Sensors (Basel) 15, 214–233 (2015)
Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 779–788. Las Vegas, NV (2016)
Parr, J.: Modern kitchen, good home, strong nation. Technol. Cult. 43, 657–667 (2002)
Surmann, A.: The evolution of kitchen design: a yearning for a modern stone age cave. In: van der Meulen, N., Wiesel, J. (eds.) Culinary Turn: Aesthetic Practice of Cookery, pp. 47–56. Transcript Verlag (2017)
Samsung connected hub for smart appliances (2022). https://www.samsung.com/us/explore/family-hub-refrigerator/connected-hub/. Accessed Feb 23 2022
The rise of kitchen technology and what to expect in future homes (2021). https://www.hippo.com/blog/future-of-kitchen-technology. Accessed 23 Feb 2022
Smriti, R.: A quick glimpse into the future of kitchen technology (2021). https://techversions.com/blogs/a-quick-glimpse-into-the-future-of-kitchen-technology/. Accessed 23 Feb 2022
Prospero, M.: Bosch’s projector turns your countertop into a touchscreen (2019). https://www.tomsguide.com/us/bosch-pai-smart-kitchen-projector,news-29077.html. Accessed 23 Feb 2022
Gomez-Rodriguez, M., Kober, J., Schölkopf, B.: Denoising photographs using dark frames optimized by quadratic programming. In: 2009 IEEE International Conference on Computational Photography (ICCP), pp. 1–9. IEEE (2009)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Jenkin, M., Hogan, F.R., Siddiqi, K., Tremblay, JF., Baghi, B., Dudek, G. (2024). Interacting with a Visuotactile Countertop. In: Filipe, J., Röning, J. (eds) Robotics, Computer Vision and Intelligent Systems. ROBOVIS 2024. Communications in Computer and Information Science, vol 2077. Springer, Cham. https://doi.org/10.1007/978-3-031-59057-3_23
Download citation
DOI: https://doi.org/10.1007/978-3-031-59057-3_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-59056-6
Online ISBN: 978-3-031-59057-3
eBook Packages: Computer ScienceComputer Science (R0)