A Paper-Based Keyboard Using ArUco Codes: ArUco Keyboard

Abstract

Object tracking in computer vision can be done either by using a marker-less or marker-based approach. Computer vision systems have been using Fiducial markers for pose estimation in different applications such as augmented reality [5] and robot navigation [4]. With the advancements in Augmented Reality (AR), new tools such as AugmentedReality uco (ArUco) [6] markers have been introduced to the literature. ArUco markers, are used to tackle the localization problem in AR, allowing camera pose estimation to be carried out by a binary matrix. Using a binary matrix not just simplifies the process but also increases the efficiency. As a part of our initiative to create a cost-efficient, 24/7 accessible, Virtual Reality (VR) based chemistry lab for underprivileged students, we wanted to create an alternative way of interacting with the virtual scene. In this study, we used ArUco markers to create a low-cost keyboard only using a piece of paper and an off-the-shelf webcam. We believe this method of keyboard will be more beneficial to the user as they can see the keys before they are typing in the corner of the screen instead of an insufficient on the screen VR keyboard or a regular keyboard where the user can’t see what they are typing with a VR headset. As potential extensions of the base system, we have also designed and evaluated a stereo camera and an IMU sensor based system with various sensor fusion techniques. In summary, the stereo camera reduces occlusion related problems, and the IMU sensor detects vibrations which in turn simplifies the KeyPress detection problem. It has been observed that use of any of these additional sensors improves the overall system performance.

Appendices

Appendix I: ArUco Code Detection Module aruco_tools.py

Appendix II: Base System minikdb_mono.py

Appendix III: IMU Based System minikbd_imu.py

Appendix IV: Stereo Camera Based System minikbd_zed.py

Appendix V: IMU Sensor Code for Arduino Uno