Multimedia Interfaces for People Visually Impaired

Abstract

In our society, there is a substantial number of visually impaired individuals. However many social mechanisms are not designed with these people in mind thus making the development of electronic assistive tools essential in order to perform basic day-to-day activities. Due to the penetration of capabilities of mobile devices, such devices have become an ideal candidate for designing solutions to aid the visually impaired. The objective of this research is to develop a multimedia user interface whose scope is to aid the visually challenged. We propose and design a product recognition system utilizing computer vision and machine learning techniques. Our system allows visually impaired individuals to identify products in grocery stores and supermarkets without any additional assistance, thus encouraging them to perform daily activities without requiring any additional help thus further promoting their independence within society. Our approach is composed of two main modules one capable of classifying grocery products using an unsupervised feature extraction methods posed by deep learning techniques while the other module is capable of recognizing products in an image using the traditionally handcrafted feature extraction algorithms. We considered multiple robust approaches to identify the one most suited for our task. Through evaluation we determined that the best approach for classification is to fine-tune a convolutional neural network pre-trained on a larger dataset. We were successful in not only surpassing our base accuracy but also obtaining an accuracy of 63 %.