Story Telling: Learning to Visualize Sentences Through Generated Scenes

Abstract

Visual learning has been helping people to store information for a longer period of time. It helps us to process information primarily through visuals and improves learning. Natural language text descriptions are given as an input, an ML model is built for image classification using Convolutional Neural Network (CNN), and the text is tokenized and given to the POS tagger. Further, the tagged objects are displayed in the form of a visual 2D scene in the Blender application. Thus, we propose a text-to-2D scene generation system which incorporates user interaction for preprocessing the output of the generated scene for the storyDB dataset. The dataset consists of abstract images required for the identification of children with autism. Our approach is an attempt to improve the memorization skill to remember and visualize the object next time when the word is heard by the child.