Insights from Exploration of Engaging Technologies to Teach Reading and Writing: Story Baker

Abstract

To engage children in learning to write, we spent several years exploring tools designed to engage children in creating and viewing stories. Our central focus was the automatic generation of animations. Tools included a digital stylus for writing and sketching, and in some cases simple robots and tangible, digitally-recognized objects. In pilot studies, children found the prototypes engaging. In 2007, a decision not to develop new hardware was made, but at today’s greatly reduced tablet cost and with more capable touch and pen technology, these experiments could inspire further research and development.

Appendix

1.1 A.1 Prototype Tablet

In 2007, we made a prototype device with a 7” 800 × 480 pixel display (Fig. 12.17a) and 4 GB solid state memory, running Windows CE on an AMD Geode 500 MHz processor. It had a FinePoint digitizer with an active stylus tethered by a retractable power cord. The device was designed to survive 3 foot drops. Its overall size was 11 × 6. 25 × 1. 4 inches, weighing 0.8 kg. By design the device looked like a painter’s palette to emphasize the stylus and creativity experience. The device also had a handle to make it easy to carry. The stylus was tethered to avoid losing it and to provide power without requiring a battery (Fig. 12.17a). A polished version of Story Baker was implemented on the device (Fig. 12.17b).

Fig. 12.17

The prototype hardware (2007). (a) The prototype device. (b) Story Baker running on the device

1.2 A.2 Sentence Structure

Figure 12.18 shows the sentence structure with some of the objects and actions. Sentences have a primary actor, action, a secondary actor, and optional background. We created an API to convert and animate a sentence (see Visual Sentence API documentation [12]).

Fig. 12.18

Animation system overview

The top row of Fig. 12.19 illustrates a generic eating animation. Beneath it is an instance of a dragon eating a banana: From left to right, the dragon faces the banana, its mouth opens, the banana enters its mouth, and the dragon chews. Chewing uses rotation and scaling, and can dynamically recolor the character; for example, the head and/or body could turn green to depict disgust or overeating. After chewing, the primary actor ejects the food to avoid trauma if the secondary actor is a favorite character. The animation is accompanied with audio to make it convincing.

Fig. 12.19

Animating a dragon eating a banana in the forest. Top: Generic eating animation, Bottom: Animating a dragon eating a banana in the forest

Figure 12.20 illustrates how our system transforms a sentence into an animation. The digital DNA, image, and audio of the primary and secondary actors are drawn from the KidWords dictionary. Behavior instructions are fetched for each actor and the resulting trajectories and sounds are modified to accord with their meta-data. A synchronization barrier ensures that the actors synchronize. For example, an actor that is kicked should not begin flying off before the kick is delivered. Since the primary actor will likely have different digital DNA than the secondary actor, they could behave quite differently (e.g., a “silly” primary actor might stumble around before kicking). Actions are defined as a sequence of behavior instructions, each of which indicates what the actor is to do, but not how: That is defined by the digital DNA. For instance, a digital instruction to move from point A to point B could be executed by striding, stumbling, or hopping. For more details, see this list of macro actions [7].

Fig. 12.20

Model for animating with digital DNA variations

Figures 12.21 and 12.22 show generic animations for “kick” and “pick up” actions, respectively, without variants due to digital DNA.

Fig. 12.21

Example of kicking

Fig. 12.22

Example of picking up

1.3 A.3 Fontlings Data Structure

In the data structure for Fontlings (Fig. 12.23), each letter (or number/special character) has a split line and a rotation point for each part (top and bottom part). Each letter has position, angle (rotation), and scaling parameter (not shown) for each part, and each letter animates independently (Top split position, Top split angle, Bottom split position, Bottom split angle, etc. in Fig. 12.23). An instruction index (Instruction Index in Fig. 12.23) tracks its position in the execution of the script. One letter or word could be in idle mode while another is happy. In this way, a specific word can highlight something while other words are alive but less dramatic. For each letter, scripts define behaviors (idle, happy, misunderstood, etc.). The Splits/Rotation Points Data Structure refers to a hash table with an entry key for each font type (Arial, Times, etc. in Fig. 12.23). It specifies rotation points and the height of the split line as a percent of the character’s height. The Behavior Scripts Data Structure is a hash table with the behavior name as entry key (Happy, Misunderstood, Idle, etc. in Fig. 12.23). It contains a script for that behavior for each character. Figure 12.23 also highlights a specific example for the font type Arial (in orange color) and the behavior Idle (in green color).

Fig. 12.23

Fontlings data structure

The data structures are initialized by the xml file Fontlings Split Init File, which contains the height of the split line for each character of a given font, and Fontlings Behavior Init File, which contains the scripts to animate each character for each behavior (see bottom of Fig. 12.23).