Bilingual Reading Experiences: What They Could Be and How to Design for Them
We introduce the idea of bilingual reading, where a document comes in two languages and the reader can choose at will on which language to focus during the reading. Between the complete ignorance of a language (where translation is the only option) and bilingualism (where translation is useless), there exists a variety of contexts of partial bilingualism where bilingual reading interfaces would prove highly useful. We first study through interviews and reviews how the bilingual reading experience is understood today. We provide an analysis framework and highlight design challenges for the design of bilingual reading appliances. We then describe a taxonomy of the different approaches available to address these challenges, analyze them in the light of our framework and show how they can be derived to sketch future bilingual reading interfaces.
KeywordsBilingual reading Mechanisms of reading Nexus of attention E-book E-reader Parallel text Text morphing Text animation
People are already bilingual (at various levels of proficiency) in most parts of the world, and have accepted e-books as viable alternatives to paper books . Publishing industry is familiar with translation processes and already distributes many resources in multiple languages: successful books are translated to reach an international market and old books and their translations have been massively digitalized. Natural language processing techniques are being developed to ease the creation of alignment structures at a massive scale  and individuals can already create such structures at reasonable costs.
With such an economical and societal importance and so little technical barriers, it is striking how little HCI works address bilingual reading. Reading has been studied many times in HCI, highlighting the variety of reading purposes and reading styles  and the importance of supporting activities [32, 37]. More general views have also been suggested about what digital reading could be . All of these works are relevant for bilingual reading, but none of them mention bilingual reading as an opportunity.
The central question of the research agenda on bilingual reading is: How could the reader benefit the most from both the original text and its translation? It is therefore essential to understand what it actually means for readers to benefit from each version of the text. Each version can satisfy different goals, e.g., the translated text is easier to read, but reading the original version allows one to practice the language. But how does the reader deal with conflicting goals such as “learning new foreign words and expressions” vs. “staying in the flow of the story”? More questions arise if one can switch at will and seamlessly between the two versions: What is the cost of language-switching decisions? What strategies inform them? What are the resulting switching patterns?
Answering these questions raises two methodological issues: First, existing materials support bilingual reading very poorly, so observing people using them might only provide limited knowledge about bilingual reading. Second, it is difficult to collect readers’ memories of past bilingual reading experiences or to rely on their ability to imagine themselves having such experiences. Indeed, most potential users of bilingual reading appliances are experts in reading in their mother tongue: they have spent many years learning how to do it and they practice their reading skills everyday (without even noticing it), up to the point that whole areas of their brain have become dedicated to reading . Reading is for them an activity that relies mostly on unconscious processes, and this holds true even for reading in a foreign language once they have mastered its writing system. These readers tend therefore to focus on remarkable events in their previous foreign-language reading experiences, which are mainly frustrating situations such as having to stop reading because of an unknown word. They have therefore a strongly biased understanding of what a bilingual reading experience could be and researchers need to find methods that let potential users reflect on their own or fictive experiences in an unbiased way.
This understanding of bilingual reading is necessary to discover what would be a right thing to design, but the complementary problem of getting the design right also brings its own challenges. Indeed, designers are also experts in reading and are therefore subject to the same biases than potential users. Moreover, text display technologies seem to have reached a stable and mature form after centuries of evolution, it is thus difficult to think out of the box and discover innovative ways of displaying text and interacting with it. In this context, proposing and evaluating a simple proof-of-concept prototype appliance might impede the discovery of alternative techniques, by focusing designers and researchers minds on the design of this prototype or some variant of it. Opening the design space to bring some room for unconventional techniques is therefore one of the main concerns that guided this work.
We describe in this paper our approach for the study of bilingual reading and discuss some early results. We focus on addressing two issues: avoiding biases in users’ examinations of their experiences, and providing support for designers while keeping the design space open. We narrow these research questions by focusing first on reading for pleasure  and especially on reading novels. We also focus on a pair of languages that share the same writing system while having both common roots and strong differences: French and English.
After setting the technical and scientific background for bilingual reading, we introduce in Sect. 3 the method that we used to investigate what a bilingual reading experience could be, and some preliminary observations. We provide in Sect. 4 a framework to describe how a bitext is displayed and read. In Sect. 5, we report on a set of design challenges that were highlighted during our interactions with potential users and analyze these challenges in the light of the previous framework. We also consider solutions proposed in other domains that address some of these challenges, and show that they follow three different approaches. Finally, we give hints about how to adapt these approaches for bilingual reading.
2.1 Current Support for Bilingual Reading
Bilingual printed books often assign a different language to the left and right side of the book, so that facing pages display corresponding parts of the text. These so-called parallel texts use a coarse granularity for the alignment: when the reader encounters an unknown word, finding its translation can be difficult as it might require to scan a big part of the facing page. Establishing a visual match at a finer granularity (e.g. paragraphs or lines) could raise the publishing costs and would still require the aligned text elements to follow the same ordering in both languages. This can be a problem with non-literal translations, where parts of text can be swapped or removed, or where sentences can be grouped differently to form paragraphs. Printed parallel texts therefore rely often on dedicated or adapted translations.
Reading a digitalized parallel text with a standard e-reader is cumbersome, as parallel texts have specific requirements for navigation between pages. Bederson et al. designed a smart-phone application to address related navigation issues in digitalized children books (including multi-lingual ones) . While their interaction techniques can be used to access and read text boxes corresponding to different languages, it does not allow the reader to quickly switch its focus of attention from one language to the other. Another solution could be to use dual-display readers such as Codex , dedicating a display to each language.
Interactive Digital Materials.
Web-based machine translation services like Google Translate are the best known and most used interfaces to read a web page originally written in a foreign language. When these tools are used to translate a full web page, hovering a translated sentence with the mouse triggers the display of a tool-tip showing the original version of this sentence. Doppel Text 1 provides a similar interface for reading classical novels as e-books, but exchanges the roles of the two languages (tool-tips show the translation). Its use of professional translations certainly allows for a better reading experience but reduces the number of available books (only 32 at the time of writing). Much more content could be made available to readers with the use of natural language processing tools that compute a (sentence-level) alignment structure. These interfaces only use very simple interactions and have ergonomic issues, such as the tool-tips being hard to distinguish from the original document. More advanced interaction techniques allow disclosing supporting materials  but have not been tested for bilingual reading. Users’ reception of these techniques being highly varied , they require a fine-tuning of the design and there is room to explore more design options.
2.2 Related Work
Monolingual Digital Reading.
Reading with digital tools has been studied for a long time: many studies have investigated the effect of display and text parameters on reading performance , or the benefits of pagination over scroll layout (which is supposed to involve lower levels of mental workload  and to make a better use of spatial memory [28, 29]), or of other visualization techniques . Other studies have investigated the role of paper and the physicality of printed books, to find that they better support navigation [24, 35] and recall of the text  than e-books.
Enhanced Foreign-Language Reading.
Some techniques have been proposed to facilitate the reading of a text in a foreign language without relying on a bi-text. For example, Jenga splits and indents sentences to make their grammar easier to grab . iDict uses eye-tracking techniques and algorithms to identify the foreign language words that cause difficulties to the reader, and automatically displays in a side pane the corresponding entry in a bilingual dictionary . Han et al. discuss the benefits of using pictures instead of bilingual dictionaries entries to avoid reading in multiple languages . In the context of written conversations, people better understand machine-translated messages when the sender highlights the key words in the message . These helping techniques and other similar ones could be used for bilingual reading, however we will argue that bilingual reading cannot be reduced to the availability of such help tools.
Generalization to Multi-representational Reading.
Studies of interest can be found in the context of multiple representations. For example, a document written in a markup language such as HTML has two natural representations: the textual one with markup code, and the graphical one displayed in the browser. The rendering process binds these representations together, defining links between elements of each representation in a similar way than the alignment structure does for languages. This kind of links has been generalized to other relations between documents such as: an original document and its annotations; successive or concurrent versions in the document edit history (the link is then obtained by a kind of diff operation); etc. . Animation has been used in such settings to help in understanding the differences between successive edits of wiki pages  or to help in understanding how a source file is compiled into a graphical representation . The later also allow graphical picking of source code, which eases the document edition and avoid disruptive searches in the code. Understanding how some text has been translated and searching the translation of some part of text are similar tasks in bilingual reading.
3 Understanding Bilingual Reading Experiences
To understand what a bilingual reading experience could be, we have conducted informal interviews with people who showed interest in this question: novel reading amateurs, foreign language teachers, professional translators and their students, linguists, and researchers in natural language processing. We interviewed around 20 people individually or in groups of less than 10, in sessions of one to four hours, with some people interviewed multiple times. The goal of these interviews was as much to get insights about bilingual reading experiences than to test methods that could be used to get such insights.
These interviews made obvious the tendency of interviewees to focus on the negative aspects of their past experiences of reading in a foreign language. They also immediately suggested solutions to their problems, and all following discussions revolved around these solutions. Almost all interviewees mentioned, for instance, the following idea: “if I encounter a word that I don’t know, I could press a button and get its translation.” None of the interviewees realized that parallel texts provided a different solution to this problem, although most of them had previous experience with this kind of material. So we had to tell them: such interventions seemed necessary to get more useful insights, and we were interested in finding what kind of intervention would be the most useful and if it could become part of a more formal collaborative design method.
The most useful interventions were the ones that allowed the interviewees to re-examine their concerns in the light of a different, familiar past experience. We used first the metaphor of subtitles in movies, which could be introduced like this: “As non-native English speakers, we like to watch Hollywood movies subtitled in our native language. This way, the actors’ voices seem more natural and we can better appreciate their acting and the director’s work. Our English is good enough to understand most of the dialogs, but we are sometime lazy or the characters use slang or have a strong accent that we cannot understand. We can then easily have a look at the subtitles to understand what is going on. Moreover, it clearly makes us practice our English speaking skills, adding the pleasure of improving ourselves to the enjoyment of being immersed in a good movie. We want to create for novels what subtitles are to movies.” This metaphor was useful to communicate a vague idea of what could be a bilingual reading experience, as well as in triggering memories about particular situations where the interviewee would switch its attention from the subtitles to the audio or conversely. However, the metaphor was not strong enough that the interviewees would spontaneously refer to it during the whole interview – movies probably seemed too far away from reading.
We also asked interviewees to analyze their concerns in the light of existing materials such as parallel texts or Doppel Text. It helped in highlighting the benefits and drawbacks of these materials and to find other ways to use them than simply looking for the translation of unknown words. We finally introduced and tested reading traces, a more formalized approach where a group is asked at the beginning of the session to read a short novel printed in the parallel text format. The reading takes less than 30 min and the sheets of the document can be detached to let the readers adopt the configuration of printed materials that they judge the most satisfying. Participants are asked to annotate the document with a pen, using their own marks, to create “a trace of what and how they read, that they could use in the following discussion, and that we could use in a later analysis”. They also write at the end of the document a description of the mark system that they used and other comments. A round table follows the reading session with each participant commenting on her experience. A group discussion ends the session. We found that asking participants to write traces of their reading activities incited them to reflect on their actions both during the reading and during the discussion.
What We Have Learned.
The first lesson is that bilingual reading should not be confused with an enhanced form of foreign-language reading, but encompasses it. For example, another important aspect of bilingual reading is to satisfy the reader’s curiosity, e.g. when she asks herself “how can one translate that pun?” or “did they translate the double entendre?” These examples show other punctual and opportunistic needs for the translation, but bilingual reading experiences can rely on a much intensive use of the two texts, in a way that redefines the reader’s engagement with the novel. We experienced this while reading parallel texts: at some point the reading turned into a game where we tried to predict how each sentence had been translated.
In our experimentation with reading traces, we observed a great variety of reading behaviors: reading the whole original text first and then the translation (or the opposite); reading each sentence or paragraph in both languages before reading the following one; switching language at each new paragraph; having a quick overview of one text to pick interesting words or expressions that the reader were interested in knowing the translation; being so immersed in the story as to forget to switch back to the first language; trying to improve the translation; marking corresponding paragraphs to ease language switching; etc. This variety of strategies reflects the diversity of readers’ skills and interests. It also highlights the role of the medium: the participants generally judged parallel text as providing a poor support for bilingual reading. It however supports reading strategies that would be hard to sustain with other techniques such as tool-tips.
4 Describing Reader’s and System’s Behaviors
While the bilingual reading experience still has to be invented, it will necessary follow a few principles imposed by the mechanisms of human visual attention and display resources manipulation. We recall these principles here and put them in the context of bilingual reading. We believe that these principles provide a common language and knowledge basis that can be used at all stages of the design process: They can be used to describe observed behaviors, whether it is to understand the bilingual reading activity or to test design solutions. They can also be used during the design phase to explore different strategies and design solutions.
4.1 Visual Selective Attention and Associated Spaces
Visual selective attention is a fundamental mechanism of human vision that is used extensively to read. It can be defined broadly as a selection mechanism, where one “object” in the visual field of the reader is selected and other objects are ignored. We will call this selected object the focus of attention, while the position of this object in the visual field or world will be called the locus of attention.2 While often used as synonyms, these concepts actually refer to elements that belong to different spaces. We will also introduce another notion, the nexus of attention, as something that connects two “objects”, permitting or calling for an attention shift from one to the other.
Locus of Attention and Display Resources.
When reading, the locus of attention is a point on a display surface where the text in focus is displayed. For our concerns, it is important to consider the diversity of possible display surfaces and their properties: sheets of paper or electronic paper, LCD screen, desktop or whiteboard surfaces where something is video-projected, etc. The notion of a display surface has become very rich and also includes virtual surfaces such as windows or coherent arrays of displays . It should not be confounded with the close notion of work surface, which is a surface in the workspace that can host display surfaces, such as a desktop or wall. Both display and work surfaces will be described as display resources.
Focus of Attention and the Representation Space.
The representation space is the space of all possible focuses of attention. It has therefore 3 axes corresponding to the properties of the object in focus, which belongs to one version of the text defined by its language, has a position in this text version, and corresponds to a linguistic level such as the word, the sentence or the paragraph. By adding a language axis, the representation space generalizes the space-scale diagrams used to discuss information spaces visualization techniques , but it uses the position in the text and the linguistic level instead of continuous axes for space and scale.
Nexus of Attention and Links in the Representation Space.
A nexus of attention connects two possible attention focuses, and can thus be understood as a bidirectional link between two points of the representation space. There are three natural kinds of nexuses: First, consecutive words, sentences and paragraphs are connected by positional links, i.e. links between points of the representation space that differ only along the position dimension. Second, elements are connected to their containers at the upper linguistic level by hierarchical links. And finally, the translation defines alignment links between points that differ along the language dimension (and potentially also along the other dimensions, e.g. a word can sometime be translated as a full sentence). The text itself can contain links between points that have the same language but can differ along the other dimensions, like footnotes, figures and references. The notion of a nexus of attention can also be extended to include links to elements in other documents, such as hyperlinks or the entry corresponding to a word in a bilingual dictionary.
4.2 Using These Spaces in the Design Process
Analyzing Readers’ Behavior.
To analyze readers’ behavior, it can be useful to know how the focus of attention changes during the reading, which can be visualized as a path in the representation space. Such information can be hard to obtain, however some works suggest that it can be usefully approximated by the parameters of the rendering process, such as the amount of scrolling [6, 20]. They can also be used to compute quantitative values that can be directly compared, such as the proportion of reading time during which a given element was visible.
Defining the Interaction.
4.3 Manipulating Limited Resources
In most situations, there is much more text to read than what can be displayed on the available display surfaces and made visible on the available work surfaces: these display resources should then be considered as limited. Visual attention can also be considered as a limited resource since there is only one focus of attention. This limitation in resources motivates the reader’s need to interact with the text, including the physical manipulation of printed books. (The way one interacts, however, should be tailored to the readers’ tasks) The management of limited resources (and particularly their reuse) has generic methods, and we discuss how they apply to attention and display resources. These methods can be used to describe observed readers’ behavior in terms of strategies used, but can also be used for a design space as we will see later.
Reuse of Work Surfaces.
Taking into account the need to reuse workspace allows one to highlight the function of some design elements that could otherwise be overlooked. For example, a book binding could be seen as only useful to maintain the sheets together, but it is also a mechanical guidance that eases the reuse of work space: when turning a page, the binding forces the new page to take the place of the previous one on the work surface. Similarly in GUIs, dragging is constrained to a single dimension to implement scrolling, while page flipping animations help in understanding how the surfaces are reused.
Spatial and Temporal Multiplexing.
Visual attention implies a chain of one-to-one mappings depicted in Fig. 3. Attention is bound to a locus on a work surface by eye movements (gaze mapping); this locus on a work surface is bound to a locus in a display surface by the physical manipulation of the display surfaces (workspace mapping); and then this locus in a display surface is bound to a point in the representation space by the composition mapping, which is controlled through the graphical user interface. Moreover, the composition mapping can itself be defined as a chaining of multiple mappings as it can use split panels, each of which correspond to a viewport in a virtual space where another composition process is held. When a space (virtual or real) is split so that a different mapping is used in each sub-space, we refer to spatial multiplexing. On the other hand, time multiplexing describes a mapping that changes in time. The total mapping must change in time to reuse resources, therefore at least one of the three mappings must change in time.
Multiplexing in Gaze Mapping.
In free attention, eye movements define the locus of attention: the gaze mapping changes in time. The remaining mappings can be fixed, allowing for spatial multiplexing strategies. In directed attention, the reader does not control the selection process, which is imposed by the rendering process. Eyes do not have to move (the gaze mapping is fixed), but the remaining mappings evolve in time. Rapid Serial Visual Presentation epitomizes this temporal multiplexing approach , although other variants are possible.
Multiplexing in Composition Mapping.
The printed book uses spatial multiplexing: each page, as a display surface, displays a fixed part of the representation space. On the other hand, the screen of an e-reader relies on temporal multiplexing: it displays different parts of the representation space at different times. However, e-readers use a text container metaphor such as a pagination metaphor that reproduces the pages of a printed book or a scroll metaphor. Their composition mappings are then better described as chains of two mappings: First, the screen is bound to a specific virtual display surface (a virtual page or a viewport on a virtual scroll) using temporal multiplexing. Then, text is placed in this virtual display surface using spatial multiplexing (different parts of the virtual display surface are bound to different words in the text). Fluid documents  and fish-eye lenses use a similar mixed strategy, however the time-multiplexed mapping introduces a geometrical deformation in addition to defining a viewport. Temporal and spatial multiplexing can also be used to describe other features: a display surface can be split using spatial multiplexing so that one part of it displays the text at another level of detail (creating an overview) or at another position (e.g. to display footnotes).
5 Designing for Bilingual Reading
5.1 Four Challenges Highlighted by the Interviews
The bilingual reading mechanisms presented in the previous section offer a complex yet incomplete model of this activity: it ignores emotional aspects and important cognitive mechanisms such as working memory, and oversimplifies user input mechanisms. This framework is however sufficient to describe the following challenges for the design of bilingual reading appliances, which were all derived from an analysis of the concerns spontaneously expressed by the people we have interviewed.
The main task in reading is to acquire the text meaning by focusing sequentially on (most of the) successive words. This task only uses one type of nexus corresponding to positional links. Other tasks can involve other types of nexuses and they can also be very important to make the reading experience rich and diversified. Obviously, language links are particularly important for bilingual reading. Hierarchical links are used to define the context, as we learned from some participants of the reading traces experiment: they marked where translated paragraphs should be split to match the original ones. Links to glossaries or to the first occurrence of a proper name were frequently requested, as well as links to external references such as dictionaries or encyclopedias.
Some tasks require considering simultaneously two elements, which imply to focus alternatively on them with frequent focus switches. An obvious example of such a task would be a comparison task, e.g. comparing the syntaxes used in a sentence and its translation. A more straightforward example for our concerns is a task where the information provided by the focus of attention is better understood in the light of another element, which becomes a second possible focus. The necessity to support this general class of tasks has been recognized for a long time in HCI: many focus + context techniques have been designed to deal with situations where the context helps to understand the focus  and compared using the notions of spatial and temporal multiplexing . These techniques use mainly the hierarchical links to define the context, but language links can also be used for bilingual reading. For instance, the translation of a sentence provides a context that can help understand the meaning of an unknown word in that sentence. We have accordingly observed people using their two index fingers to keep track of the two focuses of attention. Using our framework, we define bi-focal tasks as tasks where two points from the representation space need to be accessed quickly from one to the other and back. These two points can be connected by any kind of link defined in the document, e.g. by features such as footnotes, references and figures. They can also be defined on the fly by the reader, e.g. to support lightweight navigation such as “looking ahead in the text to preview or anticipate” .
Staying in the Flow.
Many interviewees stressed the fact that when they had to pause their reading because they could not understand a word or sentence, they felt very unsatisfied. The resulting loss of context and the non-satisfaction of their desire to advance in the story cause this feeling, but it may just be a special case of the frustration that arises when somebody’s flow of thoughts is interrupted. The challenge of creating interfaces that allow staying in the flow (as defined by Csikszentmihalyi) is well known in HCI. Bederson, for example, advise to make interfaces that encourage users to develop their skills; that reduce the need for users to consciously make connections between different interface states; that give users the feeling of being in control; and that let users set clear goals and have feedback on their progression . Reading is considered as a good example of activity that can create a state of flow, and it is easy to see how a book, taken as an interface, follows Bederson’s advices. Reading is mostly a navigating in the text: instead of using commands to follow the positional links, the reader rather sets the display resources in a state that allows her to follow these links visually and unconsciously. This principle also appears in the lightweight navigation techniques used by people who read paper magazines .
The secondary tasks discussed earlier are often supported explicitly by specific commands such as clicking on a hyper-link. Instead of providing such direct support for some tasks, a strategy that keeps the reader in the flow could then be to let her navigate in a continuous space defined by the rendering process. This way, the number of controls required is reduced and the user only needs to manipulate the view until she reaches a setting that allow her to accomplish her task. We thus see an excellent design opportunity in a more pervasive use of navigation as an interaction scheme. Relying on navigation in continuous spaces brings another opportunity, with the design of interaction techniques that change the position of the view simultaneously along multiple dimensions. Such an integral navigation has been studied for the dimensions of space and scale, with expected and measured benefits for long-range navigation [1, 15].
Breaking the Container Metaphor.
If we do not want the expression “bilingual reading” to be a paraphrase for “mostly monolingual reading where a different language is sometime used”, we probably need to escape from the box-like container metaphor. It is indeed possible to display some text without making it look like being part of a page or scroll. For instance, RSVP departs strongly from such layouts . The presentation tool Prezi 3 has been cited during the interviews because of its non-conventional text flow: it puts all slides in an unbound 2D zoomable space so that they can intersect with or contain other slides. And while these slides are not otherwise different from paper ones, transitions between slides are animated like navigation in the zoomable space, which makes viewing them a quite different experience. In our framework, container metaphors make use of the particular combination of temporal and spatial multiplexing described in Sect. 4.3. From this, a clever designer can devise other strategies by slightly modifying the bindings and spaces involved. For example, content-aware scrolling  works in a similar way than a scroll metaphor, but defines a continuous path in the document and allows the viewport’s size to change to adapt to the document’s content. The horizontal scroll  uses the spatial multiplexing strategy already used in multi-column layouts and mixes it with the temporal multiplexing strategy used by classical scrolling. These techniques illustrate how our framework opens a design space for new text display methods, which need to be explored in search of efficient techniques to display bitexts.
5.2 Analysis of Existing Approaches
To better understand the space of solutions to the previous challenges, we have additionally conducted an extended review of the HCI literature, looking for works that addressed at least one of these challenges. These works can be described as using one of three different approaches that we introduce and describe in the following sections and Fig. 3. We discuss the potential benefits and drawbacks of each approach, as well as some related technical issues that need to be addressed.
Switch on Demand.
This approach consists in having one display surface by focus (spatial multiplexing), which are synchronized so that they always display corresponding parts: when the user changes the position or level of detail in one surface, the other one is automatically updated (temporal multiplexing). This approach replaces an explicit selection of the nexus involved in a bi-focal task by a continuous use of similar nexuses, allowing using navigation instead of explicit commands. On the other hand, the approach can only be applied with nexus types that define a correspondence for each subset of the document (which is the case for language nexuses). This approach can also make it difficult to analyze the alignment structure as it is used indirectly in the rendering process. And obviously, it uses more display resources.
Spatial multiplexing can be done in two ways for this approach. First, the two display surfaces can be laid out next to each other, as a split screen or using different devices. This is the strategy used for parallel texts: considering the virtual display surfaces formed by a book’s open pages, each side corresponds to a different focus. Second, a surface can be laid on top of another, and render the second focus associated with the area hidden below. For instance, magic lenses and magnification lenses use a lens as the second focus surface; mobile augmented reality uses a display surface belonging to a different object hold between the original display and the user’s eyes.
A major drawback of parallel texts is their low granularity, as seen in Sect. 2.1. It makes it hard to locate in the second view the part corresponding to some known element of the first view, e.g. a single word. The searched part being unknown, it can only be identified using the context, which requires some interpretation and often many eye movements back and forth the two views. All techniques using synchronized views suffer from similar issues unless there is a geometrical correspondence between the two views (which is usually the case with mobile AR, but not for bilingual reading). This drawback can however become a benefit if the reader needs to access frequently the second view, as she will memorize the structure and spatial position of previously visited elements.
Magic lenses and Mobile AR provide another opportunity, as they can be used in two ways: Either the display surface of the second view is kept in a static position and scrolling in the first view triggers updates for the second view, or the display surface is moved (and eventually resized) relatively to the first view, which will also trigger updates. The second option allows the reader to specify what part of the first view she is interested in by using a spatial navigation scheme. The corresponding manipulations of the magic lens or hand-held device are however rather cumbersome and might require new developments.
Animated Transition Between Focuses.
With this approach, a single view is used. An element displayed in this view can be selected and, on demand, the whole view content is replaced by another content so that the view now displays the second focus associated to the selected element. Like synchronized views, this approach requires a type of nexus that can be defined on any subset of the document, but uses a temporal rather than spatial multiplexing scheme. When the two focal elements only differ in position or linguistic level, the transition can use animated panning and zooming. Otherwise, there might be no natural way of animating the transition. Diffamation  and Gliimpse  are notable examples showing that efficient animated transitions can however be devised for some types of textual data. These works use apparition, removal and displacement of text blocs, text lines and other graphics to animate the transition. We refer to this approach as morphing as it uses the shapes and positions of visual elements and their hierarchical relations to define the animation. These works highlight the need for advanced techniques such as animating paragraphs reflow, using curved trajectories and stabilizing the view vertically on focal elements. The benefit of these methods is to allow the user to memorize the two documents layouts, and to quickly switch from one document to the equivalent position in the other one. Because animations are deterministic, they can also be memorized to make afterward animations easier to follow. However, animations can also confuse the user if she cannot keep track of what is moving where, so the flow of animation has to be carefully tuned.
5.3 Adapting These Approaches to Bilingual Reading
Switch on Demand.
For bilingual reading, this approach might be the most obvious one and is actually used by current software (see Sect. 2.1). Because selecting text is difficult when done with the usual techniques, one may want to reduce the set of possible selections to full words, full sentences, full paragraphs, etc., taking advantage of the small number of linguistic levels. Yet, the hierarchical organization of such elements prevents the use of simple selection techniques like picking. Existing materials thus only allow the selection of sentences, reducing the granularity of the alignment structure. There is therefore room to design more efficient selection techniques. One possibility would be to reify the various linguistic levels into new in-page interactive elements that do not overlap. Such elements are already used in pop-up books to make elements appear, hide, transform or move in response to user actions. They could be used in similar ways for bilingual reading, e.g. to reveal the translation of a paragraph by raising the (virtual) flap on which it is printed. Simulations of paper folding and bending could also add realism to the space distortions introduced by techniques like fluid documents.
The synchronized views strategy used by lenses and mobile augmented reality might be inefficient for bilingual reading, as the original and translated texts can have very different layouts. The other strategy offers however many possibilities that are worse exploring, considering the diversity of behaviors that they support (as observed in the reading traces experiment). Notably, using a second device like the reader’s smartphone for the second view provides another interaction surface and opens a new space for simple gestures based on the proximity and orientation of this device.
Finally, it is possible to mix the two strategies. For instance, the two views could be defined by splitting a screen along the vertical dimension rather than along the horizontal one like parallel texts do. The top and bottom view would use different languages, and the bottom view would display the text that corresponds to what immediately follows the text displayed in the top view. The reader can then get the translation of what she just read in the bottom view simply by scrolling down, which would move the text read toward the top view where its translation would appear.
Navigation Between Languages.
Morphing text from one language to another is more complicated than what Diffamation and Gliimpse have done, because there may be no common visual element between the start and final views. There are however many opportunities to create such a morphing: Elements can be morphed differently according to their linguistic level, in the spirit of distinguished visual transitions . The morphing can better use the structure of the document by using successive steps that each address a lower linguistic level. For the animation of changes in abstract trees, this approach allows to easily understand what elements are transformed and how . The last morphing step can then replace words from the first language with their translation.
We also suggest exploring the idea of letting the user control the morphing evolution instead of following a time-controlled animation. This could bring the feeling of navigating between languages. Midway positions could also be valuable, as they expose the linguistic structure of the text, which is sometime enough to relieve ambiguities. The reader can control the animation speed, skip it entirely or go backward, depending on her needs. By coupling in a single gesture this morphing control together with the initial selection of a word to stabilize, we could get similar benefits than with the “switch on demand” approach: a precise selection of the element whose translation is wanted, at the right linguistic level. On the other hand, the interleaving of words belonging to two distinct languages could also be a source of problems for specific readers (e.g. dyslexic ones) or purposes (e.g. language teaching).
These approaches are not necessarily incompatible: they could be used together, or the reader could switch softly from one to another. For example, a tooltip created in a “switch on demand” approach could be turned into a permanent synchronized view. Interactions techniques developed to select text in a “switch on demand” approach could still work for synchronized views, highlighting the part of the second view corresponding to the selection. Similarly, the animation techniques developed for the “navigation between languages” approach could be used to solve the display coherence issues appearing when synchronized views are scrolled.
6 Future Work
In our future work, we will follow two parallel tracks. The first one is the continuation of our reflection on the design methods required to better understand how to design for bilingual reading. The second one concerns the realization of prototypes. We hope that the two tracks will converge in the longer term, e.g. in the form of technological probes.
For the realization of prototypes, the three approaches presented in Sect. 5 revealed the usefulness of three types of operations that can be used in combination: advanced text selection operations required for the “switch on demand” approach, space distortion techniques required for the “synchronized views” approach, and structured morphing strategies required for the “navigation between languages” approach. We have already started the development of a framework that provides these operations and helps exploring new solutions.
More importantly, the creation of design methods will require complementing, formalizing and evaluating the design methods that we have started to use, such as the reading traces experiment. One question that we want to answer is whether presenting to potential users the approaches introduced in Sect. 5.3 would trigger their imagination and allow them to provide even more insights about their bilingual reading experiences? Finally, the question of evaluating interfaces for bilingual reading is complex and will need further developments.
This work has been funded by the ANR CONTINT Transread project. We would like to thank Delphine Soriano for her useful and much appreciated help, as well as all the people we interviewed.
- 1.Appert, C., Fekete, J.D.: OrthoZoom scroller: 1D multi-scale navigation. In: Proceedings of CHI 2006. pp. 21–30. ACM, New York (2006)Google Scholar
- 2.Bederson, B.B.: Interfaces for staying in the flow. In: Ubiquity 2004, September 2004Google Scholar
- 3.Bederson, B.B., Quinn, A., Druin, A.: Designing the reading experience for scanned multi-lingual picture books on mobile phones. In: Proceedings of JCDL 2009, pp. 305–308. ACM (2009)Google Scholar
- 4.Braganza, C., Marriott, K., Moulder, P., Wybrow, M., Dwyer, T.: Scrolling behaviour with single- and multi-column layout. In: Proceedings of WWW 2009, pp. 831–840. ACM (2009)Google Scholar
- 5.Bucher, P., Chatty, S.: Qu’est-ce qu’une surface d’affichage?: une analyse rétrospective. In: Proceedings of IHM 2009, pp. 3–12. ACM, New York (2009)Google Scholar
- 6.Buscher, G., Biedert, R., Heinesch, D., Dengel, A.: Eye tracking analysis of preferred reading regions on the screen. In: CHI EA 2010, pp. 3307–3312. ACM, New York (2010)Google Scholar
- 7.Chang, B.W., Mackinlay, J.D., Zellweger, P.T., Igarashi, T.: A negotiation architecture for fluid documents. In: Proceedings of UIST 1998, pp. 123–132. ACM, New York (1998)Google Scholar
- 8.Chevalier, F., Dragicevic, P., Bezerianos, A., Fekete, J.D.: Using text animated transitions to support navigation in document histories. In: Proceedings of CHI 2010, pp. 683–692. ACM, New York (2010)Google Scholar
- 9.Clark, C., Rumbold, K.: Reading for pleasure: a research overview. National Literacy Trust (2006)Google Scholar
- 10.Cockburn, A., Karlson, A., Bederson, B.B.: A review of overview+detail, zooming, and focus+context interfaces. ACM Comp. Surv. 41(1), 2:1–2:31 (2009)Google Scholar
- 11.Dehaene, S.: Reading in the Brain: The New Science of How We Read. Penguin Group, New York (2009)Google Scholar
- 12.Dragicevic, P., Huot, S., Chevalier, F.: Gliimpse: animating from markup code to rendered documents and vice versa. In: Proceedings of UIST 2011, pp. 257–262. ACM (2011)Google Scholar
- 13.Elmqvist, N., Henry, N., Riche, Y., Fekete, J.D.: Melange: space folding for multi-focus interaction. In: Proceedings of CHI 2008, pp. 1333–1342. ACM, New York (2008)Google Scholar
- 14.Furnas, G.W.: A fisheye follow-up: Further reflections on focus + context. In: Proceedings of CHI 2006, pp. 999–1008. ACM, New York (2006)Google Scholar
- 15.Furnas, G.W., Bederson, B.B.: Space-scale diagrams: understanding multiscale interfaces. In: Proceedings of CHI 1995, pp. 234–241. ACM Press/Addison-Wesley Publishing Co., New York (1995)Google Scholar
- 16.Gao, G., Wang, H.C., Cosley, D., Fussell, S.R.: Same translation but different experience: the effects of highlighting on machine-translated conversations. In: Proceedings of CHI 2013, pp. 449–458. ACM, New York (2013)Google Scholar
- 17.Guilmaine, D., Viau, C., McGuffin, M.J.: Hierarchically animated transitions in visualizations of tree structures. In: Proceedings of AVI 2012, pp. 514–521. ACM, New York (2012)Google Scholar
- 18.Han, C.H., Yang, C.L., Wang, H.C.: Supporting second language reading with picture note-taking. In: CHI EA 2014, pp. 2245–2250. ACM, New York (2014)Google Scholar
- 19.Hinckley, K., Dixon, M., Sarin, R., Guimbretiere, F., Balakrishnan, R.: Codex: a dualscreen tablet computer. In: Proceedings of CHI 2009, pp. 1933–1942. ACM, New York (2009)Google Scholar
- 21.Hyrskykari, A., Majaranta, P., jouko Räihä, K.: Proactive response to eye movements. In: Proceedings of INTERACT 2003, pp. 129–136. IOS Press (2003)Google Scholar
- 22.Ishak, E.W., Feiner, S.K.: Content-aware scrolling. In: Proceedings of UIST 2006, pp. 155–158. ACM, New York (2006)Google Scholar
- 23.Jakobsen, M.R., Hornæk, K.: Transient visualizations. In: Proceedings of OZCHI 2007, pp. 69–76. ACM, New York (2007)Google Scholar
- 24.Marshall, C.C., Bly, S.: Turning the page on navigation. In: Proceedings of JCDL 2005, pp. 225–234. ACM, New York (2005)Google Scholar
- 26.Nelson, T.H.: Xanalogical structure, needed now more than ever: parallel documents, deep links to content, deep versioning, and deep re-use. ACM Comput. Surv. 31(4es) (1999). Article No 33Google Scholar
- 27.O’Hara, K.: Towards a typology of reading goals. Technical report, Xerox RXRC (1996). http://www.xrce.xerox.com/content/download/16322/117657/file/EPC-1996-107.pdf
- 28.O’Hara, K., Sellen, A., Bentley, R.: Supporting memory for spatial location while reading from small displays. In: CHI EA 1999, pp. 220–221. ACM, New York (1999)Google Scholar
- 30.Potter, M.C.: Rapid serial visual presentation (RSVP): A method for studying language processing. In: Kieras, D.E., Just, M.A. (eds.) New Methods in Reading Comprehension Research, pp. 91–118. Erlbaum, Hillsdale (1984)Google Scholar
- 31.Rainie, L., Zickuhr, K., Purcell, K., Madden, M., Brenner, J.: The rise of e-reading, April 2012. http://libraries.pewinternet.org/2012/04/04/the-rise-of-e-reading/
- 32.Schilit, B.N., Golovchinsky, G., Price, M.N.: Beyond paper: Supporting active reading with free form digital ink annotations. In: Proceedings of CHI 1998, pp. 249–256. ACM Press/Addison-Wesley Publishing Co., New York (1998)Google Scholar
- 34.Schlienger, C., Dragicevic, P., Ollagnon, C., Chatty, S.: Les transitions visuelles différenciées : principes et applications. In: Proceedings of IHM 2006, pp. 59–66. ACM (2006)Google Scholar
- 36.Subbaram, V.M.: Effect of display and text parameters on reading performance. Ph.D. thesis, The Ohio State University (2004)Google Scholar
- 37.Tashman, C.S., Edwards, W.K.: Active reading and its discontents: The situations, problems and ideas of readers. In: Proceedings of CHI 2011, pp. 2927–2936. ACM, New York (2011)Google Scholar
- 38.Tiedemann, J.: Bitext alignment. In: Hirst, G. (ed.) Synthesis Lectures on Human Language Technologies, vol. 14. Morgan & Claypool Publishers, San Rafael (2011)Google Scholar
- 39.Tucker, G.R.: A global perspective on bilingualism and bilingual education. In: Georgetown University Round Table on Languages and Linguistics 1999, p. 332 (2001)Google Scholar
- 41.Yu, C.H., Miller, R.C.: Enhancing web page readability for non-native readers. In: Proceedings of CHI 2010, pp. 2523–2532. ACM, New York (2010)Google Scholar
- 42.Zellweger, P.T., Regli, S.H., Mackinlay, J.D., Chang, B.W.: The impact of fluid documents on reading and browsing: an observational study. In: Proceedings of CHI 2000, pp. 249–256. ACM, New York (2000)Google Scholar