Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Printed material and paper documents still play an important role in modern work environments, including offices and fieldwork. People have well understood how to handle paper documents and many organizations have processes in place for distributing paper documents, implementing access control, or destroying copies. Further, the pen usage for annotating document is found to be the most direct, flexible and intuitive way [1]. At the same time we are more and more used to digital documents that offer a variety of functions. Hence, paperless dealing with information is highly unlikely, in our approach we investigate smart documents that use conductive ink and a computing component to reduce the gap between physical documents and digital functionality. The prototypes of smart documents we present in this paper can detect different physical interactions. Examples of these interactions are filling form elements with different pens, tearing up paper, punching holes, and stapling pages. Additionally, mechanisms for physical bookmarking and for detecting annotations are investigated. Detection of these interactions with physical documents offers means to active and responsive documents, and it allows reflecting these actions in their respective digital representations. The contribution of this paper is a set of mechanisms and layout designs that enable interaction with paper documents based on conductive ink printing. These mechanisms are experimentally evaluated and the layout designs are provided under creative commons license.

2 Related Work

The unlikeliness of paperless office makes it the focus of researches to maintain its affordances yet augment interactivity aspects. In this section we discuss prior work in two different areas aiming to introduce interactivity to paper form using (1) camera based systems and (2) new technologies namely conductive ink to enable printed electronics.

2.1 Interactive Paper Documents

Relevant research with the aim to add and link digital value to paper documents includes work that focuses on linking the physical and digital world and enhancing the responsiveness of paper and office tools [2–4]. Through embedding cameras onto the pen [5] or above the desktop [6, 7] for capturing pen-paper interaction, or even using scanners [8]. The Anoto penFootnote 1 contains a built-in camera to record what has been written on special paper with a preprinted pattern. Further, this pen is employed by other systems to transfer the physical editing on paper documents to the digital world [9–12]. CoScribe utilized it to combine work on digital and printed documents, where they used the digital pen to allow annotating, linking and tagging printed and digital documents as the pen interacts with both the printed documents and pen-sensitive displays [13]. In the PaperProof they extended the iPaper framework [4]. However, using a camera as the pen paper interaction tracking device introduced interactive aspect to the papers, however it restricted the interaction to a rather limited area in front of the camera.

2.2 Conductive Ink and Printed Electronics

Recently, different forms of conductive ink (paint, pens, and printers) became commercially availableFootnote 2 and electronics can nowadays be homemade. Additionally, the vast enhancement introduced by chemical sintering [14] allowed the fast and easy use of conductive ink for drawing or printing circuits using pens or off-the-shelf inkjet printers [15, 16]. Hodges et al. [17] added interactive elements through stickers. Moreover, printable electronics have been combined with other technology as described in [18] introducing new applications.

3 Smart Printed Paper Forms

Completing a paper form using a pen is generally perceived as very natural interaction. The mechanisms we propose to turn a paper document into a smart document exploit as much as possible conventional forms, shapes, methods, and tools known from common daily routines. Those include shapes and layout known from common document forms (e.g., checkboxes), office tools such as staplers, paper-clips, hole-punchers and stamps, as well as natural interaction such as ripping a document apart.

3.1 Traditional to Digital

Certain traditional forms in paper documents can easily be transferred to conductive circuit layouts. Two examples are checkboxes and signature fields. Figure 1b–c shows two checkbox layouts that are optimized for the use with pencil or pen. Their ground electrode is arranged on the outer side which makes it easy to create multiple stacked checkbox items. The signature field (Fig. 1d) demonstrates a layout which detects whether text was written inside. Most of the layouts introduced rely on the same electrical principle: two electrodes arranged in close distance that are connected when a pencil or conductive pen crosses both, significantly reducing the resistance between them which is then measured by the computing unit. The ground electrode is however shared between all conductive elements. This principle can be also applied to detect crossed out words (Fig. 1a). A pencil is often used instead of a pen if actions (e.g., crossed checkboxes) should be reversible using a rubber. This works as well for removing pencil lines on conductive elements that again increases the resistance between two electrodes. Further, a set of conventional office tools can be involved in the interaction with smart paper documents. We found a stapler to be a surprisingly reliable tool for electrically connecting circuits on multiple pages. Figure 1h depicts a layout which targets the interconnection of circuits on multiple pages. This approach allows placing the computing unit only on one page on which all circuits run together. Further tools are paper-clips that can be used as temporary selectors, a hole-puncher for setting the archival state of documents. Table 1 provides an overview of traditional mechanisms that can be also used to realize digital functionality.

Fig. 1.
figure 1

Form elements sheet demonstrating 4 mechanisms: crossing out words (a), checkboxes (b, c), signature (d), punch-hole (e), bookmarks (f), text marking (g), and staple (h).

Full size image
Table 1. Traditional mechanisms and their mapping to digital functionality
Full size table

3.2 Attaching a Computing Unit

We propose multiple mechanisms for electrically connecting a computing unit to a paper document. The connection can be either of permanent or temporal type. A permanent connection can be achieved by following the functionality of a sticker. Practically, this is realized by attaching a computing unit to a document for its whole lifespan for instance by using conductive z-tape [19]. Temporal connections can be realized in different ways:

  • Clip: Attaching a clip that uses mechanical pressure for connecting a computing unit (similar to [20]). This can be a tiny paper clip as well as a clip-board with a mechanical spring holder on top.

  • Hole-Puncher: Using a hole-puncher with embedded computing for reading out document information. The document information is read out in the moment of punching the paper.

  • Stapler: A computing unit on flexible PCB film (such as Seeduino Film) that provides connectors that are stapled onto matching connectors on a paper document.

4 Evaluation

As a proof of concept and to evaluate our mechanisms, we conducted a user study with 12 participants (4 female, 8 male). Each participant filled in a paper form with both, a pencil and a ball-pen including: 3 checkboxes and 3 radio buttons with different gaps between the printed circuit wires sizes (0.2 mm/0.5 mm/1.0 mm), cross out 2 words written in different font sizes (Myriad Pro, bold, 16pt/24pt) and sign with her/his name on a dedicated field. The order of the writing tools was counterbalanced. We additionally asked the participants to staple 10 pages to create a conductive connection among those pages. For evaluating the mechanisms of our layout library, we measured the resistance of the connection created through the pencil/pen. While pencils have been used to draw circuits on common paper they do not work well on photo paper: Only in 15 of 108 cases the pencil created a connection with a resistance smaller than 60 MΩ (M = 38.9 MΩ, SD = 14.5). The multi-page connections created by the stapler were conductive in all 108 cases. All mean resistances using a ball-pen are in a range that can be sensed by a microcontroller. By adding an appropriate resistance (ca. 200 kΩ), a voltage divider can be built for each input mechanism. In this way the checked/signed state can be sensed using digital inputs of common MCUs. The forms used in the study was stored in a normal office environment and the measurements were retaken in 6 months and they were consistent with the previously measured ones, which implies the paper’s maintainability.

5 Prototypes

As proof of concept we have created two fully functional prototypes. Each of them implements multiple mechanisms that were introduced in the last sections. Prototype 1 focuses on form elements that can detect pen-based user input on the fly. Prototype 2 demonstrates the detection of text markings, bookmarks and the interconnection of multiple pages.

5.1 Computing Unit

To measure changes in the resistance of conductive elements and to transfer this information to a host system, a computing unit and power source are required. We used the Blidgets platform that was also developed by the authors in a recent parallel project. A Blidget has the dimensions of 25 × 25 × 3 mm, a weight of 3 g which makes typical handling of a paper document remains mostly unaffected. It allows instant wireless communication with computers and smartphones via Bluetooth Low Energy (LE) and it is remotely configurable. It’s tiny and at footprint allows it to be easily connected to a circuit on a sheet of paper using for instance z-tape. We use a small 110mAh LiPo battery to power the device, but any power source between 1.8–4 V is sufficient. Software on a desktop computer equipped with a USB Bluetooth LE dongle allows to instantly monitor the state of all input ports of a Blidget. Those state changes are then mapped to specific actions related to the digital representation of the paper document. The mapping of a Blidget to a digital representation is configured once through a graphical user interface on the host computer. Afterwards modifications on the paper such as checking a box, signing a field are instantly visualized in the same user interface.

5.2 Form Elements

Our single-page prototype primarily focuses on demonstrating the feasibility of conductive form elements that allow detecting user input through a pen. As depicted in Fig. 1 it is equipped with the Blidget computing unit and fully functional. Figure 1b, c shows two circuit layouts that mimic the visual layout of checkboxes. They are optimized for electrically sensing pen strokes. One computing unit port is needed for each checkbox item as well as the common ground. The distance between the checkbox electrodes in the prototype is 0.2 mm. Another forms element is to cross out items that are not applicable, Fig. 1a two electrodes are alternately connected to the letters of each item. Also, one computing-unit port is required for each element that can be crossed out in addition to the common ground. The signature field (Fig. 1d) is a specific example for a circuit layout that is intended to detect the existence of handwritten text in input fields. It requires one input port of the computing unit. Finally, a mechanism that is not related to user input with a pen is show in Fig. 1e. A simple circuit on the left of the page is intended to detect punch holes. The software on the host computer uses this information to add an “archived” tag to the digital document.

5.3 Text Markings, Bookmarks and Multiple Pages

Research shows that, particularly for reading, paper has inherent advantages over digital documents. To state only some of them, annotating paper documents with a pen is intuitive very flexible, and smoothly integrated with reading. Moreover, paper provides for two-handed interaction and navigation and for creating flexible spatial arrangements. Our second functional prototype demonstrates mechanisms for detecting text markings, setting bookmarks and interconnecting circuits on multiple pages. For detecting text markings we introduce a simple layout that consists of a vertical ground electrode and multiple vertical section electrodes. Here the idea is that a user draws a vertical line to the left and connects the electrodes at the height of the text marking. The resolution depends on the number of used section-electrodes: in our prototype 4 electrodes are used resulting in 4 sections with text markings that can be distinguished. A more advanced circuit layout would be placed within the text area (e.g. multiple rows of electrodes) to detect markings at their original spot. For detecting bookmarks we use a simple paper-clip and an area in the upper right corner. This area consists of two generous electrodes that are electrically connected when a paper clip is attached. In this particular case the traditional functionality of the paper-clip (clipping multiple pages) is ignored and instead it is used as a temporal selector. Conductive pads on the left side and staples are used to electrically connect multiple pages. As depicted in Fig. 1h 11 electrodes allow to detect 1 bookmark and 4 text marking sections per page (2 text pages and 1 page for the computing unit). One electrode is the common ground. With the rising number of pages also rises the amount of connectors needed on the left side. Here one approach is to share the same connectors between pages and introducing a mechanism that only connects the ground electrode of the active page to the computing unit. A mechanism for encoding the page number of the active page is then required as well (e.g. binary encoding).

5.4 Equipment

For the creation of all printed circuit layouts we have used a Brother DCP-J125 printer, Mitsubishi conductiveink (NBSIJ-MU01) and Mitsubishi paper (NBWF- 3GF100) optimized for high conductivity. As ball-pen we used the Mitsubishi Uniball GEL IMPACT UM-153S and for permanently connecting the computing unit to paper we used 3 M electrically vertically conductive adhesive tape 9703.

6 Conclusion and Future Work

We presented a set of mechanisms that allow adding interactivity to paper documents and maintaining their traditional properties and affordances. Our layout library allows an easy reproduction of all used circuit layouts. An evaluation provides insights into the electrical behavior of differently designed input form layouts and the reliability of stapled multi-page connections. Two functional prototypes prove the concept of multiple proposed mechanisms and demonstrate a feasible approach for attaching a computing unit. We envision more advanced layouts and mechanisms to be created. Those could be collected in a software tool that automatically adds the desired interactive elements into documents and generates multi-page layouts as well as connectors for different computing units.