Reflection in interaction
- First Online:
- Cite this article as:
- Fruchter, R., Swaminathan, S., Boraiah, M. et al. AI & Soc (2007) 22: 211. doi:10.1007/s00146-007-0121-6
- 730 Downloads
A decision delay can translate into significant financial and business losses. One way to accelerate the decision process is through improved communication among the stakeholders engaged in the project. Capturing, transferring, managing, and reusing data, information, and knowledge in the context it is generated can lead to higher productivity, effective communication, reduced number of requests for clarification, and a shorter time-to-market cycle. We formalized the concept of reflection in interaction during communicative events among multiple project stakeholders. This concept extends Donald Schon’s theory of reflection in action of a single practitioner. We model the observed reflection in interaction with a prototype system called TalkingPaperTM. It is a ubiquitous client-server collaborative environment that facilitates knowledge capture, sharing, and reuse during synchronous and asynchronous communicative events. TalkingPaperTM bridges the paper and digital worlds. It transforms the analog verbal discours, annotated paper corporate documents, and the paper and pencil sketches into indexed and synchronized digital content that is published on and streamed on-demand from a TalkingPaperTM web server. The TalkingPaperTM sessions can be accessed by all stakeholders for rapid knowledge sharing and decision-making.
What do the Altamira cave paintings, kids’ drawings, and professional paper napkin sketches have in common? They all tell a story, but there is no voice of the storyteller. Our observations show that the most striking means of knowledge transfer from experts to novices in both education and industry settings is through the informal recounting of experiences from past projects and collaborative dialogue connecting ideas and solutions. Stories convey great amounts of knowledge and information in relatively few words together with sketches on paper or annotations on formal printed documents (Gershon and Page 2001).
A decision delay can translate into significant financial and business losses. One way to accelerate the decision process is through improved communication among the stakeholders engaged in the project. Capturing, transferring, managing, and reusing data, information, and knowledge in the context it is generated can lead to higher productivity, effective communication, reduced number of requests for clarification and time-to-market cycle. However, knowledge transfer often fails, since knowledge is not captured, it is captured in an abstract format rendering it not reusable, or there are no formal mechanisms to find and retrieve it. Valuable knowledge is lost at project transition points of the building life cycle from one phase to the next, i.e., finance, design, procure/fabricate, build, manage as the handover between different stakeholder teams takes place.
We view knowledge capture, sharing, and reuse as key steps in the knowledge life cycle (Fruchter and Demain 2002, 2005). Knowledge is created as designers collaborate on design projects using data, information, past experience and knowledge. It is captured, indexed, and stored in human memory or digital archives. At a later stage, it is retrieved and reused. Finally, as knowledge is reused it is refined and becomes more valuable. In this sense, the archive acts as a knowledge refinery.
Our ethnographic studies performed over the past decade of cross-disciplinary team at work show that a primary source of information behind design decisions is embedded within the verbal conversation among designers. Capturing these conversations is difficult because the information exchange is unstructured and spontaneous. In addition, discourse is often multimodal. It is common to augment speech with sketches as an embodyment of the mental model, or launch into a problem solving discussion tirggered by a skeched solution.
Advances in digital technology promise to assist in knowledge capture and reuse. However, the more digital content is created the more paper we print and use. Most digital content management today offer document management solutions with few answers how to capitalize on the core corporate competence, i.e., to capture, share, and reuse business critical knowledge. Digital archives store formal documents (CAD, Word, Excel, etc.) that can be easily edited, shared, searched, and archived. Knowledge reuse and externalization of tacit knowledge is not revealed by these formal documents. The knowledge creation takes place in informal concept generation and problem solving sessions in which knowledge workers gather around multiple blueprints or other documents, and engage in dialogue and paper and pencil sketching. Paper has a tactile feel; it can be easily folded or rolled and carried to meetings or site visits. It affords single or multiple users to interact and jointly annotate one or multiple documents, and more importantly; it is socially and legally accepted (Sellen 2001). However, paper is difficult to modify and expensive to distribute, archive, search, retrieve, and reuse. These limitations are very effectively supported by digital technology. Nevertheless, when inspecting for instance large CAD models on screen the current resolution of computer monitors only affords to either zoom in and see the details but loose the big picture, or zoom out and see the big picture but miss the details. Paper provides a high resolution for navigation through the content that enables users to view at a glance local details and global context.
Create content using natural idioms as communication media such as dialogue and paper and pencil sketches.
Explore and understand the context in which this knowledge was originally created.
Interact with the content in a rich, multimedia environment.
We introduce the concept of reflection in interaction during communicative events among stakeholders. The concept was formalized based on ethonographic observations. We model the observed reflection in interaction with the prototype called TalkingPaperTM.1 TalkingPaperTM represents a ubiquitous collaborative environment for collocated and mobile knowledge workers. It bridges the analog (speech, dialogue, paper and pencil sketching and annotation) and digital (audio, video, etc.) worlds to facilitate synchronous and asynchronous communicative events and support the knowledge lifecycle. We present the theoretical points of departure, and discuss evidence collected during ethnographic studies in the typical paper intensive environments such as project teamwork and the building permit process. The paper presents the problem and solution space, the TalkingPaperTM prototype, and the spectrum of interaction scenarios the prototype currently supports.
2 Theoretical points of departure
The points of departure of this research are: design theory and methodology, knowledge creation and management, and human computer interaction.
2.1 Design theory and methodology
The issue of how to capture knowledge in project design teams has received extensive attention from researchers in design theory and methodology. The value of contextual design knowledge (process, evolution, rationale) has been repeatedly recognized, but so has the additional overhead required of the designer in order to capture it. Other studies of design focused on either the sketch activity, i.e., learning from sketched accounts of design (Tversky 1999; Stiedel and Henderson 1983; Olszweski 1981; Kosslyn 1981; Goel 1995) or verbal accounts of design (Cross 1992, 1996; Dorst 1996). Some researchers have studied the relation between sketching and talking (Eastman 1969; Goldschmidt 1991). Recent studies of interactive workspaces (Ju et.al. 2004) explore capture and navigation issues related to technology-augmented interactions. To help guide the designer’s exploration of an archive of unstructured dialogue and sketch content linked to structured document databases, it will be necessary to develop a search and retrieval mechanism. Our research builds on Donald Schon’s concept of the reflective practitioner paradigm of design (Schon 1983). Schön argues that every design task is unique, and that the basic problem for designers is to determine how to approach such a single unique task. Schön places this tackling of unique tasks at the center of design practice, a notion he terms knowing in action (Schön 1983, p. 50). To Schön, design, like tightrope walking, is an action-oriented activity. However, when knowing-in-action breaks down, the designer consciously transitions to acts of reflection. Schön calls this reflection in action. In a cycle which Schön refers to as a reflective conversation with the situation, designers reflect by naming the relevant factors, framing the problem in a certain way, making moves toward a solution and evaluating those moves. Schön argues that, whereas action-oriented knowledge is often tacit and difficult to express or convey, what can be captured is reflection in action.
2.2 Knowledge creation and management
The digital age holds a great promise to assist in knowledge capture, transfer, and reuse. However, the more digital content is created the more paper we print. More importantly, we need to offer clear and distinguishing definitions and instantiations for data, information, and knowledge, rather than using them interchangably. Similar to Davenport and Prusak (1998), our research uses the following working definitions for data, information, and knowledge. Data (e.g., printed documents or digital documents of CAD, spreadsheeds, text) represent the “raw material”. This is easy to manage and store in corporate data bases or ftp sited. Nevertheless, data is not information. Information emerges during a communicative transaction between a sender and a receiver. Information is created as the sender takes data and adds meaning, relevance, purpose, value through a process of contexualization and synthesis. Neither data nor information represent knowledge. We believe and observe that knowledge is created through dialogue within or among people as they use their past experiences and knowledge in a specific context to create alternative solutions. During these dialogues knowledge is created as connections, comparisons, combinations, and their consequences are explored. It is important to note that documents do not reveal the tacit knowledge externalized during the permit checking process. They also ignore the highly contextual and interlinked modes of communication in which people generate concepts through verbal discourse and sketching.
We view the act of reflection in action as a step in the knowledge creation and capture phase of what we call the “knowledge life cycle” (Fruchter and Demian 2002, 2005). Knowledge represents an instance of what Nonaka’s knowledge creation cycle calls “socialization, and externalization of tacit knowledge”. (Nonaka and Takeuchi 1995). We build on these constructs of the knowledge lifecycle and the “socialization, externalization, combination, and internalization” cycle of knowledge transfer.
2.3 Human computer interaction
We use the scenario-based design approach (Rosson and Carroll 2001) that offers a methodology to study the current state-of-practice, describe how people use technology and analyze how technology can support and improve their activities. The scenario-based design process begins with an analysis of current practice using problem scenarios. These are transformed into activity scenarios, information scenarios and interaction scenarios. The final stage is prototyping and evaluation based on the interaction scenarios. The process as a whole from problem scenarios to prototype development is iterative.
3 Ethnographic evidence
Reduce hidden work (i.e., less coordination and rework).
Improve communication and knowledge transfer among the stakeholders, and
Decrease response time and decision delays.
The typical workspace of a professional expert at the SJ RDA (e.g., architect, engineers) comprises a large desk mainly used to spread out drawings, tons of tracing paper and sketches, a rack with all the drawings and tracing paper sketches of on-going projects.
SJ RDA experts constantly trace over the blueprints sent by the client several times using tracing paper to understand all the intricacies of the drawings—a process representing reflection-in-action.
If a SJ RDA expert had an issue he/she wanted to think about or a new idea it was quicker to capture it by sketching on tracing paper than to make a computer model.
Blueprints are checked, annotated, and traced over to understand how they correlate.
During meetings SJ RDA experts from different disciplines gather around a large meeting desk with multiple blueprints and other sets of documents (calculations, spreadsheets, docs) that they annotate, sketch on, correlate to identify problems, discuss key issues, make recommendations, request changes.
Sketches and tracing paper drawings are archived for future reference during meetings and re-use of good ideas. Nevertheless, it is very hard to search and find relevant material through the paper archive.
The final decisions, recommendations, and requests for changes are summarized in a text document and sent to the client. Nevertheless, all the discourse, arguments, and rationale behind these items are not provided. This leads to: (1) multiple requests for clarifications sent to the agency by the client and the project team members, and (2) delays in the permit process and project progress triggered by coordination and rework efforts.
4 Reflection in interaction
This research adopts a scenario-based approach (Rosson and Carroll 2001) to the design of human-computer interaction. The premise behind scenario-based methods is that descriptions of people using technology are essential in analyzing how technology can support and improve their activities. We address the following research question:
What are the governing principles of and how can we map the natural paper-based environment into a digital interactive environment that emulates the dynamic and complex interactions among multiple participants, documents, and input devices (e.g., pens)?
Number of participants; from single to multiple participants.
Number of documents or paper artifacts; from one to multiple documents, and
Number of input devices used to sketch or mark up the documents; from one to multiple input devices (e.g., pens, markers).
a single participant marking up multiple document with one input device/pen,
multiple participants marking up one document with multiple devices/pens, etc.
Naming the relevant factors in the studied design.
Framing the problem in a specific domain.
Making moves towards a solution, i.e., often modifying the design solution to address some of the identified problems, and
Evaluating the moves or proposed modifications.
Identifying the relevant factors in all considered disciplines through exploratory sketching and discussion.
Correlating these factors across disciplines and documents.
Discussing and exploring alternatives across disciplines.
Assessing alternatives and their implications.
Capturing, sharing, and reusing knowledge created in cross-disciplinary, collaborative, teams is critical to increase the quality of the product, reduce time-to-market and cost. Concept generation and development occur most frequently in informal media where design capture tools are the weakest. This statement has strong implications for the capture and reuse of design knowledge because conceptual design generates the majority of initial ideas and directions that guide the course of the project. Sketching is a natural mode for designers to communicate ideas in highly informal activities such as brainstorming sessions, project reviews. Often, the sketch itself is merely the vehicle that spawns discussion about a particular design issue. Thus, from a design perspective, capture of both the sketch itself and the discussion that provides the context behind the sketch are important. It is interesting to note that today’s state-of-practice or best practices are not captured and knowledge is lost when the whiteboard is erased or the paper napkin sketch is tossed away. With all the advances in computing today people still prefer to have a conversation and use paper and pencil sketches to communicate and capture ideas. Our observations show that during communicative events there is a continuum between discourse and sketching as ideas are explored and shared. We assert that a primary source of knowledge behind design decisions is embedded within the verbal conversation among designers. The link between dialogue and sketch provides a rich context to express and exchange knowledge. This link becomes critical in the process of knowledge sharing, retrieval and reuse to support the user’s understanding of the shared information and assessment of the relevance of the retrieved content with respect to the task at hand. Nevertheless, paper is a media hard to share, exchanged, and reuse, and does not capture the discourse among users. The moment you lost the paper sketch the ideas are lost.
This research address the communication, coordination, and cognition needs defined by: (1) the need to bridge the analog paper and discourse world with the digital world. Such a bridge can improve the knowledge transfer and reuse over the life cycle of the building, (2) the increasing complexity defined by the three dimensional problem space–multiple participants, multiple documents, multiple pens (as input devices) posed by the paper-intensive reflection in interaction process in which the stakeholders engage in dialogue and sketching activities. Note that this three dimensional problem space is applicable to both the paper and digital worlds.
5 TalkingpaperTM environment
TalkingPaperTM aims to empower the project stakeholders and engage them in productive collaborative synchronous and asynchronous teamwork by leveraging the best of the all worlds paper, digital multimedia, and networked communication. TalkingPaperTM is building on our experience in developing multimedia knowledge capture innovative technologies, e.g. RECALLTM technology (Fruchter and Yen 2000)2, as well as commercial technologies such as AnotoTM paper, digital pens (e.g., by Nokia, Logitech, Maxell), cell phones, Bluetooth communication, and GSM/GPRS network services. To date, related digital pen pilot efforts and trial projects focus on text document mark-up (Guimbretiere 2003) and forms automation for different sectors such as healthcare, service and support companies, government organizations, education testing agencies, pharmaceutical research companies, e.g., forms for clinical studies, forms for customer data and signature, weekly report forms for service engineers, etc. (e.g., Dai Nipon Printing and Hitachi Corporation in Japan; HP, USA).
Quickly sketch and explain their ideas using paper and pencil.
Quickly find (mentally) reusable items, and
Remember the context of each item, therefore understand it and reuse it effectively.
Create and capture to publish and share reusable content.
Find reusable content, and
Understand this content in the original context it was created.
Enables seamless transformation of the informal analog content, such as dialogue and paper and pencil sketches, into digital sketch objects indexed and synchronized with the streamed digital audio of a TalkingPaperTM session. This conversion process takes place in real-time, with high-fidelity, and least overhead to the participants.
Supports knowledge reuse by allowing the user to understand the content in the context it originated, i.e., interactive replay of indexed digital audio-sketch rich multimedia content that captures the creative human activities of concept generation through dialogue and paper and pencil sketching. The TalkingPaperTM sessions are automatically uploaded to a TalkingPaperTM web server that was developed to archive, share, and stream these sessions on-demand.
The create and capture activity is supported by TalkingPaperTM client application through a high-fidelity, interactive, integrated multi-user, analog-to-digital multimedia conversion process. This process converts all sketches and annotations on the AnotoTM paper into digital sketch objects that are synchronized with the speech from the digital audio channel and the documents form the corporate datatbase that were printed on AnotoTM pages. The indexing and synchronization based on the time stamps of the different channels (i.e., digital audio and sketch).
The TalkingPaperTM client application allows real time publishing of the dialogue and sketches/annotations to a TalkingPaperTM web server. The sharing of content is supported by streaming the selected session from the TalkingPaperTM web server. It automatically synchronizes digital audio-sketch episodes with the corresponding document that was printed on the AnotoTM paper used in that session.
The find activity is supported by an integrated digital audio-sketch search engine provided by I-Dialogue at the macro enterprise database search level (Yin and Fruchter, presented in this volume), and by user driven search or selection at the micro session level. In this case, the user selects specific items from a sketch or annotation on a page TalkingPaperTM to trigger play back from that item on.
The understand activity is supported by TalkingPaperTM that allows the user to replay a selected content item or full session in the context it was created. The verbal discourse provides the rationale behind the sketch and annotations on the document pages. This is achieved as the TalkingPaperTM web server streams the selected: (1) session, (2) sketch/annotation item from within a session, or (3) flag that marks up a sketch or annotation on the displayed TalkingPaperTM page indicating a specific person’s ID.
The first TalkingPaperTM prototype addressed the scenario of a single participant marking up one document page with one input digital pen.
After testing that our concept and algorithm work we developed next prototype that allowed to run multiple parallel sessions of TalkingPaperTM client applications in which single participants mark up a different document page each having their own digital pen.
The next prototype considered parallel sessions of single participants each marking up multiple document pages with their own digital pen. It is important to note that from a cognition point of view it is very natural for the human to switch from one page to the other, or one document to the other, as he/she marks up and explains issues of correlated items on different pages or documents in the analog paper and pencil world. However, it was a complex task to model and implement this cognitive activity in the digital world. To accomplish this goal we developed and implemented a printing and synchronization algorithm that keeps track and knows what document page was printed on which Anoto page. The algorithm further indexes and synchronizes the pages and their printed document content with the digital sketch/annotation objects and audio stream. Once the session is published on the TalkingPaperTM web server, it streams and recreates the same look and feel of the sketching or annotation activity and verbal discourse, as well as the sequence of marked up pages that occured in the analog paper and pencil world.
We continued the rapid prototyping cycle with a TalkingPaperTM system that enables a single user to mark up multiple pages with multiple pens. This setting allows users to assign different semantic meanings to each digital pen having a different colors. For instance, we emulated a real world setting in which structural engineers check detailing blue prints using red to mean “changes to be made,” green to mean “correct detail,” yellow to mean “checked item,” etc. In this case the user that replays the session from the TalkingPaperTM web server can immediately see the color coded annotations, for instance all the items that need to be changed marked with red, replay a selected red sketch or annotation and listen to the rationale behind the requested modification.
The next TalkingPaperTM prototype implemented the reflection in interaction scenario with multiple participants marking up one document printed on a single Anoto page sharing a single digital pen. This emulates the real world case where multiple experts examine, discuss and mark-up changes and issues on one large blue print (e.g., an architectural floor plan or a structural system for a future building).
The latest implemented prototype models the interaction scenario that engages multiple participants correlating and marking up items on multiple documents printed on different Anoto pages, each using their own digital pen. This is the most complex scenario. The implementation allows for multiple parallel sessions (i.e., project teams or groups to meet and use TalkingPaperTM to capture their discourse and annotations/sketches). Allowing each participant to use their own digital pen provides a unique identifier for each person. Participants can assign their own name to the pen ID. The person’s unique identifier (e.g., their name or the pen number) can be used during search and replay. During the streamed playback of a TalkingPaperTM session the user can enter a query such as “Joe” or “Mary and Joe” and TalkingPaperTM will flag with different colored, small markers the locations where Joe or Mary have sketched or annotated the document and spoken. The user can select any of these flags on screen and TalkingPaperTM will start playing from that point.
In the first case participants who are in their office or in a meeting room use: (1) a TalkingPaperTM client application that runs on a PC connected through LAN to the TalkingPaperTM web server, (2) the digital pen(s), (3) the cell phone to push the pen strokes to the TalkingPaperTM web server, and (4) the Anoto paper that can be a blank page for sketching or contain printed document pages that can be annotated.
For the asynchronous mobile solution the knowledge worker uses: (1) TalkingPaper client application that runs on the cell phone that is connected to the TalkingPaper web server via GSM/GPRS, (2) the digital pen, and (3) the Anoto paper that can be a blank page for sketching or can contain printed document pages that can be annotated.
An extension of Schon’s concept of the reflection in action of a practitioner to a multi-practitioner reflection in interaction paradigm.
A formalization of the complex interactions in a “multiple participants, multiple documents, multiple pens” interactive multimedia workspace that bridges the paper and digital worlds, and
TalkingPaperTM prototype as an analog-to-digital content conversion processor of rich multimodal communications in support of the knowledge life cycle i.e. create, capture, index, store, search, find, retrieve, share and reuse knowledge.
This project is sponsored by MediaX and PBL Lab at Stanford, Nokia Inc., and Cisco Inc.,.