Human-Engaged Computing: the future of Human–Computer Interaction
Debates regarding the nature and role of Human-Computer Interaction (HCI) have become increasingly common. This is because HCI lacks a clear philosophical foundation from which to derive a coherent vision and consistent aims and goals. This paper proposes a conceptual framework for ongoing discussion that can give more meaningful and pertinent direction to the future of HCI; we call the proposed approach Human-Engaged Computing (HEC). HEC is a focused yet adaptable philosophical approach which aims to establish “synergized interactions between engaged humans and engaging computers for high level wisdom which enhances our human survival probability and our full potential as humans”. In this paper, HEC is described through five perspectives—definition, components, principles, case studies and benefits. The paper concludes by suggesting future directions for HEC.
KeywordsHuman-Engaged Computing HCI Synergized interaction (or synergism ) Symbiosis Antibiosis Human capacities Human potentials Human engagement
The initial trigger for this work was a note written by Douglas Engelbart who wrote to the first author in 2006: “Let’s focus our HCI attention on increasing human capabilities to develop, integrate and understand the knowledge required for improving society’s survival probability”. This challenged the authors to consider the philosophical ground and practical goals of HCI.
Varying views and assessments of the nature of HCI have led to useful discussion within the HCI community on the essential nature of HCI and what constitutes good HCI research (Bødker 2006; Harrison 2007; Kostakos 2015; Kuutti and Bannon 2014; Rogers 2012).
However, little work has addressed the need or development of a philosophical basis that would be capable of generating a reasoned, coherent and adaptable way forward. In a review of HCI, Rogers stated: “there is no longer a coherent set of aims or goals...it seems anything goes and anyone can join in.” (Rogers 2012, p. xii). Some argue that HCI lacks a mainstream topic and school of thought Kostakos (2015). These debates extend beyond the HCI community where some commentators wonder whether HCI is “a discipline” (Blackwell 2015), some even doubting the need for the HCI field (van der Veer 2013) and some colleagues from other disciplines criticizing HCI activities as mere “condiments”.
As for the development of a philosophical framework, the guiding principles should always favor the human over the computer. Technologies have largely been used to automate tasks to achieve greater efficiency and productivity. However, this has not led to intrinsically better human outcomes. Some of today’s leading IT experts share similar views. Shneiderman (2015) said that future technologies should encourage “trust, empathy, and responsibility”. Stephen Hawking, Bill Gates, and Elon Musk have all expressed concerns on the negative impact of super AI. HCI directions should be channeled through a profound and inclusive view of humanity, not because humanity is considered to be superior but because humanity exercises unique and extraordinary capacities to enhance or harm everything it touches whether of a physical or moral nature.
In the perspective of Human–Computer “Interaction”, the consequence of human significance not fully considered is an imbalance in the synergy between humans and computers. To achieve high-level wisdom in solving complex real-world problems, we assert the need to go beyond current paradigms (e.g., Human-Centered Computing) to a more synergized interaction paradigm between humans and technologies. We chose the word “synergize” as we deem that the tight integration between the two “components”, human and computer, can produce more far-reaching effects than the sum of their individual effects.
Thus, we propose a conceptual framework for HCI—Human-Engaged Computing (HEC) (Ren 2016). HEC aims to motivate HCI outcomes toward synergized interactions between engaged humans and engaging computers for high level wisdom which enhances our human survival probability and our full potential as humans (see Fig. 1).
HEC seeks to enhance the pertinence of HCI by promoting researcher and innovator consciousness of human significance. This is achieved by consciously referencing the significance of HCI to questions like “What does it mean to be human? ” and “What is the role of technology in affirming human significance”. HEC does not seek to establish that significance but to recognize and respect it as found in respective groups. In this way, HCI and digital technological sectors in general may more naturally honor the variety of human traditions, and avoid evident trends toward technological despotism while at the same time making its own work more relevant and thus more marketable in newly appreciated demographic sectors.
2 Situating HEC
This section situates HEC in relation to current frameworks and concepts producing a view of HEC in the general context of technological progress.
2.1 The four waves of HCI and the future
The history of HCI can be considered in four waves. The first wave focused on human factors in a fixed context (Bødker 2006; Harrison 2007). The second wave was influenced by Card and Moran’s book on “The Psychology of Human-Computer Interaction” (Card et al. 1983) where cognitive science (memory, perception, and motor control) were the main themes. The third wave was influenced by the notion of Human-Centered Computing (HCC) introduced by Kling and Star (1998) in 1997. Since then, HCC has evolved to contain multiple research foci, such as addressing the social (e.g., CSCW) and emotional aspects (e.g., user experience) of human computer interactions (Rogers 2012). The fourth wave is governed by two fields: positive psychology (well-being beyond humans defined as needy) and cognitive neuroscience (human irrationality), which leads to calls to consider an even more “human-centered perspective” (Bannon 2011) such as human values (Borning 2012) and ethical aspects (Rogers 2012, embodiment (Kuutti and Bannon 2014; Harrison 2007), and well-being (Calvo and Peters 2014).
We agree with Jaworski (2012) who writes: “I gave serious consideration to the Western scientific-materialistic worldview—our underlying belief system, which has prevailed in the West for over two hundred years. I believe that this belief system is no longer adequate for the issues our society is facing; that an historic shift is now occurring; and that a more comprehensive worldview is emerging.” One emerging trend is the recent surge of interest in human wholeness, well-being, mindfulness, eastern perspectives on contrasts and compliments by contrast with comparisons and opposites. Such perspectives have received a lot of attention from big tech companies such as Apple, Facebook and Google [e.g., see Wisdom 2.0 Wisdom (2014) and Wisdom Stockholm conferences Stockholm (2015)].
Thus, we need to rethink the current HCI paradigm. HEC shares similar sentiments with the current (fourth) wave of HCI towards a more holistic consideration of human activities and the human context. It aims at establishing synergized interactions (or synergism) between engaged humans and engaging computers for high level wisdom enhancing our human survival probability and human potential beyond mere survival. It considers insights from Eastern philosophies and world views such as wholeness, the right (also called optimal or golden) balance, and “antibiosis”, i.e., the adverse effects of imbalanced considerations between humans and computers. Importantly, HEC is a philosophical tool by which HCI can view and asses itself and its traditions from the outside. HEC can be described as the promotion of a greater HCI emphasis on human engagement, where human engagement includes the explicit development of human capacities and their conscious application in HCI design and interactions.
Pioneers like Licklider and Douglas Engelbart seem to have had no intentions of displacing or neglecting human potentials, human responsibilities, human motivation or human guidance with regard to machine functions. As Licklider (1960) wrote in his article: “Men will set the goals, formulate the hypotheses, determine the criteria, and perform the evaluations. Computing machines will do the routinizable work that must be done to prepare the way for insights and decisions in technical and scientific thinking”. Despite the now politically incorrect use of the noun ‘men’ and the exponential growth in computing capacity and innovation, HEC regards this emphasis on the moral aspects of human engagement as vitally important to human significance and thus to the relevance and viability of the field. This mirrors Douglas Engelbart’s statement (Institute DE 2013): “You have to deal with both sides of the whole organization, i.e., the people and the machines, we need to find a way where both sides are going to co-evolve. The human system side has just evolved organically without explicit attention. Then when technology started to erupt with explicit focus, it was driving the world and the rest of it followed”.
Thus, the conscious development and synergism of both the Human and the Computer aspects of HCI can lead to more harmonious, efficient and effective output that will not only enhance human survival prospects but contribute to more positive development.
2.2 Human-computer symbiosis
An even earlier insight about intelligence and “man-machine symbiosis” comes from Wiener (1948). He thought humans should be in the loop in control systems: “Many people suppose that (computing machines) are replacements for intelligence and have cut down the need for original thought.... This is not the case...”. Human-Computer Symbiosis (Licklider 1960), Augmenting Human Intellect (Engelbart 1962), and Human Computer Integration (Farooq and Grudin 2016) also envisioned humans and computers working together to solve problems. The terms ‘symbiosis’, ‘augmentation’ and ‘Integration’ are closely related to the HEC notion of “synergized interaction between humans (H) and digital devices/computers(C)”, i.e., humans and computers working together according to their respective properties and according to their respective significance and considered impact on human stewardship.
HEC redresses the neglect and diminishment of the human contribution in past expressions of “symbiosis” which have resulted in adverse effects on human significance, the atrophy of human skills and perceived losses in terms of the value of human effort. Where such adverse effects on humans are not taken into account, symbiosis is seriously compromised and the usefulness of HCI along with it. Symbiosis, as applied in traditional HCI thinking is like ‘Yang’ divorced from ‘Yin’. In more balanced perceptions (e.g., biology), the term “symbiosis” includes due consideration not only of the positive potentials but also of the negative impacts of the symbiotic relationship. In the light of these considerations, we introduce the notion of “antibiosis” as an essential aspect of HCI development, research, evaluation and design.
By including “antibiosis” within the philosophical domain of HEC considerations, we intend to expand the researcher/developer’s consciousness beyond the internal relationships of traditional HCI into ever more practical human pertinence.
2.3 Human-centered perspective
There exist a number of conceptual themes in related studies on human-centered perspectives of HCI. Examples of these include HCC (Kling and Star 1998) which focuses on developing technologies that better support user activities, and value sensitive design (VSD) (Borning 2012) which focuses on developing improved technologies by understanding user values. Although these viewpoints are human-centered, they primarily focus on how to develop better technologies using a human-centered approach.
By contrast, HEC focuses on a wider ecosystem or context including humans, technologies, and nature or existence in general; this is towards not only making better technologies but also focusing on the meaning behind the system’s existence. HEC does not impose meaning on individuals or groups, it seeks to acknowledge, respect, integrate with and ‘exploit’ the respective meanings that are treasured by diverse individuals and groups. HEC focuses on how to develop engaged humans whose capacities are activated, engaged and progressively enhanced, and how to develop engaging computers whose role is to enhance and complement human capacities in their significant contexts. HEC intends to maintain full and explicit attention to both human capacities and technological capabilities, and most importantly, to develop interactions with the right balance between them according to human priorities. We believe this is the key to effective synergized interaction and to the more efficient realization of HCI’s developing potential.
2.4 Positive computing
Positive computing (Calvo and Peters 2014) focuses on developing technologies to enhance human well-being. Positive Computing and our framework share similar views on human potential and development. While positive computing is a perspective of technological design, HEC is a high-level philosophy that consistently asks What is the essential existential meaning of humans? and How do technologies support humanity defined according that meaning? Understanding that meaning is variously nuanced, HEC inspires adaptable dynamic philosophical reflection for HCI research and development and opens new dimensions in innovation that are pertinent to groups and individuals.
2.5 UX and human engagement
Drawing on the concepts common to the UX frameworks (Harbich 2008; Jordan 2000; Laurel 1993; McCarthy and Wright 2004; Norman 2003; O’Brien and Toms 2008), past works have studied engagement from the perspective of appearance, emotions, and meanings. The growing recognition of engagement studies can be attributed to three global changes in the past decades. First, our common assumption that humans are essentially rational has changed as researchers have proven otherwise (e.g., Ariely 2010; Kahneman 2011). In HEC, this does not mean that humans are essentially irrational; it means that humans are “not only rational”, that values, actions, decisions and plans derive from all aspects of the human constitution. This paradigm shift has had a radical impact on research and industry where emotion, motivation and decision-making attract a lot of research and industrial attention. Second, problems such as deficiencies in motivation and attention span (Carr 2011), due to the prolonged use of technology have had a significant effect in many areas of human interaction. The diminished capacity of humans to sustain engagement for its own sake and on the basis of necessity alone makes our call for the study of mature engagement the more pressing. Third, the world is facing many problems (e.g., sustainability, poverty, education), some of which significantly impact human survival (Shneiderman 2011). There are calls for new trends and futurists (Randers 2012) predict that humans will face an unsustainable future if no action is taken. These trends drive the emergence of interest in developing human capacities in many areas and we believe that the health and prosperity of the HCI sector are dependent on new and more responsible trends guided by an overarching adaptable philosophical rationale accompanied by ongoing dialogue across many disciplines.
“Human Engagement” in the context of HEC refers to a fully engaged human exercising their full capacities. In that sense, the traditional notion of “user engagement” is quite different to our notion of human engagement. The HEC notion of human engagement is also related to the concept of flow (Csikszentmihalyi 1990). In such a flow state, the user is exercising his/her full potential and is absorbed in the task with undivided attention and is thus free to give maximum expression to the capabilities and capacities of the user and the device in that particular task.
The HEC notion of human engagement would also require practitioners to consider the particular meaning and significance to users of their religion, philosophy, habits, aspirations, culture, biological properties wisdom and perception of the world, and so on. Individual differences may become increasingly significant when we consider human engagement.
2.6 HEC looking at HCI from the outside
While HCI is an interdisciplinary domain drawn from sociology, psychology and other such fields, HCI researchers have their own paradigm in the HCI community (Kuhn 1962).
Figure 2 describes HEC through a philosophical lens which lies outside HCI but also surrounds HCI and infuses it with potential meaning. It shows that current interaction output is limited by the current narrow vision which is dominated by technical efficiency and novelty, and overly biased towards convenience and ease-of-use. Through more holistic considerations, total environmental awareness and reconsideration of human capacities, potentials and engagement, HCI and design output will be greatly expanded with exciting new dimensions and directions.
The HCI field cannot derive its highest meaning, greatest significance or greatest impact by reference to itself, its traditions or any ‘ungrounded’ goals; it derives meaning from its context and its relevance to the ‘systems’ and environment that it serves which are essentially distinct from its own traditions. This is described by the law of innovation of Bill Joy (co-founder of Sun Microsystems) which states that “Innovation will occur. It will occur elsewhere” (Goldman and Gabriel 2005). This means that the institution that cannot reform its traditions by reference to its purpose in wider existence will lose its innovative relevance. Relevance requires external, contextual meaning from which any institution may derive, adjust, discipline and correct its own meaning, significance, theory and practice. Just as “no man is an island”, so also each and every entity derives meaning through its relationship to others and to existence as a whole. This is also the case with any discipline, practice or institution, including HCI; its usefulness and sustainability derive from its respect for the human context that gives rise to it.
2.7 HEC values
Responding to problems and limitations in current interaction, four core values emerge and help to define priorities that HEC values: (1) synergy, (2) balance, (3) wholeness, and (4) improvement of improvement.
Synergy HEC considers that synergized interaction between humans and technologies can produce amazing and powerful results. Technologies alone cannot solve social problems because they lack subtle human factors. On the other hand, HEC considers that humans alone cannot solve social problems efficiently because humans cannot scale well. By pairing humans who possess many tremendously powerful capacities, and technologies which excel in efficiency and scalability, they together can achieve incredible results.
Balance HEC considers that it is important to search for the right matchup between human capacities and technological capabilities. In the current situation, there remains a tendency to focus on devices and interaction techniques more than on human capacities, the consequence of which is to introduce a detrimental imbalance between humans and technologies. In other words, what currently hinders the development of greater synergized interaction is the imbalance between humans and technologies because synergized interaction implies the full development of both humans and technologies, without neglecting either. Restoring balance through the fuller consideration of humans will be necessary for the realization of greater synergized interaction. The concept of balance in HEC is nothing new, because the ancient Chinese Chung Yung or Doctrine of the Mean represents the value of balance (Fung 1997), and many works have shared a similar view regarding the importance of balance [e.g., (Haidt 2006)].
Wholeness HEC considers that it is important to adopt a holistic view of the whole of nature and the whole human race both as individuals and collectively. HEC first asks what humanity is and what is its place and function in the nature of things. A holistic approach should include an understanding of the body, mind and the spirit, and acknowledge that non-material needs are as important as material needs. We talk a lot about intelligence and efficiency, but less about other soft-skills (e.g., mindfulness, self-control, empathy, responsibility) which are equally important in determining successful outcomes.
It is important to note that a truly holistic approach implies organic unity and organic functionality, i.e., it does not treat human beings, humanity or existence as assemblages of attributes, properties or affordances. HEC seeks to context analytical and deconstructionist methods within as broad a sense of existence as possible while understanding and respecting the fact that humanity is pluralistic in expression. Harmony does not exclude pluralism; true harmony demands it.
Improvements of improvements HEC regards that mere improvement is not sufficient. Rather, HEC seeks to understand how to sustain long-term improvement, i.e., to understand how we can improve the way we improve, a notion pioneered by Engelbart (1962). Based on this notion, one distinguishing feature of HEC is not just about the development of interfaces, but more importantly, the development of the mindset of researchers and designers. The notion is that better people make better researchers which make better interfaces which make better interactions which make better outcomes for the whole of nature. To support the development of this mindset, an initial set of principles and values is needed. HEC therefore seeks to improve our approaches to improvement. HEC accepts that “improvement” is never “finished” but that the way we improve, innovate and make progress can always be done better, more consistent with human significance, human potential and human habitats.
3 Human-Engaged Computing
Figure 3 shows the overview of HEC. It describes the synergized interaction between high-level human capacities (engaged humans) and high-level technological capacities (engaging computers). In order to reach truly synergized interaction, we should maintain a good proportional balance between humans and computers. In fact, we argue that there is a current imbalance between humans and computers in favor of computers and to the atrophy of human capabilities and human well-being.
We describe HEC in a framework from five perspectives as follows: (1) definition, (2) components, (3) principles, (4) case studies, and (5) benefits. Each of these perspectives offers a different way of viewing Human-Engaged Computing.
4 Perspective 1: definition
The definition of “Human” in HEC refers especially to human capacities and capabilities in the context of human significance within the whole ecosphere but also taking into account individual and respective group self-significance.
Sen’s formulation of human capability contains two parts: functioning and freedom. Functioning refers to the various things a person may value doing/being or have reasons to do/be, e.g., being happy, being moral, being creative, doing a good job. Humans require freedom to achieve these various things no less than to achieve in matters of simple mechanistic utility. This freedom has two aspects – the ability to act on what matters (one’s goal/objective) and the opportunity to choose to act from among various good possibilities (choice). In other words, human capabilities provide the opportunity and the ability to produce personal and respectively relevant functioning according to individual and group distinctives. This approach includes and promotes what humans are capable of, what humans want to be capable of, what humans should be capable of and what humans choose to be pre-emptively and presumptuously excluded from.
“Engaged-” refers to the engagement and activation of human capacities (e.g., mindful attention in a task, flow). It can be described as a high, advanced level of experience where users are deeply absorbed and involved in the task at hand with undivided attention. During this state, one would perform to their maximum potential with little or no awareness of fatigue. Inner capabilities such as subjective learning efficiency, creativity, and problem-solving capability are maximized. Furthermore, engagement is multifaceted and situated (Winograd and Flores 1987), composed of bodily actions, emotion, and thinking (Dewey 1934). The notion of flow experience (Csikszentmihalyi 1990; van Schaik et al. 2012) can be used to describe the extreme level of engagement, in which a person is completely involved in a particular activity. In such a state, the participant does not even make conscious judgments about the degree or quality of his/her involvement; he/she “just does it”.
Past studies indicated the clear association between Need Satisfaction and engagement (Desmet and Pohlmeyer 2013; Hassenzahl and Tractinsky 2006; Sheldon et al. 2001). The Need Satisfaction approach argues that humans are driven by intrinsic needs, in which, when these needs are fulfilled, one would become engaged and intrinsically motivated (Desmet and Pohlmeyer 2013; Hassenzahl et al. 2010; Sheldon et al. 2001; Van Schaik and Ling 2012). The Needs Satisfaction approach makes a fundamental assumption that by satisfying high levels of human need which are enduring (never-changing, always present) across tasks and activities, one would achieve high levels of engagement. Some such needs include the need of pleasure(s) (Hassenzahl et al. 2010; Vorderer et al. 2004), the need of well-being (Desmet and Pohlmeyer 2013), the need of competence, autonomy and relatedness (Deci et al. 2000), and the need of self-actualization and purpose (Maslow 1943; Pink 2009). HEC assumes a more fundamental human capacity which is the neglected potential to engage out of mere necessity and to find satisfaction in the state of unmediated task engagement itself. In other words, engagement is itself the optimum state of satisfaction which can be cultivated for its own sake when attention to the task is not divided by irrelevant incentives and added distractions.
“Computing” in HEC refers to high-level technological capabilities which are developed with sensitive reference to humanity and the ecosphere as described above. The result of such a synthesis will be more highly synergized with productive interaction, greater relevance, more sustainable outcomes, more wholesome outcomes and more enduring trends.
Figure 4 situates synergized interaction with the current interaction paradigm. In the current interaction paradigm (Fig. 4, top), the larger proportion of input comes from computers, while the input channel from humans remains relatively small and under-developed. Synergized interaction seeks to restore balance through the fuller consideration of human capacities. Figure 4, bottom shows the synergized interaction paradigm where there is an optimal match between humans and technologies which together create synergized interaction.
Technology is most effective when it is synergized with capable, engaged human forces. As a prominent technologist who was associated with over 50 projects relating to various social problems, (Toyama 2015) stated that technologists often overlook the fact that receptive, capable and engaged humans are needed for any technology to be fully exploited and fully beneficial to users.
Synergized interaction in HEC includes consideration of Eastern insights into the nature of humans, humanity and existence as a whole, especially with regard to meaning, the relationship of discrete ‘things’ in existence considered holistically and other aspects that might be referred to as ‘noetic’ and culturally specific in nature.
HEC would therefore reflect certain notions of Eastern philosophy, such as the Yijing (or I Ching) theory of complementary partnerships rather than opposed forces, and the Chung Yung theory of ‘golden balance’ (Fung 1997). If Humans (H) are seen as ‘Yang’ and Computers (C) as ‘Yin’, then their synergized interaction combines in a cooperative manner, H and C as a whole, and the (symbiotic) effect of the composite will be greater than the sum of its parts. H and C are complementary aspects, rather than conflicting. We cite these examples because they represent fundamental insights and deep cultural sentiments that influence a large proportion of the human population. If ignored in the conceptualization, development and application of HCI R&D and innovation, HCI cannot know its proper place, cannot offer its best service and will never realize ongoing market potential.
To achieve synergized interactions, the key is to consider the notion of balance. In HEC, balance means Chung Yung—that is, just right (Fung 1997, p. 172). Balance is not an equal amount of time, expense, investment, significance and importance given to both sides of a symbiotic relationship of unlike participants; balance is considered to be the independent consideration and equal proportional treatment of both participants (H & C) according to their kind. To achieve synergized interaction with true harmony, it is important to establish this kind of balance between H and C.
5 Perspective 2: components
HEC is comprised of three components as shown in Fig. 6: (1) engaged humans, (2) engaging computers, (3) synergized interaction.
5.1 Engaged humans
Engaged Humans refer to humans whose capacities are engaged and progressively enhanced by their engagement.
There are at least four levels of human capacities to be considered: (1) biomechanical (physical), e.g., motor control, (2) noetic (intellectual), e.g., memory, logic, problem-solving, (3) social, e.g., trust, empathy, morals, and (4) emotional/metaphysical, e.g., intuition, self-control, mindfulness, spiritual. These four levels of capacities need to be progressively understood, enhanced, activated and engaged. Difficulties in evaluating and defining aspects of human consciousness and experience cannot be allowed to disqualify those aspects from serious consideration in technological R&D and application where such aspects of human consciousness shape the cultures, values and experiences of individuals and groups. To ignore their influence is to ignore the market.
Engaged Humans operate at peak efficiency in terms of task accomplishment and the personal energy consumption required for the task. By contrast, the divided, distracted or only partially engaged human is deficient in efficiency regarding task accomplishment, quality and duration and wasteful in terms of personal energy consumption. We do not consider a human to be engaged if he/she is content (satisfied) with the abandonment of his/her mind, body, spirit, awareness, responsibility or personal development in the interaction.
It is important to note that our notion of engagement is much wider than the traditional notion of engagement which refers to the users’ time-on-task or clicks-per-page (Harden 2009), or attractiveness (Chapman 1997), or positive affect (O’Brien and Toms 2008), or task-dependence (Laurel 1993). The fullest human engagement, in our terms, refers to a state of consciousness where human capacities are fully developed and exploited, regardless of what tasks or activities are encountered in the moment. This implies that human engagement in its purest form is not or does not need to be subject to the nature of the task (e.g., fun or boring, difficult or easy, profitable or not) because mature and pure engagement is its own reward. The inner state of engagement, rather than any external condition, is itself the source of human satisfaction. In this state there is no judgment of the task either positive or negative, a state that is well known in eastern practices such as mindfulness meditation and the ‘no-mind’ marshal-arts of the east; in western psychology, this state is variously described as ‘flow’, ‘being in the zone’, ‘absorption in the moment’, ‘undivided presence’ and the like. In other words, pure human engagement cannot be produced by external factors such as unrelated incentives or secondary novelties that divide the mind by requiring value judgements that do not pertain directly to the fundamental task or its goal.
The degree to which engagement becomes dependent on secondary incentives is the minimum degree to which engagement in the task is destroyed; it becomes engagement in secondary incentives. This is immaturity and it diminishes the human’s ability to flourish under responsibility. The highest form of human engagement is ‘complete union with the task’. Motivation by secondary incentives supports the diminishment of the moral power of human engagement and of comprehensive awareness to robotic participation in someone else’s meaning and purpose. Real, direct and comprehensive engagement can be cultivated by e.g., the ‘on task’ application of cultivated mindfulness. However, many programs and applications militate against unmediated (direct) engagement. From an eastern perspective, the term ‘mindfulness’ does not signify ‘a full mind’ (e.g., of techniques and instructions) but fundamental emptiness, a whole presence and awareness that is fully sensitive to the conditions and requirements of the present situation which includes spontaneous application of embodied skills. These perceptions and states of consciousness are well known in the east, e.g., “The Unfettered Mind” (Soho 2012).
These observations are very significant because they are counter-intuitive to the practice of adding enticements which attract the attention and demand constant conscious but unrelated on-the-job calculations. Far from diminishing awareness and attention, unity with the task produces efficiency, the conservation of energy and natural, task relevant satisfaction. In fact, this kind of engagement is more likely to require a minimalist approach to design where distracting factors are removed rather than attractive factors being added. Such considerations are also important when we consider the wider detrimental effects, i.e., antibiosis consequences that may not be directly related to the task at hand or the particular app that is under development. For example, dependent engagement in secondary incentives may get a task done while diminishing a child’s capacity to “just do it” because it needs to be done. The ability to act directly upon necessity is an essential human survival skill.
Engaged Humans are also considered at the collective level. Collective engagement is shared human engagement such as teamwork or global cooperation where the workflow is most efficient due to group communication and characteristics like understanding, trust, common cause, intuition, spontaneous response and desire towards resolution of conflict and the like.
The effect of human engagement on synergized interaction is threefold. Firstly, without human engagement, human capacities cannot be fully activated. Second, without human engagement, powerful technologies will not be developed, realized or used to their full potential. Third, without the prior two capacities being fully developed, it is impossible to produce a highly synergized state where humans and technologies are tightly “integrated”, synchronized and fully functional. Without human engagement, despite powerful technologies and talented humans, these two factors will fall short of their respective potentials and will not reach their full capacity.
While engagement is usually considered relevant to those who use devices and interfaces, HEC considers that engagement is no less important for those who imagine, design and build new devices and interfaces. In fact, engagement is a requirement for life in general.
5.2 Engaging computers
Engaging computers are computers and devices (including software/systems) that honor, engage, and enhance human capacities. They are cost-effective and ecologically friendly. Engaging Computers draw humans into focused attention by eliminating distractors from, e.g., interface design. The merits of technologies in HEC are evaluated by their capacity to engage human capacities with a view to human survival and human enhancement, i.e., human capacities should be enhanced not diminished or displaced. Augmentation needs to preserve and enhance human significance and human skill.
One key challenge in the development of engaging computers is to properly account not only for first-order positive effects but also for potential and actual negative second-order effects. The notion of ‘engaging computers’ asks “What should we do if one technology increases human capability in one field but decreases other capabilities in other fields? ” and “Are we sufficiently aware of such effects?” For example, using games can improve immediate learning for children, but children may slowly lose their natural capacity to self-motivate and to be content with simple absorption. HEC recommends the appropriate apportioning of attention to human and technological input and capacities on a case-by-case basis such that users experience interactions as whole persons or at least without the diminishment or atrophy of capabilities and potentials that are necessary for human maturity and fuller human development.
5.3 Synergized interactions
The term, “synergized interactions” refers to effective synergy between engaged humans and engaging computers. In other words, synergized interaction refers to effective task performance accomplished through the seamless integration of human capacities and device affordances performing as a single entity, with single focus and common purpose.
Synergized interaction can be initiated by the features and benefits of computers, by interfaces and software (e.g., video games) that engage humans to do required or desired activities/tasks, and also by human perceptions such as need, pleasure or necessity, factors which cause them to search for appropriate tools to achieve their goals, satisfy their needs or satiate their desires. It is also possible that synergized interaction may occur through a combination of these.
In order to achieve synergized interaction, an appropriate balance must be achieved, i.e., proportional attention according to the nature and priority of humans and the nature of devices. (1) Computers should not diminish the effectiveness and significance of what humans already have and what humans can do, (2) the respective capacities of humans and computers should be blended and exploited in such a manner that the most efficient outcome can be achieved in the particular task while human integrity is maintained and perhaps enhanced, (3) computers (devices) can be designed not only to meet human needs but to positively enhance human capabilities and capacities, i.e. beyond human need and into the range of human potentials and possibilities.
Balance is considered to be disrupted when computers displace what humans are, have and can do because of a lack of consideration to the priorities of the case. One good example is automation. In her “Ironies of Automation” Lisa Bainbridge (Bainbridge 1983) noted that even if it may seem that automation is making workers more productive, second-order effects may be that users lose motivation to use technology, unlearn their skills, and avoid responsibility. The Accenture survey of January (2016) may well confirm such a finding where one of the consequences of possible over indulgence in personal tech devices seems to be boredom with those very devices and the subsequent demand for ever more novel features. Furthermore, even as computers are becoming ubiquitous, more productive and smarter, the sad side-effect is that many users are becoming less proficient in social and noetic skills as a consequence. Not only is this unhealthy for those users, it is also unhealthy for HCI because synergized interaction cannot be fully achieved if humans are dumbed down to any extent at all.
Balance can be achieved when humans and computers promote and complement one another in true holistic synergism. For example, it is often difficult for scientists to tell from a gene sequence what the final protein will look like. Using serious games, Cooper (2011) asked the human participants to arrange the molecules using their exploratory and creative capacities, while asking the computers to take care of all complex calculations. By such cooperative working, the synergy between non-experts and computers outperformed pure super-AI. Another example is PauseAble (Cheng 2015; Salehzadeh et al. 2017), a recently developed, popular application for mindfulness training. Using embodied interaction, it asks users to relax by putting a finger on the touch screen and mindfully tracking a moving icon with their finger. The application is effective because the computer only detects the users loss of focus and gives feedback to that effect. The app is significant and especially pertinent because it specifically encourages, exercises and enhances human attention. The transfer of implied mindfulness principles to all app designs is an important aspect of HEC practice.
6 Perspective 3: principles
HEC enables new and broader criteria for evaluating HCI research and development based on its goal of developing fully synergized interactions between fully considered human and technological capabilities respectively. HEC recommends three design principles that can help redress neglect of the human aspect of the HCI syntheses: (1) thorough consideration of all human capacities, (2) identification of all technological hindrances to human expression, growth and capacities, (3) development of ways to trigger, enhance, exploit and, where necessary restore human capacities.
6.1 Consider all human capacities
We have to comprehensively define, evaluate, prioritize and exploit all individual and collective human capacities. We cannot properly augment humans when we have not properly understood them. By understanding both latent and active human capabilities we will also be able to identify and more accurately determine the extent to which they have been compromised and suppressed, how and why this has come about, what we can do to restore them via more nuanced design principles and guidelines. Each individual and composite capacity should be fully defined, evaluated, and integrated into HCI R&D practice.
The full consideration of human capacities should also cover the human capability approach proposed by Sen (1999) concerning functioning and freedom, as described in Sect. 4. Furthermore, even if we suppose, as some do, that human meaning is no more than a construct of the human imagination, the fact is that such meaning and many various expressions of meaning move whole cultures and great masses of people to act in multitudes of ways that are considered sacrosanct by those groups and they should therefore be considered as very significant factors in the evaluation of all technological innovation. People are, after all ‘the market’ and what matters to them must be of primary concern and interest to designers and innovators.
Considering the human side, we need to comprehend the following – (1) diminishing human capacities, (2) underexplored human capacities.
Diminishing Human Capacities. Human capacities have been dramatically diminished and this trend is expected to continue unless something of the nature of HEC is inaugurated. Our world has entered an astonishingly fast-paced era where it is difficult for many people to keep pace with the changes and where our experience of time itself has changed. This experience is still unfamiliar to the vast majority of humans. As a consequence, human capacities have been somewhat superseded, compromised, atrophied, and otherwise neglected. These negative effects are well documented as evidenced by the general decrease in the human attention span (Carr 2011; National Center for Biotechnology Information 2014), the fact that our brain mass has shrunk measurably (Hawks 2010), and human motivation and initiative have not improved (Bernard et al. 2005).
Such atrophy has huge implications for society. Its impact is most apparent when considered collectively, i.e., each instance of atrophy leads to the atrophy of collective human capacities, which leads to the atrophy of collective synergized interaction that is essential to address many existing problems such as pollution, sustainability issues, or even peace.
We are often individually ignorant of the fact that this is happening. This may be because technology makes us seem “smarter” even as our capacities are actually diminishing by neglect in favor of device functions. Furthermore, diminishing human capacities may be difficult to identify because their causes and effects are distant in time and space, i.e., they are not noticed during the actual moment of interaction. While we are occupied in interactions we are not aware of the fact of capacity and capability atrophy much less what causes it. If this continues to be the case our natural capacity to think mindfully, to evaluate, to assert ourselves and to feel empathy can be expected to decrease; this will further hinder the development of more deeply synergized interactions (see Fig. 7). Greater synergized interaction requires explicit attention to both the Human and the Computer aspects of HCI practice (see Fig. 8).
Underexplored human capacities Humans possess many powerful inner capacities which traditional interaction has neglected and preferred to displace rather than exploit. Traditional interaction limits human capacities to perception, cognition and motor control (Card et al. 1983). Fuller synergized interaction can only be achieved when all human capacities receive proper attention. A primary motivation for the proposal of HEC is to greatly enhance device development, to expand its potential via fully developed positive symbiotic relationships with the whole human entity.
6.2 Identify all hindrances to human capacities
The removal of hindrances to human capacities along with respectful and proper cognizance of cultural and individual differences and distinctives are essential considerations towards greater synergized interaction and output. By revisiting such hindrances we may glean more mature and productive insights and guidelines towards better design and innovation.
It is essential to synergize technological capabilities via a fuller appreciation of what human beings are, what they can contribute to the innovation and what is required to maintain and develop their humanity with all its functioning freely available and accessible.
It is said that “Every technological innovation comes at a human cost” but does this really have to be the case. HEC says, “No it does not have to be the case”. Human capacities are often displaced by devices which can do the work for us (Carr 2011; Goleman 2013; Hawks 2010). For example, our natural ability to remember things diminishes because of hard-drives and digital address books, certain of our interpersonal abilities and conversational skills are diminished by digital social networks, and our ability to focus is hampered by many artificial distractions and task-irrelevant incentives.
For instance, we have commonly used technologies to automate tasks in order to achieve greater efficiency and higher levels of productivity. This did not necessarily lead to greater human outcomes and in some cases it diminished human capacities and banished many skills. This bias prevents the development of better computing tools that could better augment human capacities at the individual and collective levels and both these realms are necessary for the resolution of truly difficult human problems, i.e., augmentation over automation (Norman 2009).
As another example, digital games may help children to engage and learn on one level (Gee 2003), but there may be a negative impact that slowly erodes the child’s capacity to engage at a more subtle and profound level, e.g., the child may slowly lose human capacities to self-motivate, to think clearly or to concentrate, etc. This is not to say that games are not effective, but designers should consider how to mitigate negative impacts on human capacities, and not regard the achievement of immediate and limited goals as adequate in and of themselves. This is crucial because students may gain some specific skills, but lose other more important ones, e.g., human capacities that are essential for the whole of life, or those that are essential for solving truly difficult problems, or those that are essential to perform tasks when devices are not available.
HEC recognizes that such costs may be necessary as trade-offs in some circumstances, but HEC recommends that all such trade-offs should be fully considered by both designers and users so that responsible decisions can be made. Technologies should not hinder the activation and development of human capacities and measures may be taken to compensate for losses. Such considerations promote the appropriate and parallel development of both human capacities and technological capabilities, which can lead to seamless synergized interactions towards healthier outcomes for individuals and for society.
Interaction output cannot be maximized via the current narrow vision which is dominated by infatuation with technical potentials and novelty and while it is overly biased toward convenience and ease-of-use at the expense of human well-being. On this point Doug Engelbart is famously attributed with the observation that users of new technologies ought to expect there to be a learning curve in just the same way as when we graduated from riding tricycles to riding bicycles. Through more holistic considerations and awareness and a reconsideration of human capacities, potentials and engagement, including the capacity to learn and to adapt, interaction and design output will be greatly expanded in surprising new directions.
Technology may be considered morally neutral, however the fact that humans add moral (or immoral) status to the use of devices is a matter of primary importance because it reminds us that digital artifacts do not exist in a vacuum; they are applied by morally active entities and according to various perceptions of morality and immorality. Devices are exploited by moral beings for good or for ill. Corruptly motivated people do bad things with morally neutral devices. Thus we may consider the implications for digital design in the context of human responsibility as individual users, as social beings, and as global citizens. Bardzell (2009) offers a similar notion: “Note that ethics is all but irrelevant if a design is just a tool, because ethical agency is situated squarely in the user. But if designs persuade people, or reshape everyday life, they can in that limited sense be understood to exercise agency and have an ethical dimension.”
When technologies are designed with holistic consideration of humans in mind, they can have an impact in support of all human endeavors whether solving difficult problems or adding to the positive qualities of life.
6.3 Identify ways to increase human capacity and capabilities
As all human capacities and their hindrances are progressively identified and evaluated, it will be important to determine ways in which technology can activate and enhance inner human capacities along with motor skills.
To increase human capacities, it is necessary to understand how enhancement happens so that we can improve our ability to improve as proposed by Engelbart (1962). This principle is quite counter-intuitive, i.e., instead of focusing only on specific problems where we typically narrow down the problems, breakthroughs come in the other direction, i.e., by broadening the perspective to account for an even wider context.
Instead of targeting specific human capacities, it is essential to improve the human ability to improve human capacities as whole persons. We contend that human engagement, being the exercise of all appropriate and required human capacities in the task at hand leads to the spontaneous enhancement of human capabilities overall. By consistently improving our ways to engage humans, human capacities are naturally maintained and enhanced.
If humans are not engaged because technologies are not designed to honor and engage them for who they are and what they offer, greater synergized interaction will not be achieved and the outcomes will be debilitating to a greater or lesser extent. Thus, the goal is whole conscious engagement brought about through device design that understands humans in their living contexts. Reconsideration of balance between H and C will bring with it many guiding principles for design, especially in terms of priorities in design, but also regarding far wider and greater functional possibilities and aims. When technologies are designed with holistic consideration of humans in mind, they can have an impact in support of all human endeavors whether solving difficult problems or adding to the positive qualities of life.
Neuroscience has found that our brain is malleable and plastic, i.e., our brain is consistently changing its circuitry in response to various experiences (Chaney 2006). This neuroplasticity suggests that our various human capacities can be enhanced via appropriate use, training and intervention, but it also means that inappropriate application or the lack of use can stunt growth and diminish the possibilities. Nevertheless, technologies can be designed that enhance and expand human capacities, e.g., cognitive abilities, focus, mindfulness, motivation and so on. When technology is designed appropriately and intelligently, it can supply cost-effective, scalable solutions and increase real human significance and human efficiency.
7 Perspective 4: Case studies
We present three HCI case studies showing how enhancement of human capacities can be achieved: (1) awareness, (2) motivation, and (3) recognizing human potentials (see Fig. 9).
First, we can enhance human capacities by raising knowledge and awareness. Engagement can be frustrated due to a lack of understanding and knowledge. Technologies can be used to provide biofeedback to and about the user (e.g, quantified-self), or feedback on the relevant environment (e.g., energy consumption). When humans become more self-aware of their behaviors and environment, they can take action to improve and compound the enhancement of their capacities. Mindful apps such as PauseAble (Cheng 2015; Salehzadeh et al. 2017) can directly train humans to focus and sustain their attention for longer periods.
Second, we can also enhance human capacities by increasing motivation and willpower. Engagement may be obstructed due to a lack of willpower. Technologies can be designed to make the path to achievement of a task clear and efficient. When a technology assists a human to be productive and to complete valued tasks efficiently there arises a sense of fulfillment and self-worth. Technologies can also be used as motivational tools to engage humans, e.g., using the motivational elements of games can enhance motivation to learn programming (Sims et al. 2014).
The first two approaches, i.e. awareness and motivation, indirectly enhance human capacities through inner engagement, i.e., humans have the ability to engage their own capacities when they have adequate understanding and a desire to engage in valued tasks.
Thirdly, we need to specifically “re-cognize” human potentials. Latent, dormant and neglected human capabilities can be vivified, revivified and energized when wisely supplemented by technological affordances. Some human capacities and human methods of enhancement are naturally limited, regardless of knowledge or motivation. In such cases, technologies can facilitate enhancement processes. Human speed, accuracy and reach are traditionally considered to have upper limits, though even these ‘barriers’ may be challenged in the future. In the meantime and by way of example, crowdsourcing enables speedy human communication and sharing among great numbers of people; this reach and efficiency cannot currently be achieved without the aid of technologies. These platforms allow humans to collectively engage as large groups of users for important purposes. Nevertheless, while technology currently supplements perceived human limitations the industry needs to allow for potential expansion of the innate human skills and potentials bandwidth.
8 Perspective 5: benefits
As noted earlier, the output of synergized interaction will be greater than the output of humans or computers considered respectively, and also greater than the output of the sum of these contributing parts. Due consideration of both will lead to the fullest realization of human and technological potentials. In general, the identification and exploitation of human and device engagement factors for HCI will enhance all occasions and spheres where digital devices are designed to augment human activities towards worthy goals. The effects of engagement that are applied through devices will naturally enhance and influence associated fields of endeavor because tasks that are approached with deep engagement will be achieved more efficiently.
At the business level, by integrating engagement features in product and device design, higher customer satisfaction and higher commercial demand may be expected. At the social level, tasks/goals that require high-level engagement, such as education, creativity-based tasks and personal development training would be accomplished with greater enthusiasm and stronger motivation. On a global scale, applications/interfaces that aim to address social problems will also benefit from their engaging functionality.
9 Evidence of HEC in current research
We have shown some related work on frameworks and concepts relating to HEC. We now present other similar efforts from different domains which aim to reach the same goal of understanding and exploitation of engaged humans and enhanced human capacities (Fig. 10).
9.1 Human wisdom
There is increasing interest in the field of wisdom and human development (e.g., see Wisdom 2.0 Wisdom (2014) and Wisdom Stockholm conferences Stockholm 2015). Another significant trend is Collective Intelligence (MIT 2015) proposed by MIT, which tries to answer how we can work together such that we are collectively more intelligent than any one person. However, intelligence is only one small part of all available human capacities and collected and collated information does not necessarily equate to wisdom. It may just as readily be more aligned with despotism and ideological impositions.
Futurist studies predict that the future could be structured by mathematical models. Other futurists (e.g., Houle 2012; Randers 2012) expect more attention to be paid to human consciousness, human development, and human purpose.
9.2 Human augmentation
Human augmentation (Alicea 2018) refers to technologies that enhance human capability. Often these technologies refer to biotech and nanotechnology such as cochlear implants that enhance our senses or limb devices that enhance muscle capability. Human augmentation can also broadly refer to any technologies that expand human capabilities including human data analytics that provide information about the body, brain-computer interfaces, AR/VRs.
In the view of HEC where human capability is enhanced through a synergistic synthesis between humans and machines, human augmentation has the potential to empower and improve human health and quality of life. In cases where the augmentation is needed due to weakness in a human capability, the adoption of augmentation should be considered and accounted for in the interest of the development of the human factor.
Nevertheless, HEC takes a long-term, human-first, cautious approach. HEC insists that all enhancements be done with a good reason being sure to allow for the expansion and strengthening of human capabilities. Any augmentation that comes with a tradeoff which diminishes other human capabilities or for any other material purpose should be questioned. For example, “Does such augmentation displace or diminish human potentials? ” or “Before we do apply of these augmentations, is there any other way that could lead to a better and more sustainable outcome? ” HEC, in this regard, is about orienting the work in favor of human wellbeing and significance, and not materialism, productivity or efficiency only.
9.3 From super AI to human-engaged AI
One major trend is Super-Intelligence (Super AI) where “The Singularity is Near” is touted by Kurzweil (2006). While some are very excited by the prospect of this “singularity”, many great minds of the digital revolution (e.g., Hawking, Gates, Musk, Wozniak) have expressed serious concerns about the impact of Super AI on human safety and human significance. We think that super-AI is most useful when full attention and appreciation of both human significance and technological affordances are well accounted for and when human well-being is explicitly understood and applied as the priority in the synthesis. A CHI Panel concerning integration versus powerful tools (Farooq et al. 2017) has discussed related issues. Following the HEC framework (Ren 2016), “human-engaged AI” has been proposed (Ma 2018).
9.4 Video games
Video games have received much research attention due to their ability to engage users deeply. Serving as experimental tools, different factors of engagement in digital games have been studied, e.g., the concept of fun (Hunicke 2004; Lazzaro 2004), immersion and presence (Calleja 2007; Ermi and Mäyrä 2005; Jennett et al. 2008), emotion (Freeman 2003; Lazzaro 2004), embodiment (Bayliss 2010; Benford and Bowers 1995; Gee 2008), effects of various types of input (Birk and Mandryk 2013; LaViola and Litwiller 2011), cognitive enhancement for the elderly (Anguera et al. 2013; Niksirat 2017), as well as understanding how different types of users engage differently (Heeter 2011; Jenson et al. 2007). We believe some of these findings can be synthesized and adapted to general engagement principles.
9.5 Product design, behavioral economics, neuroscience
Product design has a strong influence on practical engagement theories, e.g., Norman’s emotional model (Norman 2003), Jordan’s pleasure model (Jordan 2000), McCarthy and Wright’s felt experience model (McCarthy and Wright 2004), and Hassenzahl’s UX model (Hassenzahl and Tractinsky 2006). For instance, Norman (2003) explicitly mentioned the notion of “emotional engagement” which is composed of three layers of brain processing: visceral (appearance), behavioral (functionality) and reflective (experience). Jordan (2000) associated engagement to four kinds of pleasures: physio, psycho, socio, and ideo. Relating engagement to user experience, McCarthy and Wright (2004) introduced four threads of user experience: sensual, emotional, spatio-temporal, and compositional. In other words, user experience can be derived from our sensory responses, emotional responses, and responses related to time and space, and a composite of these. On the other hand, Hassenzahl’s UX model (Harbich 2008) explicitly described three facets of experience: experiential, affective and aesthetical. These theories are consistent with Dewey (1934) notion of holistic engagement in which action, emotion, and thinking are all embodied within an engaged activity.
Other emerging areas that shape product design are behavioral economics and cognitive neuroscience. Behavioral economics investigates how social, cognitive, and emotional factors affect customer decisions (Ariely 2010; Brafman and Brafman 2008; Kahneman 2011). Cognitive neuroscience (Ariely 2010; Kahneman 2011) is a field that consistently strives to understand human cognitive capacities through the analysis of the human brain. Some recent findings include the irrationality of humans, the effects of technological distractions and also of multitasking on human capacities, and how human capacities can be enhanced through meditation (Tang et al. 2015), video games (Anguera et al. 2013), and exercise (Kirk-Sanchez and McGough 2013).
9.6 Positive psychology
Human engagement factors have also been associated with intrinsic motivation and needs. Perhaps the two most influential empirically-proven theories are flow theory (Csikszentmihalyi 1990) and self-determination theory (Deci et al. 2000; Ryan et al. 2006). Flow theory describes eight engagement factors: clear goals, balance between skills and challenges, merging of action and awareness, concentration, autonomy, loss of consciousness, time distortion, and autotelic experience. In interactive technologies, researchers (Hassenzahl and Tractinsky 2006) found several factors affecting flow including artefact complexity, task complexity, intrinsic motivation, usability, and aesthetics. Self-Determination Theory describes three primary intrinsic drives that motivate humans to do something: autonomy, competence, and relatedness. Other slightly different motivational theories include McClelland’s Theory of Motivation (McClelland 1988) which describes the three important motivational drivers as achievement, affiliation and power, and Pink’s theory (Pink 2009) which describes the three motivational drivers as autonomy, mastery and purpose, and the need of pleasures (Jordan 2000). Fogg’s behavioral model (Fogg 2009) describes the three elements that cause behaviors to change: motivation, ability (skills) and triggers (e.g., events, availability of tools).
Positive psychology has long studied human capacities, namely human motivation and human psychological needs. It is concerned with positive human development assuming that needs have been met and while needs are being met. It may be contrasted with a pessimistic view of the nature and condition of humanity. Positive psychology adds the dimension of growth above and beyond human needs, dealing with human potentials that can be developed in addition to the consideration of survival needs and problems.
In recent years HCI has recognized the importance of applications that specifically enhance human capacities and capabilities. A growing body of researchers leverage crowdsourcing or game-based techniques to help users in education (Gee 2003), health (Theng et al. 2012), and problem-solving (von Ahn and Dabbish 2004). There is also a growing body of researchers which aims to better understand human capacities by including more physiological measurements such as EMG, EEG, heart rate and skin conductance.
The recent increase in published literatures in HCI and allied fields about engagement confirms the surge in interest in this area. In line with this trend, we argue that it is timely for the HCI community to promote and conduct HEC related research with specific relevance to HCI interests. Indeed, some senior researchers have been rethinking the relationship between humans and computers (or technologies) from different angles (Law et al. 2015; Ren 2016; Farooq and Grudin 2017; Farooq et al. 2017; Niksirat et al. 2018). A CHI panel (Farooq et al. 2017) and two CHI SIGs (Niksirat et al. 2018; Farooq and Grudin 2017), have discussed related notions.
10 Research directions
10.1 Adopting theories from other fields
HEC can benefit from the well-established tradition of HCI adopting and adapting theories and frameworks from other fields. While HEC has the potential to generate new theories and frameworks, we would do well to adopt theories and frameworks from other areas and investigate how they may apply to HCI. For example, the field of positive psychology investigates notions such as flow, mindfulness, and emotional intelligence beyond the limitations of need. Various fields from eastern practice and philosophy offer a large corpus of theories and frameworks that can give insight into human significance and inner life management which can enhance the study and development of HCI within a broad HEC framework. In similar ways, many other branches of academics (sociology, human development, philosophy of education) will be very informative as cross-disciplinary sources, guides and correctives in the quest for a more holistic approach to technological development. Cross fertilization and collaborations with such sectors will help to inform HEC and shape HCI into the future.
10.2 Measuring human capacities and human engagement
Previous work primarily focuses on some external engagement factors such as appearance, fun and emotions, but less on inner engagement factors including mindfulness, compassion, etc. Further research will be needed to focus on such factors which are important toward achieving synergized interactions. For example, sustained human engagement is initiated at the subconscious level. The traditional analytical “reverse engineering approach” may prove inadequate to the task. New methodologies are essential to understand this level of consciousness and its implications for HCI design.
Studies of individual differences in personality, gender and culture etc., should be enlarged to include wider factors affecting human engagement. Longitudinal studies of engagement (Kujala et al. 2011) and large-scale engagement (social engagement) should be prioritized. For example, although neuroscience is considered one of the leading fields regarding decision making and human engagement, the current state of research is still limited, as neuroscience is concerned with decisions made in the moment, while the understanding of long-term engagement or large-scale engagement requires further study.
There is a strong bias towards a material only interpretation of consciousness within the neuroscience community. The introduction of eastern insights into the nature of consciousness could provide a productive counterbalance to this materialistic bias. By specifically sourcing from such disciplines, new fundamental theories, guidelines and models regarding engagement should greatly advance interaction design. Indeed, these are intrinsic to the agenda, role and work of HEC. A CHI SIG concerning the core concept of engagement has discussed related issues (Niksirat et al. 2018).
10.3 Developing engaging technologies
From an HEC perspective, technologies play three roles in the achieving of synergized interaction: (1) the preservation of human capacities, (2) the enhancement of human capacities, and (3) the adaptation of human capacities. All place priority on human capacities rather than on device potentials considered in isolation.
The current interaction aspect has been too narrowly focused on limited human capacities including memory, perception and motor control. Technologies should strive to adapt to fuller human capacities even where these cannot be measured directly, i.e., even if they can only be measured by their effects. For example, in a similar way to how Menus were developed by adapting human capacities in terms of memory, the potential of technologies will further increase when wider range of human capacities is considered and exploited.
10.4 Computational aesthetics
Aesthetics are influential in how willingly, how comfortably and how efficiently humans interact with objects, devices, ideas, and systems. Although the HCI community strives to dig out knowledge, principles and guidelines of aesthetics for interface and interaction design, there still remains much room to seek an epistemological corpus which should include the notion, human factors and the quantification of aesthetic aspects (Chen et al. 2018).
Conventional empirical studies lack the ability to quantify the aesthetic perspective of design due to its subjectivity, thus the conclusions are difficult to define and seem to be too fuzzy to be applied to practical design. Computational aesthetics is expected to fill this gap from at least two levels. From the practical level, tools will be developed to help designers address aesthetic design issues, and to optimize the interface design accordingly. From the engaging level, researchers should aim at understanding the aesthetic perception of users, and help them form or develop aesthetic capabilities. We expect to understand the commonality among different modalities and thus seek a computational basis and evaluate aesthetic interaction. This is expected in the future research agenda if the views from the disciplines including philosophy, HCI, neuroscience, cognitive science, linguistics collaborate and discuss the promising direction of computational aesthetics.
10.5 Integrating east–west awareness
Understanding how to achieve engagement requires a holistic understanding of humans. One of the radical approaches to understanding humans is to consider the integration of east and west knowledge. Studies of yoga therapy (Caplan et al. 2013) and herbalism (Freeman 2011) focused on integrating east-west human wellbeing approaches. Others have attempted to integrate eastern philosophies such as ancient Chinese insights and practices (e.g., Yijing or I Ching, Chan or Zen) and apply them to management, self-development and business (Taguchi 2012). Given the holistic nature of eastern awareness and the analytical nature of western awareness, considered together they may provide direction and correction to each other with far greater effect and efficiency in meeting real human needs. Based on this thinking, a CHI workshop about leveraging and integrating eastern and western knowledge, practice and experience has discussed related issues (Law et al. 2015).
10.6 Refining the framework
The proposed framework containing values, principles and components serves mainly as a logical starting point for discussion at this point. This framework demands refinement and redefinition. For example, HEC would benefit from the further refinement and elaboration of HEC values, principles and components in collaboration with a number of relevant disciplines. An international workshop has been held among researchers from various disciplines.2
10.7 Expanding the HEC design space
HEC is a thinking model for viewing synergized interactions. It generates insights about novel possibilities for HCI, as well as new ways to manifest mature human capacities and engagement by sourcing from as comprehensive a view of existence as possible in each generation and each situation.
The HEC design space as applied to HCI and any area in relation to the relationship between humans and technologies is large and relatively unexplored. The application of HEC will be via systems/interfaces/designs that—(1) adhere to HEC values and principles which are not doctrinaire but intended to promote sensitivity to people as individuals and in their respective groups, (2) encourage the increase of human capacities through engagement to help humans achieve their goals, (3) consciously avoid the displacement of human capabilities while expanding human capacities, (4) leverage existing advances in technology.
With the proposed framework of HEC, the progressive unfolding and rediscovery of human capacities, and the application of HEC in practice, we believe new directions, new efficiencies and better values will be applied to R&D resulting in a better world for all.
This paper proposes the theme of synergized interaction between humans and technologies under the appellation—HEC which specifically identifies, promotes, enhances and synergizes innate human capacities and technological capabilities by maximizing synergism and minimizing antibiosis. We call for a coherent understanding which gives priority to human outcomes, with technologies serving humans as stewards in nature.
We have argued that a holistic consideration of human capacities is essential if greater synergy between humans and computers is to be achieved. Among the benefits of greater synergy would be seamless human-technology interactions, clearer and more relevant goals for design, more pragmatic and sustainable outcomes, more personal and social satisfaction in both the process of interaction and the outcomes in human life.
This paper does not aim or pretend to provide detailed answers, but to open the box on these kinds of issues. This paper indicates the discussions we want to have. The value of our paper is to provoke and to imply details so that a variety of people may see from their respective circumstances what can and should be done in detail. The very requirement for research and tested solutions is what has driven us to propose the establishment of HEC as an important research frame. We require a conceptual framework that is sensitive to the times, adaptable to respective circumstances and values, globally and variously relevant and in touch with the evolution of human and device potentials.
The initial work of this concept—HEC was done in the year 2013. Since then, many remarkable colleagues have given their valuable comments and support to this work and to our understanding of the relationship between humans and computers. They include Effie L-C Law, Ed H. Chi,Shengdong Zhao, Antti Oulasvirta, Huatong Sun, Jonathan Grudin, Umer Farooq, Torkil Clemmensen, Ann Light, Ying Leung, Hirohide Yamada, Peng Cheng, Kavous Salehzadeh Niksirat, Sayan Sarcar, Hirskyj-Douglas Ilyena, and all members in Ren Lab and the Center for Human-Engaged Computing at Kochi University of Technology. To all others who are not mentioned here, we express our heartfelt gratitude. To my dear friend and distinguished colleague Antti Oulasvirta, who provided the many valuable suggestions needed for high competence and his extraordinary contribution to this work. To my PhD student Chen Wang for redrawing some of the figures.
- Accenture: Consumers are Bored with Today’s Tech and Nervous about Tomorrow’s (2016). Retrieved January 5, 2016 from http://www.businessinsider.com/accenture-consumers-not-interested-in-internet-of-things-devices-2016-1
- Alicea, B.: An integrative introduction to human augmentation science. arXiv preprint arXiv:1804.10521 (2018)
- Ariely, D.: Predictably irrational: the hidden forces that shape our decisions. Harper Perennial (2010)Google Scholar
- Bardzell, J.: Interaction criticism and aesthetics. In: In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2357–2366 (2009)Google Scholar
- Bayliss, P.: Videogames, interfaces, and the body: the importance of embodied phenomena to the experience of videogame play. Ph.D. thesis, RMIT University (2010). PhD. ThesisGoogle Scholar
- Benford, S., Bowers, J.: User embodiment in collaborative virtual environments. In: Proc. CHI 1995, pp. 242–249 (1995)Google Scholar
- Birk, M., Mandryk, R.L.: Control your game-self : effects of controller type on enjoyment, motivation, and personality in game. Proc. CHI 2013, 685–694 (2013)Google Scholar
- Bødker, S.: When second wave HCI meets third wave challenges. In: Proc. NordiCHI 2006, pp. 1–8 (2006). https://doi.org/10.1145/1182475.1182476
- Borning, A., Muller, M.: Next steps for value sensitive design. In: Proc. CHI 2012, pp. 1125–1134 (2012). https://doi.org/10.1145/2207676.2208560
- Brafman, O., Brafman, R.: Sway: the irresistible pull of irrational behavior. Crown Business (2008)Google Scholar
- Calleja, G.: Revising immersion: a conceptual model for the analysis of digital game involvement. In: Proc. DiGRA 2007, Situated Play, pp. 83–90 (2007)Google Scholar
- Calvo, R., Peters, D.: Positive computing: technology for wellbeing and human potential. The MIT Press (2014)Google Scholar
- Caplan, M., Portillo, A., Seely, L.: Yoga Psychotherapy: the integration of western psychological theory and ancient yogic wisdom. J. Transpers. Psychol. 45(2), 139–158 (2013)Google Scholar
- Card, S.K., Newell, A., Moran, T.P.: The psychology of human–computer interaction. L. Erlbaum Associates Inc. (1983)Google Scholar
- Carr, N.: The shallows: what the internet is doing to our brains. W. W. Norton & Company (2011)Google Scholar
- Chaney, W.H.: A workbook for the dynamic mind (read 3 times faster-remember 10 times more). Houghton Brace (2006)Google Scholar
- Chapman, P.: Models of engagement: intrinsically motivated interaction with multimedia learning software. University of Waterloo (1997)Google Scholar
- Chen W., Sayan S.: Approaching aesthetics on user interface and interaction design. In: ISS’18 Proceeding, pp. 481–484 (2018)Google Scholar
- Cheng, P.: PauseAble (2015). Retrieved October 23, 2015 from http://pauseable.com/
- Collins, J.: Good to great: why some companies make the leap...and others don’t . HarperBusiness (2011)Google Scholar
- Cooper, S.: A framework for scientific discovery through Video Games. Ph.D. thesis, University of Washington (2011)Google Scholar
- Csikszentmihalyi, M.: Flow: the psychology of optimal experience. Harper Perennial (1990)Google Scholar
- Deci, E.L., Ryan, R.M.: The “What” and “Why” of Goal Pursuits: Human Needs and the Self-Determination of Behavior 11(4), 227–268 (2000)Google Scholar
- Desmet, P.M., Pohlmeyer, A.E.: Positive design: An introduction to design for subjective well-being. International Journal of Design 7(3) (2013)Google Scholar
- Dewey, J.: Art as experience. Perigree (1934)Google Scholar
- Engelbart, D.: Augmenting human intellect: a conceptual framework. SRI Summary Report AFOSR 3223 (1962)Google Scholar
- Ermi, L., Mäyrä, F.: Fundamental components of the gameplay experience: analysing immersion. Proc. DIGRA 2005 (2005)Google Scholar
- Farooq, U., Grudin, J.: Human–computer integration. ACM Interactions, pp. 27–32 (2016)Google Scholar
- Farooq, U., Grudin, J.: Paradigm shift from human computer interaction to integration. In: Proceedings of the CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI 17), Denver, USA., pp. 1360–1663 (2017)Google Scholar
- Farooq, U., Grudin, J., Shneiderman, B., Maes, P., Ren, X.: Human computer integration versus powerful tools. In: Proceedings of the CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI 17), Denver, USA., pp. 1277–1282 (2017)Google Scholar
- Fogg, B.: A behavior model for persuasive design. In: Proc. Pervasive 2009, pp. 40:1– 40:7 (2009). https://doi.org/10.1145/1541948.1541999
- Freeman, D.: Creating emotion in games: the craft and art of emotioneering. New Riders (2003)Google Scholar
- Freeman, J.: Integrated East/West Herbalism. Lulu (2011)Google Scholar
- Fung, Y.l.: A short history of chinese philosophy. Free Press (1997)Google Scholar
- Gee, J.P.: What video games have to teach us about learning and literacy. Palgrave Macmillan Trade (2003)Google Scholar
- Goldman, R., Gabriel, R.P.: Innovation happens elsewhere: open source as business strategy. Morgan Kaufmann (2005)Google Scholar
- Goleman, D.: Focus: The hidden driver of excellence. Harper (2013)Google Scholar
- Haidt, J.: The happiness hypothesis: finding modern truth in ancient wisdom. Basic Books (2006)Google Scholar
- Harbich, S., Hassenzahl, M.: Affect and emotion in human-computer interaction. chap. Beyond Task Completion in the Workplace: Execute, Engage, Evolve, Expand, pp. 154–162. Springer (2008)Google Scholar
- Harden, L., Heyman, B.: Digital engagement: internet marketing that captures customers and builds intense brand loyalty (2009)Google Scholar
- Harrison, S., Tatar, D., Sengers, P.: The three paradigms of HCI. In: Proc. Alt. CHI 2007, pp. 1–8 (2007)Google Scholar
- Hawks, J.: If modern humans are so smart, why are our brains shrinking? Discover (2010)Google Scholar
- Heeter, C., Lee, Y.H., Medler, B., Magerko, B.: Beyond player types: gaming achievement goal. In: Proc. Sandbox 2011, pp. 43–48 (2011)Google Scholar
- Houle, D.: Entering the shift age. Sourcebooks (2012)Google Scholar
- Hunicke, R., Leblanc, M., Zubek, R.: MDA: A formal approach to game design and game research. In: Proc. AAAI-04 Workshop, pp. 1–5 (2004)Google Scholar
- Institute, D.E.: The MIT/Brown Vennevar Bush Symposium (2013). Retrieved March 9, 2013 from http://www.dougengelbart.org/events/vannevar-bush-symposium.html
- Jaworski, J.: Source: The inner path of knowledge creation. Berrett-Koehler Publishers (2012)Google Scholar
- Jenson, J., de Castell, S., Fisher, S.: Girls playing games: rethinking stereotypes. In: Proc. FuturePlay 2007, pp. 9–16 (2007)Google Scholar
- Kahneman, D.: Thinking. Farrar, fast and slow (2011)Google Scholar
- Kuhn, T.S.: The Structure of Scientific Revolutions. University of Chicago Press, Chicago (1962)Google Scholar
- Kurzweil, R.: The singularity is near: when humans transcend biology. Penguin Books, London (2006)Google Scholar
- Kuutti, K., Bannon, L.J.: The turn to practice in HCI: towards a research agenda. In: Proc. CHI 2014, pp. 3543–3552 (2014). https://doi.org/10.1145/2556288.2557111
- Laurel, B.: Computer as Theatre. Addison-Wesley, (1993)Google Scholar
- LaViola Jr., J.J., Litwiller, T.: Evaluating the benefits of 3d stereo in modern video games. In: Proc. CHI 2011, pp. 2345–2354 (2011)Google Scholar
- Law, E.L.C., Silpasuwanchai, C., Ren, X., Bardzell, J., Clemmensen, T., Liu, Y.: Leveraging and Integrating Eastern and Western Insights for Human Engagement Studies in HCI. In: Proc. CHI EA 2015. ACM (2015). To appear. http://eastwest.xrenlab.com
- Lazzaro, N.: Why we play games: Four keys to more emotion without story. In: Game Developer’s Conference, pp. 1–8 (2004)Google Scholar
- Licklider, J.C.R.: Man-Computer Symbiosis. IRE Transactions on Human Factors in Electronics pp. 4–11 (1960)Google Scholar
- Ma, X.: Towards human-engaged ai. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI-18), pp. 5682–5686 (2018)Google Scholar
- McCarthy, J., Wright, P.: Technology as experience. The MIT Press, Cambridge (2004)Google Scholar
- MIT: MIT Center for Collective Intelligence (2015). Retrieved August 14, 2015 from http://cci.mit.edu/
- National Center for Biotechnology Information: Attention Span Statistics. The Associated Press, Tech. rep. (2014)Google Scholar
- Niksirat, K.S., Silpasuwanchai, C., Ren, X., Wang, Z.: Towards cognitive enhancement of the elderly: A ux study of a multitasking motion video game. In: Proceedings of the CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI 17), Denver, USA., pp. 2017–2024 (2017)Google Scholar
- Niksirat, S.K., Sarcar, S., Sun, H., Law, E., Clemmensen, T., Bardzell, J., Oulasvirta, A., Silpasuwanchai, C., Light, A., Ren, X.: approaching engagement towards human-engaged computing. In: Proc. CHI EA 2018 (2018)Google Scholar
- Norman, D.: The design of future things. Basic Books (2009)Google Scholar
- Norman, D.A.: Emotional design: why we love (or hate) everyday things. Basic Books (2003)Google Scholar
- Pink, D.H.: Drive: the surprising truth about what motivates us. Riverhead Hardcover (2009)Google Scholar
- Randers, J.: 2052: A global forecast for the next forty years. Chelsea Green (2012)Google Scholar
- Rogers, Y.: HCI Theory: Classical, Modern, and Contemporary. Morgan & Claypool Publishers, San Rafael (2012)Google Scholar
- Salehzadeh Niksirat, K., Silpasuwanchai, C., Mohamed Hussien Ahmed, M., Cheng, P., Ren, X.: A Framework for interactive mindfulness meditation using attention-regulation process. In: Proc. CHI 2017, pp. 2672–2684 (2017). https://doi.org/10.1145/3025453.3025914
- Sen, A.: Commodities and capabilities. Oxford University Press, Oxford (1999)Google Scholar
- Shneiderman, B.: Human values and the future of technology: a declaration of empowerment. In: Proceeding of CQL ’90 Proceedings of the conference on Computers and the quality of life, pp. 1–6 (1990)Google Scholar
- Shneiderman, B.: Humanizing Technology: A History of Human-Computer Interaction (2015). Available: http://bits.blogs.nytimes.com/2015/09/07/humanizing-technology-a-history-of-human-computer-interaction/, Accessed on September 11, 2015
- Sims, Z., Bubinski, R.: CodeAcademy (2014). Retrieved June 3, 2014 from http://codeacademy.org/
- Soho, T.: The Unfettered Mind – Writings of the Zen Master to the Sword Master. Shambhala (2012)Google Scholar
- Stockholm, W.: Where Consciousness Meets Business, Politics and Life (2015). Retrieved August 14, 2015 from http://wisdomstockholm.com/
- Taguchi, Y.: The Managerial Ideas from the East. Babel Corporation (2012)Google Scholar
- Theng, Y.L., Chua, P.H., Pham, T.P.: Wii as Entertainment and Socialisation Aids for Mental and Social Health of the Elderly. In: Proc. CHI 2012, pp. 691–702. ACM (2012)Google Scholar
- Toyama, K.: Geek Heresy. PublicAffairs (2015)Google Scholar
- von Ahn, L., Dabbish, L.: Labeling images with a computer game. In: Proc. CHI 2004, pp. 319–326. ACM (2004)Google Scholar
- van Schaik, Paul Ling, J.: Human computer integration. Human-Computer Interaction 27(3), 199–234 (2012)Google Scholar
- Wiener, N.: Cybernetics: or control and communication in the animal and the machine. MIT Press, Cambridge (1948)Google Scholar
- Winograd, T., Flores, F.: Understanding Computers and Cognition: A New Foundation for Design. Addison-Wesley Longman Publishing Co., Inc. (1987)Google Scholar
- Wisdom: Wisdom 2.0: Living with Awareness, Wisdom, and Compassion (2014). Retrieved August 14, 2015 from http://www.wisdom2summit.com/