1 Introduction

The emergence of augmented and virtual reality and the prevalence of mobile and wearable devices begins the era of augmented human, digital-self based on user observation has emerged in the virtual world. The digital-self becomes a concrete alternative to the intangible human, and becomes the separate entity from the original. The digital-self have been used for different services to optimize and manage users and in this purpose form of the digital-self have to be quantified.

For this reason, the Quantified Self (QS) movement is proposed in 2000’s and numerous augmenting-human services with the QS have been made based on the self-tracking. Because QS pursue an acquisition of the quantified status of user’s and monitoring collected data, the QS has many possibilities in enhancing human ability. Based on this self-surveillance, the QS is being used the various field of user personalization service, especially in the health-care area. The rapid growth of sales of fitness tracker business in recent statistics proves this trend (http://www.businessinsider.com).

There are two factors for the prevalence of the QS. First of all, the emergence of smartphones enables self-tracking of users. The spread of these smart devices with different embedded sensors involve the accumulation of various sensor data and has been the basis of the QS movement. Under this circumstance, the electronic device including wearable devices [1] and IoT [2] are expected to enlarge future applications of the QS. On the other hand, as personal and small data [3], the QS could guarantee the confidentiality of user information. In contrast to other fields of big data that are suffering fundamental difficulties with privacy or public access issues [4], the QS has been spotlighted as an alternative platform of the Big Data that can access long-term individual information without privacy issues [3]. Those two characteristics contribute to the rapid growth of commercialization of the QS applications.

As academic research, there are different purpose on adapting QS in studies. First of all, QS is widely accepted in fields of health-care applications for its intrinsic characteristic, which is tracking the status of the user. After the potential for medical applications is introduced [5], QS has been regarded as an alternative for personalized medical service [6]. Several recent proposals for the health-care applications have based on the QS [7, 8]. Except for these health-care applications, the QS is also adapted in the field of education to support the learning [9], and its potential is also considered in the field of gamification [10].

However, current research and services on QS still have several limitations. First of all, because the data collection process mainly depends on the mobile and wearable devices, limited status of the user considered for the QS [8]. Most studies are confined to utilizing the location and device usage information, and some recent studies infer the social status of the user from the SNS status. However, still other types of the user information such as emotional status have been tried rarely, and especially, inferring and exploiting the user context is still in a stalemate. Moreover, most QS studies and services are limited to the reactive and prompt response provisioning [11] based on the historical records of the user. Although those studies and services can provide the instant help to users, only a few studies consider proactive service that based on the prediction of the user status. The most fundamental problem is that there is no clear future direction and universal consensus on the goals to be achieved through the QS.

To overcome issues mentioned above, we presented a concept of the Holistic Quantified Self (HQS) framework to support the augmented human. The ultimate goal of an augmented human is containing the digital self which is supporting the user’s capabilities. To fulfill the concept of it, we present the HQS and its framework based on our previous conceptual framework [12]. HQS which consists of the quantified value of physical, emotional, social, and behavioral status of the user provides the comprehensive information of the user. Based on the HQS, the HQS framework can interact with the possible service provider and controls electronic devices to provide the personalized service without the release of the user information. Throughout this framework, not only the private issue of user information is resolved, but also proactive in-situ context-aware service can be made.

In the followings, we discuss the definition of the augmented human and HQS with its generation in Sect. 2. After that, we propose the HQS framework for augmented human in Sect. 3. In Sect. 4, we introduce our work-in-progress prototype of the framework. Finally, we conclude this paper by presenting the future work in Sect. 5.

Fig. 1.
figure 1

Digital self in Digilog era

Full size image

2 Augmented Human and Holistic Quantified Self

2.1 Digitalized Self and Augmented Human

With the development of technology and the arrival of augmented human life, the concept of digital-self in the dual space which consists of the physical space and the virtual space is recently begun to be discussed. Initially, digital-self was a concept proposed for personalization service for online marketing, but it started to be used more actively due to the emergence of augmented/virtual reality and the prevalence of mobile and wearable devices. These two technological advances have had a significant impact on the digital self (see Fig. 1). First, by observing all the behaviors of the user through the mobile and wearable devices, various service providers can be possible to utilize the user’s features, which is extracted from the user observation data (Externalization of the self) [13]. Also, digital-self of the user (ex. Avatar) in augmented/virtual reality generated by the process mentioned above becomes separated from the user oneself (Separation of the self) [13].

From this point of view, the body becomes the border of the externalization and separation of the self and therefore, plays an important role where the self is expressed. The body is the subject of interaction with others, mediating the physical and mental worlds, anchoring between the real space and the virtual space with smart devices [13]. To reflect these aspects, the digital self should be a comprehensive self that contains (1) body, (2) interaction with others, (3) degree of connection with the user’s virtual world, and (4) mental aspects such as emotion and propensities. Moreover, in perspective of the digital self, the body is an object of enhanced self-management, self-control, and self-regulation [14]. Considering the essential characteristics of the digital self which monitors and optimizes the performance of the user, the desirable form of the alternative digital self should be the QS [13].

From the aforementioned discussion about the digital self, we propose a concept of it as the ultimate goal of the digital-self. Traditionally, the human augmentation has been discussed primarily on physical enhancement based on the mechanics and the bionics. There have been numerous examples such as the implantable, the elective bionics, the body modification, the stem cells, and the cog-enhancers. However, in this study, we focus on the virtual enhancement of the human augmentation. Recently, the fusion of virtual reality and physics reality have presented new possibilities for human augmentation. Numerous augmented reality (AR) applications are typical examples of assisting the human intelligence and perception.

In this study, based on our previous study [15], we define that the AH is a human whose physical, intellectual, and social ability are enhanced by the augmented/virtual reality and the smart ICT technology. To fulfill AH, we have to include three augmentations for human: the augmentation of the body augmentation, the brain augmentation, and the social augmentation. First of all, the body augmentation is techniques that enhance human body ability by supporting human reaction, interaction, and spatiotemporal movement. For example, based on the user’s physical and emotional state tracking and body management can be provided. Also, the user’s intention can be expressed beyond the spatial constraints through the IoT’s-based remote services. Next, the brain augmentation is techniques of enhancing the ability of the brain by expanding the scope of human perception and recognition, by supporting decision-making process, and by assisting memorization For instance, history analysis on user behavior, in-situ information, contents, and service can be provided. Finally, the social augmentation is techniques to enhance social ability by supporting empathy, communication, and collaboration. User interaction analysis based management service for a persistent relationship can be a starting point. Moreover, to promote social relations of the user, information and experience of users can be selectively shared, and in case of conflict of interest, trend analysis and negotiation process can be supported based on the past interactions analysis.

2.2 Holistic Quantified Self

Based on the aforementioned discussion, we propose the HQS as an alternative of the comprehensive digital self in the DigiLog era and as a core to realize the AH. However, before we propose the HQS, brief history of the QS have to be mentioned to demonstrate the needs of the HQS. After the concept of QS is proposed through the QS movement in 2007 (quantifiedself.com), there have been numerous QS-based approaches made on personalization field. Because the QS acquires information about oneself by itself, there is less difficulties in data acquisition process due to personal information extrusion, which enables reliable database of users based on long-term data acquisition. Therefore, the QS was spotlighted as an alternative for the big data because of not only the QS can avoid the privacy issue, but also its unique characteristic, which is information of an individuals [16]. However, confined scope of the data acquisition and difficulties on analyzing contextual information has been an obstacle to the expansion of QS applications [8]. Consequently, whole potential of the QS is confined to the straightforward applications in the health-care fields [17].

In addition to technical limitations and legal constraints, limitation of QS also is caused by its market-oriented characteristics. Because business-initiated QS applications are limited by marketability, focus of the QS applications have become fragmented for particular needs. As mentioned above, commercialization of the QS has progressed to a considerable extent in the health-care field with fitness trackers, but the fact that only a few research for context awareness which enables more in-depth services exists implies this situation.

To address these limitations and To consider the aforementioned characteristics of the comprehensive digital self, we propose the HQS based on our previous conceptual study [12]. In this study, we define the HQS as physical, emotional, social, and digital behavioral quantified status of the users for certain time period (see Fig. 2(a)) that is quantified from the assorted sensor data from user devices. As HQS data is accumulated by, common patterns of HQS data can be recognized and this unique patterns of certain user can be regarded as identification of the user. We define this unique pattern of user as ‘E-personality’ of the user and expect that in-situ context-aware service can be made through this E-personality. Moreover, accumulated HQS data can be utilized to predict the future status of the user. Through the time series analysis of HQS data, one can predict the future status of user and it would enable proactive service to user.

Fig. 2.
figure 2

(a) Example of the HQS and (b) the E-personality (Red Pattern) (Color figure online)

Full size image

Process for the HQS generation as follows (see Fig. 3). First, raw sensor signal from the available embedded sensors of user device should be collected to find the user status. However, because these raw sensors signal may not only noisy but also can be not directly relevant with the user status in itself. Therefore, to acquire basic meaningful information of the user status, it should be extracted from raw sensor data through signal processing. We define this basic meaningful information as fundamental index and this fundamental indexes becomes basis for the HQS element. Through the status mapping process between the Fundamental index and the HQS, HQS Elements can be quantified. Finally, through the pattern analysis on these HQS elements, E-personality, common HQS patterns of the user, is identified and can be utilized for personalized service.

Fig. 3.
figure 3

Process for generating the HQS and the E-personality

Full size image

To collect the raw sensor signals, available sensors have to be surveyed in advance. Recent user devices (i.e. mobile and wearable devices) have its own sensor specification (accelerometer, gyroscope, and magnetometer, GPS, light sensor, camera, heartbeat sensor and so on) and these sensors have to be able to access with released API. After surveying the availability of sensors, storage and transmission method of these raw sensors signals have to be determined. Through these steps, raw sensors signals can be collected by user devices and would be used to build the fundamental index.

Fig. 4.
figure 4

Structure diagram of the HQS framework

Full size image

Fundamental index consists basis for the HQS as basic meaningful data of the users. To extract the fundamental index, method of signal processing and its time window have to be determined for each raw sensor signal. For stable operation in user device, time complexity and space complexity of each signal processing method have to be considered. In this context, because the acquisition cycle is directly connected with the battery consumption, time window of the signal process is determined elaborately. Moreover, power consumption of each sensor has to be considered for stable operation in user devices. For example, accelerometer is typical low-power sensors, but GPS requires much more battery consumption than the accelerometers. Based on these find-tune fundamental index calculation, the HQS can be generate for given time period.

The HQS indicate the 4 types status of user with such as physical, emotional, social, and digital behavioral status for given period. Each status contains the multiple elements and each HQS element can be calculated from multiple fundamental indexes. In other words, relation between the fundamental indexes and HQS elements are n:1 relation. As the available sensor information and fundamental index are diversified, the HQS elements will be expanded. Each HQS elements requires its own minimal measuring period and basic unit time of HQS is 24 h. Therefore, whole HQS data have 24 h of unit time period.

As common pattern of HQS (See Fig. 2(b)), E-Personality is represented the unique characteristics of the user. E-personality is represented by a vector consists of quantified values from each HQS elements. Therefore, as HQS elements are diversified, length of E-personality is also increased. One common misunderstanding for the E-personality is that E-personality represents intuitive characteristics of user such as demographic information, age, gender and so on. However, there might exists some tendencies on value of E-personality according to condition and type of user. However, E-personality would not always have distinctive patterns for stereotype of user such as “women of twenties age”. As HQS data accumulated, there can be several E-personalities can be extracted according to the status of the user. Therefore, E-personality can support the personalized service.

3 HQS Framework for Augmented Human

In this chapter, we propose the HQS framework for Augmented Human. Purpose of the HQS framework is to provide proactive, context-aware and personalized service and information to the user. Through the HQS framework, capability of the user can be enhanced and the Augmented Human would be realized. Structure of the HQS framework is represented in Fig. 4. In addition to the aforementioned HQS generation process, the framework contains personal cloud for the HQS management and analysis, user-representative agent for personalized service. Details of each component and workflows are described as follows.

At first, multimodal sensor data collector performs the user observation through the embedded sensors of user devices. Because there could be multiple sensors from user’s wearable and mobile device involved, there should be a hub for the assorted sensors. Therefore, mobile device takes a role of hub for data acquisition and control the available user devices to receive sensor signals. Because computation powers of current wearable devices are limited, it is favorable to perform the HQS generation process in mobile devices rather than in wearable devices. Exceptionally, if the computation power of mobile device is not enough and the size of HQS elements data is affordable, then one could consider adaptation of remote server, although it is not desirable in terms of privacy issue.

After the sensor signals are acquired, the HQS quantifier calculate the HQS elements according to the aforementioned process. For privacy issues, raw sensor data and the fundamental index are closed to outside. According to the characteristics of HQS elements, each HQS element is calculated periodically with its own period. Finally, calculated HQS is provided to the HQS processor.

The HQS processor handles analysis and management of the HQS. The HQS manager accumulate the HQS data and backup the data into the HQS bank in personal cloud. The HQS bank serves as a private repository for huge amount of the HQS data which is accumulated over a long period of observation. Through these accumulated historical records of the HQS, E-personality analyzer can perform the pattern recognition process to extract the E-personality to build user prediction model. Through these steps, proactive service can be made by the user-representative agent and current E-personality is provided to user-representative agent.

The user-representative agent provides the personalized service to user based on the E-personality sent from the HQS processor. To provide the service, the agent interacts with the environment which consists of various services providers, IoT’s & actuators and agents of other users. Based on the E-personality data, the agent can select the appropriate service from various service providers and can controls the available IoT’s and actuators without revealing personal information. Moreover, the agents can negotiate with the agent of other users to provide better suggestions and conditions to users. In this way, the user-representative agent provides in-situ, context-aware service to the user and it would change the current paradigm from provider-oriented push service to the user-oriented pull service.

4 Prototype: HQS Framework

According to the framework, we have been developing the prototype of the HQS framework for personal companion. Using android smartphone (LG G6 - Android 7.0 Nougat) and smartwatch (LG Watch Sports - Android Wear 2.0), we have developed the multimodal sensor collector and the HQS quantifier through the android applications. This application monitors the user status in real-time and generate the HQS data for user (See Fig. 5).

Fig. 5.
figure 5

Current android application prototype for the HQS framework

Full size image
Fig. 6.
figure 6

Process for facial expression recognition

Full size image

Through the embedded sensors of the devices, multimodal cues of the user are collected. In this prototype, we exploit step counter, camera, call log, SMS/MMS log, and application usage Log from the Android API, and PPG sensor (for heartbeat rate) from the Android Wear API. The heartbeat rate captured from the smartwatch is sent to the smartphone through the Bluetooth pairing. In smartphone, whole collected sensor signals are converted into the fundamental index. Following Table 1 describes the details of the fundamental index.

Table 1. List of the fundamental index
Full size table

According to characteristic of each index, each index has its own acquisition cycle. Among the information we can obtain from the multimodal sensors, some information is valid only by individual sensor signal, but other information may valid through the trend and flow of values. In addition, there exists intermediate information that both individual value itself and tendency are important. The former requires to acquire the data when needed, but the latter requires the related sensors to be always active. For hybrid case, appropriate mix of both method is required. However, in any case of that real-time acquisition is not important or costly, we choose periodic acquisition or event-based acquisition such as user commends.

Especially, for facial expression, we perform the vision-based facial expression recognition process (see Fig. 6). We define 3 types of emotional classes of facial expression such as positive, neutral, and negative. To get a status of facial expression of user, we conduct facial expression recognition using a RGB image which is taken during unlocking a smartphone. For recognition of user’s facial expression, we first capture an RGB image from smartphone and detect a facial region. Then, we use the local binary pattern (LBP) representation [18] for encoding the detected facial region of interest (RoI). Finally, we perform facial expression classification using support vector machine (SVM) [19] and the result of classification is stored with the timestamp of when the image is taken.

Table 2. List of HQS elements. * : Daily Elements
Full size table

After the fundamental index is accumulated, HQS elements are calculated from these fundamental index. Fundamentally, basic unit time windows of most HQS elements are one hour, however, some daily elements are meaningful when the value of 24 h are collected. Following table (see Table 2) contains the information of the HQS elements.

For physical information, we calculate the step count histogram and exercise rate. First, step count histogram is hourly histogram of the pedometer and average heart rate. Through step count histogram, users can confirm their amount of movement by hour and it can be utilized the implicit context information for user’s status such as exercising or commuting. As its daily elements, exercise rate which is sum of the all step counts in a day and daily average heartbeat rate, represents user’s daily amount of activity.

For emotional information, we calculate emotion type with facial expression and heartbeat rate. When we take the facial expression during unlock the screen, PPG sensor in smartwatch also takes the user’s heartbeat rate at the same time. This heartbeat rate compliments the vision-based facial expression recognition to infer emotional status of the user.

For social information, we calculate interaction pattern that is top 10 score interaction frequency. To calculate the degree of the interaction, for each acquaintance the user interacts with, weight sum for each interaction mode (i.e. Call, SMS/MMS) and time period of interaction is made. To secure privacy, this score stored without the name of acquaintance. Through this element, we can only identify the user’s style of interaction. For example, one can have many interactions with chosen few, the other have relatively uniform interaction with whole his/her acquaintances. As its daily element, Outgoing is sum of all interaction scores which means the degree of interaction with all acquaintance for that day. Through the outgoing element, the user can confirm degree of interaction he/she made in that day.

For digital behavior, we calculate the activity preference. This element shows the which applications are mostly used by the user. According to the mode of usage (i.e. foreground, background) weight are multiplied to the usage time period. So the user can confirm which usage of each applications and can infer type of activity is mostly performed through his/her smartphone.

Currently, we are accumulating the HQS and extracting the E-personality based on the pattern recognition of the HQS. We expects that the learning process will reveals correlations between certain HQS pattern and specific reactions of the user. These specified HQS patterns could be identified with E-personality and will be utilized in personalized service in user-representative agent.

Fig. 7.
figure 7

HQS Framework based Personal Companion. * CE : Consumer Electronics

Full size image

5 Future Work: Social Augmentation and Personal Companion

As a future work, we will develop a Personal Companion as the user-representative agent in the HQS Framework. Figure 7 depicts the structure of the ongoing Personal Companion. Through the IoT-based consumer electronic device which contains the service agent and can migrate and learn with each other, the Personal Companion provides the possible service content and requests the appropriate service based on the user’s personality. For this process, the Personal Agent analyzes accumulated HQS with user’s response through deep learning method, then identify the specific HQS pattern which is highly correlated with the certain response as E-Personality. For private issues, those E-personality data should be transmitted to the trusted consumer electronic devices only. To perform this task, we will establish the remote server and personal cloud that accumulates the HQS. Through the remote server, we can reduce the computation load of the smartphone to generate the HQS and personal cloud will enable the management and time-series analysis of the accumulated HQS data. This time analysis will enable the proactive context-aware service to the user. To provide the service, the Personal Companion should be able to control the available IoT’s and actuators for users. To handle this task, we are considering to utilize the IFTTT (https://ifttt.com/) which is web-based service controls assorted IoT’s as intermediate platform.

After whole components are developed, we will perform user evaluation on the HQS framework. Currently, we are also building a test-bed for the HQS framework, which aims at a futuristic residential space and includes assorted IoT devices and actuators. By developing user scenarios that change the indoor environment and recommend the appropriate service according to the user’s E-personality, the evaluation on the HQS framework can be made.

6 Conclusion

In this paper, we present our HQS framework for the augmented human. We state the definition of the HQS, the concept of augmented human, and present the procedure to generate the HQS. After than we present the HQS framework and our work-in-progress prototype and our future work. Through the HQS framework, the human augmentation which enhance the body, the brain, and the social ability of the user would be enhanced.

As ultimate goal of our study, we expect to contribute to quality of social life through the development of the Personal Companion. Despite it’s significant future value, enhancement of the social ability of users has not been studied so far. The Personal Companion aims to enhance the ability to sympathize by sharing user experience, and to enhance user’s social ability by providing mutual respect and negotiation. Finally, we expect that recommendation on deliberative decision can be made. Moreover, we expects to solve the problem of human alienation by providing personalized services and enhancing the social ability of the users, which has been criticized as a side effect of the technology development.