Confucius computer: a philosophical digital agent for intergenerational philosophical play
- First Online:
- Cite this article as:
- Cheok, A.D., Edirisinghe, C. & Karunanayaka, K. Pers Ubiquit Comput (2017) 21: 327. doi:10.1007/s00779-016-0995-8
Confucianism is commonly defined as “... a system of philosophical, ethical and political thought based on the teachings of Confucius,” which originated through the teachings of Confucius during the sixth-century BCE. It is a way of life or a philosophy of human nature that considers human relationships as the foundation of the society. Confucius teachings had highly influenced the development of several cultures in Asia, making Confucianism an intangible cultural heritage. In this paper, we are re-acquainting users with an intangible heritage that is part of their everyday, by developing a system that permits experiencing Confucius teachings virtually and interactively. The system can measure philosophical intent of the human and generate meaningful philosophical answers. It is also aimed for intergenerational sharing of Confucius heritage through a simple interactive process with the virtual sage making the experience enjoyable and entertaining. Previous research in natural language processing (NLP) mainly focused on the understanding and delivering of human natural language accurately. In this research, we explored how to apply NLP to model the knowledge and teachings of Confucius, through the natural conversation between human and computer. This virtual Confucius, a chat agent that generates outputs based on Confucius teachings, using a series of algorithms and techniques to improve the matching accuracy between user input and computer output, introduces a novel way of interacting with intangible cultures. Our user evaluation results revealed that there is a positive correlation between relevance and enjoyment, finding their experiences interacting with virtual Confucius very encouraging. Adults who participated in experiencing the virtual Confucius together with their children believed that this system has the potential to improve intergenerational interactions through shared play.
KeywordsPhilosophical conversation Linguistic computation Accuracy measurement Intangible cultural heritage Virtual Confucius Chat agent
In 2003, UNESCO declared the Convention for the Safeguarding of the Intangible Cultural Heritage, which defined the intangible cultural heritage as practices, representations, expressions, knowledge and skills that communities or individuals recognize as part of their cultural heritage1. The significance of intangible cultural heritage is, according to UNESCO, not the cultural manifestation itself but rather the wealth of knowledge and skills that is transmitted through it from one generation to the next2. Confucianism is commonly defined as system of philosophical, ethical and political thought based on the teachings of Confucius ( cited in ), which has been communicated as part of everyday living of several Asian cultures, from as far back as from sixth-century BCE. It is an intangible cultural heritage that has been integrated in to common practices, relationships and morals.
How can individuals re-familiarize themselves with an intangible cultural heritage within smart environs? "Creating intelligent cultural spaces"  is one of the innovative ways of recreating interactive spaces to immerse in intangible cultural heritage. This project created a digital interactive system, where individuals can interact with Confucius through his teachings and experience a cultural heritage within smart environs. In the code of ethics in Confucianism, relationships between elders and youngsters are highly valued and revered . Taking this aspect into account, considering the growing distance between old and young in this digital era, this project encourages sharing of experiences of Confucius heritage between elders and youngsters.
Thus, there are key significant aspects around which this research project has been developed: the intergenerational sharing and communication and use of modern digital interactive technology to create intangible cultural heritage experiences.
Intergenerational communication is viewed as an activity that enables interaction or exchange of verbal and nonverbal symbols between any two generations, involving sharing of skills, knowledge and experience between the grandparents or parents, and children. Play has many proven benefits for intergenerational communication, evident in prior studies. However, there is limited research which examines play between grandparents, parents and children, let alone interactive play systems that facilitate intergenerational communication. Prior literature and our own initial survey with grandparents, parents and children reviewed that there are currently limited digital play and entertainment systems which are designed for their simultaneous consumption, despite their eagerness to participate in those activities together. In this research, cultural play is defined as engaging in a play activity which allows the user to experience the core aspects of his or her culture. Design-oriented research approach was employed in this research, to develop the research prototypes, while involving the intergenerational users throughout the design process. The research prototypes were carefully engineered to meet the requirements of the users.
There have been encouraging results on the use of modern communication technology in bridging intergenerational gap, such as digital story telling, empowering both parties through communication across generations . This motivates us to design a new form of cultural play system, where users can explore cultural values and teachings through digital media. We have created Confucius Chat, a philosophical conversational agent which models Confucius knowledge and teachings of Confucius, that allows parents and children to share information. Contemporary users could significantly benefit from this interactive and personalized advice from virtual Confucius, which is not possible in passive media, such as the printed text. This philosophical conversational technology could be used to model philosophers in different cultures.
Recent research on intergenerational communication across cultures has indicated that people in Asian nations construed older family and non-family members as less accommodating than did people in Western nations . It is possible that the lack of understanding of the grandparents’ culture, for example the ethic of filial piety, makes salient age-group identities and, thereby, triggers intergroup processes . In support of this argument, studies have also shown that strong traditional Confucian norms, such as filial piety and elders’ contribution to family harmony, have resulted in youths having more positive images of old age, closer psychological proximity and thus more respectful communication with older adults in the East [3, 9, 11]. Thus, it is important that the young generation could interact with new media that promotes ancient philosophies and culture, so that they will have better understanding and communication with the older people.
2 Literature review
There is an emerging trend in entertainment research known as cultural computing, which allows the user to experience an interaction that is closely related to the core aspects of his or her cultural heritage . Similarly, Tosa et al.  think of cultural computing as a method for cultural translation that uses scientific methods to represent the essential aspects of a cultural heritage.
Human communication is originally something cultivated in an environment comprising localities, national customs and language. Therefore, the fruits of these cultures have strong roots in their unique histories. [...] Now, as the computer society covers the earth, the task that computers must take on is the clear and accurate intercommunication between local and global cultures. Toward that end, it is first necessary for those involved with computer technology to bring to life local characteristics
For example in ZENetic Computer , the user inputs the elements that he/she wants in his/her sansui painting. Based on the user input, the system then tries to infer the user’s internal consciousness and generates a story that the user can "enter" via the computer display. The user can respond to objects presented by the interactive system by manipulating input media, such as a virtual calligraphy brush or rake of a Zen rock garden, on-screen images or simply by clapping hands. By exerting effort to link the fragmentary stories, the user interactions help to decrease the gap between daily self and hidden self. The system aims to allow users to experience a virtual unification of their daily self and their unconscious self into a recreated conscious self through this dialogue with the system.
In another research project, ALICE , Salem et al took inspiration from Alice in Wonderland project  and created an augmented reality (AR) narrative with intelligent agents acting as characters who lead the user through virtual and real locations, moral choices and emotional states. ALICE is designed to provoke self-reflection on unconscious cultural issues such as logic and reasoning, self and ego, selfishness and selflessness. This is achieved by giving users the opportunity to occupy and experience any of these mental and emotional positions as they move along the plot of Alice in Wonderland.
ZENetic Computer and ALICE projects used interactive storytelling and compelling visual to bring users through specific intangible cultural contents. On the other hand, we would like to take a more open-ended approach, to allow the parents and children the flexibility to ask a wide range of questions, and the system would reply with the most relevant answer from the knowledge database, hence presenting the cultural content directly to the user. Our cultural play system employs natural language processing methods to analyze the user’s input sentence, for example the keywords, the sense of the keywords and their corresponding topics. At the same time, the system models Confucian knowledge and teachings, by engaging Confucius scholars to provide data set for classifying each of the Confucius database entries. The system will then retrieve the most relevant entry from the database based on the proximity of the input sentence and the scholars’ classification of the entry. In addition, k-nearest neighbor training algorithm, a text classification method, is used to improve the accuracy of the system retrieval.
Confucius Chat offers an alternative approach to existing virtual chat agents, for example ELIZA, Hex and Jabberwacky. Existing chat agents use simple pattern-matching approaches and employ tricks to cover up the failure of understanding the user input, for example frequently switching topics or rephrasing the input by replacing the first person’s pronouns to second person’s pronouns and vice versa. These approaches fail to help users gain further understanding on the topic of discussion and thus offer minimal benefits to the interaction.
3 Designing cultural play
3.1 Problem exploration
While exploring the problem of developing a computer to allow intergenerational communication through cultural components, we have also gathered suggestions for new forms of intergenerational play system from elderly and young people. The research is conducted in Southeast Asia; hence, the suggestions are mostly influenced by SE Asian intangible cultural values. Many elderly users suggested using play-like systems to let the children learn about traditional values. When asked about what they meant by traditional values, some mentioned Confucian values, Eastern values and family traditions. They have highlighted that since the children are spending substantial amount of time with electronic gadgets like computers and mobile devices, it would be beneficial if there are applications that allow them to explore cultural values that utilize their familiar devices and provide them with enjoyment. This would serve as an activity that they could do together with children. Many children also have shown clear interest in exploring intangible cultural heritage using games or interactive systems. Some of them mentioned that their parents bought educational software for them to learn traditional values. The contents are normally presented in text with illustration, comics and videos.
We carried out a survey with parents and younger siblings of university students. A total of 20 parents between the age of 46 to 53 and 15 children between the age of 11 to 16 took part in the survey. The participants were prompted on whether they would like to learn or explore intangible cultural values, for example Eastern values using interactive media, such as games and social network chat. Many parents think that the idea is interesting and may appeal to children. If available, they would like to use it together with the children. Most children reported that it would be fun to learn about the cultural values through games or chat with historical figures. The participants were asked the current method the family members communicate about traditional cultural values. Most of them replied that there is currently no method in place, and few mentioned that they talked about those issues when they arose naturally. In another question on whether they are currently reading any traditional cultural content on books or on the Internet, a few parents reported reading the books or searched for those material online and few children mentioned that they do read those books in school. A further question is on the enjoyment of reading those cultural contents. The parents reported that the activity is meaningful and enjoyable. Most children reported that the activity is not very interesting.
3.2 Design goals
Based on the observations above, we established the design goals for the cultural play as listed below:
(1) Intergenerational cultural communication Studies have shown that strong traditional Confucian norms, such as filial piety and elders’ contribution to family harmony, have resulted in youths having more positive images of old age, closer psychological proximity and thus more respectful communication with older adults in Asia. Therefore, we wanted to create a play system that would facilitate discussion of traditional Eastern cultural values between grandparents and parents with their children. In this way, children may have a better understanding of the cultural values embraced by their parents and grandparents.
(2) Dynamic interaction to explore intangible cultural heritage Currently, Eastern cultural values and teachings are communicated orally in a family and are available in the traditional media. Traditional media, for example books, animation or videos, only provides a linear presentation of the subject matter, and the user is normally a passive receiver of the information. However, children today possess high level of digital literacy and are more inclined to explore new knowledge using digital media. This motivates us to design a new form of cultural play system, where the children can explore cultural values by actively contributing to the discussion using modern digital media, together with their parents.
3.3 Design requirements
3.3.1 User needs
We considered various cultural contents which are relevant to the users, for example Confucius, Mencius, Lao Zi and traditional Chinese concept of Yin and Yang, many of which were recommended by the parents. We decided to start with Confucius as his philosophies, and values have deep influence in Asian Chinese culture. Confucianism has gained popularity in books, animation and movies. For instance, the book written by Yu Dan about Confucius  has witnessed phenomenal sales, indicating a high demand for Confucian knowledge in modern Chinese societies.
Another important factor for choosing Confucius for the content of our cultural play is that his philosophies have significant influence on family values of most Southeast Asian cultures. The significance of family can be seen from the following statement outlining the process of Confucian moral cultivation in the Da Xue (Great Learning) chapter of the Book of Rites , “Extension of knowledge lay in the investigation of things. Things being investigated, knowledge became complete. Their knowledge being complete, their thoughts were sincere. Their thoughts being sincere, their hearts were then rectified. Their hearts being rectified, their persons were cultivated. Their persons being cultivated, their families were regulated. Their families being regulated, their states were rightly governed. Their states being rightly governed, the whole kingdom was made tranquil and happy” .
Apparently, family is the first test ground beyond the individual self for a cultivated person to manifest himself before he can make an impact on the society. Furthermore, out of the traditional five cardinal interpersonal relations (father–son, husband–wife, younger and elder brother, ruler–subordinate and friends), three are family-based. Discussions on filial piety are disproportionately abundant in Confucius literature. This factor is particularly important, as our aim is to facilitate intergenerational communication by allowing the children to better understand their family values.
3.3.2 Context of use
The system is envisioned to be used in a home setting by the children and their parents. The system should be a simple application or accessible from a Web site and is available whenever the parents and children want to explore the traditional cultural values. The system should allow the users to input questions or statement to promote their discussion and reflection on the system output. To facilitate learning and discussion, a record of their interaction would be available for the users to review at later time.
3.4 Design idea generation
Modern-day children are highly exposed to digital culture. Through modern networked and social digital media such as the Twitter and Facebook, they make friends, explore new forms of entertainment and expand their knowledge. “24% of teens go online ‘almost constantly,' facilitated by the widespread availability of smart phones.”3 says PEW research center in their overview of 2015. Ninety-two percent of teenagers use their smart phones to access social networks daily. Sixty-seven percent of teen social networks say they update their page at least once a week. However, usage of such media should not be seen as only for leisure. Teenagers look to their social networks for much more than gossip and photograph sharing. To them, social networks are a key source of information and advice.
Thus, we would like to design a system in which the users can interact dynamically with a virtual historical character in a social network chat environment, as means to explore and understand traditional Eastern values. The user is no longer a passive receiver watching or reading the cultural content; instead, she will be an active inquirer engaging in stimulating dialogue with the historical giant who shares his or her values and wisdoms. In this way, the knowledge is also presented in the user’s context, which would be more meaningful and personalized.
3.5 Prototype iterations
In this section, we provide an overview of the prototype iterations from the first prototype which addressed more technical issues, through the more recently tested prototype, which supports more accurate system output and additional interaction features.
Cultural play prototype iteration 1
Features to be tested
Simple application which allow user to input a question or statement. Identify potential usability issues
Human factors issues
Confirmation that the users are able to have text-based conversation with virtual Confucius using the chat application
Context of user
Enter an input sentence to trigger a virtual Confucius reply
Artist/engineers drafted designs of the chat application and ideas of technical challenges
Meetings with users
Review of existing implementation of automated chat agent
Meetings with designers
Simple chat application implemented using Python. No graphical user interface, only plain text input and output
Uses Artificial Intelligence Markup Language (AIML) to create a database of templates of Confucius’s answers to questions which are frequently asked
Users are required to have a conversation with virtual Confucius using the chat application
They will enter a sentence and wait for the system to reply
What works well?
Text chat with virtual Confucius is intuitive
What is lacking?
We noted the limitation of pattern-matching algorithm of AIML. Sometimes when user asked a question in a different sentence style, the system failed to retrieve relevant output
Cultural play prototype iteration 2
Features to be tested
To address the limitation of AIML pattern matching, semantic similarity measurement is introduced to the chat system. A web-based graphical user interface is created with virtual Confucius portrait and matching design style
Human factors issues
Confirmation that the intergenerational users are able to use the chat application to explore cultural values
Context of user
6 pairs of Singapore Chinese parent and child participated in the study. The children are aged 7 to 11, while the parents are aged 38 to 52
Three chat interaction scenarios tested: alone, use together with peer and use together with either parent or child
Meeting with designers and engineering team
Meetings with users
Sketches of the style and layout of the graphical user interface were reviewed
Meetings with designers
Semantic similarity measurement method is introduced to overcome the limitation of simple pattern matching in the previous prototype
A Java web application was created so that users can access the system from any web browser
A database of Confucian statements from four classical texts was created
General usability issues with the chat interface and interaction between the two paired users
Tested in the laboratory setting with 12 users aged 38–52 (parents), 7–11 (children) and demonstrated at many conferences and exhibitions
Photograph and video recording of user interaction. Questionnaires involving closed- and open-ended questions. Different users’ chat histories were saved in a single text file for content analysis
What works well?
Parents and children enjoy using the system together
What is lacking?
Users find it fun and enjoyable to talk to virtual Confucius
Virtual Confucius Chat output accuracy could be improved
Cultural play prototype iteration 3
Features to be tested
Revised chat system with improvement on the retrieval accuracy
A personal chat log with feedback rating
Human factors issues
Enjoyment and usability issues of parents and children using the chat application to explore cultural values
Context of user
A total of 83 users who are parents and children took part in this study
The study was conducted during a community event
The participants either use the system alone or in pair, with a parent or child
Various machine learning and classification algorithms were reviewed
Meetings with users
Meeting with Confucius scholars
Meetings with designers
k-NN classification method is used to further improve the retrieval accuracy of the system. 5 Confucius scholars provided training and evaluation data for our system
A personal chat log, corresponding to a unique username, was created so that user can review their previous interaction with virtual Confucius and provide rating feedback on each input-output pair
Enjoyment and usability issues of parents and children using the chat application to explore cultural values
Tested in a residential area setting with 83 parents and children users
Photograph and video recording of user interaction. Questionnaires involving closed- and open-ended questions. Different users’ chat histories and their ratings on enjoyment and relevance were saved based on username
What works well?
With k-NN classification, the retrieval accuracy is improved significantly
What is lacking?
Users have reported high level or enjoyment using the system and given high rating on the relevance of the system output
to important concepts, historical persons and texts. For instance, if the user asks
“Hi. How are you?”
Confucius’s reply will be taken directly from our AIML database, which replies
“Who is Yan Hui?”
This AIML database consists of short introductory statements about the figures that appear in Confucius’s responses (mostly Confucius’s disciples), classical texts from which Confucius often quotes (like the Odes) and certain names of ancient countries and dynasties (like the state of Lu and the three dynasties of Xia, Shang and Zhou). This database will also include certain information about Confucius as an individual. Since user may be curious about Confucius as a person, they may ask about his personal particulars such as his age, his date of birth and his hometown. We gather this information from the earliest reliable historical text of Shiji by Sima Qian (ca. 110B.C.) and formulate them into Confucius’s answers. Besides, we also prepared a few series of dialog sequences, which would be initiated by virtual Confucius asking the user a question. This would make the conversation between the user and virtual Confucius more interactive. The prototype was tested with project team members and laboratory members to gather feedback and identify potential usability issues. In this prototype, we noted the limitation of pattern-matching algorithm of AIML, where sometimes when user asked a question in a different sentence style, the system failed to retrieve relevant output.
“Yan Hui is my favorite disciple.”
Since many of the passages are considerably long (especially those in the Book of Rites and the Classic on Filial Piety) and are comprised of several parts each with a distinct meaning, instead of transporting the whole paragraph of text into our database as one entry, we separate them into short phrases, but each is a self-sustained statement loaded with meaning. For instance, the opening passage of the Analects becomes three entries in our database: 1. Is it not pleasant to learn with a constant perseverance and application? 2. Is it not delightful to have friends coming from distant quarters? 3. Is he not a man of complete virtue, who feels no discomposure though men may take no note of him? In this way, our database is expanded to 2069 entries. The Confucius scholar assigned every Confucius database entries as a vector, with a combination of topics which best describe them. In this way, the system would identify the topics in user input sentence and compare with the database entries to find the closest match based on the semantic closeness of the input topics vector and the database entry vectors. We have also created a web application, so that users can access the system from any web browser. A pilot study was carried out with six pairs of parents and children to identify usability issues and their interaction experience.
“to Zhou belonged the eight officers, Bo Da, Bo Kuo, Zhong Tu, Zhong Hu, Shu Ye, Shu Xia, Ji Sui, and Ji Gua.” 
In prototype 3, the current prototype, a personal chat log, corresponding to a unique username was created so that user can review their previous interaction with virtual Confucius. We have also incorporated rating feedback on the Web site so that users can rate each input-output pair. This information is collected for future improvement of the system.
Set of family topics
Entry is related to parents
Entry is related to the children in a familial context
Entry is related to sibling
Entry makes references to the young in the general sense or as a collective group of people
Entry is related to the elderly
Entry is related to the ancestors
Entry is related to rites and ceremony
Entry is related to obedient behavior or conduct
Entry is related to either conflict or harmony in the family
Entry is related to love
Entry is related to respect or reverence
Entry is related to treat or serve another person
Entry is related to death
Entry is concerned about being morally upright
Entry is related to marriage
Entry is related to food
Entry is related to the transmission or acquisition of knowledge
Entry is related to money
Entry is related to filial (only if the entry is directly talking about filial and not for those that implied filial)
Entry is related to relatives or neighbors
Entry is related to health or well-being
Entry related to family (only for general entry that refers to family. If the entry refers to specific family members like father and mother, please choose other topics)
Entry is related to friends or peers
Five Confucius scholars, and final year undergraduate students, recommended by a faculty member who taught them during a Confucius studies module, were involved in selecting those entries that are only related to the family topic. A total of 108 database entries were selected. First, the Confucius scholars have to provide a set of topics which could be used in combination, to describe each of the 108 entries. This is an iterative process where the scholars provide a set of topics and manually check through the entries to identify new topics, which is unique to the existing pool of topics and is important to describe the database entries. The new topics are added to the pool of topics, and the process repeats until they felt that the set of topics are sufficient to describe each database entries. The final set of topics, a total of 23 topics, is shown in Table 4.
4 System description
4.1 Artificial Intelligence Markup Language retrieval
The user input sentence is first put through Artificial Intelligence Markup Language (AIML) database  to retrieve an output. The output sentence is then evaluated for its score (from 0.0 to 1.0). The score evaluation is based on the number of matched words in the input sentence, discounting randomness and a list of words that Confucius will not talk about.
maps to two templates in the AIML database. First is the template “_PLEASE” which means that if the sentence ends with the word please, the reply from this template will be used and the words before the word please will be used to search for a second template,
“What is your name please”
The reply for first template is
“WHAT IS YOUR NAME”.
and the reply for second template is
“Thank you for being polite”.
Hence the reply for the user input is
“My name is Confucius”.
For this example, there are two templates which match all the words in the input sentence; hence, the score is 1.0. The score is divided by half for the more general templates in AIML database which offers random output. If the output sentence contains any word in our forbidden word list which is a list of words Confucius will not discuss, for example God and Jesus, the score will be 0.0. If the score is above a predetermined threshold value, currently set at 0.7, the output from AIML will be used. Below the threshold value, the input sentence is passed to knowledge database retrieval step for further processing.
“Thank you for being polite. My name is Confucius.”
4.2 Knowledge database retrieval
The core of our system is a Similarity module, which calculates the highest similarity score for each keyword in user input sentence with the topics to identify a topic for each keyword. The set of topics are then used to retrieve the closest matched database entry based on k-NN method.
4.2.1 Identifying Keywords
To achieve the above, first the computer must understand the input sentence. This is not an easy task, because the user’s input is natural human language, which has very complicated structure, and even slight changes of the order of the words may alter the meaning of the sentence. Therefore, simple keyword matching will not work well and we need a more sophisticated method to analyze the meaning of the input. The user input is fed into a parser to get the grammatical structure of the sentence. Our system used the Stanford Parser  because of its speed and reliability. The last noun of each noun phrase is selected as headword  of the sentence. Usually, they are the topics the user is talking about. However, sometimes there are no nouns in the user input, or there are some important words that are not nouns. Therefore, we employ another method called inverse term frequency to find the important words. An inverse term frequency database is created by calculating the frequency of appearance of each word in a large corpus. Study shows that the more frequently used words such as "the" and "and" do not contribute much to the real meaning of the sentence, whereas the less frequent words, "loyal" and "conflict," for example, are the more important words .
With the result from the above two methods combined, the system will select three keywords, either from the headword method, or in the event that there is not enough headwords, the remaining words will be selected from inverse term frequency method. Furthermore, the user input is passed through a Word Sense Disambiguation (WSD) module, so that we not only know what are the keywords of the input, but also the meaning of these words in the context .
4.2.2 Identifying topics
We then compute the semantic similarity between these selected keywords and the 23 topics provided by the Confucius scholars. This is done using a WordNet-based similarity module WordNet::Similarity developed by Ted Pedersen. For each topic \(T_x,\) there are several topical words,
4.2.3 Identifying Confucius entry
We have five sets of data from five Confucius scholars. One data set is randomly selected and reserved as an evaluation set to evaluate the overall output accuracy of our system using the k-NN classification algorithm. Ideally, for any input sentence, the output given by our system should be the same as the one given by the domain expert, which means the system output should be as close as possible to human domain expert output. The rest of the 4 data sets are used to train the classifier.
Classification accuracy using different values of k
Furthermore, the process is repeated 10 times, repartitioning the samples in each iteration, to get a better estimate of the accuracy. The classification accuracy using different values of k is shown in Table 5 and Fig. 8. k value for the classifier is influenced by many factors, including the number of Confucius scholar data sets, agreement between the Confucius scholars’ data sets and other nonlinear system parameters. To determine the most suitable value of k for our system, cross-validation method , a well established technique to determine the value of k, is used. Based on the different values of k and their corresponding classification accuracy in the test, \(k = 3\) is selected using k-fold cross-validation method.
5 Technical results
To evaluate the performance of our system, we carried out glass-box and black-box evaluation . Glass-box evaluation attempts to look inside the system and measures how well each module performs, while black-box evaluation attempts to measure how well the system performs as a whole. We carried out the glass-box evaluation by examining each system module carefully. For black-box evaluation, the users gave their rating on relevance and enjoyment for each input-output chat entries.
5.1 Glass-box evaluation
The evaluation set is randomly selected from one of the five Confucius scholar’s data sets.
Keywords retrieval accuracy
Total number of keywords
Number of keywords matched
Accuracy (number of keywords matched/total number of keywords)
Topics retrieval accuracy
Total number of topics
Number of topics matched
Accuracy (number of topics matched/total number of topics)
k-NN method retrieval accuracy improvement (k = 3)
Total number of output entries provided
Number of output entries matched
With k-NN method, an accuracy improvement of 39.39% was observed when compared with average performance of Set 1 to 4. Comparing to the worst case Set 3, an improvement of 76.92% was observed.
5.2 Black-box evaluation
There is a positive correlation between relevance and enjoyment (r (778) = .673, p < .01), indicating that as the user ratings for relevance increase, the enjoyment ratings also increase.
5.3 Example of input-output retrieval
The system selects the keywords of the input sentence using two methods, headwords and inverse term frequency. The sentence is first fed into a parser to get the grammatical structure of the sentence. The last noun of each noun phrase is selected as headword of the sentence. Usually, they are the topics the user is talking about. However, sometimes there are no nouns in the user input, or there are some important words that are not nouns. Therefore, we employ another method called inverse term frequency to find the important words. The words with higher inverse term frequency value are considered more important in the sentence. Furthermore, the input sentence is passed through a Word Sense Disambiguation (WSD) module, so that we not only know what are the keywords of the input, but also the meaning of these words in the context. The output of WSD module is in the format word#parts of speech#sense number. For example, instability#n#3 is the noun sense number three of the word instability, which is a lack of balance or a state of disequilibrium; united#a#1 is the adjective sense number one of the word united, which is characterized by unity or joined into a single entity. The selected keywords with their corresponding senses in the sentence are instability#n#3, family#n#2 and united#a#. The details of how the keywords are selected are presented in Sect. 4.2.1. Based on the keywords, the system identifies the topics based on the semantic similarity of the keywords and topics as described in Sect. 4.2.2. In this example, the system is able to correctly determine the topics based on the input sentence. The topics identified are 22 and 9 which correspond to family and harmony topics, respectively, in Table 4. The system then finds the three most relevant database entries, based on the closest distant, as described in Sect. 4.2.3. In this case, the three closest entries belong to class 1283, 1970 and 1283. Based on the majority vote from the three closest neighbors, 1283 is the entry that will be output. The number 1283 refers to the index of the entry in the database. In this case, the output selected by the system matches the output given by the expert,
“What would cause the instability of a family and how should we stay united?”
“A family must first destroy itself before others can destroy it.”
In this research, we applied the NLP algorithms onto an intangible cultural heritage and created a virtual chat agent. It modeled the Confucius knowledge and teachings, delivering them intelligently through a natural language chat with human. To understand both the meaning and context of user’s natural language input and retrieve relevant answer, k-nearest neighbor (k-NN) algorithm was employed in the system to improve the retrieval accuracy. Five Confucius scholars were engaged to provide input-output data sets for the training and evaluation of the system. A total of 432 vectors, 4 each to describe a database entry, are obtained. To use k-NN algorithm, k value which yields the best performance is obtained using a k-fold cross-validation method. Each database entry is treated as a unique class, described by a set of vectors manually assigned by Confucius scholars. When a new input sentence is entered to the system, natural language processing methods are employed to determine the keywords and corresponding topics in the sentence. k-nearest neighbor algorithm will then determine the most relevant class the input sentence belongs to, based on the similarity of the input sentence topics and the vectors describing each database entry. The database entry corresponding to the selected class will be output. The software engineering details to build the system prototypes are presented.
We also carried out evaluations to test the system performance and the experience of users. Glass-box evaluation attempts to look inside the system and measures how well each module performs step by step. It was carried out by measuring the computation accuracy of each module carefully. Black-box evaluation is through the user rating on the feeling of relevance and enjoyment for each input-output chat entries, to examine how well the system works as a whole. From the glass-box evaluation, the system is able to identify the keywords and topics with an accuracy of 88.72 and 81.20%, respectively. Based on the input sentence provided by a Confucius scholar, the system selected output is compared to the Confucius scholar output. An accuracy of 42.59% was obtained using k-NN method. There is an accuracy improvement of 39.39% when compared with the average performance of individual scholar’s classification. For black-box evaluation, more than 70% of the users gave rating of 4 (agree) or 5 (strongly agree) for the enjoyment and relevance of virtual Confucius’s response to their input sentence or question. There is a positive correlation between relevance and enjoyment (r (778) = .673, p < 0.01), indicating that as the user ratings for relevance increase, the enjoyment ratings also increase.
Through processing the natural language input and computational matching with the database, we created a novel merging of ancient philosophy with recent media literacy through interactive cultural play. Our studies showed that users gave quite positive feedbacks on their experience with virtual Confucius. They enjoyed using it and were willing to share their stories with this virtual philosopher, just like talking with real friend. They also believed that this media could improve intergenerational interaction.
Currently, the Confucius Chat system has been extended into a mobile application, iSage (Fig. 13), which offers users advice, based on various philosophers and knowledge base. In this application, the users can choose to ask the virtual Sage on topics ranging from love, fate and many more to be expanded in the future. The topic knowledge databases were obtained from various sources. Based on the algorithm of Confucius Chat system, iSage allows user to interact with the application using natural language chat. The iSage application has been deployed on the Android Market4. We hope this work will in future be used to achieve new interactive experiences with all forms of intangible cultural heritage.
The authors would like to appreciate and acknowledge Eng Tat Khoo, Wang Xuan, Wei Liu and Wei Jun for their significant contributions to this research.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.