Keywords

1 Introduction

University education is now changing from a one-way teaching style to active learning, by which students learn various topics subjectively. As a backdrop to this transition, specialized knowledge is easily obtained from a variety of media, such as the Internet, at very low cost. In this situation, the most important thing for a university is to educate students in a self-learning style and provide a good learning environment. If this approach succeeds, students deepen their study spontaneously. In this kind of education style, the teachers’ role is not that of a transmitter of specialized knowledge but that of a facilitator who studies deeply with students and creates something new with them. From this point of view, a university classroom is a co-creative field between teachers and students.

With this kind of education, its quality depends on the communication between teacher and students. Therefore, in the future, it will be very important to evaluate the quality of the communication and to improve it. There are some previous studies that have dealt with communication between teacher and students. Katsumata et al. investigated such communication by analyzing the relationship between the voice of the teacher and the movements of students, which were captured by a video camera [1]. In this study, they showed that there was a positive correlation between the correlation coefficient of the teacher’s voice and the students’ movements with the degree of interest of the students. Watanabe et al. also investigated the relationship between the movements of a teacher and those of the students for estimating the degree of students’ understanding in the class [2, 3]. In their research, they showed that there were some periods when the teacher’s movements synchronized with the movements of students and speculated that these periods were important for evaluating the degree of students’ understanding. In another study, by Miura et al., the researchers analyzed the synchronization of body movements in a lecture-style experiment [4]. They reported that the nodding rhythm between a teacher and a student was more highly synchronized when the degree of the student’s understanding was high. Moreover, they reported that the students, who were the listeners in this experiment, synchronized their nodding 10 ms earlier than the teacher’s subconscious nodding.

In these studies, the relationship between the synchronization and the degree of students’ understanding in the class was investigated by analyzing the rhythm of body movements or of the voice. However, in each of these studies, a video camera and accelerometer sensors attached to the forehead were used for measuring the body movements of the teachers and students. In the case of a video camera, there is a limitation in the amount of detail that can be measured in the body movements of many students. Additionally, in the case of accelerometer sensors, it is difficult to attach such sensors to students’ foreheads in a real class. Therefore, in our research, we focus on the accelerometer sensor of a smartphone, whose use is widespread and which is owned by most of the students, for measuring the body movements of students in an actual classes. Recently, a group of researchers investigated the activity of an organization quantitatively using this kind of accelerometer, and they improved the activity of the organization by using the analyzed data [5].

In this study, we first developed and constructed a system to measure the body movements of students in a class using the accelerometer sensor of a smartphone. Next, we measured a seminar that was held in a university laboratory setting using this system and analyzed the data from the accelerometer sensor, utterance corpus, and body movements corpus. Using these data, we discuss the possibility of evaluating the communication between teacher and students.

2 Measuring System and Experiment

2.1 System Overview

Figure 1 shows an overview of the measuring system that was developed, composed of a server and client system.

Fig. 1.
figure 1

Overview of measuring system

Full size image

As the client system, Android and iOS devices are used, and they send the measurement data from the accelerometer sensor to the server system repeatedly using HTTP (Hypertext Transport Protocol). The server system receives the sent data and presents them to a browser in real time. Through these processes, a user can monitor the data for each student on site. In the following subsections, the client and server system are explained in more detail.

2.2 Client System

To take advantage of smartphones that are in widespread use, the client system was developed as an application for an Android or iOS device. In these devices, the application handles two tasks. One task is to measure the accelerometer value. In each device, the application measures the accelerometer X, Y, and Z values at a sampling rate of 100 Hz. After this measurement is taken, in order to acquire the norm value, the application sums the squared values for each axis and calculate square root of the result. The other task is to send the acquired data. In each device, the application sends the 50 calculated norm values and their time of measurement, including the ID of the device, to the server system every 500 ms.

Figure 2 shows the appearance of the measurement application on an Android device. There are setting windows for the IP address of the server, duration of measurement, and ID of the device. In this application, the dialogue box appears when there is an input mistake, and after the start of the measurement, the setting screen changes to another screen to prevent mistouches. In this application, as long as a user does not push the stop button, the application continues to send the measured data to the server system.

Fig. 2.
figure 2

Client system for an Android device

Full size image

2.3 Server System

The server system was essentially developed as an HTTP server. The system receives the calculated norm data from the accelerometer, which are sent from the client system, and presents them to a web browser. Apache was used as the HTTP server program and PHP was used in the development of the receiving and presentation of the data.

Figure 3 shows a picture of the accelerometer data from the students as displayed by the server system. The server system first receives the 50 data from the clients and saves them to a text file every 500 ms. The system then retrieves the last 5 s of data to make the graph of each device. In the graph, the newest data are shown at the left. In the graphing, the data are properly presented by showing the ID of the device and the time measured. In this system, not only the accelerometer’s raw data but also the frequency-analyzed data are presented to allow recognition of the characteristics of the body movements.

Fig. 3.
figure 3

Measurement data presented by the server system

Full size image

2.4 Overview of Experiment

Using the system described above, the communication between a presenter and the presenter’s audience in a seminar at the university’s laboratory was measured as an example of a lecture at the university. After measurement, the accelerometer data, corpus of utterances, and body movements were analyzed. The presentation, given by a student, was measured five times, and for each presentation there were about 20 persons comprising the audience. The presenter and 4–6 members of the audience wore Android devices for measurement, and the scene of the presenter and audience were recorded by two video cameras.

Each Android device was hung from the neck of the presenter or audience member by a strap with a plastic case. The device was adjusted to be positioned at the solar plexus in order to measure the nodding that was often observed in the class.

One video camera was placed to capture the presenter and the other was placed to capture the entire audience. In video recording, two synchronized digital clocks were placed one in front of each camera to allow the time relationships between the accelerometer data, utterances, and body movements to be readily associated. All measuring devices, including these two clocks, were synchronized to under 100 ms by a time setting application.

For the data analysis, first, a frequency analysis of all measurement data from the accelerometer was conducted to investigate the relationship between specific body movements and specific frequencies. Next, sonograms of the accelerometer data were made to investigate the temporal change of body movements of the presenter and the audience. Moreover, the correlation between sonograms was analyzed to investigate the relationship between the body movements of the presenter and of each audience. Using recorded video data, corpuses of utterances of the presenter and audiences were made, and each utterance was tagged [6] to allow an analysis of dialogue structure. In addition, corpuses of body movements of presenters and audiences were made by describing the movements one by one; these were used to analyze the relationship between the accelerometer data and body movements. In this study, there are five sets of measurement data; because of space considerations, however, one representative set of results is used for explanation in the next section.

3 Results

3.1 Frequency Distributions of Accelerometer Data

Figure 4 shows the frequency distribution of the accelerometer data of the presenter, and Figs. 5 and 6 show those of two typical audience members. These frequency data were for the entire measurement period. In this measurement, the sampling frequency was 100 Hz, but the range of the graphs is 0–15 Hz because the object of measure was a human, who cannot move so quickly.

Fig. 4.
figure 4

Frequency distribution of accelerometer data of a presenter

Full size image
Fig. 5.
figure 5

Frequency distribution of accelerometer data for audience member A

Full size image
Fig. 6.
figure 6

Frequency distribution of accelerometer data for audience member B

Full size image

In these figures, it is possible to recognize frequency peaks in the following three areas: 0–3 Hz, 3–7 Hz, and over 7 Hz. The relationship between the corpuses and the accelerometer data corresponding to these peaks was analyzed. The results show that the peaks at 0–3 Hz correspond mainly to major body movements such as the swaying of a shoulder. The peaks at 3–7 Hz correspond to small body movements such as normal nodding or a hand movement. The peaks above 7 Hz correspond to quick body movements such as laughing or speaking. In the case of Fig. 5, the audience member often nodded during the seminar. The audience member of Fig. 6, on the other hand, did not nod so much. As a result, in Fig. 5, there is a clear peak around 4–6 Hz. Additionally, in the case of Fig. 5, the audience member sometimes laughed during the seminar. As a result, there is a peak around 10 Hz that is not observed in Fig. 6 as this audience member did not laugh.

These results show that it is possible to detect specific body movements of a presenter or audience using the accelerometer of a smartphone that is hung from the neck.

3.2 Sonograms of Accelerometer Data and Relationship Between Sonograms

Figure 7 shows the raw accelerometer data and the corresponding sonogram for a presenter, and Figs. 8 and 9 show those of two audience members. The window of analysis for these sonograms was 4 s.

Fig. 7.
figure 7

Acceleration data (bottom) and corresponding sonogram (top) of a presenter (Color figure online)

Full size image
Fig. 8.
figure 8

Acceleration data (bottom) and corresponding sonogram (top) of audience member A (Color figure online)

Full size image
Fig. 9.
figure 9

Acceleration data (bottom) and corresponding sonogram (top) of audience member B (Color figure online)

Full size image

In this seminar, the presenter talked for about 6 min. From Fig. 7, it is possible to recognize this period from the raw data of the accelerometer and the corresponding sonogram. After the presentation, audience member A asked a question of the presenter, and then audience member B asked a question. In Figs. 8 and 9, it is possible to recognize these activities by the accelerometer data and the corresponding sonogram. In particular, in the sonogram, some peaks (highlighted in yellow) are observed. In the previous subsection, it was seen that some body movements appear as a frequency peak. The sonogram data suggest that some specific body movements appear intermittently.

These results suggest that it is possible to detect the active period of the body movement of a presenter and an audience during communication and also to clarify the specific movements by using the frequency data of an accelerometer.

Figures 10 and 11 show the time series of correlation coefficients between the sonograms of a presenter and those of audience members A and B, respectively. Figure 12 shows the time series of correlation coefficients between the sonograms of audience members A and B. The coefficient values were calculated from the average of 1 min of sonogram data, and the window slides every 30 s for this analysis.

Fig. 10.
figure 10

Correlation between sonogram of the presenter and that of audience member A

Full size image
Fig. 11.
figure 11

Correlation between sonogram of the presenter and that of audience member B

Full size image
Fig. 12.
figure 12

Correlation between sonograms of audience members A and B

Full size image

In Fig. 10, the correlation coefficient becomes relatively high around 7–8 min, when audience member A asked some questions of the presenter. In Fig. 11, the value becomes relatively high around 10 min, when audience member B asked some questions. Additionally, in Fig. 11, the correlation coefficient becomes relatively high around 7–8 min, when audience member A (not B) asked some questions of the presenter. As a result, in Fig. 12, the correlation coefficient between audience members A and B becomes relatively high around 7–8 min.

A tendency for a high correlation between the presenter and an audience member who is not directly uttering to the presenter is observed in later parts of the seminar. In Figs. 11 and 12, for example, the correlation coefficient becomes relatively high around 12 min and again around 14–15 min. In these parts, two other audience members asked some questions of the presenter. Audience members A and B were quiet at these times, but their correlations become high.

In these peak areas of high correlation, the corpuses of utterance data and body movements show that an audience member asked a question (tag: qh) or mentioned a suggestion (cs) or point of agreement (aa) to a presenter, or a presenter nodded in response to comments and sometimes mentioned a point of agreement (aap) to the audience.

4 Discussion and Conclusion

In this study, a measuring system using the accelerometer of a smartphone was developed to evaluate the communication between a teacher and students in the class. Moreover, the communication between a presenter and the audience in a seminar of a university laboratory was measured, and the accelerometer data and corpuses of utterances and body movements were analyzed. The results show that it is possible to detect the body movements of the presenter and audience members by the frequency characteristics of the accelerometer data and that the correlation between the frequency distribution of the presenter and that of the audience members becomes relatively high when a presenter has a discussion with them.

Our final purpose is to evaluate the quality of communication between a teacher and students using the accelerometer sensor of a smartphone in the class. The results of this measurement suggest that there is suitable potential for detecting the presenter’s movements while speaking and the audience’s movements while asking questions using the raw and frequency-analyzed accelerometer data. At present, the relationship between a specific movement and characteristics of the frequency distribution is manually associated using the corpus of movement data. However, in the future, it will be possible to detect specific body movements using a deep classifier system that is trained with the frequency data tagged with the body movements. Using such a learning method, it will also be possible to display the distribution of students’ body movements in class in real time.

The results of the analysis of sonogram correlations show that the relationship between the frequency distribution of a presenter and that of an audience member becomes similar when they interchange comments with each other. This result may come about from the synchronization of body movements between presenter and audience. In previous studies, the synchronization between the voice of a teacher and body movements of students, or the synchronization of body movements between students, has been observed when students’ level of understanding became high. In this study, we can observe the same phenomenon. In the high-correlation part of the seminar, the presenter nodded, corresponding to comments from the audience, and at the same time, an audience member nodded, corresponding to the response from the presenter. This kind of communication and synchronization may indicate a deep level of rapport.

The similarity of frequency distributions appeared not only between a presenter and audience member who have a discussion but also between the presenter and audience members with whom the presenter does not have a direct exchange. This result means that the synchronization of body movements was widespread among other audience members who were merely present in the room. This phenomenon might be important for classroom rapport. The nodding of students is important feedback for a teacher to sense the level of understanding in the class. If this feedback is widely synchronized among the class, both teacher and students could feel a sense of unity in learning something new. At the same time, this sense of unity would promote the involvement of students in the class. It is suggested that in a good class, this cycle of the synchronization of body movements, emergence of sense of unity, and subjective involvement might occur. In future work, the synchronization of body movements should be analyzed in more detail to evaluate the quality of communication between teacher and students. To carry out this analysis, subjective evaluations of teacher and students (e.g., levels of understanding, levels of interest) will be investigated and associated to the accelerometer data or corpuses of utterances and body movements. By such an analysis, the possibility of a method for class evaluation will be discussed.