Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Most nursing care facilities hold leisure activities on a daily basis. Elderly people with reduced cognitive functioning often exercise, play games, craft products, play and listen to music, do housework, and so on. Although most elderly people with memory problems cannot remember their activities, they often retain memories of the pleasant feelings associated with such activities. The care staff at such facilities often say that elderly people begin to talk significantly more, do more by themselves, and have fewer problem behaviors as a result of leisure activities. Hence, we can expect to improve their quality of life (QOL) and prevent their behavioral and psychological symptoms of dementia (BPSD).

Creative activities, including cooking, handicraft work, gardening, and ceramic artwork, provide them with a sense of accomplishment. These activities require them to execute a series of small tasks. They are quite willing to continue in such activities, as they find them enjoyable.

Most elderly people with reduced cognitive functioning also participate in music therapy, which is one of the leisure activities offered in the facilities where they attend. In Japan, they play percussive musical instruments like the tambourines and castanets in an ensemble and achieve a feeling of togetherness. Many researchers [1–4] have demonstrated that music therapy allows people with dementia to reduce their aggressive behavior, a BPSD. However, it is difficult for the participants of music therapy to achieve a sense of accomplishment. Many specialized lessons taught by a professional teacher are required to achieve technical growth in playing these musical instruments.

On the other hand, keyboard instruments have a high threshold [5]. A performer has to learn how to read music scores and hit the appropriate keys using all ten fingers. Professional teachers are needed to give them individual guidance and support. Hence, the leisure activities in nursing care houses do not include playing the piano. However, we believe that even elderly people with reduced cognitive functioning can try to play the piano when given appropriate support for performance and lessons. The most important thing is that the people in such a facility become willing to continue playing the piano with a sense of accomplishment.

There are many systems that assist people in learning how to play musical instruments. “The Piano Tutor [6]” provides computer-based instruction to beginning piano students. The system can follow people’s performances and analyze their performance and select appropriate remedial actions. “Piano Tutor for iPad [7]” is an application for learning the piano. The system teaches mistakes in performances by high-score tracking. “Piano Marvel [8]” shows where people should hit on a small picture of a keyboard. The system also shows people where they made mistakes. “Lighted Keys [9]” is a keyboard that includes the 3-Step Lesson System and lights the key the performer should hit next. In a piano learning system [10], the projector is set above a keyboard. The system refers to the next key that is to be pressed and each next key is outlined in color to provide keying information. These systems may be useful for adult piano beginners. However, it is difficult for most elderly people with reduced cognitive functioning to find the position of the correct key that the system shows on the display. They cannot learn fingering numbers and hit the keys with the correct fingers shown on the display. Our research aims to help people with reduced cognitive functioning continue being motivated in playing the piano rather than that to gain skills.

In the first step of this research, we conducted an experiment in which an elderly woman with “higher brain dysfunction” took piano lessons using educational video materials. She was asked to play an electronic piano following the instructions in the educational videos. In general, a student has a face-to-face piano lesson with a teacher. However, we employ the educational videos because we aim to construct a piano lesson support system that such elderly people can use pianos alone in the future. We show her progress on the piano and that she gained a sense of accomplishment. Then, we discuss improvements in a construction of a display and a system to provide a sense of accomplishment for people with reduced cognitive functioning.

2 Experiment 1

2.1 Participant

The participant is a woman in her 70 s with higher brain dysfunction due to a brain contusion. The higher brain dysfunction includes impaired cognitive functioning caused by a brain contusion, aprophoria, apraxia, agnosia, and dementia. The occurrence of these symptoms depends in part on her brain contusion. She has constructional and ideational apraxia and a reduction in skilled behavior due to her injured frontal lobe and temporal lobe. She cannot move her hands skillfully now, although she was a professional sewer when she was young. She may have semantic paraphasia. Her score of the revised version of Hasegawa’s Dementia Scale (HDS-R) was 14/30. In particular, the temporal orientation score was low. In activities of daily living, she needs a little help to eat a meal, change her clothes, and take a bath, although she is independent in toileting. She has not played the piano so far. She likes Hibari Misora, who was a very famous singer in Japan, and often sings her songs.

2.2 Informed Consent

The participant in this experiment was informed about the intentions of the experiment and the treatment of personal information. Moreover, she was informed that she could withdraw from the experiment at any time. No reward was prepared for her. Then, we obtained written consent from her.

The second author, who is a nurse, observed the experiment. If she had estimated that the participant was tired, felt bad, and was mentally hard, the experiment would be terminated early.

2.3 Method

Experiments were carried out three times in twelve days. Figure 1 shows that the participant played an electronic piano (YAMAHA NP-30) that has 76 keys while watching an educational video on a 27-inch display. The video showed not musical scores, but the hands of model performances. Because the target of our research is elderly people with cognitive dysfunction, it is difficult for them to learn and memorize each key position corresponding to each note. The participant was asked to play the piano as a help for us because she might be nervous to play the piano for the first time.

The musical piece used in this experiment was “Kawa no nagare no yoni (like a flowing stream),” a Hibari Misora song. The participant liked the piece very much. One of the authors wrote down the hook-line of this piece in a score. The participant could play this part using only eight white keys: G3, A3, B3, C4, D4, E4, F4, and G4. She used both hands to play the piece. Her thumbs did not need to pass through other fingers or other fingers did not need to pass over her thumbs.

As shown in Fig. 2, colorful seals were put on eight keys. Seals whose colors corresponded to each key were also placed on the participant’s and the video performer’s fingers.

The performer in the video repeated each phrase six times (model performance). First, the participant listened to the phrase one time. Then, she played each phrase five times along with the model performances like shadowing.

Fig. 1.
figure 1

The participant plays the keyboard while viewing the display.

Full size image
Fig. 2.
figure 2

Each finger is assigned a key.

Full size image

2.4 Video Educational Material

One of the authors made three educational videos for the participant’s lessons. Figure 3 shows three practice phases as examples. The hook-line was divided into many parts (phrases). Example 1 shows the start of the hook-line. The first note should be played using the participant’s left hand, and the second note should be played using her right hand. First, the participant practiced a very short phrase. Then, she practiced a slightly longer phrase. The phrase in Example 3 includes that of Example 2. After the participant practiced Example 2, she then practiced Example 3.

The videos for the second and third lessons were made after the first lesson and the second lesson, respectively. The phrases used on three lessons were the same. In the video for the second lesson, the performer said the color of the key (see Fig. 2) immediately before playing the key. For example, the performer said “Red” immediately before hitting the red key. We expect that the participant can find the correct key more quickly.

In the video for the third lesson, the way the hook-line was divided into the phrases differed from that of the first and second lessons. For the first and second lessons, we divide the hook-line into nine phrases and ten phrases for the third lesson. A shorter phrase allows the participant to practice with less of a mental burden.

Fig. 3.
figure 3

Examples of the phrases for practicing.

Full size image

2.5 Method of Analysis

We recorded the participant’s performance and counted the errors in pitches and fingerings. Rhythm is also an important element for music. The participant knew the musical piece very well because she often sang it. Therefore, she understood the correct rhythm of this piece. However, she had a difficult time physically hitting the keys in the correct rhythm. Therefore, in this experiment, we analyzed only the pitches and the fingerings.

The pitch errors are divided into “wrong,” “missing,” “extra,” and “holding keys.” Wrong means that the participant hit the wrong key. Missing means that she did not hit any keys. Extra means that she hit an extra key. The holding key means that she held down some keys since before starting to play a phrase.

Moreover, we considered a hitting a key “incorrect fingering” when she hit the key with a finger that did not correspond to the finger that the performer in the video used to hit the key.

2.6 Result

Performance. Table 1 shows the results of her performances (three lessons). The numbers of all notes that the author played in the videos are 280 (the first and second lessons) and 270 (the third lesson). The ratios of hitting the keys to the number of all notes are 80 % (224/280), 81 % (228/280), and 87 % (236/270) in the three lessons.

The number of pitch errors gradually decreased. Although “wrong” decreased in the second lesson from 43 to 19, it increased in the third lesson from 19 to 27.

“Correct keying” means the number of correct keys hit with the correct fingerings. The number in the second lesson is smaller than that of the first lesson although the number of the wrong keys hit decreased.

Table 2 shows the number of fingering errors. The number of “incorrect fingerings” increased in the second lesson from 64 to 149. Therefore, the ratio of the incorrect fingerings to the correct pitches decreased in the second lesson. The participant was apt to play the keys with one finger in the second lesson.

Observation. The participant played each phrase along with the model performances as though she was shadowing. As the lessons progressed, she continued playing the phrase even after the model performance had finished. Moreover, when she made extra type errors, she hummed a tune and replayed the correct key. She may have hit the keys while judging whether each key was correct or not relying on remembering what she heard.

She forgot that she had played the piano on the night of taking the first and second lessons. Soon after the second lesson, she said “I could play the piano a little” with a smile. After the third lesson, we observed her cleaning up the dust on the edge of the piano although the author had already cleaned the piano.

Moreover, she complained that it was difficult to find the same key as the key the performer played in the video.

Table 1. The results of the participant’s performances (Experiment 1).
Full size table

2.7 Discussion

The finding that the ratio of hitting the keys to the number of all notes was greater than 80 % shows that she played the piano with enthusiasm. Even elderly people with reduced cognitive functioning can try to play the piano with support. Moreover, the results show that the number of pitch errors gradually decreased. She said, “I could play the piano a little” after the second lesson. The motion of her cleaning up the dust on the edge of the piano may show her affection for the piano. Therefore, we recognized that she gained a sense of accomplishment through the piano lessons.

On the other hand, the number of correct notes with correct fingering decreased in the second lesson. We thought that she focused on touching the correct keys because the performer in the video said the color of the key. She had hardly watched the display, but saw the color of the keys and found the correct key one after another. Therefore, she might have hit most of the keys with her index finger.

Table 2. The results of fingering errors.
Full size table

In general, students who learn to play the piano know that the numbers from one to five are assigned to the fingers from the thumb to the little finger, respectively. Some notes in musical scores indicate the number of fingers that the piano player should use. However, it is difficult for elderly people with reduced cognitive functioning to recognize the fingering number on the score and hit the key with the finger applied to the designated number. Even if we were to indicate the next key by coloring or lighting the key [9] without a voice, she would have hit all of the keys using one finger.

Furthermore, the participant complained that it was difficult to find the same key as the key the performer played in the video. The eight keys of the piano that the participant and the performer played had colorful seals placed on them. However, it must have been difficult for her to find the same color on the piano as the key in the video because of her symptoms.

3 Experiment 2

3.1 Display Setup

In Experiment 1, it was difficult for the participant to find the same key as the key the performer played in the video. Therefore, we abandoned putting the seals on the keys as well as the participant’s and the performer’s fingers. As shown in Fig. 4, we calibrated the length of the keyboard in the video. Each key in the video corresponded to the same pitch key on the real piano. Moreover, we improved the appearance of the video so that the participant could find the correct keys easily. The model performance appeared like a mirror.

Fig. 4.
figure 4

Model performance that the participant can watch like a mirror.

Full size image

3.2 Method

Two months after Experiment 1, the same participant was asked to play the same electronic piano as in Experiment 1. The musical piece used in the experiment was the hook-line of the same piece. However, only five phrases were prepared and the time of the educational video was ten minutes.

There were two experimental conditions. In the first condition, the model performance in the video appeared as shown in Fig. 4. The performer in the video played the piano facing the participant like a mirror.

Figure 5 shows the second condition. As a comparative condition of the first condition, the performer in the video played the piano looking ahead in the same direction as the participant was playing. The experiment of the second condition was conducted after the experiment of the first condition.

Fig. 5.
figure 5

Examples of the phrases for practicing.

Full size image

3.3 Result

Table 3 shows the results of the participant’s performances. The number of all notes that the author played in the videos is 145. The ratios of hitting the keys to the number of all notes are 99 % (144/145) and 100 % (145/145), respectively.

There is a significant difference between the two conditions in the item “extra (p=0.04).” This result might be not be due to the differences between the two conditions, but rather the learning effect.

We can see the number of “wrong” errors is very high. In Experiment 1, the ratio of wrong errors to the number of keys hit are 19 % (43/224), 8 % (19/228), and 11 % (27/236) in the three lessons. On the other hand, in Experiment 2, those are 72 % ((104/144) and (104/145)) in both conditions.

Table 3. The results of the participant’s performances (Experiment 2).
Full size table

Table 4 shows the number of correct fingerings in the wrong hittings. The ratios of the number of correct fingerings to the wrong keys hit are very high. Namely, the participant hit the keys with the correct fingering although most of the keys were wrong. Fig. 6 illustrates an example in which the participant hit the wrong keys with the correct fingerings. The figures below each note indicate the number for each finger. We can see that the participant played the piano with much the same motion of fingers as that of the performer in the video.

Table 4. The number of correct fingerings when the wrong keys were hit.
Full size table
Fig. 6.
figure 6

Examples of the correct fingerings while hitting the wrong keys.

Full size image

3.4 Discussion

The result that the ratio of hitting the keys to the number of all notes is almost 100 % shows she played the piano with enthusiasm. In this experiment, we improved a composition of the video as shown in Figs. 4 and 5. However, the number of “wrong” errors was very high although most of the wrong keys were played with the correct fingerings. She might find it easier to consider the fingerings because of the calibration of the keyboard’s length. It is difficult for the participant to find the correct keys corresponding to the keys that the performer hit in the video. The first reason is that she has constructional apraxia. For example, people with constructional apraxia cannot make the reconstruct a model with blocks. Namely, they may not be good at estimating the correct key as a real object in comparison with a key on a screen image.

The second reason is that the quality of experimental materials was low. The frame of the display might interrupt her exploration of the correct keys. If the display connected to the keyboard of the piano seamlessly, she would find the keys more precisely. Moreover, it might be hard for her to determine the position of the key the performer hit in the video. We find it hard to determine whether each key is going down because the color of the keys’ surface is white and that of keys’ sides are light brown.

4 Conclusion

We aimed to construct a piano lesson support system that gives the people with reduced cognitive functioning a sense of accomplishment. We conducted an experiment in which an elderly woman with “higher brain dysfunction” took piano lessons using educational video materials. The results of Experiment 1 showed that the participant could try to play the piano in spite of being a beginner. She became skillful and gained a sense of accomplishment through the piano lessons. However, when the performer in the video said the color of the key each time, the participant hit any keys with only her index finger.

The results of Experiment 2 showed that the participant hit many of the wrong keys although she played with the correct fingerings. We thought that she might find it easier to consider the fingerings because of the calibration of the keyboard’s length. However, we considered that people with reduced cognitive functioning might find it hard to find the correct keys in comparison with keys on a screen image. The participant might have had a hard time knowing the position of the key in the video because the frame of the display interrupted her exploration, and it was hard to know when each key was going down.

As the next step, we will develop a model performance with computer graphics and use a frameless display.