Multiple Representations of the UI, Score and Scale for Musical Performance System and Score DB

Abstract

This paper describes the development and evaluation of the UIs and scores of musical performance system. The aim of this research is to provide a musical tool for elderly people and caregivers. Another aim of this research is to use the performance system and score database to conserve some non-Western music genres. The UIs are designed on a tablet PC, which look like keyboards. Four UIs with scores are evaluated by young people and elderly people: plain keyboard with staff notation score, and keyboards of note name/number/color representation with scores of the same representation. The results of the experiment indicate that the number representation would be useful to most people and the note name/color representations would be useful to some people. This system also have a UI of koto music which is one genre of Japanese traditional music. This system can be used for the explanation of the difference between the temperament and scale of koto music and those of Western music. Scores for musical performance system and scores of koto music are also discussed. This research provides a system to add melody and rhythm using score DB to the lyric input by a user. This system can also explain the melisma of koto songs and asynchronicity of the rhythms of voice part and koto part.

1 Introduction

In the field of music therapy, music listening and/or singing are commonly taking place [1,2,3]. However, it is difficult to use a musical instrument with staff notation scores. The objective of this research is to provide a musical performance system and score database to improve the quality of life of the people (especially elderly people) who have difficulty with using staff notation scores for performance [4]. Many musical performance system have been proposed and evaluated in the field of Computer Music [5,6,7,8], however, it is difficult for elderly people without musical experience to use these systems.

Today, Western music is accepted worldwide; however, the temperament and the scale of music are simplified, e.g., Twelve-tone equal temperament and the Major/Minor scales are commonly used. Another objective of this research is to use the performance system and score database to conserve some non-Western music genres. Koto music is discussed in this research.

2 UI and Score for Musical Performance

2.1 Representation of UI

This research has developed a musical performance system on tablet PCs. This system has several user interfaces: UI-1: UI like keyboard instrument, UI-2: UI with note names on keyboard, UI-3: UI with numbers on keyboard, UI-4: UI with colors on keyboard, and UI-5: Different layout of UI-3 (vertical keyboard). Figure 1 shows UI-1, Fig. 2 shows UI-2, Fig. 3 shows UI-3 and Fig. 4 shows UI-4.

Fig. 1.

UI-1: plain keyboard

Fig. 2.

UI-2: keyboard with note names

Fig. 3.

UI-3: keyboard with numbers

Fig. 4.

UI-4: keyboard with colors (Color figure online)

In UI-2, note names are written in Japanese. In UI-3, numbers: 1 to 7 represent notes: C, D, E, F, G, A and B. In UI-4, colors: red, orange, yellow, green, light blue, blue and violet represent notes: C to A. Note names, numbers, or colors with an upper/lower dot mean the pitches are one octave higher/lower.

In our previous research [4], we designed three layered keyboard as Fig. 5; however, we redesigned one layered keyboard (normal keyboard) with scrolling function because some people mentioned about the layout of keyboard in the questionnaire of previous experiment. We also decided to eliminate the UI with shapes on keyboard because the result of previous experiment showed that shapes were not recognizable as musical notation.

Fig. 5.

UI of previous version

This system provides sound sources of piano (C2 to C7) and koto (C2 to C7) in WAVE data format. Our previous system provided sound sources of piano and organ. These sounds were generated by additive synthesis. This system can read a musical score from score database which was developed in previous research, and a user can listen to the melody before performance. This system can also record user’s performance to support practice and improvisation. This system has a metronome function so that a user can play the system rhythmically.

This system was implemented by HTML, CSS and JavaScript, and it is working without perceptible time delay on Windows tablet PCs.

2.2 Representation of Score

Each user interface needs different score form: UI-1: staff notation, UI-2: note name representation, UI-3: number representation, UI-4: color representation, and UI-5: same as UI-3. Note names are written in some staff notation scores today for the beginners. Numbered notation is common in Japanese traditional music and some genres of non-Western music in the world.

The scores for UI-2, UI-3 and UI-4 are generated from musical score database (DB) which was developed by this research in Humdrum format [9]. We have extended the system to generate scores from MusicXML format data [10]. This system can display two types of scores. Type 1 scores specify the durations as the lengths of space and Type 2 scores specify the durations by using symbols as Western music. Type 1 scores are designed based on the notation of Ikuta school koto scores [11], while type 2 scores are designed based on the notation of Yamada school koto scores [12]. Figure 6 shows Type 1 score of number representation, Fig. 7 shows Type 1 score of color representation, and Fig. 8 shows Type 2 score of number representation. In these scores, Lyrics are written below note names, numbers or colors.

Fig. 6.

Type 1 score of number representation

Fig. 7.

Type 1 score of color representation

Fig. 8.

Type 2 score of number representation

2.3 Experiment on UI and Score

The experiments of previous research [4] showed that UI-2 (note name) was easy to play the system and that UI-3 (number) was easy to play and sing at the same time. It also showed that UI-4 (color) could be useful. The 16 examinees of the previous experiments were college students.

Methods.

In this research, UIs are improved and UI-1, UI-2, UI-3 and UI-4 are evaluated by elderly people and young people as follows.

• Examinees:

  32 young people of age 20–24 (24 people without experience of keyboard instruments, and 8 people with experience), students of Engineering School, 1 female and 31 male.

  27 elderly people of age 60–79 (21 people* without experience of keyboard instruments, and 6 people** with experience), people who live near our University, all female. * age 60–64: 3 people, 65–69: 5, 70–74: 10, and 75–79: 3. ** 60–64: 1, 65–69: 2 and 70–74: 3.

• Songs used for the experiment: Sakura for UI-1, Haruno-ogawa for UI-2, Yuyake-koyake for UI-3, Mushinokoe for UI-4.

• Date: November, 2018 for elderly people, December, 2018 for young people.

• Condition before the experiment: The examinees did not practice the system.

• Condition during the experiment: First, the examinees played the system using each UI and score. Next, the examinees played and sang using each UI and score.

The examinees answered the questions by rating 4, 3, 2 or 1 (4:positive, 3:mildly positive, 2:mildly negative, 1:negative) for each UI and its score after using them. Questions are as follows.

• Q1: Is the score easy? Q2: Is the UI easy?

• Q3: Is it easy to play? Q4: Is it easy to play and sing at the same time?

Results and Discussion.

The results of the experiments are as follows.

People Without Musical Experience.

The mean values of questions for four UIs and scores answered by young people (24 people without experience of keyboard instruments) are shown in Table 1, and those answered by elderly people (21 people without experience) are shown in Table 3. Table 1 indicates that UI-2 and its score (note name representation), UI-3 and its score (number representation) and UI-4 and its score (color representation) might be easier to recognize and to use than UI-1 and its score (staff notation) for the people who are not familiar with musical performance. While, Table 3 indicates that UI-2, UI-3 and UI-4 might be easier to use than UI-1, however all UIs and scores may be almost the same to recognize.

Table 1. Mean values of evaluation of UI-1/2/3/4 by young people without experience

Table 2. T-ratios of t-test for UI-1/2/3/4 by young people without experience

Table 3. Mean values of evaluation of UI-1/2/3/4 by elderly people without experience

Paired sample t-test was used for the comparison of the mean values of each question for two UIs. UI-1 and UI-2, UI-1 and UI-3, UI-1 and UI-4, UI-2 and UI-3, UI-2 and UI-4, and UI-3 and UI-4 are compared. In the experiment of 24 young people, the degrees of freedom is 23, and the critical value for significance level of 0.05 (two-tailed test) is 2.07. T-ratios of those comparisons are shown in Table 2. Table 2 shows that there is a significant difference between the mean values of UI-1 and UI-2/UI-3/UI-4 in Q3. This indicates that UI-2, UI-3 and UI-4 would be easier than UI-1 to play the system. Table 2 also shows that there is a significant difference between the mean values of UI-1/UI-2 and UI-3/UI-4 in Q4. This indicates that UI-3 and UI-4 would be easier than UI-1 and UI-2 to play and sing at the same time. The experiment of previous research showed the same result for UI-1, UI-2 and UI-3, and we confirmed that the number representation would useful to play and sing at the same time. Since the sequence of numbers (1, 2, 3 … 7) is acquired by everyone, the spatial recognition of the number sequence on keyboard should be easy for most people. On the other hand, the experiment of this research showed that the color representation would be also useful to sing and play at the same time. In this system, the layout of colors on keyboard is based on the spectrum, so the spatial recognition of the color sequence might be easy for some people.

Paired sample t-test was also used in the experiment of 21 elderly people. The degrees of freedom is 20, and the critical value for significance level of 0.05 (two-tailed test) is 2.09. T-ratios of UIs’ comparisons are shown in Table 4. The result indicates that UI-2 and UI-3 would be easier than UI-1 to play the system. It also indicates that UI-3 would be easier than UI-1 to play and sing at the same time, however, it cannot indicate that UI-3 would be easier than UI-2 to play and sing at the same time. Note names would conflict with songs when young people use UI-2 and its score to play and sing at the same time; however, elderly people could sing without reading the lyric because they knew the songs (old Japanese songs) better than young people.

Table 4. T-ratios of t-test for UI-1/2/3/4 by elderly people without experience

People with Musical Experience.

The mean values of questions for four UIs and scores answered by 8 young people with experience of keyboard instruments are shown in Table 5. Table 5 indicates that UI-1 and its score (staff notation) would be useful for these people. Paired sample t-test was used for the comparison of the mean values of Q4 for UI-1 and UI-3. The critical value for significance level of 0.05 (two-tailed test) is 2.36, and t ratio is 1.93, therefore, this data cannot show the difference because the data is small.

Table 5. Mean values of evaluation of UI-1/2/3/4 by young people with experience

The mean values of Q4 for UI-1 answered by 8 young people with musical experience (3.13 in Table 5) and 24 young people without experience (2.13 in Table 1) are compared using Welch’s t-test. The critical value for significance level of 0.05 (two-tailed test) is 2.20 and t ratio is 2.24, therefore, this data can show the difference. UI-1 and its score (staff notation) would be useful for the people with experience of keyboard instruments. The mean values of questions for four UIs and scores answered by 6 elderly people with experience of keyboard instruments are not analyzed because the data is too small.

About the Performance.

The mean values of Q3 and Q4 in Tables 1 and 3 for four UIs are also compared using paired sample t-test. In the experiment by 24 young people without experience of keyboard instruments, the degrees of freedom is 23 and the critical value for significance level of 0.05 (two-tailed test) is 2.07. T-ratios of the comparisons of mean values of Q3 and Q4 (in Table 1) are shown in Table 6. Tables 1 and 6 indicate that UI-2 (note name) would be easy to play but difficult to play and sing, and that UI-4 (color) would be difficult to play but easy to play and sing. In the experiment by 21 elderly people without experience, the degrees of freedom is 20 and the critical value for significance level of 0.05 (two-tailed test) is 2.09. T-ratios of the comparisons of mean values of Q3 and Q4 (in Table 3) are shown in Table 7. Tables 3 and 7 indicate that UI-2 would be easy to play but difficult to play and sing.

Table 6. T-ratios of t-test of Q3 and Q4 by young people without experience

Table 7. T-ratios of t-test of Q3 and Q4 by elderly people without experience

Choice of One UI and Score.

The examinees also answered the question: Which UI and score do you choose to play and sing? The numbers of people who chose each UI and score are as follows.

• (1) 24 young people without experience of keyboard instruments

  staff notation: 2, note name: 4, number: 8, color: 10

• (2) 21 elderly people without experience of keyboard instruments

  staff notation: 5, note name: 7, number: 4, color: 5

• (3) 8 young people with experience of keyboard instruments

  staff notation: 4, note name: 2, number: 0, color: 2

• (4) 6 elderly people with experience of keyboard instruments

  staff notation: 2, note name: 0, number: 3, color: 1

Conclusion of the Experiment.

The analysis of results shows that number representation would be useful to play and sing at the same time for the people who are not familiar with musical performance. The analysis also shows that note name/color representation would be useful for some people. The analysis shows that staff notation would be difficult for the people without musical experience; however, some of them prefer staff notation to other representations. On the other hand, staff notation would be useful for the people with musical experience; however, some of them prefer note name, number or color representation. Therefore, we decided to provide these four UIs and scores in our system.

3 Temperament, Scale and UI for Non-Western Music

This research added some functions to our system to conserve non-Western music genres. In this research we discuss koto music (one genre of Japanese traditional music) as non-Western music.

3.1 Temperament and Scale

Temperament.

Koto music uses Pythagorean tuning system. In koto music, the tuning system had been strictly defined; however people today often use 12-tone equal temperament. Therefore, our system provides 12-tone equal temperament and Pythagorean system so that users can choose one of them.

The temperament of koto music was defined in the author’s research in 2001 [13]. We extracted the intervals between two notes sequentially from the koto score data, and found that the semitones and whole tones were restricted. We defined the interval used for the semitone as “x” and the interval not used for the semitone as “y”. The sequence of these intervals (x and y) was determined as shown in Fig. 9. We calculated the frequency-ratios of tones: x = 256/243, y = 2187/2048, xy = 9/8. Figure 9 shows that the interval between D and D# is 256/243 (semitone), D and E is 9/8 (whole tone), D and G is 4/3 (perfect fourth), D and A is 3/2 (perfect fifth), and D and D’ is 2 (one octave).

Fig. 9.

The temperament of koto music

Scale.

Users of this system can also study the scales other than the Major or Minor scales. Seven different scales are theoretically possible, and it is said that six different scales were used in medieval music. Koto music uses a scale which is different from Major or Minor scales. Our system provides an explanation about five scales: the Major, Minor, Japanese court music (Ryo and Ritsu) [14, 15], and koto music so that users can know the difference. The sequence of intervals (semitones and whole tones) is as follows, where “s” is semitone and “w” is whole tone.

• Major: w w s w w w s

• Minor: w s w w s w w

• Japanese court music (Ryo): w w s w w s w

• Japanese court music (Ritsu): w s w w w s w

• Japanese traditional music (Miyako-bushi): s ww w s ww

  • (Miyako-bushi is basically a pentatonic, where “ww” is a major third.)

• koto music: s w w w s w w

In Major and Minor scales, there are keynotes. If the scale is Major and the keynote is C, it is called C Major. There are 15 keynotes in Major and Minor scales. In koto music, there are also keynotes. Since the intervals (frequency-ratios) of two sequential tones are not uniform (256/243 or 2187/2048), there are six keynotes theoretically. The author analyzed the frequency of each note appeared in koto score data because tunings cannot specify the keynotes, and found the five keynotes. We call them koto-modes as follows [16].

• D koto-mode: D D# F G A A# C D’ (mostly used in Hira-choshi tuning)

• G koto-mode: G G# A# C D D# F G’ (mostly used in Kumoi-choshi tuning)

• A koto-mode: A A# C D E F G A’

• C koto-mode: C C# D# F G G# A# C’

• E koto-mode: E F G A B C D E’

3.2 Representation of UI

Our system provides the UI for koto music as Fig. 10, which is similar to UI-5 (vertical version of UI-3: number representation), with string numbers on the keyboard. Koto has 13 strings and each string is numbered: 1, 2 … 10, To, I, Kin (To, I and Kin are written in Chinese characters, and represent 11, 12 and 13). There is already a koto performance system (iKoto) on iPad [17], which has a UI that looks like a real koto with 13 strings. Since the aim of our system is the explanation about the difference between koto music and Western music, we developed the UI as keyboard.

Fig. 10.

The UI of koto music for explanation

The UI of Fig. 10 is for D koto-mode (Hira-choshi tuning). The uses could play this system using koto scores, however, there are several restrictions in this version. Basically, users cannot use left-hand playing techniques, e.g., “Tsuyo-oshi” (pushing the string to make a pitch whole tone higher) can be replaced by playing the key marked as tsuyo-oshi (black triangle mark in Fig. 10) and “Yuri” (vibrato) cannot be played. Users cannot use normal right-hand playing techniques either, e.g., “Sha” (playing 2 tones at almost the same time) can be replaced by chord using multi-touch.

4 Score DB

4.1 Songs for Musical Performance

Score DB.

We have been working on the DB of public domain songs. We have encoded 17 Japanese Children’s songs and 10 English Children’s songs in Humdrum format. We are now converting them to MusicXML format. We would like to add more songs to DB, however, we can’t use most pieces of popular music because of the copyright. On the other hand, there are huge resources of Classical music pieces (mainly instrumental music) and there are some sites which provide free DBs.

Adding Rhythm and Melody to a Lyric.

We have been developing the following system: When a user input a lyric to the system, the system adds melody and rhythm to the lyric using Classical music pieces. There are some systems which generate melody and rhythm to a lyric [18], however, our approach is reusing huge resources of Classical music. In this system, the rhythm (sequence of durations) is determined by the number of syllables and the parameters specified by the user. First, we developed the system for haiku (Japanese seventeen syllable poem). Since haiku has 5 syllables, 7 syllables and 5 syllables, we developed the system using rhythmic patterns of 5 notes and 7 notes, then we revised the system using algorithm to generate rhythms for 5 notes and 7 notes. Our system generates rhythms in four-four time. Examples of the rhythm for 5 notes are (quarter, quarter, 8th, 8th, quarter), (quarter, 8th, 16th, 16th, half), and so on, and examples of the rhythm for 7 notes are (8th, 8th, quarter, 8th, 8th, 8th, 8th), (quarter, 8th, 8th, 16th, 16th, 8th, quarter), and so on. On the other hand, the melody (sequence of pitches) is cut from the original piece and the starting point is specified by the user. Figure 11 shows the system which adds rhythm and melody to haiku.

Fig. 11.

The system for adding rhythm and melody to the lyric

In this research, we revised the system to generate rhythms for any numbers of notes to adapt the system for general poems. We made a DB of rhythmic patterns from score DB of Classical music in order to retrieve the rhythmic patterns for the numbers of syllables input by a user. For example, if a user inputs a poem: 6 syllables, 8 syllables and 7 syllables, the system retrieves rhythmic patterns for 6 notes, 8 notes and 7 notes. We used the score DB of Mozart’s string quartets, which was developed by CCAHR, Stanford University [19].

Questionnaire.

We have asked examinees who attended the experiment for the evaluation of UIs and scores. The questions and the numbers of answers are as follows.

• (1) Do you want to use the system which adds melody to Haiku?

  • Young people: Yes 13, No 19

  • Elderly people: Yes 11, No 16

• (2) Which is easy to write: Haiku? or Poem?

  • Young people: Haiku 10, Poem 22

  • Elderly people: Haiku 12, Poem 15

4.2 DB for Non-Western Music

We are interested in conserving non-Western music genres using musical performance system and score DB. Today, koto songs are not common even among the people who play the koto. People prefer instrumental koto music. There are two main problems in koto songs.

• (1) There are variations of the melody of voice part because koto music had been an oral tradition. Koto songs have melisma mostly in each syllable, and the variations appear in the melisma.

• (2) The rhythms of koto part and voice part are different (not synchronizing) as other genres of Japanese traditional music.

This system explain these problems using examples of koto scores.

Koto Scores.

We have score data of the koto part and voice part of following songs: Sumiyoshi (1792 tones in koto part), Sakuragari (1724 tones), Enoshima (1413 tones), Kogo (1463 tones), Yuya (1503 tones), Shikinonagame (1627 tones). Two versions notated by Nakanoshima [20] and Ito [21] were encoded. Figure 12 shows examples of score data files. The koto scores are notated using string numbers. In the score data of Nakanoshima’s Voice part, “51 1/8 11 # ke” means “bar 51, 8th note, string 11 with #, syllable ke”. The tuning defines the pitch of each string, e.g., Hira-choshi tuning defines the pitches of the 1st to 13th strings as follows: D4 G3 A3 A#3 D4 D#4 G4 A4 A#4 D5 D#5 G5 A5, where middle C is C4. The pitches are converted from string numbers into MIDI note numbers, where middle C is 60, and a semitone is 1.

Fig. 12.

An example of koto score data file

Melisma.

Figure 13 shows examples of melismas in the voice part of Nakanoshima score and Ito score. In Fig. 13, the first pitch of each syllable is notated in MIDI note number and the melisma is notated by relative pitches. In Nakanoshima score of Fig. 12, the melisma of syllable “ke” is “11# 10 11## 11# 8”, so it is converted to “76 74 77 76 69”, and then it is represented as “76: −2 3 −1 −7” in Fig. 13. Figure 13 shows that there are differences between Nakanoshima’s melisma and Ito’s melisma. A melisma is composed of patterns, e.g., “−1−2 2” is the combination of down pattern (−1) and vibration (−2 2).

Fig. 13.

Examples of melisma

The differences of melismas are classified as follows [22].

• The first pitches of the melisma are different.

• The last patterns of the melisma are different or additional patterns are attached to the end of melisma.

• The middle patterns of the melisma are different or additional patterns are attached to the middle of melisma.

Rhythm.

Figure 14 shows an example of the correspondence between koto part and voice part. Usually, the melody and rhythm of both parts are not synchronizing. The asynchronicity of rhythm is especially difficult for the people who are not familiar with Japanese traditional music. In the 50th bar of Fig. 14, string 11 with # (pushing the string to make a pitch semitone higher) is plucked at the beginning of the bar, and the syllable “ti” is pronounced in pitch 11# (the same pitch as the koto part) after 8th note from the beginning of the bar. Usually, the voice part follows the koto part; however, the voice part sometimes precedes the koto part.

Fig. 14.

An example of rhythms of koto part and voice part

5 Conclusions and Future Work

5.1 Conclusions

Musical Performance System and Scores.

This research provides musical performance system and score display system. The performance system and display system have several representations of UI and score for the people who are not familiar with musical performance. The UIs are redesigned and developed on a tablet PC, which look like keyboards with note names/numbers/colors on the keys. Scores are notated by note names/numbers/colors.

Four UIs and scores are evaluated in the experiment: a plain keyboard and staff notation score, and keyboards of note name/number/color representation and scores of the same representation. Examinees are 32 young people (24 people without experience of keyboard instruments, and 8 people with experience) and 27 elderly people (21 people without experience of keyboard instruments, and 6 people with experience). The mean values of each question for two UIs are compared by t-test. The results of the experiment indicate that the number representation would be useful to most people and the note name/color representations would be useful to some people.

Public domain score DB is used to display scores in several representations. Score DB of Classical music is used to add the melody and rhythm to the lyric input by a user of this system.

System for Non-Western Music and Scores.

This performance system also has a UI of koto music and it can be used to conserve koto music which is one genre of Japanese traditional music. This system explains the difference between the temperament and scale of koto music and those of Western music.

A koto score DB is used to conserve koto music along with the performance system. This system explains the melisma of koto songs and asynchronicity of the rhythms of voice part and koto part.

5.2 Future Work

Future work includes following problems. We have to improve the system on score DB for musical performance, which adds melody and rhythm to the lyric input by a user. DB of rhythmic patterns and the method to retrieve the melody from score DB should be enhanced. Also, the system should be evaluated by users.

We have to add some functions to the system for koto music, e.g., playing techniques. The explanation of melisma and rhythm of koto songs should be improved. Our system explains the difference between scales, so we are planning to do an experiment to know if people can recognize the difference. The analysis of asynchronicity of the rhythms will be challenging research.