A Semantic Text Retrieval for Indonesian Using Tolerance Rough Sets Models

Abstract

The research of Tolerance Rough Sets Model (TRSM) ever conducted acted in accordance with the rational approach of AI perspective. This article presented studies who complied with the contrary path, i.e. a cognitive approach, for an objective of a modular framework of semantic text retrieval system based on TRSM specifically for Indonesian. In addition to the proposed framework, this article proposes three methods based on TRSM, which are the automatic tolerance value generator, thesaurus optimization, and lexicon-based document representation. All methods were developed by the use of our own corpus, namely ICL-corpus, and evaluated by employing an available Indonesian corpus, called Kompas-corpus. The endeavor of a semantic information retrieval system is the effort to retrieve information and not merely terms with similar meaning. This article is a baby step toward the objective.

This work was partially supported by the Polish National Science Centre grant DEC-2012/05/B/ST6/03215.

Appendices

Appendix

A Weighting Scheme: The TF*IDF

Salton and Buckley summarised clearly in their paper [47] the insights gained in automatic term weighting and provided baseline single-term-indexing models with which other more elaborate content analysis procedures can be compared. The main function of a term-weighting system is the enhancement of retrieval effectiveness where this result depends crucially on the choice of effective term-weighting systems. Recall and Precision are two measures normally used to assess the ability of a system to retrieve the relevant and reject the non-relevant items of a collection. Considering the trade-off between recall and precision, in practice compromises are normally made by using terms that are broad enough to achieve a reasonable recall level without at the same time producing unreasonably low precision.

Salton and Buckley further explained that, with regard to the differing recall and precision requirements, three main considerations appear important:

• 1. Term frequency (tf). The frequent terms in individual documents appear to be useful as recall-enhancing devices.

• 2. Inverse document frequency (idf). The idf factor varies inversely with the number of documents $df_t$ to which a term t is assigned in a collection of N documents. It favors terms concentrated in a few documents of a collection and avoids the effect of high frequency terms which are widespread in the entirety of documents.

• 3. Normalisation. Normally, all relevant documents should be treated as equally important for retrieval purposes. The normalisation factor is suggested to equalise the length of the document vectors.

Table A.1. Term-weighting components with SMART notation [39]. Here, $tf_{t,d}$ is the term frequency of term t in document d, N is the size of document collection, $df_t$ is document frequency of term t, $w_i$ is the weight of term t in document i, u is the number of unique terms in document d, and CharLength is the number of characters in the document.

Table A.1 summarises some of the term weighting schemes together with the mne-monic which is sometimes called SMART notation. One example of the mnemonic is lnc.ltc. The first triplet (i.e. lnc) represents the weighting combination for the document vector, while the second triplet (i.e. ltc) represents the weighting combination for the query vector. For each triplet, it describes the form of tf component, idf component, and normalization component being used. Thus, mnemonic lnc.ltc means that the document vector employs log-weighted term frequency, no idf for collection component, and cosine normalisation, while the query vector employs log-weighted term frequency, idf weighting for collection component, and cosine normalisation. Equation A.1 is the common weighting scheme used for a term in a document, i.e. mnemonic ntn, which is called TF*IDF weighting scheme.

$$\begin{aligned} w_{t,d} = tf \cdot idf = tf_{t,d} \cdot \log \frac{N}{df_t} \end{aligned}$$

(A.1)

B Document Ranking Method: The Cosine Measure

Manning et al. [39] stated that cosine similarity is fundamental to IR systems that use any form of vector space scoring. Given a query vector and a set of document vectors in a high dimensional space, we may rank the documents by comparing the angle between the query vector and each document vector; the smaller the angle, the more similar the vectors. In linear algebra, the angle $\theta$ between two vectors, $\overrightarrow{x}$ and $\overrightarrow{y}$, can be measured as follows:

$$\begin{aligned} \overrightarrow{x} \cdot \overrightarrow{y} = |\overrightarrow{x}| * |\overrightarrow{y}| * cos(\theta ) \end{aligned}$$

(B.1)

where $\overrightarrow{x} \cdot \overrightarrow{y}$ represents the dot product while $|\overrightarrow{x}|$ and $|\overrightarrow{y}|$ represent the lenght of the vectors. The dot product $\overrightarrow{x} \cdot \overrightarrow{y}$ of two vectors is defined as $\sum _{j=1}^{M}x_{j} * y_{j}$ and the Euclidean length of a vector $|\overrightarrow{x}|$ is defined as $\sqrt{\sum _{j=1}^{M}(x_{j})^2}$. Thus, formula (B.2) can be used to measure the similarity between a query vector Q and a document vector D:

$$\begin{aligned} similarity(Q, D) = \frac{\sum _{j=1}^{M}w_{qj} * w_{dj} }{\sqrt{ \sum _{j=1}^{M}(w_{qj})^2 * \sum _{j=1}^{M}(w_{dj})^2}} \end{aligned}$$

(B.2)

C The Corpora

1.1 C.1 ICL-Corpus and WORDS-Corpus

Our original corpus, called ICL-corpus, consists of 1,000 first emails of Indonesian Choral Lovers (ICL) Yahoo! Groups and are formatted as of the Text REtrieval Conference (TREC) format [20]. Therefore our test collections consist of three parts (a set of documents, a set of information needs, and relevance judgments) and all documents are marked up in a TREC-like format, i.e. each document is marked up by $<$DOC$>$ and $<$/DOC$>$ tags, the document number is marked up by $<$DOCNO$>$ and $<$/DOCNO$>$ tags, the subject of email is marked up by $<$SUBJECT$>$ and $<$/SUBJECT$>$ tags, the date of email is marked up by $<$DATE$>$ and $<$/DATE$>$ tags, the sender is marked up by $<$FROM$>$ and $<$/FROM$>$ tags, and the text body is marked up by $<$TEXT$>$ and $<$/TEXT$>$ tags.

We worked with two choral experts intensively in the annotation process in order to construct the information needs and relevance judgments for our testbed. The annotation process consisted of two tasks which were a) topic assignment, where the human experts assigned topic(s) for each document within the original corpus; and b) keywords determination, where they determined terms considered as highly related with the topic(s) given. The annotation process aimed to grasp how the topic(s) could be assigned to a particular document which was mainly described by the keywords determined. We take benefit from these keywords as the list of terms closely related with the topic of document, as well as the document itself, and assume that the other terms not listed are less important terms. The first step of topic assignment yielded 127 topics and the keywords determination yielded a new corpus, called WORDS-corpus.

Fig. C.1.

The content of corpora. Picture on the left is an example of ICL-corpus document which consists of original document, while picture on the right is an example of WORDS-corpus document which consists of keywords given by human expert manually for particular ICL-corpus document, i.e. in this case, the ICL-corpus document with number “DR-480” which is shown on the left.

Consult Fig. C.1 to see the content of both corpora. Notice that the main difference between documents in ICL-corpus and WORDS-corpus lies in the text body, i.e. the document of ICL-corpus consists of a body of emails while the document of WORDS-corpus consists of keywords defined by human experts. Figure C.2 shows the relationship between both corpora.

Fig. C.2.

Corpus relationship. The WORDS-corpus was yielded by human expert in annotation process over ICL-corpus.

Table C.1. List of topics. This is a list of 127 topics of ICL-corpus and the total number (document frequency) of relevant documents for each topic with ID 0 to 35.

As we mentioned above, the topic assignment yielded 127 topics of which many have low document frequency; 81.10 % of them have document frequency $<$ 10 and 32.28 % of them have document frequency 1. We further processed the 127-topics, as it is shown by Tables C.1 and C.2, and came up with 28 topics as listed in Table C.3. Thus, we have two version of relevance judgments a) relevance judgment which consists of 127 topics; and b) relevance judgment which consists of 28 topics.

Table C.2. List of topics. This is a list of 127 topics of ICL-corpus and the total number (document frequency) of relevant documents for each topic with ID 36 to 126.

Table C.3. List of topics. This is a list of 28 topics of ICL-corpus and the total number (document frequency) of relevant documents for each topic.

For the 127-topics, distribution of topics is showed by Table C.4 while list of topics with document frequency $\ge$ 10 is showed by Table C.5. For all the tables here, ID column defines the topic identifier, Topic column is the topic in Indonesian, and DF column is the document frequency or total number of relevant documents with regard to the topic.

Table C.4. Topic distribution. This table shows the total number of topic which has document frequency $<$ 10 out of 127 topics.

Table C.5. List of topics. This table presents topics of ICL-corpus with document frequency $\ge$ 10 out of 127 topics.

Refer to the TREC format, Fig. C.3 is an example of relevance judgment file while Fig. C.4 is an example of the information needs file. For the relevance judgment file, the first column defines the topic identifier, the third column defines the document identifier, and the fourth column defines the relevancy, i.e. 1 if the document is relevant to the topic, and 0 otherwise. The second column is an arbitrary string and in this case brings no information. The information needs file consists of topics (string between $<$TITLE$>$ and $<$/TITLE$>$ tags) with its description (string between $<$DESC$>$ and $<$/DESC$>$ tags) and narrative (string between $<$NARR$>$ and $<$/NARR$>$ tags). It follows the TREC format, thereby marked up by some tags in which each topic is enclosed by $<$TOP$>$ and $<$/TOP$>$ tags.

Fig. C.3.

The relevance judgment file. This picture is an inset of the relevance judgment file. Respectively, column 1 to 4 are the topic identifier, random string, document identifier, and document relevancy with topic.

Fig. C.4.

The information needs file. This picture is an inset of the information needs file.

Annotation Process

We have mentioned above that the annotation process consisted of two tasks, namely topic assignment and keywords determination, and yielded WORDS-corpus and two lists of topics (127-topics and 28-topics). This was a collaborative work with three choral experts in which four phases were carried out as it is presented in Fig. C.5.

Fig. C.5.

Annotation process. The annotation process had four phases: determination, revision, decision, and categorization.

First of all, the first expert did the topic assignment and keyword determination for 1,000 documents of ICL-corpus. Considering his work (namely Result_1), we did the same thing and came with a different result (namely Result_2). Based on Result_1 and Result_2, the first expert did the revision of his previous result and produced the new result (namely Result_3). The second expert made a decision (Result_4) by analyzing Result_1, Result_2, and Result_3.

On this stage, we had 127 topics and decided to make the list smaller by categorizing it. Thus, we analyzed those topics and agreed on 28 topics. Refer to the 28-topics, the third expert reassigned each documents of ICL-corpus.

In addition to the construction process, this is another main difference of our corpus with an Indonesian corpus made by Jelita Asian from Kompas newswire articles (Kompas-corpus)⁴². In ICL-corpus, each document must be assigned by at least one topic while in Kompas-corpus it is not the case, i.e. there are documents that are not designated to any topics.

1.2 C.2 WIKI_1800

WIKI-1800 is a corpus consists of 1,800 text documents in music domain which are the short abstract of Indonesian Wikipedia articles⁴³. The full version of the corpus consists of 85,601 short abstracts in variety of topics and was downloaded from DBpedia⁴⁴. The WIKI_1800 employed in this study was obtained by filtering out the 85,610 abstracts specifically based on music domain which was conducted by our third expert.

Figure C.6 shows a small chunk of WIKI-1800 document. Each document is represented as an RDF triple notation which contains three components (i.e. subject, predicate, and object), plus the URL of the Web page. In Fig. C.6, the $<$ http://dbpedia.org/resource/Indonesia_Raya $>$, which acts as the subject, is an URI reference to the resource of Indonesia Raya. The $<$ http://www.w3.org/2000/01/rdf-schema#comment $>$ (or rdfs:comment for short), which acts as the predicate, is an URI reference that refers to the property used to provide a human-readable description of a resource; R rdfs:comment L states that L is a human-readable description of R [48]. Therefore, the string inside the quotes next to the rdfs:comment is the human-readable description of Indonesia Raya, which is actually the short abstract of the Indonesia Raya article. Finally, the $<$ http://id.wikipedia.org/wiki/Indonesia_Raya# $>$ is the URL that will go to the Web page of Indonesia Raya.

Fig. C.6.

WIKI-1800. An example of WIKI-1800 document.

1.3 C.3 The Choral Experts

In data preparation of our study, we worked in collaboration with three people who have experiences in choral for years. They were Agastya Rama Listya, Kristoforus Kuntarahadi, and Inke Kusumastuti; in Appendix C.1, we called them the first expert, second expert, and third expert respectively. Figure C.7 displays the pictures of them.

Fig. C.7.

The choral experts. The choral experts involved in annotation process of our study: (1) Agastya Rama Listya, (2) Kristoforus Kuntarahadi, and (3) Inke Kusumastuti.

Agastya Rama Listya was born in Yogyakarta on February 18, 1968, and now is living in Salatiga, Central Java, Indonesia. He obtained his Bachelor of Arts in Theory and Music Composition from the Indonesian Arts Institute, Yogyakarta, Indonesia, in 1992. In 2001, he received his Master of Sacred Music in Choral Conducting from Luther Seminary and St. Olaf College, Minnesota, USA. He was the Dean of the Faculty of Performing Arts, Satya Wacana Christian University at Salatiga for two periods (2009–2011) and was affiliated as the committee member of Badan Kerjasama Gereja-Gereja se-Salatiga (2007–2010), Lembaga Pengembangan Pesparawi Daerah Jawa Tengah (2007–2010), and Badan Pembina Seni Mahasiswa Indonesia Jawa Tengah (2008–2010). Agastya has published 7 books, 6 articles in journals, and 16 essays. He is a productive music composer and arranger in which many of his choral works were performed by numerous choirs in Indonesia. He is also an active choral coach of a number of choirs where under his direction have made some prominent achievements regionally, nationally, and internationally. Individually, he was the winner of 4 different national choral composition contests during 1998–2009 and the winner of Yazeed Djamin Award for Piano Composition Contest in 2006. Agastya Rama Listya’s name was included in the 30th Pearl Anniversary of Marquis Who’s Who in The World (November 2012).

Kristoforus Kuntarahadi was born in Yogyakarta on January 14, 1979. He is now a staff in the office of Bishop’s Conference of Indonesia, in Jakarta. He was the student of several well-known Indonesian vocalists and chorister, i.e. Avip Priatna, Lucia Kusumawardhani, Yoseph Chang, and Tommy Prabowo. He has been an active singers in some choirs since 1990, including the famous Indonesian choir, Batavia Madrigal Singers in Jakarta, and the tenor solo performer in some concerts. He obtained several achievements on regional singing festival during 1993–1997. Nationally, as a classical singer, he was the runner-up of Bintang Radio dan Televisi (a national radio and television singing competition) in 1995 and the third prize winner of PEKSIMINAS V (a national singing competition for student) in 1999. He received an award from Governor of Yogyakarta as an outstanding vocal artist in 1997.

Table C.6. List of topics. This is a list of 20 topics of Kompas-corpus and the document frequency, DF, of relevant documents for each topic.

Inke Kusumastuti is a medical doctor and currently continuing her education in Psychiatry in Udayana University, Denpasar, Bali. She was born in Blitar on April 17, 1986. She did not receive any formal education in music specifically, but she is practically a motivated self-learner when it comes to singing. She got numerous prizes in individual regional singing contests since she was in elementary school (1992–2001). In 2001–2004 she was involved in a band as the vocalist and the band won several regional competitions. In 2003, she experienced to be a cafe singer for a year. After that, while pursuing her medical education, she had been an active sopranos in some choirs, including the Eternal Choir, a well-known semi-professional small choir in Yogyakarta. As a chorister, she was involved in numerous concerts and choral competitions and received some achievements. In 2007, she followed a conducting workshop given by Andrew deQuadros in the First Asian Choir Games and, in 2010, she joined a choral clinic given by Marc Anthony Carpio, a choirmaster of Phillippine Madrigal Singers. Recently, in 2012, she got the third prize winner in Bintang Radio RRI Jember (a singing contest conducted by national radio of Indonesia at Jember). Her favorite artist is The Real Group, a world-acclaimed Swedish-based a capella group, which has significantly shaped her current music interest, and her dream is to able to employ music as part of therapy for people with mental disorders.

1.4 C.4 Kompas-Corpus

Kompas-corpus [11] is a set of newswire articles collected from a known Indonesian newspaper Kompas⁴⁵ published between January and June 2002. It consists of 3,000 documents constructed by following the TREC format, thereby accompanied by a file of information needs and a file of relevance judgments. There are 20 topics chosen by a native speaker after reading each documents in order to represent the user information needs. Those topics are listed in Table C.6 as well as the total number of relevant documents for each topic. Out of 3,000, only 433 documents are assigned topic(s).