Skip to main content
SpringerLink
Log in

Comparing machine and human performance for caller’s directory assistance requests

  • Published:
International Journal of Speech Technology Aims and scope Submit manuscript

Abstract

To understand how to systematically construct the machine models for automating Directory Assistance (DA) that are capable of reaching the performance level of human DA operators, we conducted a number of studies over the years. This paper describes the methods used for such studies and the results of laboratory experiments. These include a series of benchmark tests configured specifically for DA related tasks to evaluate the performance of state-of-the-art and commercially-available Hidden Markov Model (HMM) based Automatic Speech Recognition (ASR) technologies. The results show that the best system achieves a 57.9% task completion rate on the city-state-recognition benchmark test. For the most frequently-requested-listing benchmark test, the best system achieves a 40% task completion rate.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Allauzen, C., Mohri, M., & Saraclar, M. (2004). General indexation of weighted automata—application to spoken. In Proc. the workshop on interdisciplinary approaches to speech indexing and retrieval at HLT/NAACL (pp. 33–40) 2004.

  • Alphonso, I., & Chang, S. (2006). Saliency parsing for automated directory assistance. In Proc. INTERSPEECH, 2006.

  • Billi, R., Canavesio, F., & Rullent, C. (1998). Automation of telecom Italia directory assistance service: field trial results. In Proc. 1998 IEEE 4th workshop of interactive voice technology for telecommunications applications, 1998.

  • Boves, L., Jouvet, D., Sienel, J., De Mori, R., Bechet, F., Fissore, L., & Laface, P. (2000). ASR for automatic directory assistance: the SMADA project (2000). In Proceedings of ASR, 2000.

  • Jan, E. E., Maison, B., Mangu, L., & Zweig, G. (2003). Automatic construction of unique signature and confusable sets for natural language directory assistance applications. In Proc. INTERSPEECH, 2003–Eurospeech, 2003.

  • Kamm, C. A., Yang, K.-M., Shamieh, C. R., & Singhal, S. (1994). Speech recognition issues for directory assistance applications. In Proc. 1994 IEEE 4th workshop of interactive voice technology for telecommunications applications, 1994.

  • Lennig, M., Bielby, G., & Massicotte, J. (1995). Directory assistance automation in Bell Canada: trial results. Speech Communication, 17(3–4), 227–234.

    Article  Google Scholar 

  • Levin, E., & Mané, A. M. (2005). Voice user interface design for automated directory assistance. In Proc. INTERSPEECH, 2005–Eurospeech (pp. 2509–2512) 2005.

  • Lindgaard, G., & Caple, D. (2001). A case study in interactive keyboard design using participatory design techniques. Applied Ergonomics, 32, 71–80.

    Article  Google Scholar 

  • Meng, H.-M., Wai, C., & Pieraccini, R. (2003). The use of belief networks for mixed-initiative dialog modeling. IEEE Transactions on Speech and Audio Processing, 11(6), 757–773.

    Article  Google Scholar 

  • Natarajan, P., Prasad, R., Schwartz, R., & Makhoul, J. (2002). A scalable architecture for directory assistance automation. In Proc. ICASSP (Vol. 2, pp. 21–24) 2002.

  • Parthasarathy, S., Allauzen, C., & Munkong, R. (2005). Robust access to large structured data using voice form-filling. In Proc. INTERSPEECH, 2005.

  • Pieraccini, R., & Huerta, J. (2005). Where do we go from here? Research and commercial spoken dialog systems. In Proc. the 6th SIGdial workshop on discourse and dialogue, Lisbon, Portugal, September 2005.

  • Rudolph, J. (2008). The Paisley Group’s NDA performance IndexSM finds North American directory assistance is “best of the bests”. In The Paisley Group press release, November 11, 2008.

  • Scharenborg, O., Sturm, J., & Boves, L. (2001). Business listings in automatic directory assistance. In Proc. Eurospeech-2001 (pp. 2381–2384) 2001.

  • Schramm, H., Rueber, A., & Kellner, A. (2000). Strategies for name recognition in automatic directory assistance systems. Speech Communications, 31(4).

  • Seide, F., & Kellner, A. (1997). Towards an automated directory assistance information system. In Proc. Eurospeech (Vol. 3, pp. 1327–1330) 1997.

  • Sethy, A., & Narayanan, S. (2002). A syllable based approach for improved recognition of spoken names. In Proceedings of the ISCA pronunciation modeling workshop, 2002.

  • Williams, J. D., & Witt, S. M. (2004). A comparison of dialog strategies for call routing. International Journal of Speech Technology, 7(1), 9–224.

    Article  Google Scholar 

  • Yu, D., Ju, Y.-C., Wang, Y.-Y., Zweig, G., & Acero, A. (2007). Automated directory assistance—from theory to practice. In Proc. INTERSPEECH, 2007.

Download references

Author information

Authors and Affiliations

  1. AT&T Labs—Research, Florham Park, USA

    Harry M. Chang

Authors
  1. Harry M. Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Harry M. Chang.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Chang, H.M. Comparing machine and human performance for caller’s directory assistance requests. Int J Speech Technol 10, 75–87 (2007). https://doi.org/10.1007/s10772-009-9020-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10772-009-9020-1

Keywords

Search

Navigation