International Journal of Speech Technology

, Volume 10, Issue 2–3, pp 75–87 | Cite as

Comparing machine and human performance for caller’s directory assistance requests

  • Harry M. ChangEmail author


To understand how to systematically construct the machine models for automating Directory Assistance (DA) that are capable of reaching the performance level of human DA operators, we conducted a number of studies over the years. This paper describes the methods used for such studies and the results of laboratory experiments. These include a series of benchmark tests configured specifically for DA related tasks to evaluate the performance of state-of-the-art and commercially-available Hidden Markov Model (HMM) based Automatic Speech Recognition (ASR) technologies. The results show that the best system achieves a 57.9% task completion rate on the city-state-recognition benchmark test. For the most frequently-requested-listing benchmark test, the best system achieves a 40% task completion rate.


Speech recognition Directory assistance Spoken dialog systems Automated telephone operator services System performance 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Allauzen, C., Mohri, M., & Saraclar, M. (2004). General indexation of weighted automata—application to spoken. In Proc. the workshop on interdisciplinary approaches to speech indexing and retrieval at HLT/NAACL (pp. 33–40) 2004. Google Scholar
  2. Alphonso, I., & Chang, S. (2006). Saliency parsing for automated directory assistance. In Proc. INTERSPEECH, 2006. Google Scholar
  3. Billi, R., Canavesio, F., & Rullent, C. (1998). Automation of telecom Italia directory assistance service: field trial results. In Proc. 1998 IEEE 4th workshop of interactive voice technology for telecommunications applications, 1998. Google Scholar
  4. Boves, L., Jouvet, D., Sienel, J., De Mori, R., Bechet, F., Fissore, L., & Laface, P. (2000). ASR for automatic directory assistance: the SMADA project (2000). In Proceedings of ASR, 2000. Google Scholar
  5. Jan, E. E., Maison, B., Mangu, L., & Zweig, G. (2003). Automatic construction of unique signature and confusable sets for natural language directory assistance applications. In Proc. INTERSPEECH, 2003–Eurospeech, 2003. Google Scholar
  6. Kamm, C. A., Yang, K.-M., Shamieh, C. R., & Singhal, S. (1994). Speech recognition issues for directory assistance applications. In Proc. 1994 IEEE 4th workshop of interactive voice technology for telecommunications applications, 1994. Google Scholar
  7. Lennig, M., Bielby, G., & Massicotte, J. (1995). Directory assistance automation in Bell Canada: trial results. Speech Communication, 17(3–4), 227–234. CrossRefGoogle Scholar
  8. Levin, E., & Mané, A. M. (2005). Voice user interface design for automated directory assistance. In Proc. INTERSPEECH, 2005–Eurospeech (pp. 2509–2512) 2005. Google Scholar
  9. Lindgaard, G., & Caple, D. (2001). A case study in interactive keyboard design using participatory design techniques. Applied Ergonomics, 32, 71–80. CrossRefGoogle Scholar
  10. Meng, H.-M., Wai, C., & Pieraccini, R. (2003). The use of belief networks for mixed-initiative dialog modeling. IEEE Transactions on Speech and Audio Processing, 11(6), 757–773. CrossRefGoogle Scholar
  11. Natarajan, P., Prasad, R., Schwartz, R., & Makhoul, J. (2002). A scalable architecture for directory assistance automation. In Proc. ICASSP (Vol. 2, pp. 21–24) 2002. Google Scholar
  12. Parthasarathy, S., Allauzen, C., & Munkong, R. (2005). Robust access to large structured data using voice form-filling. In Proc. INTERSPEECH, 2005. Google Scholar
  13. Pieraccini, R., & Huerta, J. (2005). Where do we go from here? Research and commercial spoken dialog systems. In Proc. the 6th SIGdial workshop on discourse and dialogue, Lisbon, Portugal, September 2005. Google Scholar
  14. Rudolph, J. (2008). The Paisley Group’s NDA performance IndexSM finds North American directory assistance is “best of the bests”. In The Paisley Group press release, November 11, 2008. Google Scholar
  15. Scharenborg, O., Sturm, J., & Boves, L. (2001). Business listings in automatic directory assistance. In Proc. Eurospeech-2001 (pp. 2381–2384) 2001. Google Scholar
  16. Schramm, H., Rueber, A., & Kellner, A. (2000). Strategies for name recognition in automatic directory assistance systems. Speech Communications, 31(4). Google Scholar
  17. Seide, F., & Kellner, A. (1997). Towards an automated directory assistance information system. In Proc. Eurospeech (Vol. 3, pp. 1327–1330) 1997. Google Scholar
  18. Sethy, A., & Narayanan, S. (2002). A syllable based approach for improved recognition of spoken names. In Proceedings of the ISCA pronunciation modeling workshop, 2002. Google Scholar
  19. Williams, J. D., & Witt, S. M. (2004). A comparison of dialog strategies for call routing. International Journal of Speech Technology, 7(1), 9–224. CrossRefGoogle Scholar
  20. Yu, D., Ju, Y.-C., Wang, Y.-Y., Zweig, G., & Acero, A. (2007). Automated directory assistance—from theory to practice. In Proc. INTERSPEECH, 2007. Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  1. 1.AT&T Labs—ResearchFlorham ParkUSA

Personalised recommendations