Toward Effective Processing of Information Graphics in Multimodal Documents: A Bayesian Network Approach

  • Sandra Carberry
  • Stephanie Elzer
Part of the Studies in Computational Intelligence book series (SCI, volume 96)

Information graphics (non-pictorial graphics such as bar charts and line graphs) are an important component of multimodal documents. When information graphics appear in popular media, such as newspapers and magazines, they generally have a message that they are intended to convey. This chapter addresses the problem of understanding such information graphics. The chapter presents a corpus study that shows the importance of taking information graphics into account when processing a multimodal document. It then presents a Bayesian network approach to identifying the message conveyed by one kind of information graphic, simple bar charts, along with an evaluation of the graph understanding system. This work is the first (1) to demonstrate the necessity of understanding information graphics and taking their communicative goal into account when processing a multimodal document and (2) to develop a computational strategy for recognizing the communicative goal or intended message of an information graphic.


Bayesian Network Perceptual Task Communicative Signal Information Graphic Conditional Probability Table 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    James Alty and Dimitrios Rigas. Exploring the use of structured music stimuli to communicate simple diagrams: The role of context. International Journal of Human-Computer Studies, 62(1):21–40, 2005.CrossRefGoogle Scholar
  2. 2.
    Jeffrey Bigham, Ryan Kaminsky, Richard Ladner, Oscar Danielsson, and Gordon Hempton. Webinsight: Making web images accessible. In Proceedings of the Eighth International ACM SIGACCESS Conference on Computers and Accessibility, pages 181–188, 2006.Google Scholar
  3. 3.
    Ben Bradshaw. Semantic based image retrieval: A probabilistic approach. In Proceedings of the 8th ACM International Conference on Multimedia, pages 167–176, 2000.Google Scholar
  4. 4.
    Sandra Carberry. Plan Recognition in Natural Language Dialogue. ACL-MIT Press Series on Natural Language Processing. MIT, Cambridge, Massachusetts, 1990.Google Scholar
  5. 5.
    Sandra Carberry. Techniques for plan recognition. User Modeling and User-Adapted Interaction, 11(1–2):31–48, 2001.zbMATHCrossRefGoogle Scholar
  6. 6.
    Eugene Charniak and Robert Goldman. A bayesian model of plan recognition. Artificial Intelligence Journal, 64:53–79, 1993.CrossRefGoogle Scholar
  7. 7.
    Daniel Chester and Stephanie Elzer. Getting computers to see information graphics so users do not have to. In Proceedings of the 15th International Symposium on Methodologies for Intelligent Systems, pages 660–668, 2005.Google Scholar
  8. 8.
    Herbert Clark. Using Language. Cambridge University Press, Cambridge, 1996.Google Scholar
  9. 9.
    Robert F. Cohen, Arthur Meacham, and Joelle Skaff. Teaching graphs to visually impaired students using an active auditory interface. In SIGCSE ’06: Proceedings of the 37th SIGCSE technical symposium on Computer science education, pages 279–282, 2006.Google Scholar
  10. 10.
    Marc Corio and Guy Lapalme. Generation of texts for information graphics. In Proceedings of the 7th European Workshop on Natural Language Generation EWNLG’99, pages 49–58, 1999.Google Scholar
  11. 11.
    Wayne Daniel. Applied Nonparametric Statistics. Houghton Mifflin, Boston, 1978.zbMATHGoogle Scholar
  12. 12.
    Stephanie Elzer. A Probabilistic Framework for the Recognition of Intention in Information Graphics. PhD thesis, University of Delaware, Newark, DE 19716, 2006.Google Scholar
  13. 13.
    Stephanie Elzer, Sandra Carberry, Daniel Chester, Seniz Demir, Nancy Green, Ingrid Zukerman, and Keith Trnka. Exploring and exploiting the limited utility of captions in recognizing intention in information graphics. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, pages 223–230, 2005.Google Scholar
  14. 14.
    Stephanie Elzer, Nancy Green, Sandra Carberry, and James Hoffman. A model of perceptual task effort for bar charts and its role in recognizing intention. User Modeling and User-Adapted Interaction, 16(1): 1–30, 2006.CrossRefGoogle Scholar
  15. 15.
    Stephanie Elzer, Edward Schwartz, Sandra Carberry, Daniel Chester, Seniz Demir, and Peng Wu. A browser extension for providing visually impaired users access to the content of bar charts on the web. In International Conference on Web Information Systems, pages 59–66, 2007.Google Scholar
  16. 16.
    Robert Futrelle. Summarization of diagrams in documents. In I. Mani and M. Maybury, editors, Advances in Automated Text Summarization, pages 403–421. MIT, Cambridge, 1999.Google Scholar
  17. 17.
    Robert Futrelle and Nikos Nikolakis. Efficient analysis of complex diagrams using constraint-based parsing. In Proceedings of the Third International Conference on Document Analysis and Recognition, pages 782–790, 1995.Google Scholar
  18. 18.
    Graphpad software. quickcalcs: Online calculators for scientists (2002).
  19. 19.
    Nancy Green, Giuseppe Carenini, Stephan Kerpedjiev, Joe Mattis, Johanna Moore, and Steven Roth. Autobrief: An experimental system for the automatic generation of briefings in integrated text and graphics. International Journal of Human-Computer Studies, 61(1):32–70, 2004.CrossRefGoogle Scholar
  20. 20.
    E. Hovy and C.-Y. Lin. Automated text summarization in summarist. In I. Mani and M. Maybury, editors, Advanced in Automatic Text Summarization, pages 81–94. MIT, Cambridge, 1999.Google Scholar
  21. 21.
    A. Kennel. Audiograf: A diagram-reader for the blind. In Second Annual ACM Conference on Assistive Technologies, pages 51–56, 1996.Google Scholar
  22. 22.
    Stephan Kerpedjiev and Steven Roth. Mapping communicative goals into conceptual tasks to generate graphics in discourse. In Proceedings of the International Conference on Intelligent User Interfaces, pages 60–67, 2000.Google Scholar
  23. 23.
    Inderjeet Mani and Mark Maybury, editors. Advances in Automatic Text Summarization. MIT, Cambridge, 1999.Google Scholar
  24. 24.
    Daniel Marcu. The rhetorical parsing of unrestricted texts: A surface-based approach. Computational Linguistics, 26(3):395–448, 2000.CrossRefGoogle Scholar
  25. 25.
    Mark Maybury, editor. Intelligent Multimedia Information Retrieval. MIT, Cambridge, 1997.Google Scholar
  26. 26.
    David K. McGookin and Stephen A. Brewster. Soundbar: exploiting multiple views in multimodal graph browsing. In NordiCHI ’06: Proceedings of the 4th Nordic conference on Human-computer interaction, pages 145–154, 2006.Google Scholar
  27. 27.
    Marie-Francine Moens, Roxana Angheluta, and Jos Dumortier. Generic technologies for single and multi-document summarization. Information Processing and Management, 41(3):569–586, 2005.CrossRefGoogle Scholar
  28. 28.
    Jane Morris and Graeme Hirst. Non-classical lexical semantic relations. In Proceedings of the HLT Workshop on Computational Lexical Semantics, pages 46–51, 2004.Google Scholar
  29. 29.
    Ani Nenkova. Automatic text summarization of newswire: Lessons learned from the document understanding conference. In Proceedings of National Conference on Artificial Intelligence (AAAI), pages 1436–1441, 2005.Google Scholar
  30. 30.
    Norsys Software Corp.: Netica, 2005.Google Scholar
  31. 31.
    Raymond Perrault and James Allen. A Plan-Based Analysis of Indirect Speech Acts. American Journal of Computational Linguistics, 6(3–4):167–182, 1980.Google Scholar
  32. 32.
    Rameshsharma Ramloll, Wai Yu, Stephen Brewster, Beate Riedel, Mike Burton, and Gisela Dimigen. Constructing sonified haptic line graphs for the blind student: First steps. In Proceedings of the 4th ACM Conference on Assistive Technologies, pages 17–25, 2000.Google Scholar
  33. 33.
    Martin Rotard, Sven Knödler, and Thomas Ertl. A tactile web browser for the visually disabled. In HYPERTEXT ’05: Proceedings of the sixteenth ACM conference on Hypertext and hypermedia, pages 15–22, 2005.Google Scholar
  34. 34.
    Barry Schiffman, Inderjeet Mani, and Kristian Concepcion. Producing biographical summaries: Combining linguistic knowledge with corpus statistics. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics, pages 450–457, 2001.Google Scholar
  35. 35.
    Rohini K. Srihari, Zhongfei Zhang, and Aibing Rao. Intelligent indexing and semantic retrieval of multimodal documents. Information Retrieval, 2(2):1–37, 2000.CrossRefGoogle Scholar
  36. 36.
    W3c: Web accessibility initiative.
  37. 37.
    Steven Wall and Stephen Brewster. Tac-tiles: Multimodal pie charts for visually impaired users. In Proceedings of the 4th Nordic Conference on Human-computer Interaction, pages 9–18, 2006.Google Scholar
  38. 38.
    Xiaojun Wan, Jianwu Yang, and Jianguo Xiao. Towards an iterative reinforcement approach for simultaneous document summarization and keyword extraction. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, pages 552–559, 2007.Google Scholar
  39. 39.
    Jen-Yuan Yeh, Hao-Ren Ke, Wei-Pang Yang, and I-Heng Meng. Text summarization using a trainable summarizer and latent semantic analysis. Information Processing and Management, 41(1):75–95, 2005.CrossRefGoogle Scholar
  40. 40.
    Jin Yu, Ehud Reiter, Jim Hunter, and Chris Mellish. Choosing the content of textual summaries of large time-series data sets. Natural Language Engineering, 13:25–49, 2007.CrossRefGoogle Scholar
  41. 41.
    Jin Yu, Jim Hunter, Ehud Reiter, and Somayajulu Sripada. Recognising visual patterns to communicate gas turbine time-series data. In Proceedings of 22nd SCAI International Conference on Knowledge-Based Systems and Applied Artificial Intelligence (ES2002), pages 105–118, 2002.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Sandra Carberry
    • 1
  • Stephanie Elzer
    • 2
  1. 1.Department of Computer ScienceUniversity of DelawareNewarkDE
  2. 2.Department of Computer ScienceMillersville UniversityPennsylvaniaPA

Personalised recommendations