Error Prediction with Partial Feedback

  • William Darling
  • Cédric Archambeau
  • Shachar Mirkin
  • Guillaume Bouchard
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8189)

Abstract

In this paper, we propose a probabilistic framework for predicting the root causes of errors in data processing pipelines made up of several components when we only have access to partial feedback; that is, we are aware when some error has occurred in one or more of the components, but we do not know which one. The proposed error model enables us to direct the user feedback to the correct components in the pipeline to either automatically correct errors as they occur, retrain the component with assimilated training examples, or take other corrective action. We present the model and describe an Expectation Maximization (EM)-based algorithm to learn the model parameters and predict the error configuration. We demonstrate the accuracy and usefulness of our method first on synthetic data, and then on two distinct tasks: error correction in a 2-component opinion summarization system, and phrase error detection in statistical machine translation.

Keywords

error modeling user feedback binary classification EM 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Ritchie, D.M.: The evolution of the unix time-sharing system. Communications of the ACM 17, 365–375 (1984)CrossRefGoogle Scholar
  2. 2.
    Ritter, A., Clark, S., Mausam, Etzioni, O.: Named entity recognition in tweets: An experimental study. In: Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pp. 1524–1534. Association for Computational Linguistics, Edinburgh (2011)Google Scholar
  3. 3.
    Ly, D.K., Sugiyama, K., Lin, Z., Kan, M.Y.: Product review summarization from a deeper perspective. In: Proceedings of the 11th Annual International ACM/IEEE Joint Conference on Digital Libraries, JCDL 2011, pp. 311–314. ACM, New York (2011)Google Scholar
  4. 4.
    Finkel, J.R., Manning, C.D., Ng, A.Y.: Solving the problem of cascading errors: approximate bayesian inference for linguistic annotation pipelines. In: Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, EMNLP 2006, pp. 618–626. Association for Computational Linguistics, Stroudsburg (2006)CrossRefGoogle Scholar
  5. 5.
    Blair-goldensohn, S., Neylon, T., Hannan, K., Reis, G.A., Mcdonald, R., Reynar, J.: Building a sentiment summarizer for local service reviews. In: NLP in the Information Explosion Era (2008)Google Scholar
  6. 6.
    Lu, Y., Zhai, C., Sundaresan, N.: Rated aspect summarization of short comments. In: Proceedings of the 18th International Conference on World Wide Web, WWW 2009, pp. 131–140. ACM, New York (2009)CrossRefGoogle Scholar
  7. 7.
    Zhai, Z., Liu, B., Zhang, L., Xu, H., Jia, P.: Identifying evaluative sentences in online discussions. In: AAAI (2011)Google Scholar
  8. 8.
    MacCartney, B., Grenager, T., de Marneffe, M.C., Cer, D., Manning, C.D.: Learning to recognize features of valid textual entailments. In: Proceedings of the Main Conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, HLT-NAACL 2006, pp. 41–48. Association for Computational Linguistics, Stroudsburg (2006)CrossRefGoogle Scholar
  9. 9.
    Cunningham, H., Humphreys, K., Gaizauskas, R., Wilks, Y.: Software infrastructure for natural language processing. In: Proceedings of the Fifth Conference on Applied Natural Language Processing, ANLC 1997, pp. 237–244. Association for Computational Linguistics, Stroudsburg (1997)CrossRefGoogle Scholar
  10. 10.
    Lamb, A., Paul, M.J., Dredze, M.: Separating fact from fear: Tracking flu infections on twitter. In: Proceedings of NAACL-HLT, pp. 789–795 (2013)Google Scholar
  11. 11.
    Neal, R.M.: Connectionist learning of belief networks. Artificial Intelligence 56(1), 71–113 (1992)MathSciNetMATHCrossRefGoogle Scholar
  12. 12.
    Marciniak, T., Strube, M.: Beyond the pipeline: discrete optimization in nlp. In: Proceedings of the Ninth Conference on Computational Natural Language Learning, CONLL 2005, pp. 136–143. Association for Computational Linguistics, Stroudsburg (2005)CrossRefGoogle Scholar
  13. 13.
    Koller, D., Friedman, N.: Probabilistic Graphical Models: Principles and Techniques. MIT Press (2009)Google Scholar
  14. 14.
    Bottou, L.: Large-scale machine learning with stochastic gradient descent. In: Lechevallier, Y., Saporta, G. (eds.) Proceedings of the 19th International Conference on Computational Statistics (COMPSTAT 2010), Paris, France, pp. 177–187. Springer (August 2010)Google Scholar
  15. 15.
    Liang, P., Klein, D.: Online em for unsupervised models. In: Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, NAACL 2009, pp. 611–619. Association for Computational Linguistics, Stroudsburg (2009)CrossRefGoogle Scholar
  16. 16.
    Fayyad, U., Reina, C., Bradley, P.S.: Initialization of iterative refinement clustering algorithms. In: Proceedings of the Fourth International Conference on Knowledge Discovery and Data Mining, pp. 194–198 (1998)Google Scholar
  17. 17.
    Wilson, T., Wiebe, J., Hoffmann, P.: Recognizing contextual polarity in phrase-level sentiment analysis. In: Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, HLT 2005, pp. 347–354. Association for Computational Linguistics, Stroudsburg (2005)CrossRefGoogle Scholar
  18. 18.
    Nenkova, A., Vanderwende, L., McKeown, K.: A compositional context sensitive multi-document summarizer: exploring the factors that influence summarization. In: SIGIR 2006: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 573–580. ACM, New York (2006)CrossRefGoogle Scholar
  19. 19.
    Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)MATHGoogle Scholar
  20. 20.
    Xiong, D., Zhang, M., Li, H.: Error detection for statistical machine translation using linguistic features. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, ACL 2010, pp. 604–611. Association for Computational Linguistics, Stroudsburg (2010)Google Scholar
  21. 21.
    Ueffing, N., Ney, H.: Word-level confidence estimation for machine translation. Comput. Linguist. 33(1), 9–40 (2007)MATHCrossRefGoogle Scholar
  22. 22.
    Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., Herbst, E.: Moses: Open source toolkit for statistical machine translation. In: Proceedings of ACL, Demo and Poster Sessions (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • William Darling
    • 1
  • Cédric Archambeau
    • 1
  • Shachar Mirkin
    • 1
  • Guillaume Bouchard
    • 1
  1. 1.Xerox Research Centre EuropeMeylanFrance

Personalised recommendations