Applying Piecewise Approximation in Perceptron Training of Conditional Random Fields

  • Teemu Ruokolainen
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7619)


We show that the recently proposed piecewise approximation approach can benefit conditional random fields estimation using the structured perceptron algorithm. We present experiments in noun-phrase chunking task on the CoNLL-2000 corpus. The results show that, compared to standard training, applying the piecewise approach during model estimation may yield not only savings in training time but also improvement in model performance on test set due to added model regularization.


Conditional Random Field Standard Training Perceptron Algorithm Piecewise Approximation Graph Inference 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Lafferty, J., McCallum, A., Pereira, F.C.N.: Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In: Proceedings of the Eighteenth International Conference on Machine Learning, pp. 282–289 (2001)Google Scholar
  2. 2.
    Collins, M.: Discriminative training methods for hidden markov models: Theory and Experiments with Perceptron Algorithms. In: Proceedings of the ACL 2002 Conference on Empirical Methods in Natural Language Processing, vol. 10, pp. 1–8. Association for Computational Linguistics (2002)Google Scholar
  3. 3.
    Tsochantaridis, I., Joachims, T., Hofmann, T., Altun, Y.: Large margin methods for structured and interdependent output variables. Journal of Machine Learning Research 6, 1453–1484 (2005)MathSciNetzbMATHGoogle Scholar
  4. 4.
    Zhang, Y., Clark, S.: Syntactic processing using the generalized perceptron and beam search. Computational Linguistics 37(1), 105–151 (2011)CrossRefGoogle Scholar
  5. 5.
    Sutton, C., McCallum, A.: Piecewise training for structured prediction. Machine Learning 77(2), 165–194 (2009)CrossRefGoogle Scholar
  6. 6.
    Sontag, D., Meshi, O., Jaakkola, T., Globerson, A.: More data means less inference: A pseudo-max approach to structured learning. In: Lafferty, J., Williams, C.K.I., Shawe-Taylor, J., Zemel, R., Culotta, A. (eds.) Advances in Neural Information Processing Systems 23, pp. 2181–2189 (2010)Google Scholar
  7. 7.
    Samdani, R., Roth, D.: Efficient decomposed learning for structured prediction. In: Proceedings of the 29th International Conference on Machine Learning (2012)Google Scholar
  8. 8.
    Srikumar, V., Kundu, G., Roth, D.: On amortizing inference cost for structured prediction. In: Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 1114–1124 (2012)Google Scholar
  9. 9.
    Vapnik, V.: An overview of statistical learning theory. IEEE Transactions on Neural Networks 10, 988–999 (1999)CrossRefGoogle Scholar
  10. 10.
    Freund, Y., Schapire, R.: Large margin classification using the perceptron algorithm. Machine Learning 37(3), 277–296 (1999)zbMATHCrossRefGoogle Scholar
  11. 11.
    Daumé III, H.: Practical structured learning techniques for natural language processing. PhD thesis, University of Southern California (2006)Google Scholar
  12. 12.
    Kim, T., Sang, E., Buchholz, S.: Introduction to the conll-2000 shared task: Chunking. In: Proceedings of the 2nd Workshop on Learning Language in Logic and the 4th Conference on Computational Natural Language Learning, vol. 7, pp. 127–132. Association for Computational Linguistics (2000)Google Scholar
  13. 13.
    Marcus, M., Marcinkiewicz, M., Santorini, B.: Building a large annotated corpus of english: The penn treebank. Computational Linguistics 19(2), 313–330 (1993)Google Scholar
  14. 14.
    Bishop, C.: Pattern recognition and machine learning, vol. 4. Springer, New York (2006)zbMATHGoogle Scholar
  15. 15.
    Frey, B., MacKay, D.: A revolution: Belief propagation in graphs with cycles. Advances in Neural Information Processing Systems, 479–485 (1998)Google Scholar
  16. 16.
    Wainwright, M.J., Jaakkola, T.S., Willsky, A.S.: Map estimation via agreement on trees: message-passing and linear programming. IEEE Transactions on Information Theory 51, 3697–3717 (2005)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 1222–1239 (2001)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Teemu Ruokolainen
    • 1
  1. 1.Department of Information and Computer ScienceAalto UniversityAaltoFinland

Personalised recommendations