Textual classification of SEC comment letters

Abstract

This study examines the impact of SEC comment letters on future financial reporting outcomes and earnings credibility. Naïve Bayesian classification identifies comment letters associated with future restatements and write-downs. An investor attention-based quantitative measure of importance, using EDGAR downloads, also predicts these outcomes. Disclosure-event abnormal returns, revenue recognition comments, and the number of letters in a conversation appear to be useful quantitative metrics for classifying importance in certain settings. This study also documents trends in comment letter topics over time and identifies topics associated with the textual and quantitative classifications of importance, providing insights into the factors that draw investor attention and that relate to future restatements and write-downs. Innocuous comment letters are associated with improvements in earnings credibility following comment letter reviews.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3

Notes

  1. 1.

    Examples of short seller research that uses issues raised in comment letters include presentations by Greenlight Capital on Green Mountain Coffee, Pershing Square on Herbalife, and Prescience Point on Boulder Brands. http://online.wsj.com/public/resources/documents/EinhornGMCRpresentation_Oct2011_VIC.pdf. Retrieved 7 September, 2020.http://factsabout-herbalife.com/wp-content/uploads/2013/01/Who-wants-to-be-a-Millionaire.pdf. Retrieved 1 February 2013.https://www.presciencepoint.com/research/research-archives/boulder-brands-inc-bdbd/. Retrieved 7 September, 2020.

  2. 2.

    Care needs to be taking cleaning the raw EDGAR log file data set to accurately count comment letter downloads, which are usually filed as PDF documents. Ryans, J., 2017. Using the EDGAR log file data set. Working paper, London Business School.

  3. 3.

    As described previously, the use of multi-word features such as bigrams, which are used in this study, can allow for some preservation of word order in naïve Bayesian analysis.

  4. 4.

    See Securities and Exchange Commission. Staff Observations in the Review of Executive Compensation Disclosure. September 10, 2007. http://www.sec.gov/divisions/corpfin/guidance/execcompdisclosure.htm. Accessed 7 September 2020.

References

  1. Antweiler, W., & Frank, M. Z. (2004). Is all that talk just noise? The information content of internet stock message boards. Journal of Finance, 59(3), 1259–1294.

    Google Scholar 

  2. Ball, R., & Brown, P. (1968). An empirical evaluation of accounting income numbers. Journal of Accounting Research, 6(2), 159–178.

    Google Scholar 

  3. Bao, Y., & Datta, A. (2014). Simultaneously discovering and quantifying risk types from textual risk disclosures. Management Science, 60(6), 1371–1391.

    Google Scholar 

  4. Bartov, E., Givoly, D., & Hayn, C. (2002). The rewards to meeting or beating earnings expectations. Journal of Accounting and Economics, 33(2), 173–204.

    Google Scholar 

  5. Beaver, W. H. (1968). The information content of annual earnings announcements. Journal of Accounting Research, 1968, 67–92.

    Google Scholar 

  6. Bens, D. A., Cheng, M., & Neamtiu, M. (2016). The impact of SEC disclosure monitoring on the uncertainty of fair value estimates. The Accounting Review, 91(2), 349–375.

    Google Scholar 

  7. Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.

    Google Scholar 

  8. Bloomfield, R. J. (2002). The “incomplete revelation hypothesis” and financial reporting. Accounting Horizons, 16(3), 233–243.

    Google Scholar 

  9. Bozanic, Z., Dietrich, J. R., & Johnson, B. A. (2017). SEC comment letters and firm disclosure. Journal of Accounting and Public Policy, 36(5), 337–357.

    Google Scholar 

  10. Brown, S. V., Tian, X., & Tucker, J. W. (2018). The spillover effect of SEC comment letters on qualitative corporate disclosure: Evidence from the risk factor disclosure. Contemporary Accounting Research, 35(2), 622–656.

    Google Scholar 

  11. Bryan, S. H. (1997). Incremental information content of required disclosures contained in management discussion and analysis. The Accounting Review, 72(2), 285–301.

    Google Scholar 

  12. Carhart, M. M. (1997). On persistence in mutual fund performance. Journal of Finance, 52(1), 57–82.

    Google Scholar 

  13. Cassell, C. A., Dreher, L. M., & Myers, L. A. (2013). Reviewing the SEC’s review process: 10-K comment letters and the cost of remediation. The Accounting Review, 88(6), 1875–1908.

    Google Scholar 

  14. Christensen, H. B., Hail, L., & Leuz, C. (2013). Mandatory IFRS reporting and changes in enforcement. Journal of Accounting and Economics, 56(2–3), 147–177.

    Google Scholar 

  15. Collins, D. W., & Kothari, S. P. (1989). An analysis of intertemporal and cross-sectional determinants of earnings response coefficients. Journal of Accounting and Economics, 11(2), 143–181.

    Google Scholar 

  16. Cunningham, L. M., Johnson, B. A., Johnson, E. S., & Lisic, L. L. (2019). The switch up: An examination of changes in earnings management after receiving sec comment letters. Contemporary Accounting Research, 37(2), 917–944.

    Google Scholar 

  17. Dagan, I., Feldman, R., Hirsh, H., (1996). Keyword-based browsing and analysis of large document sets, Proceedings of the Fifth Annual Symposium on Document Analysis and Information Retrieval (pp. 191–208). University of Nevada.

  18. Davis, A. K., Piger, J. M., & Sedor, L. M. (2012). Beyond the numbers: Measuring the information content of earnings press release language. Contemporary Accounting Research, 29(3), 845–868.

    Google Scholar 

  19. De Franco, G., Vasvari, F. P., Vyas, D., & Wittenberg-Moerman, R. (2013). Debt analysts’ views of debt-equity conflicts of interest. The Accounting Review, 89(2), 571–604.

    Google Scholar 

  20. Dechow, P. M., Ge, W., Larson, C. R., & Sloan, R. G. (2011). Predicting material accounting misstatements. Contemporary Accounting Research, 28(1), 17–82.

    Google Scholar 

  21. Dechow, P. M., Huson, M. R., & Sloan, R. G. (1994). The effect of restructuring charges on executives’ cash compensation. The Accounting Review, 69(1), 138–156.

    Google Scholar 

  22. Dechow, P. M., Lawrence, A., & Ryans, J. (2016). SEC comment letters and insider sales. The Accounting Review, 91(2), 401–439.

    Google Scholar 

  23. Dechow, P. M., Sloan, R. G., & Sweeney, A. P. (1995). Detecting earnings management. The Accounting Review, 70(2), 193–225.

    Google Scholar 

  24. DeFond, M. L., & Jiambalvo, J. (1991). Incidence and circumstances of accounting errors. The Accounting Review, 66(3), 643–655.

    Google Scholar 

  25. Drake, M., Johnson, B., Roulstone, D., & Thornock, J. (2019). Is there information content in information acquisition. The Accounting Review, 95(2), 113–139.

    Google Scholar 

  26. Drake, M. S., Roulstone, D. T., & Thornock, J. R. (2015). The determinants and consequences of information acquisition via EDGAR. Contemporary Accounting Research, 32(3), 1128–1161.

    Google Scholar 

  27. Duro, M., Heese, J., & Ormazabal, G. (2019). The effect of enforcement transparency: Evidence from SEC comment-letter reviews. Review of Accounting Studies, 24(3), 780–823.

    Google Scholar 

  28. Dyck, A., Morse, A., & Zingales, L. (2010). Who blows the whistle on corporate fraud? Journal of Finance, 65(6), 2213–2253.

    Google Scholar 

  29. Dye, R. A. (1985). Disclosure of nonproprietary information. Journal of Accounting Research, 23(1), 123–145.

    Google Scholar 

  30. Dyer, T., Lang, M., & Stice-Lawrence, L. (2017). The evolution of 10-K textual disclosure: Evidence from latent Dirichlet allocation. Journal of Accounting and Economics, 64(2–3), 221–245.

    Google Scholar 

  31. Engelberg, J. E., & Parsons, C. A. (2011). The causal impact of media in financial markets. Journal of Finance, 66(1), 67–97.

    Google Scholar 

  32. Feldman, R., Dagan, I., (1995). Knowledge discovery in textual databases KDT. Proceedings of the First International Conference on Knowledge Discovery and Data Mining (pp. 112–117). Association for Computing Machinery.

  33. Feldman, R., Govindaraj, S., Livnat, J., & Segal, B. (2010). Management’s tone change, post earnings announcement drift and accruals. Review of Accounting Studies, 15(4), 915–953.

    Google Scholar 

  34. Feroz, E.H., Park, K., Pastena, V.S., (1991). The financial and market effects of the SEC’s accounting and auditing enforcement releases. Journal of Accounting Research 29(Supp.), 107–142.

  35. Francis, J., LaFond, R., Olsson, P., & Schipper, K. (2005). The market pricing of accruals quality. Journal of Accounting and Economics, 39(2), 295–327.

    Google Scholar 

  36. Francis, J., Nanda, D., & Olsson, P. (2008). Voluntary disclosure, earnings quality, and cost of capital. Journal of Accounting Research, 46(1), 53–99.

    Google Scholar 

  37. Francis, J. R. (2011). A framework for understanding and researching audit quality. Auditing: A Journal of Practice & Theory, 30(2), 125–152.

    Google Scholar 

  38. Gietzmann, M. B., & Pettinicchio, A. K. (2013). External auditor reassessment of client business risk following the issuance of a comment letter by the SEC. European Accounting Review, 23(1), 57–85.

    Google Scholar 

  39. Hayn, C. (1995). The information content of losses. Journal of Accounting and Economics, 20(2), 125–153.

    Google Scholar 

  40. Heese, J., Khan, M., & Ramanna, K. (2017). Is the SEC captured? Evidence from comment-letter reviews. Journal of Accounting and Economics, 64(1), 98–122.

    Google Scholar 

  41. Hennes, K. M., Leone, A. J., & Miller, B. P. (2014). Determinants and market consequences of auditor dismissals after accounting restatements. The Accounting Review, 89(3), 1051–1082.

    Google Scholar 

  42. Hirshleifer, D., & Teoh, S. H. (2003). Limited attention, information disclosure, and financial reporting. Journal of Accounting and Economics, 36(1), 337–386.

    Google Scholar 

  43. Hoberg, G., & Lewis, C. (2017). Do fraudulent firms produce abnormal disclosure? Journal of Corporate Finance, 43(1), 58–85.

    Google Scholar 

  44. Holthausen, R. W., & Verrecchia, R. E. (1988). The effect of sequential information releases on the variance of price changes in an intertemporal multi-asset market. Journal of Accounting Research, 26(1), 82–106.

    Google Scholar 

  45. Hribar, P., & Jenkins, N. T. (2004). The effect of accounting restatements on earnings revisions and the estimated cost of capital. Review of Accounting Studies, 9(2–3), 337–356.

    Google Scholar 

  46. Hribar, P., Kravet, T., & Wilson, R. (2014). A new measure of accounting quality. Review of Accounting Studies, 19(1), 506–538.

    Google Scholar 

  47. Huang, A. H., Lehavy, R., Zang, A. Y., & Zheng, R. (2018). Analyst information discovery and interpretation roles: A topic modeling approach. Management Science, 64(6), 2833–2855.

    Google Scholar 

  48. Huang, A. H., Zang, A. Y., & Zheng, R. (2014). Evidence on the information content of text in analyst reports. The Accounting Review, 89(6), 2151–2180.

    Google Scholar 

  49. Johnston, R., & Petacchi, R. (2017). Regulatory oversight of financial reporting: Securities and exchange commission comment letters. Contemporary Accounting Research, 34(2), 1128–1155.

    Google Scholar 

  50. Jung, W. O., & Kwon, Y. K. (1988). Disclosure when the market is unsure of information endowment of managers. Journal of Accounting Research, 26(1), 146–153.

    Google Scholar 

  51. Karlgren, J., Cutting, D., (1994). Recognizing text genres with simple metrics using discriminant analysis. Proceedings of the 15th conference on Computational Linguistics (pp. 1071–1075). Association for Computational Linguistics.

  52. Kessler, B., Numberg, G., Schutze, H., (1997). Automatic detection of text genre, Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics (pp. 32–38).Association for Computational Linguistics.

  53. Kinney, W. R., & McDaniel, L. S. (1989). Characteristics of firms correcting previously reported quarterly earnings. Journal of Accounting and Economics, 11(1), 71–93.

    Google Scholar 

  54. Kinney, W. R., Palmrose, Z. V., & Scholz, S. (2004). Auditor independence, non-audit services, and restatements: Was the US government right? Journal of Accounting Research, 42(3), 561–588.

    Google Scholar 

  55. Kormendi, R., & Lipe, R. (1987). Earnings innovations, earnings persistence, and stock returns. Journal of Business, 60(3), 323–345.

    Google Scholar 

  56. Kothari, S. P., Li, X., & Short, J. E. (2009a). The effect of disclosures by management, analysts, and business press on cost of capital, return volatility, and analyst forecasts: A study using content analysis. The Accounting Review, 84(5), 1639–1670.

    Google Scholar 

  57. Kothari, S. P., Shu, S., & Wysocki, P. D. (2009b). Do managers withhold bad news? Journal of Accounting Research, 47(1), 241–276.

    Google Scholar 

  58. La Porta, R., Lopez-de Silanes, F., & Shleifer, A. (2006). What works in securities laws? Journal of Finance, 61(1), 1–32.

    Google Scholar 

  59. Lang, M., & Lundholm, R. (1993). Cross-sectional determinants of analyst ratings of corporate disclosures. Journal of Accounting Research, 31(2), 246–271.

    Google Scholar 

  60. Larcker, D. F., & Zakolyukina, A. A. (2012). Detecting deceptive discussions in conference calls. Journal of Accounting Research, 50(2), 495–540.

    Google Scholar 

  61. Laurion, H., Lawrence, A., & Ryans, J. (2017). U.S. audit partner rotation. The Accounting Review, 92(3), 209–237.

    Google Scholar 

  62. Law, K. K. F., & Mills, L. F. (2015). Taxes and financial constraints: Evidence from linguistic cues. Journal of Accounting Research, 53(4), 777–819.

    Google Scholar 

  63. Lawrence, A., Sloan, R., & Sun, Y. (2013). Non-discretionary conservatism: Evidence and implications. Journal of Accounting and Economics, 56(2), 112–133.

    Google Scholar 

  64. Leuz, C., & Wysocki, P. (2016). The economics of disclosure and financial reporting regulation: Evidence and suggestions for future research. Journal of Accounting Research, 54(2), 525–622.

    Google Scholar 

  65. Lewis, D.D., (1998). Naïve (Bayes) at forty: The independence assumption in information retrieval. Machine learning: ECML-98 (pp. 4–15). Springer.

  66. Li, F. (2008). Annual report readability, current earnings, and earnings persistence. Journal of Accounting and Economics, 45(2), 221–247.

    Google Scholar 

  67. Li, F. (2010). The information content of forward-looking statements in corporate filings. A naïve Bayesian machine learning approach. Journal of Accounting Research, 48(5), 1049–1102.

    Google Scholar 

  68. Liu, L. L., Raghunandan, K., & Rama, D. (2009). Financial restatements and shareholder ratifications of the auditor. Auditing: A Journal of Practice & Theory, 28(1), 225–240.

    Google Scholar 

  69. Livnat, J., & Mendenhall, R. R. (2006). Comparing the post–earnings announcement drift for surprises calculated from analyst and time series forecasts. Journal of Accounting Research, 44(1), 177–205.

    Google Scholar 

  70. Ljungqvist, A., & Qian, W. (2016). How constraining are limits to arbitrage? Evidence from a recent financial innovation. Review of Financial Studies, 29(8), 1975–2028.

    Google Scholar 

  71. Loughran, T., & McDonald, B. (2011). When is a liability not a liability? Textual analysis, dictionaries, and 10-Ks. Journal of Finance, 66(1), 35–65.

    Google Scholar 

  72. Loughran, T., & McDonald, B. (2016). Textual analysis in accounting and finance: A survey. Journal of Accounting Research, 54(4), 1187–1230.

    Google Scholar 

  73. Loughran, T., McDonald, B., (2017). The use of EDGAR filings by investors. Journal of Behavioral Finance 18(2), 231–248).

  74. Mendenhall, R.R., Nichols, W.D., (1988). Bad news and differential market reactions to announcements of earlier-quarters versus fourth-quarter earnings. Journal of Accounting Research 26(Supp.), 63–86.

  75. Mosteller, F., & Wallace, D. L. (1984). Applied Bayesian and classical inference. New York: Springer-Verlag.

    Google Scholar 

  76. Naughton, J. P., Rogo, R., Sunder, J., & Zhang, R. (2018). SEC monitoring of foreign firms’ disclosures. Review of Accounting Studies, 23(4), 1355–1388.

    Google Scholar 

  77. Palmrose, Z. V., Richardson, V. J., & Scholz, S. (2004). Determinants of market reactions to restatement announcements. Journal of Accounting and Economics, 37(1), 59–89.

    Google Scholar 

  78. Pang, B., Lee, L., Vaithyanathan, S., (2002). Thumbs up?: Sentiment classification using machine learning techniques, Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing . (pp. 79–86). Association for Computational Linguistics.

  79. Peterson, K. (2012). Accounting complexity, misreporting, and the consequences of misreporting. Review of Accounting Studies, 17(1), 72–95.

    Google Scholar 

  80. Quinn, K. M., Monroe, B. L., Colaresi, M., Crespin, M. H., & Radev, D. R. (2010). How to analyze political attention with minimal assumptions and costs. American Journal of Political Science, 54(1), 209–228.

    Google Scholar 

  81. Ramanna, K., & Watts, R. L. (2012). Evidence on the use of unverifiable estimates in required goodwill impairment. Review of Accounting Studies, 17(4), 749–780.

    Google Scholar 

  82. Robinson, J. R., Xue, Y., & Yu, Y. (2011). Determinants of disclosure noncompliance and the effect of the SEC review: Evidence from the 2006 mandated compensation disclosure regulations. The Accounting Review, 86(4), 1415–1444.

    Google Scholar 

  83. Sandler, L., (2013). Muddy waters secret China weapon is on SEC website. Bloomberg News. https://www.bloomberg.com/news/articles/2013-02-19/muddy-waters-secret-china-weapon-is-on-sec-website. Accessed 7 September 2020.

  84. SEC, (2015). FY 2016 congressional budget justification, FY 2016 annual performance plan, and FY 2014 annual performance report. U.S. Securities and Exchange Commission. http://www.sec.gov/about/reports/secfy16congbudgjust.shtml. Accessed 7 September 2020.

  85. Talley, E., & O’Kane, D. (2012). The measure of a MAC: A machine-learning protocol for analyzing force majeure clauses in ma agreements. Journal of Institutional and Theoretical Economics, 168(1), 181–201.

    Google Scholar 

  86. Teoh, S. H., & Wong, T. (1993). Perceived auditor quality and the earnings response coefficient. The Accounting Review, 68(2), 346–366.

    Google Scholar 

  87. Tetlock, P. C. (2007). Giving content to investor sentiment: The role of media in the stock market. Journal of Finance, 62(3), 1139–1168.

    Google Scholar 

  88. You, H., & Zhang, X. J. (2009). Financial reporting complexity and investor underreaction to 10-K information. Review of Accounting Studies, 14(4), 559–586.

    Google Scholar 

Download references

Acknowledgments

This paper is based on my dissertation from the University of California at Berkeley. I especially thank my dissertation committee members: Patricia Dechow (chair), Alastair Lawrence, Panos Patatoukas, Richard Sloan, and Stephen Davidoff Solomon. I have received valuable comments and suggestions from Russell Lundholm (editor), two anonymous reviewers, John Barrios, Robert Bartlett, Stefano DellaVigna, Paul Fischer, Miles Gietzmann, Mark Huson, Greg Miller, Lillian Mills, Miguel Minutti-Meza, Reining Petacchi, Gordon Phillips, Lakshmanan Shivakumar, Shyam Sunder, Phil Stocken, İrem Tuna, Anastasia Zakolyukina, Luigi Zingales, and workshop participants at Cornell University, Dartmouth College, IESE, London Business School, the University of California at Los Angeles, the University of Texas at Austin, the University of Toronto, Yale University, the 2015 FARS Conference, 2015 EAA Annual Congress, 2015 PCAOB Conference, and the 2018 Carnegie Mellon University Accounting Mini-Conference.

Author information

Affiliations

Authors

Corresponding author

Correspondence to James P. Ryans.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1. Naïve Bayes classification terms over time

Table 11 Naïve Bayes classification terms over time

Appendix 2. Variable definitions

Table 12 Variable definitions

Appendix 3. Supplemental latent Dirichlet allocation analysis

Table 13 Top terms associated with comment letter topics
figure4
figure5

These figures present examples of the variation in standard letter language from a pre- and post-2010 comment letter, which the latent Dirichlet allocation topic analysis identifies within the letters as two different topics. Depending on the scope of the review and if there are requests for revisions, these sections of the comment letters will exhibit additional variation.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Ryans, J.P. Textual classification of SEC comment letters. Rev Account Stud 26, 37–80 (2021). https://doi.org/10.1007/s11142-020-09565-6

Download citation

Keywords

  • SEC comment letters
  • Financial reporting quality
  • Enforcement
  • Text classification
  • Restatements
  • Write-downs
  • Naïve Bayes
  • Latent Dirichlet allocation

JEL

  • D78
  • G14
  • G18
  • G38
  • M41
  • M48