This study examines the impact of SEC comment letters on future financial reporting outcomes and earnings credibility. Naïve Bayesian classification identifies comment letters associated with future restatements and write-downs. An investor attention-based quantitative measure of importance, using EDGAR downloads, also predicts these outcomes. Disclosure-event abnormal returns, revenue recognition comments, and the number of letters in a conversation appear to be useful quantitative metrics for classifying importance in certain settings. This study also documents trends in comment letter topics over time and identifies topics associated with the textual and quantitative classifications of importance, providing insights into the factors that draw investor attention and that relate to future restatements and write-downs. Innocuous comment letters are associated with improvements in earnings credibility following comment letter reviews.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Examples of short seller research that uses issues raised in comment letters include presentations by Greenlight Capital on Green Mountain Coffee, Pershing Square on Herbalife, and Prescience Point on Boulder Brands. http://online.wsj.com/public/resources/documents/EinhornGMCRpresentation_Oct2011_VIC.pdf. Retrieved 7 September, 2020.http://factsabout-herbalife.com/wp-content/uploads/2013/01/Who-wants-to-be-a-Millionaire.pdf. Retrieved 1 February 2013.https://www.presciencepoint.com/research/research-archives/boulder-brands-inc-bdbd/. Retrieved 7 September, 2020.
Care needs to be taking cleaning the raw EDGAR log file data set to accurately count comment letter downloads, which are usually filed as PDF documents. Ryans, J., 2017. Using the EDGAR log file data set. Working paper, London Business School.
As described previously, the use of multi-word features such as bigrams, which are used in this study, can allow for some preservation of word order in naïve Bayesian analysis.
See Securities and Exchange Commission. Staff Observations in the Review of Executive Compensation Disclosure. September 10, 2007. http://www.sec.gov/divisions/corpfin/guidance/execcompdisclosure.htm. Accessed 7 September 2020.
Antweiler, W., & Frank, M. Z. (2004). Is all that talk just noise? The information content of internet stock message boards. Journal of Finance, 59(3), 1259–1294.
Ball, R., & Brown, P. (1968). An empirical evaluation of accounting income numbers. Journal of Accounting Research, 6(2), 159–178.
Bao, Y., & Datta, A. (2014). Simultaneously discovering and quantifying risk types from textual risk disclosures. Management Science, 60(6), 1371–1391.
Bartov, E., Givoly, D., & Hayn, C. (2002). The rewards to meeting or beating earnings expectations. Journal of Accounting and Economics, 33(2), 173–204.
Beaver, W. H. (1968). The information content of annual earnings announcements. Journal of Accounting Research, 1968, 67–92.
Bens, D. A., Cheng, M., & Neamtiu, M. (2016). The impact of SEC disclosure monitoring on the uncertainty of fair value estimates. The Accounting Review, 91(2), 349–375.
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.
Bloomfield, R. J. (2002). The “incomplete revelation hypothesis” and financial reporting. Accounting Horizons, 16(3), 233–243.
Bozanic, Z., Dietrich, J. R., & Johnson, B. A. (2017). SEC comment letters and firm disclosure. Journal of Accounting and Public Policy, 36(5), 337–357.
Brown, S. V., Tian, X., & Tucker, J. W. (2018). The spillover effect of SEC comment letters on qualitative corporate disclosure: Evidence from the risk factor disclosure. Contemporary Accounting Research, 35(2), 622–656.
Bryan, S. H. (1997). Incremental information content of required disclosures contained in management discussion and analysis. The Accounting Review, 72(2), 285–301.
Carhart, M. M. (1997). On persistence in mutual fund performance. Journal of Finance, 52(1), 57–82.
Cassell, C. A., Dreher, L. M., & Myers, L. A. (2013). Reviewing the SEC’s review process: 10-K comment letters and the cost of remediation. The Accounting Review, 88(6), 1875–1908.
Christensen, H. B., Hail, L., & Leuz, C. (2013). Mandatory IFRS reporting and changes in enforcement. Journal of Accounting and Economics, 56(2–3), 147–177.
Collins, D. W., & Kothari, S. P. (1989). An analysis of intertemporal and cross-sectional determinants of earnings response coefficients. Journal of Accounting and Economics, 11(2), 143–181.
Cunningham, L. M., Johnson, B. A., Johnson, E. S., & Lisic, L. L. (2019). The switch up: An examination of changes in earnings management after receiving sec comment letters. Contemporary Accounting Research, 37(2), 917–944.
Dagan, I., Feldman, R., Hirsh, H., (1996). Keyword-based browsing and analysis of large document sets, Proceedings of the Fifth Annual Symposium on Document Analysis and Information Retrieval (pp. 191–208). University of Nevada.
Davis, A. K., Piger, J. M., & Sedor, L. M. (2012). Beyond the numbers: Measuring the information content of earnings press release language. Contemporary Accounting Research, 29(3), 845–868.
De Franco, G., Vasvari, F. P., Vyas, D., & Wittenberg-Moerman, R. (2013). Debt analysts’ views of debt-equity conflicts of interest. The Accounting Review, 89(2), 571–604.
Dechow, P. M., Ge, W., Larson, C. R., & Sloan, R. G. (2011). Predicting material accounting misstatements. Contemporary Accounting Research, 28(1), 17–82.
Dechow, P. M., Huson, M. R., & Sloan, R. G. (1994). The effect of restructuring charges on executives’ cash compensation. The Accounting Review, 69(1), 138–156.
Dechow, P. M., Lawrence, A., & Ryans, J. (2016). SEC comment letters and insider sales. The Accounting Review, 91(2), 401–439.
Dechow, P. M., Sloan, R. G., & Sweeney, A. P. (1995). Detecting earnings management. The Accounting Review, 70(2), 193–225.
DeFond, M. L., & Jiambalvo, J. (1991). Incidence and circumstances of accounting errors. The Accounting Review, 66(3), 643–655.
Drake, M., Johnson, B., Roulstone, D., & Thornock, J. (2019). Is there information content in information acquisition. The Accounting Review, 95(2), 113–139.
Drake, M. S., Roulstone, D. T., & Thornock, J. R. (2015). The determinants and consequences of information acquisition via EDGAR. Contemporary Accounting Research, 32(3), 1128–1161.
Duro, M., Heese, J., & Ormazabal, G. (2019). The effect of enforcement transparency: Evidence from SEC comment-letter reviews. Review of Accounting Studies, 24(3), 780–823.
Dyck, A., Morse, A., & Zingales, L. (2010). Who blows the whistle on corporate fraud? Journal of Finance, 65(6), 2213–2253.
Dye, R. A. (1985). Disclosure of nonproprietary information. Journal of Accounting Research, 23(1), 123–145.
Dyer, T., Lang, M., & Stice-Lawrence, L. (2017). The evolution of 10-K textual disclosure: Evidence from latent Dirichlet allocation. Journal of Accounting and Economics, 64(2–3), 221–245.
Engelberg, J. E., & Parsons, C. A. (2011). The causal impact of media in financial markets. Journal of Finance, 66(1), 67–97.
Feldman, R., Dagan, I., (1995). Knowledge discovery in textual databases KDT. Proceedings of the First International Conference on Knowledge Discovery and Data Mining (pp. 112–117). Association for Computing Machinery.
Feldman, R., Govindaraj, S., Livnat, J., & Segal, B. (2010). Management’s tone change, post earnings announcement drift and accruals. Review of Accounting Studies, 15(4), 915–953.
Feroz, E.H., Park, K., Pastena, V.S., (1991). The financial and market effects of the SEC’s accounting and auditing enforcement releases. Journal of Accounting Research 29(Supp.), 107–142.
Francis, J., LaFond, R., Olsson, P., & Schipper, K. (2005). The market pricing of accruals quality. Journal of Accounting and Economics, 39(2), 295–327.
Francis, J., Nanda, D., & Olsson, P. (2008). Voluntary disclosure, earnings quality, and cost of capital. Journal of Accounting Research, 46(1), 53–99.
Francis, J. R. (2011). A framework for understanding and researching audit quality. Auditing: A Journal of Practice & Theory, 30(2), 125–152.
Gietzmann, M. B., & Pettinicchio, A. K. (2013). External auditor reassessment of client business risk following the issuance of a comment letter by the SEC. European Accounting Review, 23(1), 57–85.
Hayn, C. (1995). The information content of losses. Journal of Accounting and Economics, 20(2), 125–153.
Heese, J., Khan, M., & Ramanna, K. (2017). Is the SEC captured? Evidence from comment-letter reviews. Journal of Accounting and Economics, 64(1), 98–122.
Hennes, K. M., Leone, A. J., & Miller, B. P. (2014). Determinants and market consequences of auditor dismissals after accounting restatements. The Accounting Review, 89(3), 1051–1082.
Hirshleifer, D., & Teoh, S. H. (2003). Limited attention, information disclosure, and financial reporting. Journal of Accounting and Economics, 36(1), 337–386.
Hoberg, G., & Lewis, C. (2017). Do fraudulent firms produce abnormal disclosure? Journal of Corporate Finance, 43(1), 58–85.
Holthausen, R. W., & Verrecchia, R. E. (1988). The effect of sequential information releases on the variance of price changes in an intertemporal multi-asset market. Journal of Accounting Research, 26(1), 82–106.
Hribar, P., & Jenkins, N. T. (2004). The effect of accounting restatements on earnings revisions and the estimated cost of capital. Review of Accounting Studies, 9(2–3), 337–356.
Hribar, P., Kravet, T., & Wilson, R. (2014). A new measure of accounting quality. Review of Accounting Studies, 19(1), 506–538.
Huang, A. H., Lehavy, R., Zang, A. Y., & Zheng, R. (2018). Analyst information discovery and interpretation roles: A topic modeling approach. Management Science, 64(6), 2833–2855.
Huang, A. H., Zang, A. Y., & Zheng, R. (2014). Evidence on the information content of text in analyst reports. The Accounting Review, 89(6), 2151–2180.
Johnston, R., & Petacchi, R. (2017). Regulatory oversight of financial reporting: Securities and exchange commission comment letters. Contemporary Accounting Research, 34(2), 1128–1155.
Jung, W. O., & Kwon, Y. K. (1988). Disclosure when the market is unsure of information endowment of managers. Journal of Accounting Research, 26(1), 146–153.
Karlgren, J., Cutting, D., (1994). Recognizing text genres with simple metrics using discriminant analysis. Proceedings of the 15th conference on Computational Linguistics (pp. 1071–1075). Association for Computational Linguistics.
Kessler, B., Numberg, G., Schutze, H., (1997). Automatic detection of text genre, Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics (pp. 32–38).Association for Computational Linguistics.
Kinney, W. R., & McDaniel, L. S. (1989). Characteristics of firms correcting previously reported quarterly earnings. Journal of Accounting and Economics, 11(1), 71–93.
Kinney, W. R., Palmrose, Z. V., & Scholz, S. (2004). Auditor independence, non-audit services, and restatements: Was the US government right? Journal of Accounting Research, 42(3), 561–588.
Kormendi, R., & Lipe, R. (1987). Earnings innovations, earnings persistence, and stock returns. Journal of Business, 60(3), 323–345.
Kothari, S. P., Li, X., & Short, J. E. (2009a). The effect of disclosures by management, analysts, and business press on cost of capital, return volatility, and analyst forecasts: A study using content analysis. The Accounting Review, 84(5), 1639–1670.
Kothari, S. P., Shu, S., & Wysocki, P. D. (2009b). Do managers withhold bad news? Journal of Accounting Research, 47(1), 241–276.
La Porta, R., Lopez-de Silanes, F., & Shleifer, A. (2006). What works in securities laws? Journal of Finance, 61(1), 1–32.
Lang, M., & Lundholm, R. (1993). Cross-sectional determinants of analyst ratings of corporate disclosures. Journal of Accounting Research, 31(2), 246–271.
Larcker, D. F., & Zakolyukina, A. A. (2012). Detecting deceptive discussions in conference calls. Journal of Accounting Research, 50(2), 495–540.
Laurion, H., Lawrence, A., & Ryans, J. (2017). U.S. audit partner rotation. The Accounting Review, 92(3), 209–237.
Law, K. K. F., & Mills, L. F. (2015). Taxes and financial constraints: Evidence from linguistic cues. Journal of Accounting Research, 53(4), 777–819.
Lawrence, A., Sloan, R., & Sun, Y. (2013). Non-discretionary conservatism: Evidence and implications. Journal of Accounting and Economics, 56(2), 112–133.
Leuz, C., & Wysocki, P. (2016). The economics of disclosure and financial reporting regulation: Evidence and suggestions for future research. Journal of Accounting Research, 54(2), 525–622.
Lewis, D.D., (1998). Naïve (Bayes) at forty: The independence assumption in information retrieval. Machine learning: ECML-98 (pp. 4–15). Springer.
Li, F. (2008). Annual report readability, current earnings, and earnings persistence. Journal of Accounting and Economics, 45(2), 221–247.
Li, F. (2010). The information content of forward-looking statements in corporate filings. A naïve Bayesian machine learning approach. Journal of Accounting Research, 48(5), 1049–1102.
Liu, L. L., Raghunandan, K., & Rama, D. (2009). Financial restatements and shareholder ratifications of the auditor. Auditing: A Journal of Practice & Theory, 28(1), 225–240.
Livnat, J., & Mendenhall, R. R. (2006). Comparing the post–earnings announcement drift for surprises calculated from analyst and time series forecasts. Journal of Accounting Research, 44(1), 177–205.
Ljungqvist, A., & Qian, W. (2016). How constraining are limits to arbitrage? Evidence from a recent financial innovation. Review of Financial Studies, 29(8), 1975–2028.
Loughran, T., & McDonald, B. (2011). When is a liability not a liability? Textual analysis, dictionaries, and 10-Ks. Journal of Finance, 66(1), 35–65.
Loughran, T., & McDonald, B. (2016). Textual analysis in accounting and finance: A survey. Journal of Accounting Research, 54(4), 1187–1230.
Loughran, T., McDonald, B., (2017). The use of EDGAR filings by investors. Journal of Behavioral Finance 18(2), 231–248).
Mendenhall, R.R., Nichols, W.D., (1988). Bad news and differential market reactions to announcements of earlier-quarters versus fourth-quarter earnings. Journal of Accounting Research 26(Supp.), 63–86.
Mosteller, F., & Wallace, D. L. (1984). Applied Bayesian and classical inference. New York: Springer-Verlag.
Naughton, J. P., Rogo, R., Sunder, J., & Zhang, R. (2018). SEC monitoring of foreign firms’ disclosures. Review of Accounting Studies, 23(4), 1355–1388.
Palmrose, Z. V., Richardson, V. J., & Scholz, S. (2004). Determinants of market reactions to restatement announcements. Journal of Accounting and Economics, 37(1), 59–89.
Pang, B., Lee, L., Vaithyanathan, S., (2002). Thumbs up?: Sentiment classification using machine learning techniques, Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing . (pp. 79–86). Association for Computational Linguistics.
Peterson, K. (2012). Accounting complexity, misreporting, and the consequences of misreporting. Review of Accounting Studies, 17(1), 72–95.
Quinn, K. M., Monroe, B. L., Colaresi, M., Crespin, M. H., & Radev, D. R. (2010). How to analyze political attention with minimal assumptions and costs. American Journal of Political Science, 54(1), 209–228.
Ramanna, K., & Watts, R. L. (2012). Evidence on the use of unverifiable estimates in required goodwill impairment. Review of Accounting Studies, 17(4), 749–780.
Robinson, J. R., Xue, Y., & Yu, Y. (2011). Determinants of disclosure noncompliance and the effect of the SEC review: Evidence from the 2006 mandated compensation disclosure regulations. The Accounting Review, 86(4), 1415–1444.
Sandler, L., (2013). Muddy waters secret China weapon is on SEC website. Bloomberg News. https://www.bloomberg.com/news/articles/2013-02-19/muddy-waters-secret-china-weapon-is-on-sec-website. Accessed 7 September 2020.
SEC, (2015). FY 2016 congressional budget justification, FY 2016 annual performance plan, and FY 2014 annual performance report. U.S. Securities and Exchange Commission. http://www.sec.gov/about/reports/secfy16congbudgjust.shtml. Accessed 7 September 2020.
Talley, E., & O’Kane, D. (2012). The measure of a MAC: A machine-learning protocol for analyzing force majeure clauses in ma agreements. Journal of Institutional and Theoretical Economics, 168(1), 181–201.
Teoh, S. H., & Wong, T. (1993). Perceived auditor quality and the earnings response coefficient. The Accounting Review, 68(2), 346–366.
Tetlock, P. C. (2007). Giving content to investor sentiment: The role of media in the stock market. Journal of Finance, 62(3), 1139–1168.
You, H., & Zhang, X. J. (2009). Financial reporting complexity and investor underreaction to 10-K information. Review of Accounting Studies, 14(4), 559–586.
This paper is based on my dissertation from the University of California at Berkeley. I especially thank my dissertation committee members: Patricia Dechow (chair), Alastair Lawrence, Panos Patatoukas, Richard Sloan, and Stephen Davidoff Solomon. I have received valuable comments and suggestions from Russell Lundholm (editor), two anonymous reviewers, John Barrios, Robert Bartlett, Stefano DellaVigna, Paul Fischer, Miles Gietzmann, Mark Huson, Greg Miller, Lillian Mills, Miguel Minutti-Meza, Reining Petacchi, Gordon Phillips, Lakshmanan Shivakumar, Shyam Sunder, Phil Stocken, İrem Tuna, Anastasia Zakolyukina, Luigi Zingales, and workshop participants at Cornell University, Dartmouth College, IESE, London Business School, the University of California at Los Angeles, the University of Texas at Austin, the University of Toronto, Yale University, the 2015 FARS Conference, 2015 EAA Annual Congress, 2015 PCAOB Conference, and the 2018 Carnegie Mellon University Accounting Mini-Conference.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix 1. Naïve Bayes classification terms over time
Appendix 2. Variable definitions
Appendix 3. Supplemental latent Dirichlet allocation analysis
These figures present examples of the variation in standard letter language from a pre- and post-2010 comment letter, which the latent Dirichlet allocation topic analysis identifies within the letters as two different topics. Depending on the scope of the review and if there are requests for revisions, these sections of the comment letters will exhibit additional variation.
About this article
Cite this article
Ryans, J.P. Textual classification of SEC comment letters. Rev Account Stud 26, 37–80 (2021). https://doi.org/10.1007/s11142-020-09565-6
- SEC comment letters
- Financial reporting quality
- Text classification
- Naïve Bayes
- Latent Dirichlet allocation