How to Code a Million Missions: Developing Bespoke Nonprofit Activity Codes Using Machine Learning Algorithms

Abstract

National Taxonomy of Exempt Entities (NTEE) codes have become the primary classifier of nonprofit missions since they were developed in the mid-1980s in response to growing demands for a taxonomy of nonprofit activities (Herman in Nonprofit and Voluntary Sector Quarterly 19(3):293–306, 1990, Barman in Social Science History 37:103–141, 2013). However, the increasingly complex nature of nonprofits means that NTEE codes may be outdated or lack specificity. As an alternative, scholars and practitioners can create a bespoke taxonomy for a specific purpose by hand-coding a training dataset and using machine learning classifiers to apply the codes to a large population. This paper presents a framework for determining training set sizes needed to scale custom taxonomies using machine learning algorithms.

Appendix: Model Fit Formulas

Metric	Formula
Sensitivity	${\text{SN}} = \frac{{{\text{TP}}}}{{{\text{TP}} + {\text{FN}}}} = \frac{{{\text{TP}}}}{{\text{P}}}$
Specificity	${\text{SP}} = \frac{{{\text{TN}}}}{{{\text{TN}} + {\text{FP}}}} = \frac{{{\text{TN}}}}{{\text{N}}}$
Precision	${\text{PREC}} = \frac{{{\text{TP}}}}{{{\text{TP}} + {\text{FP}}}}$
Recall	${\text{SN}} = \frac{{{\text{TP}}}}{{{\text{TP}} + {\text{FN}}}} = \frac{{{\text{TP}}}}{{\text{P}}}$
F1	$F_{1} = \frac{{{2} \cdot {\text{PREC}} \cdot {\text{REC}}}}{{{\text{PREC}} + {\text{REC}}}}$
Accuracy	${\text{ACC}} = \frac{{{\text{TP}} + {\text{TN}}}}{{{\text{TP}} + {\text{TN}} + {\text{FN}} + {\text{FP}}}} = \frac{{{\text{TP}} + {\text{TN}}}}{{{\text{P}} + {\text{N}}}}$
Balanced accuracy	$\begin{aligned} {\text{BA }} & = \frac{{\left( {{\text{Sensitivity}} + {\text{Specificity}}} \right)}}{2} \\ & = \left[ {\left( {\frac{{{\text{TP}}}}{{{\text{TP}} + {\text{FN}}}}} \right) + \left( {\frac{{{\text{TN}}}}{{{\text{TN}} + {\text{FP}}}}} \right)} \right]* \frac{1}{2} \\ \end{aligned}$
Error	${\text{ERR}} = \frac{{{\text{FP}} + {\text{FN}}}}{{{\text{TP}} + {\text{TN}} + {\text{FN}} + {\text{FP}}}} = \frac{{{\text{FP}} + {\text{FN}}}}{{{\text{P}} + {\text{N}}}}$

1. Source: Balanced Accuracy’s first formulation comes from Kuhn (2019). The second formulation comes from Brodersen et al. (2010). Other formulas come from Saito and Rehmsmeier (n.d.).