The economics of pre-crime interventions
- 1.3k Downloads
Several observers suggest that we may have undergone a shift from a post-crime to a pre-crime society in which the principal focus has become the pursuit of security by anticipating and forestalling future harms, rather than responding retrospectively to harms that have actually happened. This paper is about the economics of pre-crime interventions. It investigates the welfare consequences of risk assessment and early interventions to prevent individuals from engaging in criminal activities. Furthermore, it deals with the question of what constitutes an optimal application of risk assessment and early intervention. Finally, it presents three rules of thumb to identify conditions where pre-crime intervention may be welfare enhancing.
KeywordsRisk Assessment Early Intervention
In the contemporary society, there appears to be a tendency towards a “risk society”.1 Consistent with this tendency, is a change in perspectives on crime. Zedner (2007, p. 262) recently described this change: “In important respects we are on the cusp of a shift from a post- to a pre-crime society, a society, in which the possibility of forestalling risks competes with and even takes precedence over responding to wrongs done. In consequence, the post-crime orientation of criminal justice is increasingly overshadowed by the pre-crime logic of security.”2 In a pre-crime society, people demand measures to prevent “potential criminals” from engaging in criminal behaviour. For that purpose risk assessment and early intervention may be used.
Risk assessment refers to the process of estimating the likelihood that an individual will engage in criminal behaviour. In general three main approaches can be distinguished: unstructured clinical assessment, actuarial assessment, and structured clinical assessment (Philipse 2005). The unstructured clinical assessment refers to a practice where each individual professional estimates risk on the basis of a subjective selection and weighing of information. Actuarial risk assessment is based on factors that have been shown to correlate with future behaviour. A large number of risk assessment tools fall within this category. Structured clinical assessments take actuarial assessments as a starting point, but allow individual variations.
By early interventions we mean interventions to prevent high risk individuals from engaging in criminal behaviour. Early interventions may be aimed at preventing first time offences or recidivism. There are numerous programs that aim at reducing future criminal behaviour. Examples include pre-school programs and juvenile offender programs, such a multi-systemic therapy.3 Basically, these programs try to mitigate risk factors and stimulate protective factors.4
There is an extensive literature on risk taxation and early intervention, especially in the fields of criminology and forensic psychiatry. In the law and economics of crime, hardly any attention seems to be paid to pre-crime interventions. The focus is mainly on post-crime interventions, that is law enforcement. The basic idea is that law enforcement yields a welfare gain if the benefits from the deterrent and incapacitation function of sanctions exceeds the costs of law enforcement.5
In this paper we look at the case where “pre-crime interventions” are introduced given the system of “post-crime interventions”. Adding pre-crime interventions to the existing system may lead to a reduction in crime, thus reducing the harm inflicted on victims and reducing the need for law enforcement activities. Since there are both costs and benefits associated with risk assessment and early intervention, we investigate the welfare implications. More specifically: what constitutes an optimal application of risk assessment and early intervention?
This paper is organized as follows. Section 2 investigates the welfare consequences that can be expected if risk assessment is used as a guide for early intervention. More specifically, we look at the welfare implication if “pre-crime interventions” are introduced, given the existing system of “post-crime interventions”. We investigate the optimal application of risk assessment and early intervention. Section 3 makes a step from theory to practice. After discussion practical problems, this section presents three rules of thumb to identify conditions where pre-crime interventions may be welfare enhancing. Section 4 concludes.
2 Investigating welfare consequences
Ideally, risk assessment correctly predicts whether or not an individual will engage in criminal behaviour, and early intervention effectively prevents high risk individuals from engaging in criminal behaviour. In practice, however, not all predictions will be correct and not all interventions will be effective. In 2.1 we look at the accuracy of risk assessment, in 2.2 we consider the costs and benefits of early intervention. Combining insights from 2.1 to 2.2, in 2.3 we turn to the optimal application.
2.1 Accuracy of risk assessment
Numbers of true positives, false positives, false negatives and true negatives
Behaviour in the absence of precrime intervention
a + b
c + d
a + c
b + d
a + b + c + d
Without loss of generality, we normalize group size to one, that is: a + b + c + d = 1. In the absence of pre-crime intervention, the probability of crime p is equal to a + c.
The numbers of true positives, false positives, false negatives and true negatives depend on the accuracy of the risk assessment instrument. There is an extensive literature on the accuracy of risk assessment instruments, especially in the field op epidemiology and forensic psychiatry. In these fields, risk assessment instruments are inter alia used to predict illness or problematic behaviour in the future. In this literature, the accuracy of risk assessment instruments is generally discussed in terms of a pair of indices: sensitivity and specificity.6
Sensitivity refers to the ability of a test to detect criminals of the future. It is the proportion of actual positives which are correctly identified as such. This proportion is equal to a/(a + c). This proportion is often called the “true positive rate”, TPR. A fraction of the individuals that actually will engage in criminal behaviour will in the test be classified as “not criminal”. The “false negative rate”, FNR, is equal to c/(a + c).
Specificity refers to the ability of a test to detect individuals who will not engage in criminal behaviour. It is the proportion of “not criminal individuals” correctly predicted to be “not criminal”. This proportion is equal to d/(b + d). This proportion is often called the “true negative rate”, TNR. A fraction of the individuals that actually are “not criminal” will in the test by classified as “criminal”. The “false positive rate”, FPR, is equal to b/(b + d).7
The sensitivity and specificity of a test depends on the threshold level used in the test. In order to increase the sensitivity of a test, one will have to accept a decrease in selectivity. There is an unavoidable sensitivity–specificity-trade-off. In the literature this trade-off is frequently depicted by a ROC-curve (ROC = receiver operating characteristic).8
In principle, policy makers may choose between different points on the ROC-curve. From an efficiency point of view, what is the optimal choice?
The choice of a point on the ROC-curve determines the number of true positives, false positives, false negatives, and true negatives. The optimal choice depends on the costs and benefits associated with these outcomes. In 2.2 these costs and benefits are discussed.
2.2 Costs and benefits
A fraction (a + b) is considered to be high risk individuals. A fraction a/(a + b) of the high risk individuals are true positives. That is: in the absence of early interventions these individuals will actually engage in criminal behaviour. By means of early interventions, a fraction α of the true positives will be prevented from engaging in criminal behaviour: effective intervention. A fraction (1−α) will still engage in criminal behaviour: ineffective intervention.
A fraction b/(a + b) of the high risk individuals are false positives. That is: in the absence of early interventions these individuals will not engage in criminal behaviour. In the case of false positives, individuals that are not inclined towards criminal behaviour are actually treated as potential criminals. That is, false positives give rise stigmatization. A fraction β of the false positives will still refrain from criminal behaviour: unnecessary intervention. Due to self-fulfilling prophecies, a fraction (1−β) of the false positives will be induced to engage in criminal behaviour, thus increasing harm and the need for post-crime interventions: self fulfilling prophecy.
A fraction (c + d) is considered to be low risk individuals. A fraction d/(c + d) of the low risk individuals are true negatives. That is, in the absence of early interventions these individuals will not engage in criminal behaviour: true negative. A fraction c/(c + d) of the low risk individuals are false negatives. That is, in the absence of early interventions these individuals will engage in criminal behaviour: false negative.The costs associated with these six categories can be found in Table 2. Consider, for instance, the category effective intervention. Because effective intervention prevents individuals from engaging in criminal behaviour, no harm is inflicted on potential victims (CH = 0), since there is no crime there is no need for law enforcement activities (CE = 0). Consequently CS = 0. Since risk assessment and early intervention are applied, there are costs involved in diagnosis (CD = D) and intervention (CI = I). And since the individuals in this category are inclined towards criminal behaviour, no one is incorrectly treated as a potential criminal CX = 0. CX are the costs due to the fact that a person that is not inclined towards criminal behaviour is actually treated as a potential criminal (false positives). Consequently, the costs associated with effective intervention CEI are equal to D + I.
Costs associated with risk assessment and early intervention
Behaviour in the absence of precrime intervention
Self fulfilling prophecy
Fraction: (1 − α)
Fraction: (1 − β)
CS = 0
CS = S
CS = 0
CS = S
CD = D
CD = D
CD = D
CD = D
CI = I
CI = I
CI = I
CI = I
CX = 0
CX = 0
CX = X
CX = X
CEI = D + I
CII = S + D + I
CUI = D + I + X
CSP = S + D + I + X
CTP = αCEI + (1−α)CII = D + I + (1−α)S
CFP = βCII + (1−β)CSP = D + I + X + (1−β)S
CS = S
CS = 0
CD = D
CD = D
CI = 0
CI = 0
CX = 0
CX = 0
CFN = S + D
CTN = D
The first term are the costs involved in testing as such.
The second term represents the costs associated with interventions directed at “high risk” individuals.
The third term represents an increase in harm due to “self fulfilling prophecies”, i.e. false positives induced to engage is harmful behaviour.
The forth term comprises disadvantages caused to false positives.
The final term reflects costs saving (reduction in harm and law enforcement costs) due to the fact a fraction of the true positives are prevented from engaging in harmful behaviour.9
Consider the special case where perfect risk assessment instruments (b = 0 and c = 0) and perfect intervention mechanisms (α = 1, β = 1) are available. In that case, a welfare gain can be achieved if D < a(S–I), i.e. if the costs of diagnosis are smaller than the difference between avoided harm and law enforcement costs and the costs of intervention (in the high risk subgroup).
In the next section we consider the case where risk assessment instruments and intervention mechanisms are imperfect.
2.3 Optimal application
In principle, policy makers may choose between different points on the ROC-curve. This choice determines the number of true positives, false positives, false negatives, and true negatives. As summarized in Table 2, these four outcomes have different cost implications. From an efficiency point of view, what is the optimal choice?
From (7) we may infer that the optimum will move to a point situated on “north-east” of the ROC curve if α, β, p or S increases and I of X decreases (the comparative statics can be found in the appendix).13 That is: in order to be able to intervene in a lager part in the lives of “criminal individuals”, it is considered acceptable to intervene in a larger part of the life of individuals that are actually not criminal.
Parameters influencing the optimal trade-off
Optimum moves to a point on “flatter part” of the ROC-curve if
Explanation: Reason why change in parameter leads to the choice of higher TPR and FPR, i.e. higher sensitivity and lower specificity.
The larger the fraction of true positives that is actually prevented from engaging into criminal behaviour, the larger the benefits from tracing criminal individuals.
The larger the fraction of false positives that, despite of needless interventions, do not engage in “criminal behaviour”, the less the losses induced by wrongfully treating someone as a potential criminal.
The larger the fraction of a group that is inclined towards “criminal behaviour”, the smaller the numerical consequences of a given probability of false positives and the less reason to keep the false positive rate low.
The larger the size of potential harm and the costs of law enforcement, the larger the gains from timely tracing potential criminals.
The lower the costs involved in intervening in the lives of individuals identified as potential criminals, the lower the costs due to intervening in the lives of false positives.
The lower the losses due to being incorrectly treated as potential criminal X, the lower the costs due to intervening in the lives of false positives.
3 From theory to practice
In the preceding sections we considered risk assessment and early intervention from a theoretical perspective. In this section we make a step from theory to practice. First, we present an empirical illustration, then we discuss some practical problems, and finally, we present three rules of thumb.
3.1 Empirical illustration
Example ‘adolescent risk behaviour screen’
From Table 4 we may infer that a positive test results implies a 15-times higher probability of multi problem behaviour (87/239 = 36.4 vs. 19/761 = 2.5%). This result seems to produce a very strong argument in favor of intervention. It remains to be seen, however, whether this holds true. Out of 239 testing positive, 152 are false positives. If interventions are based on this test results, almost 2 out of 3 interventions will hit the wrong persons. In other words, only 1 out of 3 measures will be on target.
3.2 Practical problems
As explained above, Table 4 corresponds to a specific point on the ROC-curve representing the test. Choosing another point on the ROC-curve, implies a change in Table 4. Moving to a point on a “flatter part” of the ROC-curve, on the one hand yields an increase in sensitivity. The number of true positives will increase, thus enabling a reduction in harm. On the other hand, this change yields a decrease in specificity. The number of false positives will increase. This not only leads to unnecessary costs, it may also give rise to stigmatization and self fulfilling prophecies. Looking for the optimal point on the ROC-curve, effectively amounts to weighing the costs and benefits related to different points.
Applying (7), it is in theory straightforward to find the optimal point. In practice, it is not that simple. Finding an optimum, requires insight into the cost changes as shown in Table 2. Some of these costs can be computed rather easily; e.g. the costs of intervention as such. For several reasons, it may be more difficult to calculate the size of harm that may be prevented by early interventions. This is because criminal careers depend on the occurrence of “life-events”.16 Furthermore, during a criminal career a criminal will cause harm at different points in time. The size of harm H is in fact the present value of harm inflicted at different points in time (cf. Cohen 1998). It may be even more difficult to assess the value of stigmatization and to estimate the probability of self fulfilling prophecies.17
3.3 Rules of thumb
The optimal application of risk assessment depends on the results that will be obtained when early interventions are based on this assessment. For a given test, the choice of a threshold determines the numbers of true positives, false positives, false negatives and true negatives. And these four possible outcomes yield divergent costs and benefits.
where there are good possibilities to predict and influences future behaviour,
where there is substantial harm and prevalence, and
where the costs are low.
3.3.1 Ad 1. Good possibilities to predict and influence future behaviour
It must be possible to predict the relevant behaviour rather accurately. That is, the available test must have good scores on sensitivity and specificity. To arrive at an optimal trade-off between sensitivity and specificity, one needs to have insight into the consequences of early intervention.
Furthermore, there must be possibilities to effectively influence future behaviour. Effective instruments are characterized by the fact that a large part of the true positives abstains from criminal behaviour (high value of α), and that a large fraction of the false positives (nevertheless) does not engage in criminal behaviour (high value of β).
As explained above, an increase in α and β moves the optimum to a point on the flatter part of the ROC-curve.
It is important to note that possibilities to predict and influence behaviour change during lifetime. More specifically, these possibilities tend to change in opposite directions. For very young children, on the one hand it is hardly possible to predict whether they will develop a criminal career. On the other hand, there are good opportunities to influence behaviour. For “hard core criminals”, there are good opportunities to predict behaviour, but it is hardly possible to influence behaviour.
These considerations suggest that there may be an optimal age interval for the application of risk assessment and early intervention. This interval probably corresponds to adolescence, i.e. the phase where most criminal careers start.18
3.3.2 Ad 2. Substantial harm and prevalence
Provided there are good possibilities to predict and influence future behaviour, a more substantial reduction in expected harm can be achieved if the size of potential harm is larger, and the fraction of the group that is inclined toward “criminal behaviour” is larger (i.e. S and p are larger).19
As explained above, an increase in S and p moves the optimum to a point on the flatter part of the ROC-curve. This implies the choice of a higher sensitivity and a lower specificity. Consequently, a test that works well for a small group with a high p, will not necessarily work well for a larger group with a smaller p.
3.3.3 Ad 3. Low costs
It goes without saying, that the application of risk assessment as such should not be too expensive (low value of D), the costs involved in intervening in the lives of individuals identified as potential criminals should be modest (low value of I), and the losses due to being incorrectly treated as a potential criminal should be low (low value of X).
As explained above, a decrease in I and X moves the optimum to a point on the flatter part of the ROC-curve.
Other things being equal, benign interventions that do not lead to stigmatization and do not inflict harm are more efficient than harm inflicting interventions.
The practical implications of these rules, of course, depend on the actual possibilities to predict and influence future behaviour etc. There is an extensive literature on the application of risk assessment in order to predict the probability that an individual will commit crime in the future. This literature, combined with the rules of thumb, gives some indications on the type of cases where precrime interventions can be expected to be welfare enhancing.
The main findings of a recent meta-study on the probability that children will enter “a life of crime” (Farrington and Welsh 2007), can be summarized as follows. This probability depends on individual factors, family factors and environmental factors. Individual factors: Low intelligence and attainment, and low empathy and impulsiveness, are important risk factors for offending. Family factors: Criminal or antisocial parent, large family size, poor parental supervision, parental conflict, and disrupted families. Environmental factors: Offenders disproportionately come from deprived families, tend to have friends who are also delinquents, tend to attend high delinquency-rate schools, and tend to live in deprived areas. It is hardly possible to isolate the influence of specific factors. For one thing, multicolinearity is a serious problem. Furthermore, there may be complicated interaction effects. The probability of delinquent behaviour increases if the number of risk factors increases relative to the number of protective factors (Loeber and Farrington 2000; Stouthamer-Loeber et al. 2002).
Karoly et al. (2005, p. 114–116) present several key findings with respect to the economics of early childhood investments. First, it is possible in principle for early childhood interventions to generate short-term and longer-terms benefits that can more than offset program costs. Second, the favourable economic returns from early childhood intervention programs are not limited to smaller-scale demonstration programs. Third, favourable benefit-cost ratios are achieved for both higher cost, more-intensive programs and lower-cost, less-intensive programs. Fourth, there is some evidence that effectively targeting program services generates more-favourable economic outcomes. Finally, increased program intensity tends to be associated with declining returns.
A meta-study on predictors of adult offender recidivism leads to the following insights (Gendreau et al. 1996). The strongest predictor domains are criminogenic needs, criminal history/history of antisocial behaviour, social achievement, age/gender/race, and family factors. Less robust predictors include intellectual functioning, personal distress factors, and socioeconomic status in the family of origin. Another meta-study presents an overview of risk assessment instruments used to estimate the risk of recidivism; this study presents AUC-values (Coid et al. 2007).
There is some information on the costs (and benefits) of early intervention programs. Aos et al. (2004) conclude that there is credible evidence that certain well-implemented programs can achieve significantly more benefits than costs. There is also evidence that specific programs yield more costs than benefits. There is even evidence that some “well-meaning programs” may result in an increase in crime (Petrosino et al. 2000).
The question of whether risk assessment and early intervention will be welfare enhancing and, if so, how risk assessment should be applied, depends on a number of variables. These variables comprise the size of potential harm, the fraction of a group inclined toward “criminal behaviour”, the quality of risk assessment instruments, the quality of intervention mechanisms, and the costs imposed on false positives.
The application of risk assessment and early intervention should be reserved to cases where there are good possibilities to predict and influences future behaviour, where there is substantial harm and prevalence and where the costs are low. If these conditions are not fulfilled, the application of risk assessment and early intervention may be inefficient. Even worse, it may lead to an increase in the level of crime.
Several developments may warrant a more extensive application. First, there may be an improvement in the quality of instruments to predict and influence future behaviour. Second, there may be social developments, such as the rise of a risk society. The risk society yields an increase in the subjective valuation of harm. Furthermore, the subjective valuation of costs associated with false positives may decrease.
A caveat is in order. As indicated, the subjective valuation of harm and the valuation of costs associated with false positives may change over time. An increase in subjective harm may be used as an argument for a change in the decision threshold used in risk assessment. This would give rise to an increase in the number of false positives, thus leading to dissatisfaction. Therefore, one would expect an increase in the subjective costs associated with false positives. Consequently, one would expect an adaptation of the decision threshold in order to reduce the number of false positives. This mechanism would, in other words, give rise to “cycling of decision thresholds” (Weaver and Richardson 2006). From an efficiency point of view, it would be better to avoid this mechanism.
The term pre-crime society is derived from a story by Philip K. Dick published in 1956 about a future society where murders are prevented before they happened. The concept became well-known after Spielberg’s 2002 movie “Minority Report”. The film’s story takes place in 2054 when three ‘pre-cogs’, bio-engineered young people, are able to foresee murders. They are the principal resource of a ‘pre-crime’ unit, the chief of which is John Anderton (played by Tom Cruise), who arrest people for murders they have not yet committed. See: Wright (2008, p. 482).
Aos et al. (2004) present on overview of (the costs and benefits of) relevant programs.
The so-called risk factor prevention paradigm suggests that risk factors should be reduced while protective factors are enhanced (Farrington 2008, p. 80). Risk factors are characteristics that make it more likely that an individual will choose to violate the law, while protective factors are characteristics that make it less likely that an individual will opt for criminal behaviour. See for instance Loeber and Farrington (2000).
Sensitivity and specifity are, in fact, two conditional probabilities that characterize the accuracy of a test. Kirstein and Schmidtchen (1997) follow a similar approach in their discussion of judicial detection skill. The basic idea of their approach is taken from Heiner (1983, 1985), who refers to the literature in experimental psychology about imperfect detections of signals. In the economic literature on law enforcement, one often finds a distinction between type I errors and type II errors. In law the natural starting point is the null hypothesis that the defendant is innocent. A type I error is the rejection of a true hypothesis, i.e. the conviction of an innocent person. A type II error is accepting a false hypothesis, i.e. the acquittal of a guilty person. Thus defined, a test with a high specifity has a low type I error rate and a sensitive test has a low type II error rate. Some authors start from the null hypothesis that the defendant is guilty and the obtain definitions of type I and type II errors that the one described here. See e.g. Ehrlich (1982), Miceli (1990), Garupa (1997), and Rizzolli et al. (2009). By speaking about “the probability of mistaken acquittal” and “the probability of mistaken conviction” Polinsky et al. (2008) avoid this problem.
Note that Cdif reflects cost changes due the introduction of risk assessment and early intervention relative to the status quo. That is: relative to the existing situation where only post-crime interventions are applied. Effective pre-crime interventions economize on costs associated with post-crime inventions.
Though it is, of course, an empirical question whether the condition will ever be met, this possibility can not be ruled out a priori. (Contra: MacNeil 2005).
Note that the optimal trade-off does not depend on D, the costs of diagnosis. D reflects the costs of using a test that is characterized by a specific ROC-curve.
This point is alleged to optimize “the balance between sensitivity and specificity”, though it is not explicitly derived from an optimization procedure.
The “life-course view on the development of crime” (developmental criminology) suggests that criminal careers depend on life-events. Important changes in life (professional, military, marital) may act as turning pints in criminal careers. See inter alia Sampson and Laub (2003, 2005), Blokland and Nieuwbeerta (2005).
In empirical research often only part of these components is actually quantified. Because of the difficulties involved in calculating all the costs and benefits of early interventions, some studies focus on the costs and savings to the government. An example can be found in a study of the Perry Preschool Project: “The savings to the government ($25,437) are more than twice as large as the program costs ($12, 1418), yielding net savings to the government of $13,289).” (Greenwood et al. 2001, p. 137). A more recent studies tries to quantify costs and benefits for society at large, and reports a positive net present value (Belfield, et al. 2006).
The age-crime curve shows a significant rise in this phase (cf. Farrington 1986).
In the literature on risk management one often finds a so called “risk matrix”. In the matrix, phenomena are ordered on two dimensions: the probability of harm and the severity of harm. A large S and p correspond to a point in the upper-right corner of a risk matrix.
I gratefully acknowledge helpful comments by Ben van Velthoven, participants of the 26th annual conference of the European Association of Law and Economics (Rome, September 2009), especially Doron Teichman, and two anonymous referees.
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
- Aos, S., Lieb, R., Mayfield, J., Miller, M., & Pennucci, A. (2004). Benefits and costs of prevention and early intervention programs for youth. Olympia: Washington State Institute for Public Policy.Google Scholar
- Beck, U. (1992). Risk society: Towards a new modernity. London: Sage.Google Scholar
- Belfield, C. R., Nores, M., Barnett, S., & Schweinhart, L. (2006). The high/scope Perry Preschool program. The Journal of Human Resources, XLI(1), 162–190.Google Scholar
- Coid, J., et al. (2007). Predicting and understanding risk of re-offending: the prisoner cohort study, Research summary 6. Ministry of Justice UK.Google Scholar
- David, G. (2001). The culture of control: Crime and social order in contemporary society. Oxford: Oxford University Press.Google Scholar
- Farrington, D. P. (2008). The integrated cognitive antisocial potential (ICAP) theory. In D. P. Farington (Ed.), Integrated Developmental and Life-Course Theories of Offending (pp. 73–92). New Brunswich/London: Transaction Publishers.Google Scholar
- Farrington, D. P., & Welsh, B. C. (2007). Saving Children from a Life of Crime. Oxford: Oxford University Press.Google Scholar
- Greenwood, P. W., et al. (2001). Estimating the Costs and Benefits of Early Childhood Interventions. In B. C. Welsh, D. P. Farrington, & L. W. Sherman (Eds.), Costs and Benefits of Preventing Crime (pp. 123–148). Boulder, CO: Westview Press.Google Scholar
- Heiner, R. A. (1983). The origin of predictable behaviour. American Economic Review, 73, 560–595.Google Scholar
- Heiner, R. A. (1985). The origin of predictable behaviour: Further modeling and applications, American Economic Review. Papers and Proceeding, 75, 391–396.Google Scholar
- Jankowski, M. K., Rosenberg, H. J., Sengupta, A., Rosenberg, S. D., & Wolford, G. L. (2007). Development of a screening tool to identify adolescents engaged in multiple problem behaviours: the adolescent risk behaviour screen (ARBS). Journal of Adolescent Health, 40, 180.e19–180.e26.CrossRefGoogle Scholar
- Karoly, Lynn A., M. Rebecca Kilburn, Jill S. Cannon. (2005). Early Childhood Interventions, Rand, pp. 114–116.Google Scholar
- Miceli, T. J. (1990). Optimal Prosecution of defendants whose guilt is uncertain. Journal of Law, Economics and Organisation, 6, 189–201.Google Scholar
- Philipse, W.M.G. (2005). Predicting criminal recidivism: empirical studies and clinical practice in forensic psychiatry, Radboud University Nijmegen.Google Scholar
- Polinsky, A. Mitchell and Steven Shavell (2008). Public enforcement of law, in: Nuno Garoupa (ed.), Criminal Law and Economics, Edward Elgar, 1–59.Google Scholar
- Rizzolli, M., & Saraceno, M. (2009). Better that X guilty persons escape than that one innocent suffer, working paper department of economics University of Milan—Bicocca.Google Scholar
- Shavell, S. (1987). A model of optimal incapacitation. American Economic Review, 77, 107–110.Google Scholar
- Simon, D., & Boring, J. R. III (1990). Sensitivity, specificity, and predictive value, in: J. Willis Hurst (ed.), Clinical Methods, Butterworth.Google Scholar
- Streiner, D. L., & Cairney, J. (2007). What’s under the ROC? An introduction to receiver operating characteristic curves. The Canadian Journal of Psychiatry, 52(2), 121–128.Google Scholar
- Weiss, N. S. (2008). Clinical Epidemiology, in: Kenneth J. Rothman, Sander Greenland and Timothy L. Lash, Modern Epidemiology, Wolters Kluwer, 3rd Edition.Google Scholar
- Zweig, M. H., & Campbell, G. (1993). Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clinical Chemistry, 39, 561–577.Google Scholar