Social workers provide interventions to their clients in the expectation that they will bring about the intended change. However, estimating the expected effect for the individual client is like making an informed guess as results of an intervention´s effectiveness are typically presented at the group level (Bolton, 2004). Group-based indices of change, such as statistical significance tests and effect size, provide important information regarding mean differences but are blind to how differently individuals can respond to an intervention. Some individuals respond positively, some negatively, while others show no change. That is why best care practice must include observations both at the individual level and group level (APA Presidential Task Force on Evidence-Based Practice, 2006).

A further, related, problem is that the description of “service-as-usual” (SAU) interventions often lacks important details (Glasziou et al., 2008; Kazdin, 2013), hindering social workers’ ability to track fidelity of intentional change strategies. For example, a Swedish study documents that the vast majority (94% or more) of parental or family SAU interventions have poorly described content (Swedish Association of Local Authorities & Regions, 2022), lacking, for example, a detailed description of what outcomes to expect from the activities conducted and why.

Indicators of “clinically meaningful change” (CMC) (Wolpert et al., 2015) provide a solution to these related problems. By increasing the reliability of the measured outcomes or clarifying their importance with respect to an explicit yardstick, such indicators, first, strengthen social workers’ ability to identify what, if anything, works in any given SAU intervention for their individual clients. Second, by so doing, it motivates them to probe what aspect of a SAU intervention could have caused the change. What was the active ingredient?

Thus, social workers are empowered by the ability to assess the suitability of an intervention for each client and, if necessary, to adapt it to individual needs, thereby raising the chances of improving clients’ lives.

In this article we describe, by answering three questions, how CMC indicators systematically complement clinical social work: What is “clinically meaningful change”? What difference can the choice of indicator make for the assessment of a child’s well-being? And what is required to implement such indicators in routine social work practice?

What is Clinically Meaningful Change?

The construct of “meaningful change” in individuals came into focus at least as early as in the 1980s (Jacobson & Revenstorf, 1988) and has been defined and referred to in different ways in the literature in psychology and medicine (Ferrer & Pardo, 2014) with a correspondingly wide range of operationalization. Following Wolpert et al. (2015) we use the term “clinically meaningful change” (CMC), which is defined as changes in an individual’s outcome measures that are reliable (i.e., not due to chance or measurement error) or that are large enough to be considered ‘important’ (de Vet et al., 2006). We operationalize this definition with five CMC indicators for illustrative purposes (see Table 1).

Table 1 Overview of Selected Indicators of Meaningful Change
Full size table

For many physiologic measures, for example, blood pressure or body temperature level, a consensus has developed through research, training and experience about what changes can be considered important. In other cases, where outcomes are seemingly simple and unequivocal (e.g., attending school vs. playing truant) researchers, social workers and families find it easy to speak the same language in evaluating results. In contrast, measures of an individual’s psychosocial well-being are intrinsically more subjective, and initial states and outcomes are often measured on continuous scales. In addition, the distribution of the cases manifesting symptoms often overlaps with the range of those not doing so according to the cut-off point on the scale, owing partly to contextual differences. For example, we usually consider a normal body temperature level to be 37 degrees Celsius and take that as the cut-off point for fever. However, temperatures above 37 degrees do not necessarily indicate fever without additional contextual information such as the age of the body and normal variation in body temperature (Geneva et al., 2019). Context is similarly important for social workers using CMC indicators.

The above characteristics lead to greater variation in how to interpret intentional changes and judge their importance, which is seldom reflected in differences in raw scores on a scale. They also highlight the imperative of having a yardstick against which to compare the results of measurements of complex outcomes and accounting for their inherent unreliability. The construct of CMC responds to this imperative by describing what constitutes “significant”, “important,” or “meaningful” change (Bolton, 2004).

The Need for CMC Indicators in Social Work

While CMC indicators have broad relevance in social work, we focus on their use in systematizing out-of-home care (OHC), where the need is arguably most urgent on both ethical and moral grounds (Bergström et al., 2020). Placing children in care is expected to compensate for the shortcomings in the care provided by biological (or adoptive) parents such that children can ultimately enjoy the same quality of life as their peers not in care. Yet this expectation remains largely unfulfilled. Hardly any study has found better long-term outcomes for young adults who have grown up in OHC, regardless of outcome, comparison group, methodology, or national context (e.g., Doyle, Jr., 2007, 2008; Warburton et al., 2014). One reason could be that a placement in OHC may by itself be insufficient to address the degree of problems a foster child has. For example, a Norwegian study finds that 50% of 6–12-year-old children placed in care have at least one mental health disorder (Lehmann et al., 2013).

This calls for interventions over and above placement in care because foster parents are typically ill-equipped to deal with such issues. Consequently, social care authorities are seeking to make placements and supplemental interventions more effective. In Denmark, this has led to two social care priorities in the face of severe budget constraints: to intervene earlier and to invest in interventions with proven effectiveness (Ministry of Social Affairs & of the Interior, 2019). Determining whether interventions are effective is facilitated by the creation of structured routines, centered around the use of reliable and valid measures of the desired outcomes (Boswell et al., 2015; Mackrill & Sørensen, 2020). However, one component, CMC indicators to assess change at the individual level, is still largely missing from structured routines.

Method

Context

The decision-making process of child protection professionals is a complex process and still largely “a black box” (Munro, 2019). Routine social work practice among vulnerable children in Denmark is similar to that in other European countries such as Sweden and the Netherlands, where the quality of fact-finding in child protection services has been characterized by a lack of structure in investigations, an over-reliance on the unstructured clinical interview and poor quality of clinical judgement (Andershed & Andershed, 2016; Erens et al., 2022). The situation in foster care is exacerbated by poor communication, coordination and documentation of intentional change strategies between the many adults caring for children, including the social workers such as the statutory caseworker and the foster care consultant (Baviskar et al., 2023), and a lack of continuity caused by the high turnover rate among social workers (Danish Association of Social Workers, 2023; Radey & Wilke, 2023; Tham, 2007). Further, anecdotal evidence gave us the impression that social workers working with foster parents were often struggling to identify foster children’s developmental problems in a way that could generate actionable knowledge. Their mandatory counselling meetings often lacked substantive content on the child and focused instead on tangential or superficial issues. In sum, social workers’ working environment in Denmark is not conducive to providing detailed, accurate and reliable actionable knowledge about a child’s development nor to linking that knowledge to intentional change strategies.

This situation provided the rationale for two innovations through the intervention Me and My Foster Family (MOMP in Danish short-form; https://www.nubu.dk/forskning/momp/english/), which we recently pilot-tested in eight Danish municipalities. The first was the use of standardized screening tools by social workers in routine practice to inform their counseling to foster parents as a part of their routine practice and the second, to train social workers to analyze screening data using CMC indicators (Baviskar et al., 2023).Without such indicators, social workers would be struggling to make sense of the raw scores generated by the screening questionnaires: What scores, for example, should set off alarm bells, and which ones indicate that the child’s condition is improving?

The intended mechanism of change in the MOMP intervention is as follows: The child’s carers answer the screening questionnaires about the child sent to them by the social worker via an online platform → the social worker analyzes the data collected using CMC indicators and discusses the results with the child’s carers, leading to more detailed, accurate and actionable knowledge of the child’s development → an intentional change strategy is planned, which may include supplementary SAU interventions beyond the placement itself → the CMC on each measure is assessed against expectations in the next round of monitoring and adjustments made accordingly (see Fig. 1).

Fig. 1
figure 1

Illustration of a Cycle of Activities in the Project “Me and My Foster Family”

Full size image

Cases and Data

We present two cases featuring boys in family foster care drawn from data on 368 children collected in the MOMP study. The boys, Frederik and Joachim, were screened at baseline using foster mothers’ responses on the psychological screening instrument Strengths and Difficulties Questionnaire (SDQ) and followed up six months later. The SDQ, a widely used (sdqinfo.org) and tested measure of child mental health (e.g., Kersten et al., 2015), fulfills the need for a simple, well-documented instrument that can be used clinically by non-specialists in the primary sector (Arnfred et al., 2019). The SDQ Total Difficulties Score (TDS) goes from 0 to 40 points, with higher scores indicating greater mental health problems. This means that a decrease in scores indicates a positive development. Further, to use the CMC indicators described in this study, some descriptive statistics of the general population and SDQ norms are needed for 6–10-year-old boys in Denmark. These were obtained from Arnfred et al. (2019) and via personal communication with one of the co-authors. Note that the two cases were deliberately chosen for their SDQ scores with a view to illustrating a wide range of interpretations of CMC. In Frederik’s case, the pre- and post-intervention TDS were 21 and 14, respectively. The corresponding scores for Joachim were 18 and 13. At baseline, Frederik (example 1 in Table 2) is placed in the “very high difficulties” category of the clinical threshold. This indicates that Frederik had substantially greater mental health problems than boys of that age typically have in Denmark, a result supported by background information that Frederik had already undergone some type of psychiatric evaluation. Joachim (example 2 in Table 2) had a lower baseline score compared to Frederik, indicating a “high” level of problems.

Table 2 Use of the Selected Indicators: An Example Using the Parent-Scored Total Difficulties Score (TDS) of the Strengths and Difficulties Questionnaire for Danish Boys in the Age Group 6–10 Years
Full size table

Data from the post-meeting entry by the social worker on the online platform indicate that in Joachim’s case it was decided to implement an intervention called “MOMP Learning Intervention.” Developed as part of the MOMP project, it is a low-cost, low intensity intervention aimed at enhancing learning and well-being through a set of learning games that foster parents and the child can play together. The intervention delivers learning content in Danish and Mathematics corresponding to the guidelines of the Danish Ministry of Education for pre-school to sixth grade. It was developed because foster care consultants (FCC) lacked well-documented SAU interventions to treat the problems uncovered by the screening. In Frederik’s case, no new intervention was registered by the FCC. However, background information for Frederik indicated that he was already receiving “special-needs education,” the contents and timing of which were unspecified. In sum, different change strategies for each of the boys were agreed upon at the FCCs’ counseling meetings with their respective foster parents.

Statement of Ethics

The boys’ identities are unknown to the MOMP research team. The MOMP data-confidentiality agreement with municipalities meant that researchers were provided only with information on the children’s sex, age-group, background information on the type of foster family the child was placed in (e.g., network-care), any psychiatric assessment and extra help received at school, screening results and any post-screening interventions received. Thus, the boys are completely anonymous to anyone other than their social worker. Children’s participation in the project was subject to informed consent being given by their biological parents to the municipalities.

CMC Indicators

As we have noted above, there are many ways in which the CMC construct can be defined and operationalized—and therefore many indicators of the construct. For example, Ferrer and Pardo (2014) review eight indicators, Lavigne (2016) reviews seven, while Wolpert et al. (2015) assess four indicators. Of these, we have chosen five—Percent change, Crossing the clinical threshold, Standard deviation, Standard error of measurement and Reliable Change Index (see Table 1)—for the purposes of introducing the concept of CMC to the field of social work and illustrating our argument: How CMC indicators, combined with use of standardized screening tools, foster a more structured and transparent approach to assessing change in individual children in routine social work. It is important to note that while the use of CMC indicators is illustrated here using the SDQ scores as the outcome variable, they can be used with many other measures.

The five indicators were selected if they fulfilled the following pragmatic requirements. They should be useable at the individual level (e.g., Jacobson &Truax, 1991), and easily adaptable for evaluating social work practice (e.g., Westermark et al., 2011). Further, they should not require any data other than on the general population.

Percent Change

The percent change indicator is the percent change from an individual’s baseline score (Lavigne, 2016). Meaningful change is said to occur if the follow-up score exceeds a specified percentage of the individual’s baseline score. To be credible, the cut-off percent should be based on results from earlier research in the same area. For example, previous studies measuring changes in psychosocial symptoms, parents’ attitudes and behaviors toward their children, and child deviant behavior after interventions addressing foster children with conduct problems have used a cutoff of ≥ 30% from one measuring point to another as an indicator of meaningful change (Westermark et al., 2011).

Crossing the Clinical Threshold

Meaningful change for children in care can also be assessed using a threshold defined by a pre-set cutoff point on the scale for identifying the likelihood of a psychiatric disorder. Compared to the percent change method, the cutoff here is not defined by magnitude of change but in relation to norms defined by a reference group, typically peers in the general population. As a rule of thumb, the cutoff point is equivalent to the 9th percentile score in a general population (e.g., Goodman, 1997). If higher scores on a scale indicate the presence of more psychiatric symptoms, a score above the 90th percentile suggests the presence of a psychiatric disorder and usually a need for a more thorough clinical assessment. If a score crosses the cutoff score compared to a previous score, then there has been a crossing of the threshold in a positive or negative direction. Multiple thresholds can be used as cutoffs to obtain a more graduated scale (see our example in Table 2).

Standard Deviation

A standard deviation expresses in standard units by how much the members of a group differ from the mean value for a population on a measure. Change measured in standard deviation units can give the social worker a sense of the magnitude of change in a child’s score—and thus of whether it is meaningful—compared to the same change expressed in raw scores. This is because if scores in the reference population are theoretically distributed normally, deviations from the mean of the scores beyond a threshold expressed in standard deviation units tell you the proportion of observations that have “unusual” scores. For example, about 68% of all scores lie within one standard deviation from the mean of normally distributed scores, and about 95% lie within two deviations from the mean. For example, meaningful change could be defined as change greater than ± one standard deviation (e.g., Westermark et al., 2011).

Standard Error of Measurement

Measurement of concepts is prone to random error. Thus, any measurement will fluctuate about the correct or “true” value depending on the context. The standard error of measurement (SEM) estimates the extent to which repeated measures of an individual on the same scale tend to vary around her true score. It is defined as the deviation of an individual’s observed scores from their true score. If the measured change in raw scores is smaller than the estimated SEM, then no true change has taken place (Copay et al., 2007).

Reliable Change Index

The RCI indicates the minimum difference between a subject's pre and post-treatment scores necessary for it to be considered reliable, taking into account measurement error and its effect on the variability of scores (Jacobson & Truax, 1991). As such, it builds on the SEM by also indicating how confident we can be that true change has taken place. Whether a child has experienced reliable change (i.e., more than the unreliability of the measure would suggest might happen for 95% of subjects) is determined by examining if the difference between the follow-up and baseline scores is more than a pre-specified level. That level is a function of the initial standard deviation of the measure and its reliability.

Results: What Difference Can the Choice of CMC Indicator Make?

We answer the above question by illustrating the use of the five CMC indicators in out-of-home care (see Table 1) and examine whether changes in the TDS at follow-up, amounting to 7 points and 5 points for Frederik and Joachim, respectively, are clinically meaningful. The results are presented in Table 2 and summarized in the text below.

Percent Change (PC)

The results for Percent Change are as follows:

Frederik: A seven-point decrease from a baseline TDS of 21 is a 33.33% change, indicating a positive CMC.

Joachim: A five-point decrease from a baseline TDS of 18 is a 27.78% change, indicating no CMC.

Crossing the Clinical Threshold (CCT)

The classification of SDQ TDS values is as follows [category: range of scores (percentage of population scores in category)]: Close to average: 0–13 (80.6); values between the 80th and the 89th percentile = slightly raised: 14–17 (10.5); values between the 90th and the 94th percentile = high: 18–20 (3.7), and values above the 95th percentile = very high: 21–40 (5.3) (Arnfred et al., 2019).

Frederik: A decrease from a baseline score of 21 to 14 means that the boy’s TDS changed category from very high to slightly raised problems, indicating positive CMC.

Joachim: A decrease from a baseline score of 18 to 13 means that the boy’s TDS changed category from high to close to average, indicating positive CMC.

Standard Deviation (SD)

The standard deviation of the TDS for boys in this age group is 6.2 points (Arnfred et al., 2019 and via personal communication with one of the co-authors).

Frederik: A seven-point change in Frederik’s TDS is greater than the standard deviation of the TDS, indicating a positive CMC.

Joachim: A five-point change in Joachim’s TDS is smaller than the standard deviation of the TDS, indicating no CMC.

Standard Error of Measurement (SEM)

The SEM of the TDS at baseline is estimated to be 2.5.

The change in score for both Frederik and Joachim (by seven and five points, respectively) greatly exceeds this magnitude of measurement error, therefore there is positive CMC for both.

Reliable Change Index (RCI)

The z-score corresponding to the 90% level of significance is 1.65.

Frederik: RCI is estimated to be 2.0, which is greater than 1.65. This indicates that there is a positive CMC.

Joachim: RCI is estimated to be 1.4, which is less than 1.65, indicating no CMC has occurred.

Summing up, the two examples show how roughly the same magnitude of raw score change, seven points and five points, respectively, is assessed as clinically meaningful on all five indicators in Frederik’s case, but only on two of the five in Joachim’s case. Note that this does not tell us anything about the cause of the change, only that the symptoms of mental health problems have dropped over time, bringing each boy’s symptom load closer to that of their peers in the general population, thereby narrowing the gap, foster child by foster child.

What is Required to Implement Such Indicators in Routine Social Work Practice?

We draw on our experiences implementing CCT and RCI in the MOMP project to answer the above question. In MOMP, social workers received training in a user-friendly online platform to administer SDQ questionnaires to different respondents and to analyze the collected data using CMC indicators. For example, use of the CCT is facilitated by the online platform using color-coded figures that allow easy interpretation while the same is achieved for the RCI by a new set of graphs that automatically (using the pre-programmed RCI formula) convert the raw score change into the RCI. Social workers are then expected to use the results in their mandatory half-yearly counseling meetings with foster parents. Further, compared to the previous situation marked by the absence of standardized screening instruments, social workers need more time, at least initially, to prepare for their counselling meetings if they are to use the measurement data, both because of an increase in the amount of outcome data available and their relative complexity.

We are developing a training manual (Baviskar et al., 2023) following the Fraser et al. (2009) five-step model (see p. 70, Fig. 4.1) with teaching material on CMC consisting of a step-by-step guide and worked-out examples supported by a tailored script. MOMP training sessions varied in length from 4 to 7 h depending on the municipalities’ wishes. In addition, two-hour sessions were held on average every 8th week over an 18-month period in each municipality for training social workers in how to use screening results in their counseling meeting with foster parents including, if necessary, selection of appropriate interventions for addressing the problems revealed by the screening.

We have collected qualitative data on the use of the CMC indicators through think-aloud and semi-structured interviews with and observations of social workers and foster parents. The data suggest, overall, that the indicators can be implemented, and that they are seen as a useful input in social counseling (Danneskiold-Samsøe et al., 2020). They also suggest that the more complex the CMC indicator, the more hesitant the social workers are in using it. For example, the CCT indicator is intuitive and requires little or no statistical knowledge. In contrast, the RCI involves understanding a statistical formula or at least the basic idea behind it, which can be a challenge for social workers, who are trained almost exclusively in qualitative methods.

From a broader, social worker education perspective, use of CMC indicators should build on the basic grasp of a few key statistical concepts such as scaling, measurement error and the normal distribution. This would clarify to students, for example, why a change in raw outcome scores is not always synonymous with CMC and, more generally, curb any tendency to reify numbers. Such relevant forms of numerical literacy need to be a part of social work curricula. The skills gap related to literacy in being able to fully read and interpret statistical information in the UK identified by Teater and colleagues (2017) is by no means limited to that country, applies equally to the social work profession (Taylor, 1990), and needs urgent attention. A short-term solution is to select a CMC indicator that is easy to grasp and explain. Of the five presented here, CCT fits this description best. That can be combined with easy access to training materials and on-demand refresher courses as we have done in MOMP.

Discussion

In this article we have introduced the construct of “clinically meaningful change” (CMC), illustrated its importance for social workers trying to assess intentional change in individual clients and to probe the efficacious elements of poorly described interventions, and described the challenges in implementing CMC indicators in routine practice.

In our two cases, Joachim and Fredrik, the five CMC indicators gave different, even contradictory, results. While only used for illustrative purposes, such a situation in the real world, we would argue, is analogous to the situation where social workers must make sense of contradictory pieces of information about the child from clinical interviews with different informants. Here, too, they must use their experience and knowledge of the child’s context to make sense of such results.

Further, in order to make sense of change over time in a child, the social worker needs to have a clear idea of the intended outcome from baseline to follow-up (cf. Gottfredson et al., 2015). In Joachim’s case, the result based on RCI suggested that “MOMP Learning Intervention,” a well-described intervention, had not led to CMC, contrary to the conclusion implied by the raw-score change and by the CCT indicator. This should give the social worker reason to reconsider or at least raise questions regarding the suitability of that intervention for Joachim or probe its fidelity of implementation.

In contrast, Frederik’s positive CMC based both on the RCI and CCT suggests that his placement, or the undefined special-needs education he was receiving, or some combination of the two, had caused the positive result. As we noted above, the contents of most SAU interventions are poorly described and “special needs education” is an example of this, with form and content varying widely. This would ideally give the social worker reason to examine what aspect of this education, or some related factor, had been efficacious.

Applicability of the CMC Construct in Different Cultures

Though our example is from Denmark, we believe that the CMC construct is generalizable across cultures given the wide acceptance of the need for reliable measurement and for identifying what is an “important” change. However, its concrete operationalization can vary owing to cultural variation in survey responses (Johnson et al., 2011) used to provide input on the outcome variable for CMC indicators in the form of norm values and reliability. For example, research shows that responses to SDQ questionnaire items are different in Japan compared to in Britain (Moriwaki & Kamio, 2014) owing to different interpretation of the items, including the internal frames of reference used. This illustrates the risk of applying the norm values or reliability coefficients (needed, e.g., to estimate the CCT and RCI, respectively) from the Japanese general population to the British population or vice versa. The same risk may also apply to a marginalized section of, or ethnic minorities in, a country’s population.

In the absence of appropriate population-based data, it would be advisable to use a CMC indicator that is insensitive to population characteristics, for example, the Percent Change method, which uses only the magnitude of change from the individual child’s own baseline score as a yardstick. Each CMC indicator has its pros and cons (e.g., Lavigne, 2016) and user conditions determine the suitability of one indicator over the others.

As this article is, to the best of our knowledge, one of the first to raise the subject of clinically meaningful change in social work, our aim was simply to point out the need for CMC indicators, not to recommend a specific one. Which CMC indicator to use depends on many factors, including availability of the required statistical input (e.g., norm values), and training and infrastructure requirements, and is therefore probably a decision made by management.

Importance of CMC Indicators for Social Workers’ Motivation

More generally, CMC indicators can play a role in maintaining social workers’ work motivation. Surveys consistently indicate that social workers feel overburdened and stressed (Danish Association of Social Workers, 2023). Therefore, it is imperative to ensure that they perceive their work as meaningful. One way of doing this, as we have noted, is to empower them by giving them the tools (CMC indicators) to assess intentional change strategies in a more transparent or accurate way and, following up, to probe the effectiveness of any measures taken. Further, our data indicate that such empowerment need not come at the cost of a disproportionate rise in workload. In MOMP, output regarding CMC is automatically generated by the online platform, and it takes only about 15–20 min for the social worker to understand the results and plan how to discuss them in the counseling meeting with the foster parents.

Implications for Researchers

Our study also has implications for researchers. As we pointed out above, group-level analyses are unable to uncover variation between individuals in response to interventions. For example, one reason why some individuals experience a negative or neutral CMC could be the side effects from an intervention. Therefore, research on the impact of interventions should also include individual-level analyses using CMC indicators to gain a better idea of how many clients, considered individually, experience a meaningful change following an intervention.

Limitations

In this article we have limited our focus to the psychological well-being of children in OHC using the SDQ. However, we should emphasize that use of these indicators is not limited to this type of outcome, this target group, or to the chosen measure. Indeed, indicators of CMC should be used for all outcomes whose clinical importance at the individual level is hard to gauge owing to their inherently subjective nature, measurement on continuous scales, and overlap between symptomatic and asymptomatic cases.

Further, it is important to note that while we have presented ways of answering the question “is the measured change meaningful or important?,” none of these indicators can capture the full magnitude of “meaningfulness,” which also includes what the children themselves consider important. This implies that determining whether a change is clinically meaningful should be treated as a starting point for a follow-up meeting with the child and its carers to better identify needs and inform services. In short, the use of such indicators is necessary but insufficient for determining whether a child in OHC needs, or has benefited from, an intervention. Echoing Vostanis (2006) and O’Neill (2018) regarding use of the SDQ in social work, we maintain that these indicators are meant as an input to—not a substitute for—the social worker’s professional judgement.

Finally, our choice of CMC indicators was motivated by the MOMP project of which this study is a part and by space constraints. A more comprehensive review of CMC indicators in social work should not only include more indicators but also combine pre-post measurements with clinical observations including individual case-studies.