Skip to main content
SpringerLink
Log in

A Systematic Review and Meta-Analysis of the Diagnostic Accuracy of the Advanced Clinical Solutions Word Choice Test as a Performance Validity Test

  • Review
  • Published:
Neuropsychology Review Aims and scope Submit manuscript

Abstract

Thorough assessment of performance validity has become an established standard of practice in neuropsychological assessment. While there has been a large focus on the development and cross-validation of embedded performance validity tests (PVTs) in recent years, new freestanding PVTs have also been developed, including the Word Choice Test (WCT) as part of the Advanced Clinical Solutions Effort System. And, while the WCT’s general utility for identifying invalid performance has been demonstrated in the ensuing decade since its initial publication, optimal cut-scores and associated psychometric properties have varied widely across studies. This study sought to synthesize the existing diagnostic accuracy literature regarding the WCT via a systematic review and to conduct a meta-analysis to determine the performance validity cut-score that best maximizes sensitivity while maintaining acceptable specificity. A systematic search of the literature resulted in 14 studies for synthesis, with eight of those available for meta-analysis. Meta-analytic results revealed an optimal cut-score of ≤ 42 with 54% sensitivity and 93% specificity for identifying invalid neuropsychological test performance. Collectively, the WCT demonstrated adequate diagnostic accuracy as a PVT across a variety of populations. Recommendations for future studies are also provided.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • American Academy of Clinical Neuropsychology. (2007). American Academy of Clinical Neuropsychology (AACN) practice guidelines for neuropsychological assessment and consultation. The Clinical Neuropsychologist, 21, 209–231.

    Article  Google Scholar 

  • An, K. Y., Charles, J., Ali, S., Enache, A., Dhuga, J., & Erdodi, L. A. (2019). Reexamining performance validity cutoffs within the Complex Ideational Material and the Boston Naming Test-Short Form using an experimental malingering paradigm. Journal of Clinical and Experimental Neuropsychology, 41, 15–25.

    Article  Google Scholar 

  • Bain, K. M., & Soble, J. R. (2019). Validation of the Advanced Clinical Solutions Word Choice Test (WCT) in a mixed clinical sample: Establishing classification accuracy, sensitivity/specificity, and cutoff scores. Assessment, 26, 1320–1328.

    Article  Google Scholar 

  • Bain, K. M., Soble, J. R., Webber, T. A., Messerly, J. M., Bailey, K. C., Kirton, J. W., & McCoy, K. J. M. (2019). Cross-validation of three Advanced Clinical Solutions performance validity tests: Examining combinations of measures to maximize classification of invalid performance. Applied Neuropsychology: Adult. Advance online publication. https://doi.org/10.1080/23279095.2019.1585352

  • Bailey, K. C., Soble, J. R., Bain, K. M., & Fullen, C. (2018). Embedded performance validity tests in the Hopkins Verbal Learning Test – Revised and the Brief Visuospatial Memory Test – Revised: A replication study. Archives of Clinical Neuropsychology, 33(7), 895–900. https://doi.org/10.1093/arclin/acx111

    Article  PubMed  Google Scholar 

  • Barhon, L. I., Batchelor, J., Meares, S., Chekaluk, E., & Shores, E. A. (2015). A comparison of the degree of effort involved in the TOMM and the ACS Word Choice Test using dual task paradigm. Applied Neuropsychology Adult, 22, 114–123.

    Article  Google Scholar 

  • Bashem, J. R., Rapport, L. J., Miller, J. B., Hanks, R. A., Axelrod, B. N., & Millis, S. R. (2014). Comparisons of five performance validity indices in bona fide and simulated traumatic brain injury. The Clinical Neuropsychologist, 28, 851–875.

    Article  Google Scholar 

  • Boone, K. B., Lu, P., & Herzberg, D. (2002). The Dot Counting Test manual. Western Psychological Services.

  • Boone, K. B., Salazar, X., Lu, P., Warner-Chacon, K., & Razani, J. (2002). The Rey 15-Item Recognition Trial: A technique to enhance sensitivity of the Rey 15-Item Memorization Test. Journal of Clinical and Experimental Neuropsychology, 24, 561–573.

    Article  Google Scholar 

  • Boone, K. B. (2012). Clinical practice of forensic neuropsychology: An evidenced-based approach. Guilford Press.

  • Brennan, A. M., & Gouvier, W. D. (2006). Are we honestly studying malingering? A profile and comparison of simulated and suspected malingerers. Applied Neuropsychology, 13, 1–11.

    Article  Google Scholar 

  • Chelune, G. J. (2017). Evidence-based practices in neuropsychology. In S. C. Bowden (Ed.), Neuropsychological assessment in the age of evidence-based practice: Diagnostic and treatment evaluations (pp. 155–181). Oxford University Press.

  • Critchfield, E., Soble, J. R., Marceaux, J. C., Bain, K. M., Bailey, K. C., Webber, T. A., et al. (2019). Cognitiveimpairment does not cause performance validity failure: Analyzing performance patterns among unimpaired, impaired, and noncredible participants across six tests. The Clinical Neuropsychologist, 33(6), 1083–1101. https://doi.org/10.1080/13854046.2018.1508615

    Article  PubMed  Google Scholar 

  • Curtis, K. L., Thompson, L. K., Greve, K. W., & Bianchini, K. J. (2008). Verbal fluency indicators of malingering in traumatic brain injury: Classification accuracy in known groups. The Clinical Neuropsychologist, 22, 930–945.

    Article  Google Scholar 

  • Davis, J. J. (2014). Further consideration of Advanced Clinical Solutions Word Choice: comparison to the Recognition Memory Test-words and classification accuracy in a clinical sample. The Clinical Neuropsychologist, 28, 1278–1294.

    Article  Google Scholar 

  • Dean, A. C., Victor, T. L., Boone, K. B., Philpott, L. M., & Hess, R. A. (2009). Dementia and effort test performance. The Clinical Neuropsychologist, 23, 133–152.

    Article  Google Scholar 

  • Erdodi, L. A., Kirsch, N. L., Lajiness-O’Neill, R., Vingilis, E., & Medoff, B. (2014). Comparing the Recognition Memory Test and the Word Choice Test in a mixed clinical sample: Are they equivalent? Psychological Injury and Law, 7, 255–263.

    Article  Google Scholar 

  • Erdodi, L. A., Tyson, B. T., Abeare, C. A., Zuccato, B. G., Rai, J. K., Seke, K. R., & Roth, R. M. (2017). Utility of critical items within the Recognition Memory Test and Word Choice Test. Applied Neuropsychology: Adult, 25, 327–339.

    Article  Google Scholar 

  • Erdodi, L. A., Tyson, B. T., Shahein, A. G., Lichtenstein, J. D., Abeare, C. A., Pelletier, C. L., & Roth, R. M. (2017). The power of timing: Adding a time-to-completion cutoff to the Word Choice Test and Recognition Memory Test improves classification accuracy. Journal of Clinical and Experimental Neuropsychology, 39, 369–383.

    Article  Google Scholar 

  • Freeman, S. C., Kerby, C. R., Patel, A., Cooper, N. J., Quinn, T., Sutton, A. J. (2019). Development of an interactive web-based tool to conduct and interrogate meta-analysis of diagnostic test accuracy studies: MetaDTA. BMC Medical Research Methodology, 19, 81. https://crsu.shinyapps.io/dta_ma/

  • Grant, A. (2018). Detection of feigned symptoms of ADHD, depression, and anxiety in college students (Unpublished master’s thesis). Saint Louis, MO: Saint Louis University.

    Google Scholar 

  • Green, P. (2003). Green’s word memory test for Microsoft Windows: User’s manual. Green’s Publishing.

  • Green, P. (2004). Green’s medical symptom validity test (MSVT) for Microsoft Windows: User’s manual. Green’s Publishing.

  • Greiffenstein, M. F., Baker, W. J., & Gola, T. (1994). Validation of malingered amnesia measures with a large clinical sample. Psychological Assessment, 6, 218–224.

    Article  Google Scholar 

  • Greve, K. W., Heinly, M. T., Bianchini, K. J., & Love, J. M. (2009). Malingering detection with the Wisconsin Card Sorting Test in mild traumatic brain injury. The Clinical Neuropsychologist, 23, 343–362.

    Article  Google Scholar 

  • Guyatt, G., Rennie, D., Meade, M., & Cook, D. (Eds.). (2015). Users’ guides to the medical literature: A manual for evidence-based clinical practice (3rd ed.). AMA Press.

  • Harbord, R. M., & Whiting, P. (2009). Metandi: Meta-analysis of diagnostic accuracy using hierarchical logistic regression. The Stata Journal, 9(2), 211–229.

    Article  Google Scholar 

  • Hardy, R. J., & Thompson, S. G. (1998). Detecting and describing heterogeneity in meta-analysis. Statistics in Medicine, 17(8), 841–856. https://doi.org/10.1002/(sici)1097-0258(19980430)17:8%3c841::aid-sim781%3e3.0.co;2-d

    Article  CAS  PubMed  Google Scholar 

  • Hayward, L., Hall, W., Hunt, M., & Zubrick, S. R. (1987). Can localized brain impairment be simulated on neuropsychological test profiles? Australian and New Zealand Journal of Psychiatry, 21, 87–93.

    Article  CAS  Google Scholar 

  • Heilbronner, R. L., Sweet, J. J., Morgan, J. E., Larrabee, G. J., & Millis, S. R. (2009). American Academy of Clinical Neuropsychology Consensus Conference Statement on the neuropsychological assessment of effort, response bias, and malingering. The Clinical Neuropsychologist, 23, 1093–1129.

    Article  Google Scholar 

  • Institute of Medicine. (2015). Psychological testing in the service of disability determination: Report brief. Institute of Medicine.

  • Iverson, G. L., Lange, R. T., Green, P., & Franzen, M. D. (2002). Detecting exaggeration and malingering with the Trail Making Test. The Clinical Neuropsychologist, 16, 398–406.

    Article  Google Scholar 

  • Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., & Moher, D. (2009). The PRISMA Statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and Elaboration. PLoS Medicine, 6(7), e1000100.

    Article  Google Scholar 

  • Lippa, S. M. (2018). Performance validity testing in neuropsychology: A clinical guide, critical review, and update on a rapidly evolving literature. The Clinical Neuropsychologist, 32(3), 391–421. https://doi.org/10.1080/13854046.2017.1406146

    Article  PubMed  Google Scholar 

  • Lu, P. H., Boone, K. B., Cozolino, L., & Mitchell, C. (2003). Effectiveness of the Rey-Osterrieth Complex Figure Test and the Meyers and Meyers Recognition Trial in the detection of suspect effort. The Clinical Neuropsychologist, 17, 426–440.

    Article  Google Scholar 

  • Macaskill, P., Gatsonis, C., Deeks, J., Harbord, R., & Takwoingi, Y. (2010). Analysing and presenting results. In J. Deeks, P. Bossuyt & C. Gatsonis (Eds.), Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 1.0 (pp. 1–61). The Cochrane Collaboration. http://methods.cochrane.org/

  • Martin, P. K., Schroeder, R. W., & Odland, A. P. (2015). Neuropsychologists’ validity testing beliefs and practices: A survey of North American professionals. The Clinical Neuropsychologist, 29, 741–776.

    Article  Google Scholar 

  • Martin, P. K., Schroeder, R. W., Olsen, D., Maloy, H., Boettcher, A., Ernst, N., & Okut, H. (2020). A systematic review and meta-analysis of the Test of Memory Malingering in adults: Two decades of deception detection. The Clinical Neuropsychologist, 34, 88–119.

    Article  Google Scholar 

  • Miller, J. B., Millis, S. R., Rapport, L. J., Bashem, J. R., Hanks, R. A., & Axelrod, B. N. (2011). Detection of insufficient effort using the Advanced Clinical Solutions for the Wechsler Memory Scale, Fourth Edition. The Clinical Neuropsychologist, 25, 160–172.

  • Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & the PRISMA Group. (2008). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA Statement. PLoS Medicine, 6, e100097.

    Google Scholar 

  • Neale, A. C., Ovsiew, G. P., Resch, Z. J., & Soble, J. R. (2020). Feigning or forgetfulness: The effect of memory impairment severity on Word Choice Test performance. The Clinical Neuropsychologist. Advance online publication. https://doi.org/10.1080/13854046.2020.1799076

  • Ovsiew, G. P., Resch, Z. J., Nayar, K., Williams, C. P., & Soble, J. R. (2020). Not so fast! Limitations of processing speed and working memory indices as embedded performance validity tests in a mixed neuropsychiatric sample. Journal of Clinical and Experimental Neuropsychology, 42(5), 473–484. https://doi.org/10.1080/13803395.2020.1758635

    Article  PubMed  Google Scholar 

  • Patrick, S. D., Rapport, L. J., Kanser, R. J., Hanks, R. A., & Bashem, J. R. (2020). Performance validity assessment using response time on the Warrington Recognition Memory Test. The Clinical Neuropsychologist: Advance online publication.

    Book  Google Scholar 

  • Pearson, N. (2009). Advanced clinical solutions for WAIS-IV and WMS-IV: Administration and scoring manual. The Psychological Corporation.

  • Pliskin, J. I., De-Dios-Stern, S., Resch, Z. J., Saladino, K. F., Ovsiew, G. P., Carter, D. A., & Soble, J. R. (2020). Comparing the psychometric properties of 8 embedded performance validity tests in the Rey Auditory Verbal Learning Test, Wechsler Memory Scale Logical Memory, and Brief Visuospatial Memory Test-Revised recognition trials for detecting invalid neuropsychological test performance. Assessment. Advance online publication. https://doi.org/10.1177/1073191120929093

  • Resch, Z. J., Pham, A. T., Abramson, D. A., White. D. J., DeDios-Stern, S., Ovsiew, G. P., Castillo, L., & Soble, J. R. (2020). Examining independent and combined accuracy of embedded performance validity tests in the California Verbal Learning Test-II and Brief Visuospatial Memory Test-Revised for detecting invalid performance. Applied Neuropsychology: Adult. Advance online publication. https://doi.org/10.1080/23279095.2020.1742718

  • Rey, A. (1964). L’examen Clinique en psychologie. Presses Universitaires de France.

  • Rogers, R. (Ed.) (2008). Clinical assessment of malingering and deception (3rd ed.). Guilford Press.

  • Root, J. C., Robbins, R. N., Chang, L., & Van Gorp, W. G. (2006). Detection of inadequate effort on the California Verbal Learning Test-Second Edition: Forced choice recognition and critical item analysis. Journal of the International Neuropsychological Society, 12, 688–696.

    Article  Google Scholar 

  • Schroeder, R. W., Martin, P. K., Heinrichs, R. J., & Baade, L. E. (2019). Research methods in performance validity testing studies: Criterion grouping approach impacts study outcomes. The Clinical Neuropsychologist, 33, 466–477.

    Article  Google Scholar 

  • Soble, J. R., Alverson, W. A., Phillips, J. I., Critchfield, E. A., Fullen, C., O’Rourke, J. J. F., et al. (2020). Strength in numbers or quality over quantity? Examining the importance of criterion measure selection to define validity groups in performance validity test (PVT) research. Psychological Injury and Law, 13, 44–56. https://doi.org/10.1007/s12207-019-09370-w

    Article  Google Scholar 

  • Soble, J. R., Bain, K. M., Bailey, K. C., Kirton, J. W., Marceaux, J. M., Critchfield, E. A., et al. (2019). Evaluating the accuracy of the Wechsler Memory Scale-Fourth Edition (WMS-IV) Logical Memory embedded validity index for detecting invalid test performance. Applied Neuropsychology: Adult, 26(4), 311–318. https://doi.org/10.1080/23279095.2017.1418744

    Article  Google Scholar 

  • Sopko, K. (2012). Detecting feigned reading learning disability and Attention Deficit Hyperactivity Disorder: The effects of coaching (Unpublished master’s thesis). Saint Louis, MO: Saint Louis University.

    Google Scholar 

  • Tombaugh, T. N. (1996). Test of memory malingering. Multi-Health Systems.

  • Vickery, C. D., Berry, D. T., Inman, T. H., Harris, M. J., & Orey, S. A. (2001). Detection of inadequate effort on neuropsychological testing: A meta-analytic review of selected procedures. Archives of Clinical Neuropsychology, 16, 45–73.

    CAS  PubMed  Google Scholar 

  • Viechtbauer, W. (2007). Hypothesis tests for population heterogeneity in meta-analysis. The British Journal of Mathematical and Statistical Psychology, 60(Pt 1), 29–60. https://doi.org/10.1348/000711005X64042

    Article  PubMed  Google Scholar 

  • Webber, T. A., Critchfield, E. A., & Soble, J. R. (2020). Convergent, discriminant, and concurrent validity of nonmemory-based performance validity tests. Assessment, 27(7), 1399–1415. https://doi.org/10.1177/1073191118804874

    Article  PubMed  Google Scholar 

  • Webber, T. A., & Soble, J. R. (2018). Utility of various WAIS-IV Digit Span indices for identifying noncredible performance validity among cognitively impaired and unimpaired examinees. The Clinical Neuropsychologist, 32(4), 657–670. https://doi.org/10.1080/13854046.2017.1415374

    Article  PubMed  Google Scholar 

  • White, D. J., Korinek, D., Bernstein, M. T., Ovsiew, G. P., Resch, Z. J., & Soble, J. R. (2020). Cross-validation of non-memory-based embedded performance validity tests for detecting invalid performance among patients with and without neurocognitive impairment. Journal of Clinical and Experimental Neuropsychology, 42, 459–472. https://doi.org/10.1080/13803395.2020.1758634

    Article  PubMed  Google Scholar 

  • Whiting, P. F., Rutjes, A. W., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M., Sterne, J. A., Bossuyt, P. M., & QUADAS-2 Group. (2011). QUADAS-2: A revised tool for the quality assessment of diagnostic accuracy studies. Annals of Internal Medicine, 155(8), 529–536. https://doi.org/10.7326/0003-4819-155-8-201110180-00009

    Article  Google Scholar 

  • Young, J. C., Roper, B. L., & Arentsen, T. J. (2016). Validity testing and neuropsychology practice in the VA healthcare system: results from recent practitioner survey. The Clinical Neuropsychologist, 30, 497–514.

    Article  Google Scholar 

  • Zuccato, B. G., Tyson, B. T., & Erdodi, L. A. (2018). Early bird fails the PVT? The effects of timing artifacts on performance validity tests. Psychological Assessment, 30, 1491–1498.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychiatry, University of Illinois College of Medicine, Chicago, IL, USA

    Matthew T. Bernstein, Zachary J. Resch, Gabriel P. Ovsiew & Jason R. Soble

  2. Department of Psychology, Rosalind Franklin University of Medicine and Science, IL, North Chicago, USA

    Zachary J. Resch

  3. Department of Neurology, University of Illinois College of Medicine, Chicago, IL, USA

    Jason R. Soble

Authors
  1. Matthew T. Bernstein
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zachary J. Resch
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gabriel P. Ovsiew
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jason R. Soble
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Matthew T. Bernstein.

Ethics declarations

Conflicts of interest

The authors have no conflicts of interest to report.

Availability of data and material

All of the reviewed studies are publicly available.

Funding

The authors do not have any financial interest with the subject matter discussed in the manuscript.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bernstein, M.T., Resch, Z.J., Ovsiew, G.P. et al. A Systematic Review and Meta-Analysis of the Diagnostic Accuracy of the Advanced Clinical Solutions Word Choice Test as a Performance Validity Test. Neuropsychol Rev 31, 349–359 (2021). https://doi.org/10.1007/s11065-020-09468-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11065-020-09468-y

Keywords

Search

Navigation