Abstract
Purpose
Registers derived from administrative datasets are valuable tools in psychosis research, but diagnostic accuracy can be problematic. We sought to compare the relative performance of four methods for assigning a single diagnosis from longitudinal administrative clinical records when compared with reference diagnoses.
Methods
Diagnoses recorded in inpatient and community mental health records were compared to research diagnoses of psychotic disorders obtained from semi-structured clinical interviews for 289 persons. Diagnoses were derived from administrative datasets using four algorithms; ‘At least one’ diagnosis, ‘Last’ or most recent diagnosis, ‘Modal’ or most frequently occurring diagnosis, and ‘Hierarchy’ in which a diagnostic hierarchy was applied. Agreements between algorithm-based and reference diagnoses for overall presence/absence of psychosis and for specific diagnoses of schizophrenia, schizoaffective disorder, and affective psychosis were examined using estimated prevalence rates, overall agreement, ROC analysis, and kappa statistics.
Results
For the presence/absence of psychosis, the most sensitive and least specific algorithm (‘At least one’ diagnosis) performed best. For schizophrenia, ‘Modal’ and ‘Last’ diagnoses had greatest agreement with reference diagnosis. For affective psychosis, ‘Hierarchy’ diagnosis performed best. Agreement between clinical and reference diagnoses was no better than chance for diagnoses of schizoaffective disorder. Overall agreement between administrative and reference diagnoses was modest, but may have been limited by the use of participants who had been screened for likely psychosis prior to assessment.
Conclusion
The choice of algorithm for extracting a psychosis diagnosis from administrative datasets may have a substantial impact on the accuracy of the diagnoses derived. An ‘Any diagnosis’ algorithm provides a sensitive measure for the presence of any psychosis, while ‘Last diagnosis’ is more accurate for specific diagnosis of schizophrenia and a hierarchical diagnosis is more accurate for affective psychosis.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Perera G, Soremekun M, Breen G, Stewart R (2009) The psychiatric case register: noble past, challenging present, but exciting future. Br J Psychiatry 195(3):191–193
Jablensky A, McGrath J, Herrman H, Castle D, Gureje O, Evans M, Carr V, Morgan V, Korten A, Harvey C (2000) Psychotic disorders in urban areas: an overview of the study on Low Prevalence Disorders. Aust N Z J Psychiatry 34(2):221–236
Morgan V, Waterreus A, Jablensky A, Mackinnon A, McGrath J, Carr V, Bush R, Castle D, Cohen M, Harvey C, Gellatly C, Neil A, McGorry PD, Hocking B, Shah S, Saw S (2011) People living with psychotic illness 2010. Commonwealth of Australia, Canberra
Morgan V, Jablensky A (2010) From inventory to benchmark: quality of psychiatric case registers in research. Br J Psychiatry 197:8–10
Byrne N, Regan C, Howard L (2005) Administrative registers in psychiatric research: a systematic review of validity studies. Acta Psychiatr Scand 112:409–414
Mortensen P (1995) The untapped potential of case registers and record-linkage studies in psychiatric epidemiology. Epidemiol Rev 17:205–209
Crabbe T, Donmall M, Millar T (1999) Validation of the University of Manchester Drug Misuse Database. J Epidemiol Community Health 53:159–164
Fennig S, Craig TJ, Tanenberg-Karant M, Bromet E (1994) Comparison of facility and research diagnoses in first-admission psychotic patients. Am J Psychiatry 151:1423–1429
Goodman AB, Rahav M, Popper M, Ginath Y, Pearl E (1984) The reliability of psychiatric diagnosis in Israel’s Psychiatric Case Register. Acta Psychiatr Scand 69:391–397
Kristjansson E, Allebeck P, Wistedt B (1987) Validity of the diagnosis schizophrenia in a psychiatric inpatient register. A retrospective application of DSM-III-criteria on ICD-8 diagnoses in Stockholm County. Nord J Psychiatry 41:229–234
Löffler W, Häfner H, Fätkenheuer B, Maurer K, Riecher-Rössler A, Lützhøft J, Skadhede S, Munk-Jørgensen P, Strömgren E (1994) Validation of Danish case register diagnosis for schizophrenia. Acta Psychiatr Scand 90(3):196–203
Øiesvold T, Nivison M, Hansen V, Sørgaard KW, Østensen L, Skre I (2012) Classification of bipolar disorder in psychiatric hospital. A prospective cohort study. BMC Psychiatry 12:13
Systema S, Giel R, Ten Horn GHMM, Balestrieri M, Davies N (1989) The reliability of diagnostic coding in psychiatric case registers. Psychol Med 19:999–1006
Taiminen T, Ranta K, Karlsson H, Lauerma H, Leinonen K-M, Wallenius E, Kaljonen A, Salokangas R (2001) Comparison of clinical and best-estimate research DSM-IV diagnoses in a Finnish sample of Ž rst-admission psychosis and severe affective disorder. Nord J Psychiatry 55:107–111
Keskimani I (1991) Accuracy of data on diagnoses, procedures and accidents in the Finnish Hospital discharge register. Int J Health Sci 2:15–21
Robinson JR, Tataryn DJ (1997) Reliability of the Manitoba mental health management information system for research. Can J Psychiatry Revue Canadienne de Psychiatrie 42:744–749
Kessing L (1998) Validity of diagnoses and other clinical register data in people with affective disorder. Eur Psychiatry 13:392–398
Jin YP, Gatz M, Johansson B, Pedersen NL (2004) Sensitivity and specificity of dementia coding in two Swedish disease registries. Neurology 63:739–741
Kirkby KC, Hay DA, Daniels BA, Jones IH, Mowry BJ (1998) Comparison between register and structured interview diagnoses of schizophrenia: a case for longitudinal diagnostic profiles. Aust N Z J Psychiatry 32(3):410–414
West SL, Richter A, Melfi CA, McNutt M, Nennstiel ME, Mauskopf JA (2000) Assessing the Saskatchewan database for outcomes research studies of depression and its treatment. J Clin Epidemiol 53:823–831
McConville P, Walker NP (2000) The reliability of case register diagnoses: a birth cohort analysis. Soc Psychiatry Psychiatr Epidemiol 35:121–127
Lichtenstein P, Bjork C, Hultman C, Scolnick E, Sklar P, Sullivan P (2006) Recurrence risks for schizophrenia in a Swedish National Cohort. Psychol Med 36(10):1417–1425
Sigurdsson E, Fombonne E, Sayal K, Checkley S (1999) Neurodevelopmental antecedents of early-onset bipolar affective disorder. Br J Psychiatry 174(2):121–127
Valuri A, Morgan V, Jablensky A (2001) Deriving a research diagnosis from a mental health register. Dept of Psychiatry and Behavioural Science, University of Western Australia, Perth
Isohanni M, Makikyro T, Moring J, Rasanen P, Hakko H, Partanen U, Koiranen M, Jones P (1997) A comparison of clinical and research DSM-III-R diagnoses of schizophrenia in a Finnish national birth cohort. Clinical and research diagnoses of schizophrenia. Soc Psychiatry Psychiatr Epidemiol 32:303–308
Makikyro T, Isohanni M, Moring J, Hakko H, Hovatta I, Lonnqvist J (1998) Accuracy of register-based diagnoses of schizophrenia in a genetic study. Eur Psychiatry 13:57–62
Ruschena D, Mullen PE, Burgess P, Cordner SM, Barry-Walsh J, Drummer OH, Palmer S, Browne C, Wallace C (1998) Sudden death in psychiatric patients. Br J Psychiatry 172(4):331–336
Wallace C, Mullen P, Burgess P, Palmer S, Ruschena D, Browne C (1998) Serious criminal offending and mental disorder. Case linkage study. Br J Psychiatry 172(6):477–484
Morgan VA, Waterreus A, Jablensky A, Mackinnon A, McGrath JJ, Carr V, Bush R, Castle D, Cohen M, Harvey C, Galletly C, Stain HJ, Neil AL, McGorry P, Hocking B, Shah S, Saw S (2012) People living with psychotic illness in 2010: the second Australian national survey of psychosis. Aust N Z J Psychiatry 46(8):735–752
Castle D, Jablensky A, McGrath JJ, Carr V, Morgan V, Waterreus AJ, Valuri G, Stain H, McGuffin P, Farmer AMD (2006) The Diagnostic Interview for Psychoses (DIP): development, reliability and applications. Psychol Med 36:69–80
McGuffin PC, Farmer A, Harvey I (1991) A polydiagnostic application of operational criteria in studies of psychotic illness. Development and reliability of the OPCRIT system. Arch Gen Psychiatry 48:764–770
National Centre for Classification in Health (2010) The International statistical classification of diseases and related health problems, tenth revision, Australian modification, 7th edn. National Centre for Classification in Health, Faculty of Health Sciences, The University of Sydney, Sydney
Sim J, Wright CC (2005) The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther 2005(85):257–268
Chen G, Faris P, Hemmelgarn B, Walker RL, Quan H (2009) Measuring agreement of administrative data with chart data using prevalence unadjusted and adjusted kappa. BMC Med Res Methodol 9(5). doi:10.1186/1471-2288-1189-1185
Mackinnon A (2000) A spreadsheet for the calculation of comprehensive statistics for the assessment of diagnostic tests and inter-rater agreement. Comput Biol Med 30(3):127–134
Landis J, Koch G (1977) The measurement of observer agreement for categorical data. Biometrics 33:159–174
Bromet EJ, Kotov R, Fochtmann LJ, Carlson GA, Tanenberg-Karant M, Ruggero C, Chang S-w (2011) Diagnostic shifts during the decade following first admission for psychosis. Am J Psychiatry 168(11):1186–1194
Schwartz JE, Fennig S, Tanenberg-Karant M, Carlson G, Craig TJ, Galambos N, Lavelle J, Bromet E (2000) Congruence of diagnoses 2 years after a first-admission diagnosis of psychosis. Arch Gen Psychiatry 57:593–600
Arajärvi R, Suvisaari J, Suokas J, Schreck M, Haukka J, Hintikka J, Partonen T, Lönnqvist J (2005) Prevalence and diagnosis of schizophrenia based on register, case record and interview data in an isolated Finnish birth cohort born 1940–1969. Soc Psychiatry Psychiatr Epidemiol 40:808–816
Erdman HP, Klein MH, Greist JH, Bass SM, Bires JK, Machtinger PE (1987) A comparison of the Diagnostic Interview Schedule and clinical diagnosis. Am J Psychiatry 144:1477–1480
Acknowledgments
This work was supported by Schizophrenia Research Institute, Australia, utilising infrastructure funding from NSW Ministry of Health. L. Luo was supported by an Australian Rotary Health Research Fund Research Grant awarded to KR Laurens and M. J. Green (2010).
Conflict of interest
The authors declare that they have no conflicts of interest.
Ethical standards
The study was approved by an appropriate ethics committee and has been performed in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki and its later amendments. Linkage of data occurred only for persons providing informed consent, and was in accordance with NSW legislation and policy.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Sara, G., Luo, L., Carr, V.J. et al. Comparing algorithms for deriving psychosis diagnoses from longitudinal administrative clinical records. Soc Psychiatry Psychiatr Epidemiol 49, 1729–1737 (2014). https://doi.org/10.1007/s00127-014-0881-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00127-014-0881-5