Concordance between self-report and medical records of preventive healthcare delivery among a sample of disadvantaged patients from four aboriginal community controlled health services
- 49 Downloads
This cross-sectional study aimed to explore, among a sample of patients attending one of four Aboriginal Health Services (ACCHSs), the degree of concordance between self-report and medical records for whether screening for key healthcare items had ever been undertaken, or had been undertaken within the recommended timeframe.
Across the four ACCHSs, a convenience sample of 109 patients was recruited. Patients completed a self-report computer survey assessing when they last had preventive care items undertaken at the service. ACCHS staff completed a medical record audit for matching items. The degree of concordance (i.e. the percentage of cases in which self-reports matched responses from the medical record) was calculated.
Concordance was relatively high for items including assessment of Body Mass Index and blood pressure, but was substantially lower for items including assessment of waist circumference, alcohol intake, physical activity, and diet.
Reliance on either patient self-report or medical record data for assessing the level of preventive care service delivery by ACCHSs requires caution. Efforts to improve documentation of some preventive care delivery in medical records are needed. These findings are likely to also apply to patients in other general practice settings.
KeywordsAboriginal Indigenous Self-report Medical record Agreement Concordance Preventive care
Aboriginal Community Controlled Health Service
Body Mass Index
Diabetes Care Plan
National Key Performance Indicator
For many disadvantaged and indigenous groups, such as the Aboriginal and Torres Strait Islander peoples of Australia, the prevalence of risk factors for chronic disease, such as high blood pressure and smoking, is disproportionately high . Reductions in such risks factors are needed in order for the health of disadvantaged and indigenous groups to achieve equality with the general population . Primary care has a key role to play in assisting patients to improve their health risk status . For Aboriginal Australians, a large proportion of the population receive primary and preventive care from an Aboriginal Community Controlled Health Service (ACCHS) . ACCHSs are similar to other primary health care services, but focus on providing wholistic and culturally appropriate services to the Aboriginal communities they serve .
Evaluation of the delivery of healthcare is essential for improving the quality of health services, as such assessments can identify gaps in care. The provision of preventive care can be evaluated through medical records, or patient or provider self-report. However, the best source of information for assessing preventive care delivery remains uncertain . Patient recall can be poor for a range of clinical interactions,  yet it remains an important indicator of the effective receipt of healthcare. Patients must be able to recall healthcare advice in order to adhere to the advice after a consultation . Self-report can sometimes be less costly and more convenient than extracting data from medical record review . Medical records can also have variable accuracy as a measure of healthcare delivery, with missing data a particular concern . Despite no single validated measure (patient recall, provider recall or medical record audit) having a high and consistent level of accuracy, these measures are often used as indicators of healthcare delivery or service performance for quality improvement and other purposes . Therefore, it is important to explore the level of concordance in such assessments across populations and health settings.
Relatively little attention has been directed towards assessing measures of healthcare delivery among highly vulnerable or disadvantaged groups, where rates of comorbidity are often high, [11, 12] and factors such as poor health literacy have the potential to influence both delivery and receipt of healthcare [11, 13]. It is particularly important to explore concordance between measures of care among patient groups who experience socioeconomic disadvantage and high levels of comorbidity, such as Indigenous Australians [12, 14]. For Indigenous Australians, the regular assessment of a number of key preventive health items within primary care is recommended. Risk factors including: smoking status, Body Mass Index (BMI), waist circumference, physical activity, alcohol consumption, diet, blood pressure, and cholesterol, should be assessed opportunistically,  and every 12 months as part of an annual comprehensive health assessment (Medicare item 715) .
The aims of this study were to describe, among patients attending one of four ACCHSs, the degree of concordance between self-report and medical records for: a) whether key preventive healthcare items had ever been undertaken; and b) whether key preventive healthcare items had been undertaken within the recommended timeframe (the last 12 months).
Study design and setting
Data was collected as part of a larger ‘Chronic Care Service Enhancement Project’, involving a collaboration between ACCHSs from across NSW, the Centre for Aboriginal Health (NSW Ministry of Health), and the University of Newcastle. The larger project involved the collaborative implementation and evaluation of strategies to improve preventive and chronic disease care provided by ACCHSs in NSW. One component of the project examined the quality of data recorded by ACCHSs about care provided to patients. The level of accuracy of ACCHS medical record data was assessed by comparing data on matching clinical items collected from different sources including patient self-report and clinic medical records. Four ACCHSs agreed to take part in the data quality assessment, and only the results of this component of the larger study are reported here. The four ACCHSs were located in major cities (1 service), inner regional (2 services) and remote areas (1 service) of NSW . The size of the services ranged from approximately 500 active patients (those who had visited the service at least once in the last 2 years), up to approximately 1700 active patients. All four ACCHSs used electronic Patient Information Management Systems for data recording and storage. Data collection was staggered across services, and occurred over several months (from two to six months) at each service during 2013 and 2014. Ethics approval for this study was obtained from the Aboriginal Health & Medical Research Council (AH&MRC) Ethics Committee (approval number 863/12) and the University of Newcastle (UoN) Human Research Ethics Committee (H-2012-0100).
Participants and procedure
The four participating ACCHSs were asked to collect data about matching clinical items from consenting patients and the patient’s medical record. Staff including reception or an Aboriginal Health Worker at participating ACCHSs invited a convenience sample of approximately 30 patients per site to complete a touchscreen computer survey. Services determined which member of staff was suitable and had time to undertake recruitment. Patients were aged ≥18 yrs. Staff were asked to include a range of ages, gender, and patients with and without chronic disease. Patients attending one of the four ACCHSs for a GP appointment were asked to complete the survey immediately after their appointment. Written informed consent for survey completion, and to conduct an audit of the patient’s medical record, was sought from all participants. The survey asked patients to self-report when they were last screened for key health risk factors including: BMI, waist circumference, blood pressure, smoking status, alcohol intake, physical activity and diet, and when they last had preventive healthcare items including an annual Health Assessment (HA; Medicare item 715) and a Diabetes Care Plan (DCP; for patients with diabetes) undertaken by a healthcare provider at the service. A staff member undertook a manual search of consenting patients’ medical records and completed a standardised data sheet containing items matching those collected by the patient survey. For each item, staff were asked to indicate whether it had ever been recorded since the patient first attended the service (yes/no) and if so, the date of last assessment. No strict criteria were provided to determine whether a HA or DCP had been completed, and this may have been based on a completed template, or billed item number in the medical record. De-identified data was provided by each site to the research team.
The degree of concordance between self-report and medical records was explored separately for two comparisons: a) whether key preventive healthcare items had ever been undertaken; and b) whether key preventive healthcare items had been undertaken within the recommended timeframe (the last 12 months).
For comparison a) whether an item had ever been undertaken, self-report variables were recoded to a binomial response (yes/no). All self-report response options indicating that the healthcare item had been undertaken (response options “≤12 m ago” or “> 12 m ago”) were coded as “yes”. All other responses, including “not sure”, “never”, and any missing responses, were coded as “no”. The medical record audit included a “yes” or “no” response for whether each item had been undertaken. Any blank/missing records in the medical record audit were coded as “no”.
For comparison b) whether an item had been undertaken within the recommended timeframe (the last 12 months), self-report variables were recoded to a binomial response. A response indicating that the healthcare item had been undertaken within the last 12 months (response option “≤12 m ago”) was coded as “yes”. All other responses were coded as “no” (including “> 12 m ago”, “never”, “not sure” and any missing values). Medical record dates for when an item was last undertaken were compared against the date of survey completion. Items that had been recorded within 12 months were coded as “yes”. Items which had been recorded more than 12 months before the date of survey completion were coded as “no”. Items that were not recorded in the medical record, or where the date was missing, were also coded as “no”. Twelve months was selected as it represents the annual screening interval recommended for the healthcare items included in the survey .
As there is no accepted gold standard criterion for evaluating the accuracy of healthcare performance measures,  the level of agreement between self-reported healthcare receipt and medical records was evaluated as concordance, defined as the percentage of cases in which self-reports matched responses from the medical record . A sample size of 100 allowed the outcomes to be assessed with a 10% level of precision, assuming a prevalence rate of 50% and 95% level of confidence. Participants who self-reported that they had never smoked were not asked when their smoking status had been assessed at the medical service. Therefore only current and ex-smokers were included in the analysis of concordance about smoking status assessment. Similarly, only participants who self-reported having diabetes were asked about whether and how long ago they received a DCP and were included in the concordance analysis for a DCP.
Sample self-reported number of visits to the service in the last 12 months and prevalence of chronic disease
Prevalence n (%)
Number of visits to the service in the last 12 months
10 or more
Self-reported chronic disease
Aim a: To describe the degree of concordance between self-report and medical records for whether healthcare items had ever been undertaken
The degree of concordance between self-report and medical records for whether BMI, blood pressure and smoking status had ever been screened was relatively high (> 80%). The degree of concordance between the two data sources varied from between 60 and 80% for whether patients had ever had their waist circumference or diet assessed, or had completed a HA or DCP. Concordance was < 60% for whether patients had ever had their alcohol intake or physical activity levels assessed.
Aim b: To describe the degree of concordance between self-report and medical records for whether healthcare items had been undertaken within the last 12 months
The degree of concordance between data sources for whether preventive care items had been undertaken within the last 12 months was lower than for whether items had ever been undertaken (Aim a) across all healthcare items, except for the receipt of a DCP. Concordance between the two data sources for whether items had been undertaken within the last 12 months was above 80% for BMI and blood pressure, but was < 60% for screening of waist circumference, alcohol intake, physical activity, diet, and for completion of an annual HA.
This study is the first to compare self-report and medical record data for the provision of preventive healthcare among a sample of ACCHS patients attending one of four services in NSW. The degree of concordance between patient self-report and medical records was high for some items, including BMI and blood pressure, but substantially lower for other items such as alcohol intake, physical activity and diet assessment. The reliability of the two data sources appears to differ across healthcare items, and it is not clear which data source should be considered to be more accurate.
Preventive care items where medical record appears to be more accurate
For items where the medical record indicated that a higher proportion of the sample had been assessed compared to self-report, it is probable that the medical record is the more accurate data source, given it is highly unlikely that these items would have been fabricated in the medical record . These included the assessment of waist circumference, blood pressure, smoking status, and alcohol intake.
Preventive care items where self-report appears to be more accurate
Conversely, for items where the self-reported prevalence of assessment was higher than according to medical records, it is likely that patient self-report is the more accurate data source. In these cases, the discrepancy may be due to a failure of the healthcare provider to record or update items in the patient’s medical record . These items included assessment of physical activity and diet, and completion of an annual HA and DCP. However, for the latter two items, the higher prevalence according to self-report could be due to patients not being clear about what these specific items consist of, and thus not distinguishing these from more general care or general advice about health/ diabetes .
Reasons for discrepancies between self-report and medical records
The discrepancies between self-report and medical records are likely to be due to either errors in patient recall or problems with medical record data recording or extraction. The high degree of concordance between assessment of risk factors including BMI and blood pressure is likely due to high rates of documentation of these activities in medical records, together with good patient recall. Activities such as having height or weight measured for BMI assessment, or use of a blood pressure cuff to assess blood pressure, are distinct procedures, for which patient self-report is likely to accurately reflect clinical practice .
Lower concordance for physical activity and diet screening appears to be due to a failure to document these activities in the medical record, with substantially more patients reporting that these had been assessed than compared to their medical record. This could be the result of a lower perceived priority of these health risks by healthcare providers, and subsequent failure to record these, or because of difficulties associated with the ability of clinical information systems to record such information. The poor concordance for smoking status and alcohol intake, where patients were less likely to report these as having been assessed than compared to documentation in their medical record, could further be due to the sensitive nature of these health behaviours . Current smokers, or those drinking alcohol at higher than recommended levels, may be prone to disregard or underreport the evaluation of these behaviours. In addition, healthcare provider sensitivity about raising such issues [22, 23] may result in poorer patient recall. Given the high underlying rates of smoking and harmful alcohol use among many Aboriginal Australians,  this is a particular concern, as patients are subsequently also unlikely to recall and follow advice provided about smoking or reducing alcohol intake.
Given the unique setting of this study, there are limited data for direct comparison. In a study which compared patient and General Practitioner agreement about screening and risk status among 141 Aboriginal patients attending one ACCHS, there was reasonable agreement regarding smoking status, but poor agreement for alcohol consumption risk status, physical activity risk status, and for whether patients had been screened within the recommended timeframes for diabetes and blood pressure . In two large USA studies, patient self-report and medical record data agreement for the receipt of risk factor counselling were broadly similar to the levels of agreement reported in the current study for physical activity and diet, but lower for smoking status. Levels of agreement ranged from 53% -57%  for receipt of counselling on physical activity (compared to 49% concordance in the current study), 54% - 60%  for diet (compared to 62% concordance in the current study), while agreement for the receipt of smoking cessation counselling was only 41%  (compared to 81% concordance in the current study). However, such findings are not directly comparable as the current study examined assessment of risk factors rather than the receipt of behavioural counselling. In the USA studies, the authors suggest inconsistent physician recording of counselling interventions is due to time pressures,  the result of such care not being financially reimbursed, [6, 21] and because of physician’s undervaluing the ‘talking’ aspects of medical care . These factors are also likely to apply to assessment of healthcare items within the ACCHS setting, and indeed to general practice in Australia more broadly.
This study has a number of important limitations including the small sample size. The convenience nature of the sample may also limit the generalisability of the study findings to other ACCHSs. The sample included regular/frequently attending patients, who will have more opportunities for receiving preventive care than less regular patients, as well as greater potential for confusion in recalling timeframes- particularly for verbally delivered care, such as the assessment of diet or physical activity. The amount of missing data for some items was quite large, in particular for medical records of physical activity and diet assessment (see Additional file 1: Tables S1 and S2). As missing records were recoded as “no” for having ever/been assessed in the last 12 months, the amount of missing data will have affected the degree of concordance for these items. Unfortunately, BMI and waist circumference assessment data from one site were also lost and not able to be included in analysis, and comparisons of smoking status assessment and completion of a DCP were drawn from smaller samples of current or ex-smokers (n = 79), and those with self-reported diabetes (n = 37) respectively. Finally, self-report is subject to biases such as recall and social desirability, [25, 26, 27] as well as telescoping, where events are recalled as having occurred more recently than in reality . The medical record audits were not subject to any cross-checking or inter-rater reliability measures and staff experience and training may have varied across sites.
Implications for practice
Caution is required when relying on either self-report or medical record data alone to determine levels of preventive care service delivery in the ACCHS setting, as in other general practice settings . This is particularly the case for whether items have been undertaken within recommended timeframes. Service providers and researchers need to be aware of the potential degree of error associated with either data source. While self-report and medical records appear to be relatively accurate for determining whether patient BMI, blood pressure or smoking status screening have been undertaken within the recommended timeframe, there is likely to be substantial under- or over-reporting of delivery of other preventive care items including alcohol intake, physical activity, diet, and completion of an annual HA or DCP, according to the data source used. While there are some unique considerations within the ACCHS setting, for example difficulties in accurately assessing alcohol intake which tends to be characterised by heavy but infrequent, episodic drinking,  the findings of this study are also likely to be applicable to other Australian general practice settings more broadly.
In order to obtain a more accurate reflection of the delivery of preventive care, multiple sources of data may need to be used [18, 21]. This is of concern given that many ACCHSs rely on medical record data extraction for reporting of national Key Performance Indicators (nKPIs), which are used nationally to monitor health outcomes for Aboriginal and Torres Strait Islander Australians . nKPIs currently include smoking status, alcohol consumption, BMI and annual HA . A number of quality improvement initiatives in Australian Aboriginal primary care have recognised the need to improve data collection processes, information systems, and documentation of delivered services [20, 31, 32]. A systematic review examining the quality of electronic patient records in primary care noted that, overall, lifestyle data were generally of lower quality than prescribing data . Results of the current study reiterate the need to improve medical record documentation of preventive care, in particular for physical activity and diet. This might include staff training and the development of more user-friendly software, and/or automated system prompts to complete blank fields. Specific strategies to help improve both delivery and patient recall of smoking and alcohol risk assessment, and the receipt of behavioural counselling for these health risks, are also a high priority.
Service providers and researchers need to be aware of the potential degree of error associated with relying on either self-report or medical record data to determine levels of preventive care service delivery in the ACCHS and other general practice settings. Self-report and medical records appear to be relatively accurate for determining whether patient BMI, blood pressure or smoking status screening have been undertaken within the recommended timeframe, however there is likely to be substantial under- or over-reporting of delivery of other preventive care items including alcohol intake, physical activity, diet, and completion of an annual HA or DCP, according to the data source used. Efforts are needed to improve medical record documentation of preventive care delivery, in particular for physical activity and diet. Similarly, strategies to improve patient recall of preventive care advice regarding smoking and alcohol intake are required.
This evaluation was completed as part of the Chronic Care Service Enhancements Program funded by the Centre for Aboriginal Health, NSW Health. This project was conceptualised, developed and implemented in partnership with participating Aboriginal Community Controlled Health Services, and the Centre for Aboriginal Health, with evaluation by the University of Newcastle. We wish to thank all those involved for their contribution to the project.
This research was funded by the Centre for Aboriginal Health (NSW Department of Health). Infrastructure support was received from the Hunter Medical Research Institute. A/Prof Christine Paul was supported by a HMRI Research Fellowship and a National Health and Medical Research Council Career Development Fellowship (1061335). These funding bodies played no role in the analysis and interpretation of data; in the writing of the manuscript; or in the decision to submit the article for publication.
Availability of data and materials
Data are available from the corresponding author on request.
NN, CP, JW, JS, KW and SW made substantial contributions to the overall conception and design of the study. NN, CP, JW and JS contributed to data collection templates and procedures, and KW and SW undertook data acquisition. NN, CP and JW undertook data analysis, and all authors contributed to interpretation of the data. All authors were involved in drafting or revising the manuscript. All authors have read and approved the submitted version of the manuscript and agree to be accountable for the accuracy and integrity of the work.
Ethics approval and consent to participate
All study participants gave written informed consent to participate in this study. Ethics approval for this study was obtained from the Aboriginal Health & Medical Research Council (AH&MRC) Ethics Committee (approval number 863/12) and the University of Newcastle (UoN) Human Research Ethics Committee (H-2012-0100).
Consent for publication
The authors have no competing interests to declare.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
- 3.Harris M. The role of primary health care in preventing the onset of chronic disease, with a particular focus on the lifestyle risk factors of obesity, tobacco and alcohol. Canberra: National Preventative Health Taskforce. 2008;1:21.Google Scholar
- 5.Network NRD. Aboriginal community controlled health services: NSW RDN; 2019 Available from: https://www.nswrdn.com.au/site/acchs.
- 14.Marmot M. Social determinants and the health of indigenous Australians. Med J Aust. 2011;194(10):512–3.Google Scholar
- 15.National Aboriginal Community Controlled Health Organisation, Royal Australian College of General Practice. National guide to a preventive health assessment for Aboriginal and Torres Strait Islander people. 2nd edn. South Melbourne: RACGP; 2012.Google Scholar
- 16.Department of Health. Medicare Health Assessment for Aboriginal and Torres Strait Islander People (MBS Item 715) Canberra: Australian Government; 2016 . Available from: http://www.health.gov.au/internet/main/publishing.nsf/Content/mbsprimarycare_ATSI_MBSitem715. [cited 12 Jun 2018]
- 17.Department of Health. Doctor Connect- Locator Canberra: Department of Health; 2017. Available from: http://www.doctorconnect.gov.au/locator. [cited 12 Jun 2018]
- 18.Beckles GL, Williamson DF, Brown AF, Gregg EW, Karter AJ, Kim C, et al. Agreement between self-reports and medical records was only fair in a cross-sectional study of performance of annual eye examinations among adults with diabetes in managed care. Med Care. 2007;45(9):876–83.CrossRefGoogle Scholar
- 19.Brown JB, Adams ME. Patients as reliable reporters of medical care process: recall of ambulatory encounter events. Med Care. 1992:400–11.Google Scholar
- 27.Schröder M, editor. Retrospective Data Collection in the Survey of Health, Ageing and Retirement in Europe. SHARELIFE Methodology Mannheim: Mannheim Research Institute for the Economics of Ageing; 2011.Google Scholar
- 30.Australian Institute of Health and Welfare. National Key Performance Indicators for Aboriginal and Torres Strait Islander primary health care. Results from June 2016. National key performance indicators for Aboriginal and Torres Strait Islander primary health care series no. 4. Cat. no. IHW 177. Canberra: AIHW; 2017.Google Scholar
- 32.DiGiacomo M, Davidson P, Taylor K, Smith J, Dimer L, Ali M, Wood M, Leahy T, Thompson S. Health information system linkage and coordination are critical for increasing access to secondary prevention in aboriginal health: a qualitative study. Qual Prim Care. 2010;18:17–26.Google Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.