Background

In Australia, primary health care (PHC) covers health care unrelated to a hospital visit including health promotion, prevention, early intervention, treatment of some acute conditions, and management of chronic conditions. Through a fee-for service arrangement administered by the Australian Federal government, primary health care services are delivered by community health centres, Aboriginal community-managed medical services and allied health clinics, but the bulk of PHC is provided by general practices, the setting of focus in this study. These practices have a mix of staffing including general practitioners (GPs), practice nurses (PNs), practice managers (PMs), allied health professionals and administrative staff/receptionists.

Workforce issues in general practice, mainly the recruitment and retention of GPs and PNs, have been concerning across many parts of Australia [1,2,3,4,5] and other developed nations [6,7,8], particularly in rural and remote practices where there is added complexity [1, 7,8,9]. These workforce issues have implications for access and quality of care provided to residents in these regions and ultimately to the ongoing management of their health. In Australia, about 28% of the population live in rural and remote areas [10]. In comparison to their urban counterparts, these Australians have poorer access to, and use of, primary health care services including general practice, and also have poorer health outcomes, higher rates of injury, hospitalisations, and deaths [10].

A range of policy, legislative and industry initiatives have been utilised to address these workforce issues such as the Stronger Rural Health Strategy and the National Strategic Framework for Rural and Remote Health [1, 5, 11, 12]. Solutions have focussed primarily on offering financial and visa incentives to entice both local and overseas-trained general practice staff to relocate to rural and remote locations. Research indicates that these initiatives have been less successful than those focused on training local personnel, improving the productivity and job satisfaction of practice staff, and improving efficiencies to optimise time spent with patients [13]. One such strategy is to extend the scope of practice for existing health professionals and other general practice staff [3, 14, 15]. These initiatives are rarely evaluated [2, 5, 6]. Evaluations that measure meaningful impacts of interventions on key implementers and beneficiaries, and the cost and expected returns from investment, are vital for ensuring resources are not wasted on initiatives that are ineffective [2, 5, 6, 12, 15,16,17].

In Australia, primary health networks (PHNs) were established by the Federal Government to increase efficiency and effectiveness of medical services and improve coordination of care to ensure patients receive the right care, in the right place, at the right time [18]. In 2018, the Hunter New England and Central Coast Primary Health Network (HNECC PHN hereafter also referred to as PHN), began an initiative—subsidising administrative staff from local general practices and Aboriginal Medical Services (AMS) to undertake a Certificate IV in Medical Practice Assisting (hereafter referred to as the MPA Program) offered by UNE Partnerships, a registered training organisation associated with the University of New England (UNE). The aim of the MPA Program was to upskill administrative staff in general practices (whose main duties included tasks such as answering the phone, booking appointments and ordering supplies) to undertake selected clinical tasks and higher order administrative tasks (see list of these in Additional file 4). This would free up PNs and GPs to work at the top of their scope, thereby increasing productivity and throughput within general practices [16], improving patient care and experience, and contributing to workforce retention in primary care [19].

The MPA Program originated in 2003 at GP Partners, a division of general practice in Queensland [20]. In 2007 the Certificate IV in Medical Practice Assisting was included in the national health training package and endorsed by all Australian states and territories [21]. However, a formal evaluation of the impact from the MPA program has never been conducted.

In 2020, the Embedded Economist program funded by New South Wales Regional Health Partners and the Australian Medical Research Future Fund was implemented at HNECC PHN. The program enabled PHN staff to work with health economists and impact specialists from the Hunter Medical Research Institute to undertake an impact evaluation of the MPA Program, as it was being applied by the PHN. It was anticipated that the results would inform ongoing PHN investment in the MPA program and inform general practices and their staff considering this intervention in the future, not just in Australia, but in other developed nations facing similar workforce issues.

The primary aims of the impact evaluation were to assess the impact of the MPA Program on participating general practices and their staff, capture the economic cost of the MPA Program to the PHN, practices and students, and determine whether the monetisable consequencesFootnote 1 of the investment represented a good return on the investment. A secondary aim was to apply the Framework to Assess the Impact of Translational Health Research (FAIT) methodology to a non-research program of work and assess its suitability in effectively evaluating the impacts of a health services led workforce development program.

Methods

Setting

The setting for the impact assessment was the HNECC PHN and general practices across the Hunter, New England, and Central Coast regions in the Australian state of New South Wales who had participated in the MPA Program in 2018 and 2019. The HNECC regions cover an area of approximately 130 000 square kilometres and are home to 1.2 million residents, a quarter of whom live in rural, regional, and remote locations. The combined area has a total of 410 general practices and nine Aboriginal Medical Services; 65 of them participated in the MPA Program in 2018 and 2019 [22].

Evaluation design

FAIT was developed specifically to improve research translation and to optimise and assess the impact from research investments [23]. FAIT has been demonstrated to be an effective impact assessment tool in a range of Health and Medical Research (HMR) projects in Australia and internationally [24, 25]. It combines three validated methods of impact assessment (Payback, Economic analysis, and Narratives) to present a multidimensional, comprehensive approach to assessing the impact of research projects and programs. A detailed description of the FAIT method can be found in Additional file 1. This study represents the first application of FAIT to a non-research program of work.

Program logic model and payback

A detailed program logic model (PLM) was developed to map the pathway between the need for the MPA Program and the eventual impact of the intervention (see Fig. 1). Within FAIT, the PLM underpins all three methods.

Fig. 1
figure 1

Impact logic model and scorecard

Full size image

Next, a modification of the Payback Framework, first developed by Buxton and Hanney [26], was applied. The Payback Framework captures impact within broad domains of benefit. For the MPA project the relevant domains were selected by the evaluators (SR, RL, AS & SD) in collaboration with PHN staff (AT, NI) to reflect the anticipated benefits of the MPA program from the various perspectives. The selected domains were: knowledge advancement, capacity building, practice change, community benefits and economic impacts. Existing metrics were selected or specifically developed to capture the intended impacts of the MPA Program such as skills development, job satisfaction and workforce productivity.

Economic evaluation

A cost-consequence analysis (CCA) was used to compare costs of the MPA program to its potential consequences for general practices and for MPA students. A CCA does not limit itself to standard metrics like cost–benefit ratios. Rather it lays out the costs and consequences for decision makers to make their own subjective judgements on program cost-efficacy and whether the investment choice represents value for money.

Costs

All resources used for the MPA program were reported in monetary terms (economic costs). This included actual cash costs and the ‘opportunity costs’ of directing resources to the MPA program rather than another activity. A bottom-up approach was used to capture all resources from the perspectives of HNECC PHN, participating practices and MPA students. All resources were identified, valued, and aggregated separately as detailed in Additional file 2. Costs are presented in monetary units so that the value of different resources can be aggregated and compared. Unit costs and cost assumptions are captured in Additional files 2 and 3, and in Table 3.

Consequences

Consequences were focussed on realised and potential benefits for general practices and MPA graduates. These would also indirectly benefit the PHN whose main aim and function is to improve the delivery and quality of primary care, including general practice, within its jurisdiction. The consequences included were limited to those that were monetisable. Consequences not readily monetisable (e.g., job satisfaction and increased confidence in performing clinical tasks) are presented in their natural units and discussed within the Payback metrics (Table 2).

General practices

The main monetisable consequences were efficiencies gained through utilisation of MPA graduates in their upskilled role for half a day, every day; and opportunities for the practice to gain from increases in billable services. The administrative duties performed by the MPA graduate prior to the completion of their course (i.e. undertaking general reception and administrative duties) would be picked up by the Practice Manager or a casual administration assistant. Financial and administrative data to inform on actual level of billable services was unavailable, so modelling was informed by survey data and supplemented with expert input from PHN staff.

MPA graduates

Monetizable consequences for graduates were their potential higher remuneration following course completion. Additional wages were projected over a full year and based on survey responses and an aspirational goal of MPAs being paid $3.00 extra per hour (mid-point between the receptionist and enrolled nurse wage scale). Sensitivity analyses was used to provide the range of potential returns to both general practices and MPA graduates.

Narrative

The narrative was built from the program logic model and impacts and responses from study participants, all of whom provided consent for their qualitative responses to be included in the analysis. Comments were grouped by key themes and captured the various perspectives around impact. The results for the application of FAIT to the MPA Program are summarised and presented in a scorecard format by each method.

Data collection

Data collection methods for each component of FAIT are summarised here:

Payback: Online surveys and interviews with participants and administrative records from the HNECC PHN.

Economics: Online surveys and interviews with participants, administrative records from the HNECC PHN, and secondary data and expert input provided by the MPA Coordinator and Primary Care Improvement Officer from the PHN.

Narrative: Online surveys and interviews with participants.

Participants

Participants represented five groups of general practice staff: (1) practice managers; (2) current MPA students; (3) PN’s; (4) GP’s and (5) MPA graduates. Eligible general practices were those that were listed as having at least one MPA student enrolled between January 2018 and December 2019. Eligible practice managers, PNs and GPs were those who were working at those practices and eligible MPA students were those who were currently enrolled in the MPA Program, at the time of the survey. Eligible MPA graduates were those who had graduated by June 2020. All eligible practices were invited to participate. Of the 62 eligible practices, 36 had one current MPA student (58%) and 26 (42%) had a returned graduate. Of these 26 practices, 4 had their MPA graduate leave their practice by the time of the survey, leaving only 22 eligible MPA graduates. To supplement general practice staff views, the MPA Coordinator and Primary Care Improvement Officer from the PHN, and the Program Manager for Health Programs at UNE Partnerships were consulted due to their experience with the program and capacity to provide contextual clarity.

Surveys and interviews

Online surveys were programmed and deployed using REDCap (Research Electronic Data Capture) a secure web application for building and managing online surveys and databases [28]. Five surveys were developed, one for each participant group. All groups, except for MPA graduates, self-administered the survey. Graduate surveys were administered via telephone interview with a member of the HMRI team who entered the graduate’s answers for them. This was to enable the capture of more detailed responses with regards to the impact of the program on graduates personally. A pragmatic pilot involving two practices from the cohort was undertaken. Suggested changes were incorporated prior to deployment.

Results

A total of 57 general practice staff completed surveys (see Table 1 for details of response rates). The MPA Coordinator and Primary Care Improvement Officer from the HNECC PHN provided supplementary information. Results are presented by the three FAIT methods.

Table 1 MPA survey: participants by role
Full size table

Payback

Table 2 reports on the impacts of the MPA Program by way of impact metrics, grouped within five domains of impact—knowledge advancement, capacity building, practice change, community benefit and economic benefit. Key impacts included the proportion of MPA graduates with increased confidence to perform clinical tasks, the proportions performing various clinical tasks post-graduation, their increase in job satisfaction and the increased throughput of patients through general practices.

Table 2 Payback metrics
Full size table

Cost-consequence analysis

Costs

In 2018, when 44 students commenced and received scholarships, the total annual economic cost to the PHN for the MPA Program was $126 718, the largest item being $94 600 for scholarships (see Table 3 (1)). General practice costs (see Table 3 (2)) were separated by (a) costs during the candidature and (b) costs post candidature. Total economic cost during candidature is estimated at $22 687. During the candidature period, main costs were labour during on-the job training time, (MPA student who is backfilled by a practice manager and/or a casual administrative assistant) ($15 401) and student study leave ($3271). The total economic costs for 12 months post-graduation was estimated at $47 070; and the total economic cost for both periods was $69 756 per MPA graduate, including increased wages for the graduate with on-costs. For students, total economic costs for the 18-month course per individual was estimated at $3729 (Table 3 (3)) including personal computers, stationery, and travel to face-to-face workshops.

Table 3 Cost and monetary consequences of the MPA Program
Full size table

Potential revenues

Analysis also showed potential revenue increases for General Practices and students/graduates. However, there were no monetisable consequences for the PHN investment into the MPA Program given the PHN was never intended to be a monetary beneficiary of the MPA program. (Table 4 (1)).

Table 4 Monetisable consequences for General Practices and Medical Practice Assistants, post-graduation
Full size table

For general practices with graduated MPAs, potential extra revenues were recognised where the graduate could: (a) conduct extra revenue earning tasks as qualified by the course; or (b) replace practice nurses on lower-level tasks, freeing the nurses for higher revenue services such as creating GP management plans. Scenario analysis found potential annual gross extra revenue of $136 474 to $152 672 or a mid-point of $144 673 per practice (Table 5, Modelled Scenario A). On this basis, general practices could recoup their MPA investments within one year. Potential increases in remuneration exist for MPA graduates. If able to negotiate a gross pay increase of between $1.50 and $3.00 per hour, MPA graduates will have a gross income increase of between $2974 and $5948 per annum. Given their MPA related economic costs are $3729, prospective students can expect to recoup their full investment within 2 years of graduation. Economic costs and potential extra revenues are described in detail in the Appendices.

Table 5 Cost and consequences for the PHN, General Practices and MPA graduates
Full size table

Narrative

Table 6 presents the narrative of the MPA Program which summarises the pathway to impact from need for the MPA program through to the impacts on general practices and MPA graduates, as depicted in the Program Logic Model (Fig. 1). The narrative provides the context against which the results from the Payback and cost-consequence analysis can be interpreted.

Table 6 Narrative of the MPA Program
Full size table

Suitability of FAIT

With regards to the novel application of FAIT to a health services led program, the FAIT method was able to be applied to the MPA Program without any specific customisation over and above what would be expected from its application to a research program. Although the PLM was applied retrospectively, it proved to be useful for: (i) documenting the pathway between the MPA Program and its impact on practices, their staff, MPA students and graduates; (ii) identifying metrics that could evidence impact from the MPA Program; and (iii) raising awareness within the PHN team as to other benefits of the MPA Program that had not been previously considered such as a reduction in stress for PNs or certain measures of increased productivity such as additional patients seen in the practice per day. The economic analysis gave transparency to the economic cost involved in the delivery of the MPA Program from the perspective of the PHN, participating practices and MPA students. It was also able to project the monetary value of some of the potential consequences of the investment. The narrative articulated the pathway from the need for the MPA Program (a solution to workforce issues within rural general practice) to the impact; and expressed benefits of the MPA Program that could not be expressed in quantitative terms such as “improved relationships between MPA graduates and their colleagues and patients” and “confidence to undertake further tertiary studies- primarily in health care”.

Discussion

Impact of MPA program

The impact assessment found the MPA Program to be a feasible, economically viable strategy for rural general practices in Australia to address workforce shortages and retention issues. When appropriately managed, the program increased potential for revenue generation and improved patient care, and increased job satisfaction for MPA graduates, PNs and GPs, an outcome linked to improved workforce retention [1, 9, 29]. The MPA Program provided participating practices with a multi-skilled, flexible resource who could be used in an ad-hoc manner to fill in for PNs, where appropriate, or in a more systematic way by backfilling PNs on leave, undertaking scheduled clinical tasks that attract additional revenue for the practice, or providing more hands-on care to patients. MPA graduates were also able to free up PNs from non-revenue generating tasks to undertake more complex patient-oriented activities like health assessments and care plans, which provide more comprehensive care to patients in at risk groups and with complex needs, while generating revenue for the practice. If utilised to capacity, a practice, and its MPA graduates whose additional training was appropriately recognised with a salary increase, could recoup their respective investments in 12–24 months and practices could go on to increase their profits and clinic throughputs over subsequent years.

Qualitative findings suggested other benefits including reduced workload, reduced stress, and reduced pressure on PNs in the practice. The MPA graduates also benefited through their enhanced market value (through their upskilling and increased confidence) and increased job satisfaction associated with undertaking more meaningful tasks. In addition to existing strategies like recruiting more overseas trained doctors, and training more local doctors [13, 30], the MPA Program value adds to such initiatives through utilising an existing, non-scarce medical administration workforce to allows PNs and GPs to work at the top of their scope, relieve workload pressure during peak periods and provides education pathways for administration staff, all key factors in improving regional and remote workforce retention and maximising cost efficiency and workforce capacity [13, 16, 17]. The MPA Program would be transferable to other countries with a fee-for-service arrangement in primary care and who face the same workforce issues that prohibit adequate servicing of rural and regional populations. It is particularly useful for busy practices, those with labour shortages, and large practices where there are greater economies of scale to be gained by having an MPA who can be utilised to their maximum capacity.

Other insights about the MPA program

The impact assessment also revealed a large variation in the utilisation and remuneration of MPAs, resulting in a variance in the anticipated benefits for participating practices. This suggests that greater consistency in the utilisation and remuneration of MPA graduates and more supported collaborative integration planning involving all practice staff and the PHN, could optimise the benefits of the program. In addition to being adequately renumerated for their additional skills, MPA graduates also suggested that greater recognition of the role, a dedicated award, and greater direction on how to manage this role within the practice team would improve the overall MPA experience. Support from the PHN through providing relevant good practice models and exempler case studies of succesful MPA graduate reintegration and utilisation would be of benefit. Additionally, the impact assessment highlighted the importance of embedding impact planning and evaluation processes upfront to minimise data collection burden and costs and optimise impact. Additional file 5 summarises recommendations for optimising the benefits and impacts from the MPA Program.

Strength of the study

The comprehensive collection of evidence and its analysis, as per the FAIT methods, was a key strength of the study. The co-development of the PLM allowed the project team to fully consider all components and perspectives of the MPA Program and develop appropriate metrics to measure its impact. The Payback metrics enabled a comprehensive assessment of the full range of possible impacts of the MPA Program. The CCA gave transparency to the investment from all three perspectives (PHN, general practices and MPA students) informing future implementation and scale-up of the initiative. The narrative brought to light the benefits of the MPA course from the perspective of the MPA graduates and their voices were given prominence.

The collaborative research approach provided capacity and capability benefits to PHN staff. PHN staff improved their confidence and skills in measuring and reporting impact in a multi-dimensional way. PHN staff also co-produced the research and have benefitted from the networking and learning opportunities with HMRI staff.

Limitations and implications

COVID-19 is likely to have impacted on interview response rates, thereby limiting the representativeness of the results. New mandated infection control measures and increased demand for consultations meant that general practices and AMSs were under additional pressure between November 2020 and February 2021, the period when the survey was open. While remedial measures were taken to increase response rates (e.g., extending survey deadline and offering incentives), they remained below an ideal level. It is also likely that practices who responded to the survey represented practices that had a more positive MPA experience than non-responding practices. This could have biased the results in favour of greater positive impact from the MPA program.

COVID-19 also impacted MPA students in the 2019 and 2020 cohorts increasing their workload and their course time, thus reducing the number of participating practices with returning MPA graduates. COVID-19 also limited MPA graduates’ abilities to fully utilise their skills given the high volume of phone consults during this period. The timing of this impact assessment did not provide sufficient lag to realise longer term benefits of the MPA Program such as reduction in workforce shortages.

Potential monetisable consequences used in the modelling were not exhaustive. According to survey results and expert input, MPAs can also increase gross revenue through paid quality improvement activities (e.g. undertaking Practice Incentive Program (PIP) quality improvement measures including height and weight measurements for all patients) which attract a government payment [27] that many practices currently forgo due to a lack of nurses’ time to complete such activities. These other potential consequences required a large number of assumptions to support the modelling, and consequently were not included.

The absence of Aboriginal and/or Torres Strait Islander MPA students, MPA graduates, or staff from Aboriginal Medical Services gives no visibility to the impact of the MPA Program on their operations and means that their perspectives are missing from this study. This limits the applicability of these findings to Australia’s First Nation peoples and the services controlled by their communities. Low numbers of Aboriginal MPA students in the course is thought to be partially due to a preference for attaining the Aboriginal Health Worker qualification rather than the MPA qualification and is an area worthy of further review and discussion. The absence of direct feedback from patients, consumers and caregivers is a further limitation that was unable to be addressed within the resources and timing of this study. Future evaluations of the MPA Program should consider the inclusion of patient perspectives and views to provide a more holistic evaluation.

Conclusions

The HNECC PHN initiative and resultant outcomes and impacts associated with regional general practice administrative staff undertaking the MPA Program have been significantly beneficial to a number of participating practices Critically, the impact assessement of the MPA program indicates there is room for improving utilisation of MPA graduates in the workplace and that the program has broader long term potential to mitigate the impacts of regional workforce shortages and low retention, improve primary practice efficiences and gross revenue and further grow and upskill the regional health workforce.