The use of a modified Delphi technique to develop a critical appraisal tool for clinical pharmacokinetic studies

Background Critical appraisal aids in assessing the quality of scientific literature, which is central to the practice of evidence-based medicine. Several tools and guidelines are available for critiquing and assessing the quality of specific study types. However, limited guidance exists for critical appraisal of clinical pharmacokinetic studies. Aim We aimed to achieve experts’ consensus regarding the quality markers for clinical pharmacokinetic studies in an attempt to develop a critical appraisal tool. Method Quality markers related to clinical pharmacokinetic studies, were derived from the published literature and categorized according to manuscript reporting domains (abstract, introduction/background, methodology, results, discussion, and conclusion). Questions that aid in appraising pharmacokinetic studies were formulated from these quality markers. Experts were involved in a modified Delphi process to achieve a consensus regarding the formulated questions. The proposed tool was pilot tested on 30 recently published clinical pharmacokinetic studies. Inter-observer agreement was measured to determine the reliability of the included items. Results Twenty-five experts consented to participate. Three rounds of a modified Delphi survey were required to generate a consensus for a 21-item tool aimed at appraising the quality of clinical pharmacokinetic studies. When applied to 30 recently published clinical pharmacokinetic studies, most items scored fair to moderate levels of agreement (61.90–95.24%). Conclusion The clinical pharmacokinetic critical appraisal tool (CACPK) developed in this study consisted of 21 items aimed at helping an end-user to determine the quality of a pharmacokinetic study. Further studies are warranted to reaffirm the validity and reliability of the CACPK tool. Supplementary Information The online version contains supplementary material available at 10.1007/s11096-022-01390-y.

tools. A study may meet the required aspects for reporting, but may fail to meet the expected quality standards. Furthermore, some reporting checklists may not assess important dimensions of quality related to the study design, conduct, analysis, clinical relevance, and result validity [9]. Additionally, currently published critical appraisal tools are not highly specific to determine the methodological quality and validity of clinical pharmacokinetic studies. Most of the available tools lack specific items that help in analyzing the published articles in depth [10].
To our knowledge, two studies were conducted to assess the quality of reporting clinical pharmacokinetic studies [11,12]. The first study was a systematic review, which evaluated the quality of reporting of pharmacokinetic studies of antibiotics in patients with sepsis receiving continuous renal replacement therapy [11]. In this systematic review, the researchers found that none of the identified articles during their systematic search reported the full set of parameters that help end-users interpret the reported results. Furthermore, 20% of the published pharmacokinetic trials did not contain the fundamental pharmacokinetic parameters [11]. Consequently, reporting guidelines for clinical pharmacokinetic studies (The ClinPK Statement) were issued to guide investigators in reporting clinical pharmacokinetic studies. The developers designed a yes/no checklist that was composed of 24 items to guide researchers in reporting the minimum required information in clinical pharmacokinetic studies [12]. While offering a valuable guideline for reporting the findings of clinical pharmacokinetic studies, The ClinPK Statement has its drawbacks and did not cover some important dimensions of quality. Another tool, Grading and Assessment of Pharmacokinetic-Pharmacodynamic Studies (GAPPS), was recently published to assess the strength of evidence extracted from pediatric PKPD antibiotic studies [13].
A critical appraisal tool with broad application that aids clinicians in appraising and assessing the quality of published clinical pharmacokinetic studies does not exist. Developing an understanding of quality markers for clinical pharmacokinetic studies that clinicians consider important would aid in the creation of a tool aimed at assessing the quality of published clinical pharmacokinetic studies.

Aim
We aimed to achieve an expert consensus regarding the quality markers of clinical pharmacokinetic studies with an attempt to develop a critical appraisal tool through a modified Delphi technique.
• Developing a critical appraisal tool to evaluate clinical pharmacokinetic studies will help end-users to apply evidence-based medicine to clinical practice and future research. • The clinical pharmacokinetic critical appraisal tool (CACPK) developed in this study consisted of 21 items aimed at helping end-users to appropriately assess the quality of pharmacokinetic studies. • The currently developed CACPK tool shall undergo periodic improvement in order to keep pace with changes in the literature and to maintain the practical benefits to the field of clinical pharmacokinetics.

Introduction
The practice of evidence-based medicine (EBM) integrates individual clinician's expertise and experience with the best available clinical evidence from empirical research [1,2]. Central to the practice of EBM is the critical appraisal of published literature, in which the relevance, quality, and the trustworthiness of a published study is systematically assessed [2]. The critical appraisal process helps in assessing the validity, reliability, and quality of the published scientific knowledge. Therefore, several generic and specific critical appraisal tools for varying study designs were developed. Design-specific critical appraisal tools contain items that critique the methodological quality of the study design [3]. Generic critical appraisal tools help in appraising quantitative and qualitative studies in general [4]. Accordingly, the critical appraisal process aids in assessing the quality of the study results, and how they are interpreted and applied in health policymaking, therapeutic decision, and clinical practice [5]. Therefore, selecting the most appropriate critical appraisal tool is essential for the application of evidencebased practice [6].
The application of pharmacokinetic principles in clinical settings is considered an integral part of pharmaceutical care provision by the pharmacist [7]. Clinicians aim to improve a patient's response to the drug dosage regimen and to minimize toxicity by designing an individualized dosage regimen through the application of clinical pharmacokinetic concepts [8]. This process is aided by literature review of available evidence extracted from clinical pharmacokinetic studies, which have to meet high standards of quality to ensure the delivery of safe and effective therapeutic regimen [9].
There are many developed and published critical appraisal tools and reporting quality checklists to guide researchers to assess the quality of the published research. Reporting checklists, however, are not synonymous to critical appraisal

Modified Delphi survey
A modified Delphi process was utilized via an online survey platform (SurveyMonkey®) in order to build consensus amongst experts regarding each item considered for inclusion in the tool. An inventory of quality markers to evaluate pharmacokinetic studies [14] was used to identify potential items through a systematic approach. These items were revised scientifically and linguistically, reworded, and reduced after several iterations with the research team members prior to being sent to the study participants. A large number of participants could be included in the rounds through using modified Delphi design because of the lack of geographical restrictions. Participants could express their opinions freely, as their answers were anonymous. Therefore, a dominance that might occur during face-to-face interactions and any bias introduced by moderators were avoided. Panelists in this study were recruited from different countries and practice settings like academia, industry, clinical practice, and regulatory authorities. Thus, each participant could share their perspective in the field of clinical pharmacokinetics and help in generating new ideas that helped in broadening the knowledge base of other participants.

Modified Delphi process
Round 1 of this modified Delphi process utilized a questionnaire comprising potential identified candidate items. Experts were presented with individual candidate items and were requested to state their level of agreement that the item should be included using a 5-point Likert scale (1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = strongly agree). For each candidate item, participants were given the option to provide free-text comments to support their rating or to suggest changes to the item. Upon completion of the potential items generation, participants were given the option to nominate additional items for consideration.
Consensus was set a priori at 75% agreement with any one of the available outcomes (inclusion, exclusion, modification). Inclusion was met if 75% or more of the participants rated an item as agree or strongly agree. Exclusion was met if 75% or more of participants rated an item as disagree or strongly disagree. Item modification was considered if neither inclusion nor exclusion were met after which the item was modified based on participants' feedback. If after a modification, an item still did not meet the requirements for inclusion or exclusion and the percentage change in consensus was less than 15%, it was excluded from the final tool to be used for critical appraisal of clinical pharmacokinetic studies (CACPK). For each additional Delphi round following round 1, participants were given the

Ethics approval
This study was approved by the Qatar University Institutional Review Board: QU-IRB 970-E/18. Data were analyzed using SPSS version 24 (IBM SPSS® Statistics for Windows, version 24; IBM Corp, Armonk, NY, USA).

Use of pre-selected set of preliminary quality markers
A pre-selected set of preliminary quality markers of clinical pharmacokinetic studies identified systematically [14] was used to build a questionnaire, comprising potential identified candidate items into a design-specific critical appraisal tool. These items were categorized into relevant study sections, including title, abstract, background, methods, results, discussion, conclusion and other.

Sampling and selection of Delphi experts
A purposive sampling method was used to select participants who met the study's eligibility criteria. Potential participants were approached as expert panelists if they were considered as any of the following: • academic experts with a designation that reflects their direct involvement in clinical pharmacokinetics as evidenced in their research and teaching portfolios. • clinicians who had experience in the application of clinical pharmacokinetic principles or the provision of therapeutic drug monitoring in their clinical practice. These individuals should have experience in interpreting the findings of clinical pharmacokinetic studies and applying these to their patients. • pharmaceutical industry researchers with years of experience in relevant positions involving clinical pharmacokinetic research. • individuals in regulatory bodies who assess clinical pharmacokinetic studies when making decisions for their respective health authorities, e.g. FDA.
Potential participants were requested to state their area of expertise related to clinical pharmacokinetics, number of years in the field of clinical pharmacokinetics, current geographical location of practice and a global self-assessed competence in the field of clinical pharmacokinetics (none, little, average, advanced). know, and Not applicable' were grouped together. Cohen's Kappa was used to assess level of agreement between raters [15]. Kappa values of less than 0 were rated as less than the chance of agreement; 0.01-0.20, slight agreement; 0.21-0.40, fair agreement; 0.41-0.60, moderate agreement; 0.61-0.80, substantial agreement; 0.81-0.99, almost perfect agreement. Average time to complete each assessment was assessed for feasibility of the tool.

Participants
One hundred and nineteen potential expert panelists, identified by the research team or through snowballing, were invited to take part in this study. Personal invitations were sent via email. Twenty-five panelists agreed to participate in the modified Delphi process. The sociodemographic characteristics of the panelists of the modified Delphi are shown in Table 1.
The average level of experience of the participants in the field of clinical pharmacokinetics was 8.5 years with 56% self-identified as clinicians with relevant practice experience in clinical pharmacokinetics. Around 96% of the participants rated themselves with an average or advanced level of competence in clinical pharmacokinetics.
Forty items were identified through systematic search of the literature and were included in the first round of the Delphi. Two additional control items that were not deemed relevant to clinical pharmacokinetics were included to gauge the integrity of the participant responses for a total of 42 items (Supplement 1). At the end of round 1, the panel blinded aggregate comments from all other participants for each item. As the aim of the study was to create a tool to be used to assess the quality of clinical pharmacokinetic studies, the researchers sought to gauge the participants' opinion of an effective means of rating each item included in the tool. Consequently, an additional section of the survey asked participants to rate various scales of measurement that could be used to assess each item in the finalized tool. These initially included options of "yes/no" and a 4-point scale ranging from "poor, fair, good, to excellent". Authors were also open to additional options presented during the modified Delphi. The experts were also provided a freetext box to suggest additional rating systems. Participants were given four weeks to complete each Delphi round with reminder emails being sent after three weeks. All participants' responses were kept anonymous from the other participants.

Pilot testing
The CACPK final tool was applied to 30 clinical pharmacokinetic studies in order to test validity and feasibility of the tool in evaluating published literature. Ten studies were purposively selected each from three journals that focus on clinical pharmacokinetics (Clinical Pharmacokinetics, International Journal of Pharmacokinetics, Journal of Pharmacokinetics and Pharmacodynamics). Articles included randomized clinical pharmacokinetic trials, evaluations of drug interactions, population pharmacokinetic studies, and bioequivalence studies. Studies including less than five participants were excluded from the pilot testing. Application of the quality tool was performed by four end-users (AS, SP, OR, KW) in duplicate. Included articles were divided equally at random amongst raters so that each rater evaluated 15 articles and each article was evaluated twice. Raters were also asked to record the time to complete the quality assessment of each article evaluated. Outcomes of 'No, I don't    software to perform the statistical analysis appropriate to achieve the study objectives?) and 21 (were appropriate summary statistics to describe centrality and variance used to document the pharmacokinetic results?) scored substantial to almost perfect agreement. Kappa values could not be calculated for items 2 and 5 as all respondents answered 'yes' with 100% agreement. Items 3 and 10 scored less than the chance of agreement.
Applying this CACPK final tool to clinical pharmacokinetic studies took an average of 28.5 min (range 14-90 min). No raters expressed any difficulty in understanding any of the items or applying these to the studies.

Statement of key findings
The CACPK tool (Supplement 3) developed in this study offers new understanding of quality markers that should be utilized for appraising published clinical pharmacokinetic studies via pharmacokinetics experts consensus. Together with the CACPK tool, two additional appendices (Supplements 4 and 5) were created in case further clarification is needed about items 7 and 15 in the tool, respectively. This tool with its two appendices guide researchers through answering 21 questions to determine the quality of published clinical pharmacokinetic studies. The tool was developed in a format similar to other critical appraisal tools like a measurement tool to assess systematic reviews (AMSTAR) to facilitate the appraising process and was composed of four sections: appraising background, appraising study design and experimental methods, appraising applied statistics, and appraising results [16].
were amended based on participants' comments and were re-entered into round 2 (Fig. 1).
In round 2, 28 items were presented to the participants for consideration. At the end of the round, 6 items (21.4%) reached consensus for inclusion in the CACPK final tool, while 19 items (67.9%) reached consensus for exclusion. The remaining three items (10.7%) were reformulated based on participant's feedback and were included in round 3.
Round 3 comprised of 3 items in which all (100%) reached consensus for inclusion into the CACPK final tool. Consensus rates for each of the included item along with the round of the modified Delphi in which they were approved are summarized in Table 2. Participant consensus regarding the choice of rating of each item in the CACPK final tool was for items to be rated as 'Yes, No, I don't know, and Not applicable'.
The CACPK final tool included 21 items that met consensus for inclusion ( Table 3). The tool was divided into five sections: background, design, sampling, applied statistics, and results. The final rating scale that was used to help the end-users to assess the quality of published clinical pharmacokinetic studies was "Yes, No, I Do Not Know, and Not Applicable".

Pilot testing
Thirty recently published articles focused on clinical pharmacokinetics were assessed for quality (Supplement 2) using the quality assessment tool developed through the modified Delphi (CACPK final tool). A majority of items (12/21) were scored at fair to moderate agreement. Of note, items 13 (were sample storage conditions appropriate and described in a manner that could be accurately replicated?), 16 (were the essential pharmacokinetic parameters required to make the results applicable in clinical settings addressed?), 18 (were the chosen statistical tests and  researchers, individuals in academic and industrial sectors and policymakers). The inclusion of different stakeholders allowed us to enrich our tool with different perspectives from different end-users.

Further research
Further research can be conducted to further assess the validity, reliability, and overall utility of the tool. Research should also be conducted to determine how the presence or absence of most important markers of quality for clinical pharmacokinetic studies affects study results and interpretation. We acknowledge that the field of clinical pharmacokinetics is vast and that end-users of clinical pharmacokinetic research extends to a vast array of clinicians. It is therefore unlikely that we included expert panelists from all areas relevant to clinical pharmacokinetics.

Conclusion
This study aimed to achieve expert consensus regarding preidentified set of quality markers for the appraisal of clinical pharmacokinetic studies. Through this modified Delphi process, a list of questions gauging the overall quality of clinical pharmacokinetic studies was developed that allowed for critical appraisal. Work presented in this study provides a critical appraisal tool (CACPK) with broad application to clinical pharmacokinetic studies.

Acknowledgements
The authors would like to thank the members of the expert panel who participated in this study for their contributions during the modified Delphi technique meetings. We appreciate all of their time, effort, opinions, comments, and suggestions that contributed greatly to the refinement of the CACPK tool. The authors would like to thank Dr. Ahmed Awaisu for his assistance in editing the final version of the manuscript.
Authors' contributions All authors participated in the concept and planning of the work described; acquisition, analysis and interpretation of the data; and in the drafting, critical revision, and approval of the final version of the manuscript.
Funding No sources of funding were used to assist in this study.
Open Access funding provided by the Qatar National Library.
Code Availability Not applicable.

Strengths and weaknesses
The development of a critical appraisal tool for clinical pharmacokinetic studies adds much value to the existing literature as it complements currently available checklists by assessing dimensions of quality of clinical pharmacokinetic studies. This tool, with further testing of reliability and validity, can be potentially used to guide clinicians and policymakers to evaluate the quality of published articles to take clinical decisions and develop policies by applying EBM. Furthermore, stakeholders who work in the academic sector can use it to teach students how to appraise this type of study.
Despite these strengths, we need to acknowledge the limitations of this study. Participants' identification was anonymous as researchers, which prevented the ability to correlate responses to professional experience. Furthermore, we received a declining response rate with each round of the survey leading to overemphasized response from a smaller number of participants in the latter rounds of the study.

Interpretation
The aim of developing the CACPK tool was to assess the quality of studies, which has several dimensions including study design, conduct, analysis, clinical relevance, results validity and quality of reporting. While other clinical pharmacokinetic guidelines have focused on assessing the quality of reporting of clinical pharmacokinetic research, these tools do not directly address whether the methods used were of high quality. This tool thus assesses the appropriateness of the execution of methods used in carrying out the desired clinical pharmacokinetic analysis.
Pilot testing of the clinical pharmacokinetic critical appraisal tool amongst four individuals resulted in a majority (12/21) of items on the tool scoring fair to moderate agreement. While these four individuals did have considerable experience in the application of clinical pharmacokinetics, this result may not be replicated if clinicians with little to no understanding of clinical pharmacokinetic principles use this tool. Application of this tool in assessing the quality of a clinical pharmacokinetic study did however appear to be reasonably feasible in regards to time requirement with the average study taking under 30 minutes to appraise.
To our knowledge, this study is among the few ones to identify quality markers of clinical pharmacokinetic studies for developing a specific critical appraisal tool with broad application. One of the main strengths of this study was the robustness of the methodology used to develop clinical pharmacokinetics critical appraisal tool. In the modified Delphi process, we were keen to recruit experts who represent all clinical pharmacokinetic stakeholders (clinicians, Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons. org/licenses/by/4.0/.