Development and validation of the ASPIRE-VA coaching fidelity checklist (ACFC): a tool to help ensure delivery of high-quality weight management interventions

Abstract

Practical and valid instruments are needed to assess fidelity of coaching for weight loss. The purpose of this study was to develop and validate the ASPIRE Coaching Fidelity Checklist (ACFC). Classical test theory guided ACFC development. Principal component analyses were used to determine item groupings. Psychometric properties, internal consistency, and inter-rater reliability were evaluated for each subscale. Criterion validity was tested by predicting weight loss as a function of coaching fidelity. The final 19-item ACFC consists of two domains (session process and session structure) and five subscales (sets goals and monitor progress, assess and personalize self-regulatory content, manages the session, creates a supportive and empathetic climate, and stays on track). Four of five subscales showed high internal consistency (Cronbach alphas > 0.70) for group-based coaching; only two of five subscales had high internal reliability for phone-based coaching. All five sub-scales were positively and significantly associated with weight loss for group- but not for phone-based coaching. The ACFC is a reliable and valid instrument that can be used to assess fidelity and guide skill-building for weight management interventionists.

Appendices

Appendix 1: ASPIRE-VA coaching fidelity checklist (ACFC) ordered by subscale

Session structure

Scoring:

• 0: Did not cover = this topic or focus point did not happen at all

• 1: Partially covered = this happened to some extent with the individual, or in a group setting, occurred for some but not all group members, all of the time (e.g., interventionist facilitated discussion, but only among certain members of the group)

• 2: Fully covered = the topic was met fully for individual or all participants in a group

Set goals and monitor progress

• __a—Health Coach determines level of self-monitoring compliance (i.e., average daily step counts, Stoplight levels)

• __b—Health Coach prompts review of goal attainment (i.e., review of goals versus actual)

• __c—Health Coach elicits discussion about successes/failures since last session and initiates problem-solving approach when necessary to address barriers

• __d—Health Coach prompts participant(s) to identify specific small change goals for the next week (a “menu of ideas” for steps, Stoplight colors, and/or current week topic)

• __e—Health Coach and participant(s) come to agreement on small change goals

Assess and personalize self-regulatory content

• __a—Health Coach elicits discussion of a psycho-educational topic that develops a self-management skill or changes cognitions

• __b—Health Coach assesses participant(s)’ knowledge of topic and degree of relevance to their weight management practices

• __c—Health Coach customizes session content to individual/group situation to increase knowledge or skills

• __d—Health Coach assesses participant(s)’ self-confidence/readiness to follow-through on plan

• __e—Health Coach helps participant(s) identify barriers to success and problem-solve possible solutions to these contingencies

Session process

Scoring:

• 0: Did not demonstrate = this process objective or component was not demonstrated at all

• 1: Inconsistently demonstrated = this happened to some extent, but not or all group members, all of the time

• 2: Demonstrated consistently through entire session = objective was demonstrated consistently and appropriately throughout the entire session

• 9: NA

Manage the session

• __a—Health coach came prepared and organized

• __b—Time was allocated appropriately in order to cover the appropriate content focus points for that session

• __c—Health coach delivered didactic material in a matter of fact and friendly way

• __d—Health coach facilitated discussion and interaction using open-ended questions, affirmations, reflections, summaries

• __e—Health coach elicits clarification of participant(s)’ engagement by seeking feedback about didactic content

Stay on track

• __a—Health coach addressed process (tangential) issues but did not allow them to disrupt content agenda

• __b—Health coach modulated distractions (e.g., side bar conversations, interruptions by family members)

Create a supportive and empathetic climate

• __a—Health coach avoided judgmental feedback on participant(s) contributions

• __b—Health coach responded empathically and accurately to individual or group member behavior (verbal, nonverbal)

Appendix 2. ASPIRE-VA coaching fidelity checklist (ACFC) items ordered for clinical use

Session core elements

Scoring:

• 0: Did not cover = this topic or focus point did not happen at all

• 1: Partially covered = this happened to some extent, but not or all group members, all of the time (e.g., interventionist facilitated discussion, but only among certain members of the)

• 2: Fully covered = the goal was met fully for all participants

Checking in on self-monitoring and goal attainment

• __a—Interventionist determines level of self-monitoring compliance (i.e., average daily step counts, Stoplight levels)

• __b—Interventionist prompts review of goal attainment (i.e., review of goals versus actual)

• __c—Interventionist elicits discussion about successes/failures since last session and initiates problem-solving approach when necessary to address barriers

Assess and personalize self-regulatory content

• __a—Interventionist elicits discussion of a psycho-educational topic that develops a self-management skill or changes cognitions

• __b—Interventionist assesses participant(s)’ knowledge of topic and degree of relevance to their weight management practices

• __c—Interventionist customizes session content to individual/group situation to increase knowledge or skills

Action planning and session wrap-up

• __a—Interventionist prompts participant(s) to identify specific small change goals for the next week (a “menu of ideas” for steps, Stoplight colors, and/or current week topic)

• __b—Interventionist assesses participant(s)’ self-confidence/readiness to follow-through on plan

• __c—Interventionist helps participant(s) identify barriers to success and problem-solve solutions to these contingencies

• __d—Interventionist and participant(s) come to agreement on small change goals

Notes for coach:

Scoring:

• 0: Did not demonstrate = this process objective or component was not demonstrated at all

• 1: Inconsistently demonstrated = this happened to some extent, but not or all group members, all of the time

• 2: Demonstrated consistently through entire session = objective was demonstrated consistently and appropriately throughout the entire session

• 9: NA

Manage the session

• __a—Interventionist came prepared and organized

• __b—Time was allocated appropriately in order to cover the appropriate content focus points for that session

• __c—Interventionist delivered didactic material in a matter of fact and friendly way

• __d—Interventionist facilitated discussion and interaction using open-ended questions, affirmations, reflections, summaries

• __e—Interventionist elicits clarification of participant(s)’ engagement by seeking feedback about didactic content

Stay on track

• __a—Interventionist addressed process (tangential) issues but did not allow them to disrupt content agenda

• __b—Interventionist modulated distractions (e.g., side bar conversations, interruptions by family members)

Create a supportive and empathetic climate

• __a—Interventionist avoided judgmental feedback on participant(s) contributions

• __b—Interventionist responded empathically and accurately to individual or group member behavior (verbal, nonverbal)

Notes for coach:

Appendix 3. ASPIRE-VA coaching fidelity checklist (ACFC) guidelines for assigning ratings—items ordered for clinical use

Heading information

• a. Session date: enter date of session being rated

• b. Interventionist: enter initials of interventionist

• c. Rater: enter initials of the individual assigning the ratings for the session

• d. Session number: enter the session/topic number or week (e.g., week 1)

• e. Duration: enter the number of minutes from start to finish

• f. Mode: enter group or phone

There should be no not applicable (N/A) for any core element. To minimize subjectivity, this checklist is designed to determine whether a core element occurred. Rather than using a dichotomous rating of covered or not covered, partially covered is added to refer to situations where the interventionist begins to cover a point but is diverted from proceeding by a tangential issue or other reason.

Checking in on self-monitoring and goal attainment

• __a—Interventionist determines level of self-monitoring compliance (i.e., average daily step counts, Stoplight levels)

  • 0: Did not cover = The interventionist did not ask the patient to provide average weekly attainment of step counts and Stoplight goals based on their ASPIRE tracking log.

  • 1: Partially covered = Interventionist began to assess self-monitoring compliance but failed to (1) obtain at least step count and stoplight goals; (2) did not collect data from all group members; (3) or is diverted off topic and did not come back to compliance check

  • 2: Fully covered = basic logging compliance was obtained from participant or all group participants

• __b—Interventionist prompts review of goal attainment (i.e., review of goals versus actual)

  • 0: Did not cover = The interventionist did not prompt/or review participant(s)’ actual behavior versus planned goal

  • 1: Partially covered = Interventionist did not fully review actual vs. planned goal attainment by (1) only reviewing progress for one of two key behaviors (step counts, red goal, green goal); (2) did not collect data from all group members; (3) or was diverted off topic and does not come back to compliance check

  • 2: Fully covered = Interventionist obtained/reviewed participant(s)’ self-reported goal attainment relative to goals for phone participant or all group participants

• __c—Interventionist elicits discussion about successes/failures since last session and initiates problem-solving approach when necessary to address barriers

  • 0: Did not cover = The interventionist did not elicit participant(s)’ perspective on specific barriers or facilitating factors to goal attainment to identify strategies to overcome problems to successful goal attainment in the future; nor did the participant volunteer these barriers out of natural group/phone discussion.

  • 1: Partially covered = Interventionist elicited participant(s)’ perspective on specific barriers or facilitating factors to goal attainment to identify strategies to overcome problems to successful goal attainment in the future but did not obtain specific response from participants(s) because interventionist: (1) did not collect data from all group members; (2) or was diverted off topic and did obtain sufficient understanding of barriers/feedback to offer reinforcement to facilitators and/or problem solving response to barriers. Score as “1: if participant(s) voluntarily discussed issue but interventionist did not elicit further elaboration or problem-solving for participant(s) and passively permitted unguided discussion of barriers.

  • 2: Fully covered = Interventionist elicited participant(s)’ perspective on specific barriers or facilitating factors to goal attainment to reinforce successes and to identify strategies to overcome problems to successful goal attainment in the future for phone participant or all group participants

Assess and personalize self-regulatory content

• __a—Interventionist elicits discussion of a psycho-educational topic that develops a self-management skill or changes cognitions

  • 0: Did not cover = The interventionist did not engage participant(s) with key questions that stimulated a discussion of a self-management session topic that was designed to foster development of specific behavioral skills or awareness of psychosocial topic for self-regulation of weight (i.e., no specific ASPIRE session module was discussed). An undesirable example would be to ask an open-ended question about a topic but not follow-up with substantive prompts and queries that customize the topic to participant or group interests or needs (see minimum duration).

  • 1: Partially covered = The interventionist engaged participant(s) with key questions to initiate a discussion about a self-management session topic that is designed to foster development of a specific behavioral skills or awareness of psychosocial topic for self-regulation of weight (i.e., a specific ASPIRE session module was discussed). However, after discussion is initiated, substantive engagement is curtailed due to discussion of tangential issues, interventionist changed topic before participant(s) adequately expressed how topic pertains to their life situation, and/or participant(s) replies with short answers or denied relevance.

  • 2: Fully covered = The interventionist fully engaged participant(s) in a discussion of a self-management session topic that is designed to foster development of a specific behavioral skills or awareness of psychosocial topic for self-regulation of weight (i.e., a specific ASPIRE session module was discussed). Interventionist skillfully uses key open-ended questions to initiate participant dialogue using techniques like elicit–provide–elicit, chunk–check–chunk, summaries, and reflective listing that encouraged participant(s) to explore and draw own conclusions as to the personal relevance of content. “Substantive discussion” should last at least 5 min in phone session and at least 30 min (phase I groups) or 15–30 min (phases II and III groups).

• __b—Interventionist assesses participant(s)’ knowledge of topic and degree of relevance to their weight management practices

  • 0: Did not cover = The interventionist did not attempt to elicit specific thoughts, assumptions, meanings, or behaviors related to the current ASPIRE content in order to customize the material in a way that was relevant to participant(s) or to help overcome resistance to exploring the topic.

  • 1: Partially covered = The interventionist used appropriate techniques (e.g. open-ended questions, reflections, metaphors) to elicit specific thoughts, assumptions, meanings, or behaviors related to the current ASPIRE content but had difficulty establishing a focus or focused on thoughts/behaviors that were irrelevant to the participant(s)’ life situation.

  • 2: Fully covered = The interventionist skillfully focuses on key thoughts, assumptions, behaviors, etc. that made the topic relevant to participant(s)’ weight loss efforts and which offered an opportunity for change to help support progress towards healthy weight loss. Interventionist also asked for participant(s) understanding of session content, asking permission to share information or advice to correct misperceptions or inaccuracies.

• __c—Interventionist customizes session content to individual/group situation to increase knowledge or skills

  • 0: Did not cover = The interventionist made no effort to customize ASPIRE content in participant workbook to information elicited from participant(s) about topic relevance (e.g., thoughts, beliefs, available resources/constraints, etc.) and workbook exercises are not modified to encourage participant(s) to consider topic as another strategy to help achieve weight loss goals.

  • 1: Partially covered = The interventionist appropriately customizes discussion of ASPIRE content in participant workbook to information elicited from participant(s) about topic relevance but does not explore or encourage participant(s) to take actionable steps to integrate this information into their weight loss efforts.

  • 2: Fully covered = The interventionist appropriately customizes discussion of ASPIRE content in participant workbook to information elicited from participant(s) about topic relevance and supports participant(s)’ efforts to take actionable steps to integrate this new content information into their weight loss efforts.

Action planning and session wrap-up

• __a—Interventionist prompts participant(s) to identify specific small change goals for the next week (a “menu of ideas” for steps, Stoplight colors, and/or current week topic)

  • 0: Did not cover = The interventionist did not help participant(s) identify a list of potential ideas for daily or weekly behavior changes (i.e., menu of ideas) that could be implemented before next ASPIRE session.

  • 1: Partially covered = The interventionist either appropriately helped some participant(s) but not all participants summarize a list of patient-cited ideas for change and choose goals (group only), or interventionist chose goal for patient rather than patient choosing own goal.

  • 2: Fully covered = The interventionist skillfully focuses helped participant(s) summarize a list of possible goals for behavior change (from current or prior sessions) and helped participant(s) identify specific, a behavior to be implemented before the next ASPIRE session. Interventionist may also query whether there was a potential goal behavior not on the menu of ideas list to be added.

• __b—Interventionist assesses participant(s)’ self-confidence/readiness (motivation) to follow-through on plan

  • 0: Did not cover = The interventionist took no action to elicit participant motivation (self-confidence, readiness, interest using 1-10 rulers or verbal inquiry) to implement goals selected for next ASPIRE session.

  • 1: Partially covered = The interventionist elicited participant motivation to implement goals selected for next ASPIRE session but did not use 1–10 rulers to help identify barriers, beliefs, or self-confidence to carry out action steps to goal.

  • 2: Fully covered = The interventionist elicited participant motivation and ability to implement goals selected for next ASPIRE session by using 1-10 rulers to help identify barriers, benefits, and solutions depending on participant response.

• __c—Interventionist helps participant(s) identify barriers to success and problem-solves possible solutions to these contingencies

  • 0: Did not cover = The interventionist took no action to use problem-solving theory model to help participant(s) identify strategies to overcome potential barriers cited to implementing their goals before the next ASPIRE session. Patient action steps were vague and did not fit into a coherent strategy to help participant(s) achieve specific ASPIRE goals for weight loss.

  • 1: Partially covered = The interventionist successfully elicited potential barriers/challenges to attaining short-term goals and encouraged participant(s) to identify strategies to overcome potential barriers cited by participant(s) to implementing their goals before the next ASPIRE session but no agreement on specific steps participants will take for these contingencies.

  • 2: Fully covered = The interventionist successfully elicited potential barriers/challenges to attaining short-term goals and encouraged participant(s) to identify strategies to overcome potential barriers cited by participant(s) to implementing their goals before the next ASPIRE session and encouraged exploration of hypothetical scenarios of what steps needed to be taken to be successful at making a change and a elicited a set of specific actions that participant(s) will take to overcome goals to achieve their goal (what, when, where). When applicable, interventionist asks permission to share advice on what others have done or to provide expert information.

• __d—Interventionist and participant(s) come to agreement on small change goals

  • 0: Did not cover = There was vague or poor agreement in action steps (measureable goal) behaviors to be implemented by participant in interim period before next session. Recorded goals may be interventionist initiated with tacit acceptance by participant(s).

  • 1: Partially covered = Interventionist and participant identified ASPIRE goals to be implemented before next session and tracked with monitoring log. However, interventionist did not summarize these goals, how they will be achieved, and tracked and verified for understanding and accuracy with participant(s) (e.g., “have I gotten that right?”)

  • 2: Fully covered = Interventionist and participant collaboratively identified ASPIRE goals to be implemented before next session and tracked with monitoring log. Interventionist summarized the action planning steps to be implemented and elicited verification from participant that what he/she has described is accurate. Also, checks “Is there anything else?”

Session conduct that promotes high-quality delivery

Manage the session

• __a—Interventionist came prepared and organized

  • 0: Did not demonstrate = Interventionist presents in a disorganized, flustered, or anxious state. Session lacks focus, direction, and interventionist is unable to speak authoritatively about planned session content or the personal details of individual participant(s).

  • 1: Inconsistently Demonstrated = Interventionist was able to deliver sections of session well (i.e., check-in, content, or action planning) but appears flustered and confused delivering information customized to individual session or participant during one section of session (e.g., lack of familiarity or ability to customize session content with outline or lack of participant progress in ASPIRE (participant knowledge).

  • 2: Demonstrated consistently = Interventionist is able to confidently lead the session, providing a focused structure to session that provided participant(s) with clear sense of the session agenda, customized data (taking from general to personally specific), and provided session specific information/or advice when needed that fit within the personal situation/goal progress of participant(s).

• __b—Time was allocated appropriately in order to cover the appropriate content focus points for that session

  • 0: Did not demonstrate = Interventionist made no attempt to structure session time. Session seemed aimless/without focus.

  • 1: Inconsistently Demonstrated = Session had some focus (topic), but interventionist had problems with structure or pacing (e.g., too little structure, too slowly paced, or too rapidly paced and ending early without adequate topic discussion).

  • 2: Demonstrated consistently = Interventionist used time efficiently by tactfully limiting tangential or unproductive discussion and by pacing the session as rapidly as needed for the participants(s) to cover check-in, session content, and action planning. Adequate time was given to ensure topics were adequately covered to participant(s) satisfaction and understanding.

• __c—Interventionist delivered didactic material in a matter of fact and friendly way

  • 0: Did not demonstrate = Interventionist relied primarily on persuasion, lecturing, or debating participant. Interventionist did more talking than participant and expert role of telling participant(s) (imposing viewpoint) put participant(s) on the defensive.

  • 1: Inconsistently Demonstrated = Interventionist for the most part, helped participant(s) see new perspectives through guided discovery (engaged discussion with peers, considering alternatives, weight pros/cons, etc., asking permission to share advice or info) rather than taking adversarial or educational orientation. Used opened-ended questions appropriately but on occasions, relied too heavily on persuasion or advice giving than self-discovery.

  • 2: Demonstrated consistently = Interventionist was skillful in using guided discovery to engage participant(s) to explore challenges with weight loss, consider new information/skills, and to draw their own conclusions. Participants were treated as experts about their own behavioral change interests and were treated with respect, sincerity, and positive regard/optimism. Interventionist achieved a good balance between skillful use of open-ended questioning and reflective listening, and use of information provision strategies to stimulate engagement in intervention program.

• __d—Interventionist facilitated discussion and interaction using open-ended questions, affirmations, reflections, and summaries

  • 0: Did not demonstrate = Interventionist demonstrated poor use of patient centered communication techniques (MI). Participant exhibited a tendency to talk more than patient, relied on closed-ended questions, and had difficulty demonstrating active, empathetic listening skills to encourage active discussion (affirmations, reflections, and summaries).

  • 1: Inconsistently Demonstrated = Interventionist demonstrated a basic but inconsistent ability to use patient centered communication skills to foster active discussion of session components. Interventionist regularly used open-ended questions to initiate discussion but had difficulty using affirmations, complex reflections, and summaries to skillfully guide discussion, elicit deeper participant reflection regarding a response, or to clarifying understanding.

  • 2: Demonstrated consistently = Interventionist displayed skillful use of MI techniques to engage participant(s) in collaborative discussion regarding their lifestyle changes. Interventionist relied on reflections to obtain deeper reflection from open-ended questions. Reflections were frequently paired with affirmations to convey empathy and understanding and summaries were used periodically during session to review key points and verified both interventionist and participant(s) understanding of content.

• __e—Interventionist elicits clarification of participant(s)’ engagement by seeking feedback about didactic content

  • 0: Did not demonstrate = Interventionist did not ask for feedback to determine participant(s)’ understanding of, or response to a session or goal setting objective.

  • 1: Inconsistently Demonstrated = Interventionist elicited some feedback from participant(s), regarding satisfaction/buy-in to session topics. Interventionist inconsistently responded to participant(s) indications of discounting, resisting, or minimizing value of content.

  • 2: Demonstrated consistently = Interventionist was adept at eliciting and responding to verbal and non-verbal feedback through session (e.g., elicited reactions to session, regularly checked for participant reactions (reflections, summaries)) to insure learning atmosphere is positive and participant(s) are receptive to information being presented.

Stay on track

• __a—Interventionist addressed process (tangential) issues but did not allow them to disrupt content agenda

  • 0: Did not demonstrate = Interventionist made no attempt to redirect discussion of topics unrelated to ASPIRE or psycho-behavioral self-management topics for weight maintenance/loss. Session rambled and some participants became visibly irritated by digressions from main topic or monopolization of discussion by one or two participants.

  • 1: Inconsistently Demonstrated = Interventionist covered the primary focus points related to check-in, session content, and action planning but at times, Interventionist had problems with participants getting off-topic by telling unrelated stories or discussing issues unrelated to ASPIRE or current discussion point raised by Interventionist that had little focus or benefit to other participants.

  • 2: Demonstrated consistently = Interventionist respectfully acknowledged participants’ contributions to discussion topics but skillfully redirected discussion of tangential issues by asking permission to revisit the topic later in the session (or after session) or by cutting the participant off respectfully. Session remained focused on key topics generated by session content or problems encountered with self-monitoring (check-in) or action planning.

• __b—Interventionist modulated distractions (e.g., side bar conversations, interruptions by family members)

  • 0: Did not demonstrate = Interventionist made no attempt to use group management skills to cut off side conversations that interrupted a participant’s response to a discussion topic or to remind disruptive participants’ that their behavior (e.g. interrupting a speaker) violates the group’s rules. On phone sessions, interventionist made no attempt to ask participant to manage disrupting elements in background (pets, TV or radio noise, or conversations with people in background).

  • 1: Inconsistently Demonstrated = Interventionist was moderately successful at using group management skills to cut off side conversations or disruptive participant’s actions that violate the group’s rules. On phone sessions, interventionist was mostly successful at managing disrupting background noises or factors.

  • 2: Demonstrated consistently = Interventionist respectfully acknowledged participants’ contributions to discussion topics but skillfully asked permission to modulate distractions or informs disruptive group members of inappropriate behaviors. Session is focused and respectful of participant(s) rights.

Create a supportive and empathetic climate

• __a—Interventionist avoided judgmental feedback on participant(s) contributions

  • 0: Did not demonstrate = Interventionist demonstrated inability to keep personal opinions or advice out of session and was not able to demonstrate acceptance of patient struggles, warmth, and genuine concern for participant(s). Interventionist seemed hostile, demeaning, unnecessarily critical, or some way destructive to the participant.

  • 1: Inconsistently Demonstrated = Interventionist was not destructive to participant(s)’ welfare but at times, exhibited statements that undermined patient trust that the interventionist would be helpful, optimistic, warm, concerned, and empathetic. At times, interventionist appeared inpatient, insincere, critical, or had difficulty conveying confidence in participant(s)’s ability to make changes.

  • 2: Demonstrated consistently = Interventionist displayed high levels of warmth, concern, confidence, genuineness, and professionalism in helping participant(s) work through painful issues without conveying judgment or superiority.

• __b—Interventionist responded empathically and accurately to individual or group member behavior (verbal, nonverbal)

  • 0: Did not demonstrate = Interventionist repeatedly failed to understand what the patient explicitly said and consistently missed the literal and implied emotions underlying a point. Had difficulty reflecting or rephrasing what patient said and often missed more subtle communication (e.g., tone and sarcasm of participant(s) comments).

  • 1: Inconsistently Demonstrated = Interventionist was usually able to reflect or rephrase (s) participant’s perspective as noted by what but often missed more subtle communications that need to be reflected by reflections of emotional content or what was not said. (But often unaware of shifts in level of participant impatience conveyed by sarcasm, digressions, tone of voice).

  • 2: Demonstrated consistently = Interventionist displayed ability to understand participant(s)’ perspective thoroughly and was skillful at expressing this understanding through appropriate verbal reflections and non-verbal responses to participant(s). Participants talked less and used more reflections, encouraging statements (uh-huh, yes, I see,), and voice to convey genuine interest and sympathetic understanding to participant discussion.