Introduction

Healthcare research in general, and eHealth studies specifically, increasingly value and require the involvement of different stakeholder groups because this can increase the success of a project, through aligning the project with what stakeholders want. One approach that includes active involvement of stakeholders is Action Research (AR), where stakeholders become co-researchers of a project (Reason and Bradbury 2013). Other elements of AR include that 1) the project is conducted in cycles of planning, action and reflection; 2) it takes place within the context that is studied (e.g., in a community); and 3) it aims to make a change in practice and to extend scientific knowledge at the same time. However, a literature review on eHealth AR projects showed that there is a lack of clear knowledge sharing, making it difficult for others to shape their project and to learn from others (Oberschmidt et al. 2022). Research frameworks can be a useful starting point for getting familiar with a certain field, approach or methodology. Such frameworks also support transferability of knowledge or comparability between studies, as they can support researchers to work in a more standardised way.

Table 1 Provides an overview of different frameworks for stakeholder involvement, eHealth and AR. Stakeholder involvement has been the topic of various research frameworks outside the context of eHealth, each focusing on another discipline or aspect of the involvement. The frameworks provide useful categorisations to clearly describe for example the nature or timepoints of involvement in a project. While this can be very helpful to describe involvement in a standardised way, there is little practical knowledge in these frameworks about how to set up and conduct stakeholder engagement.

Table 1 Overview of frameworks for stakeholder involvement, eHealth and AR including their main findings
Full size table

Similarly, several eHealth frameworks include the involvement and roles of stakeholders, but not as key element of their framework. While these frameworks in some cases provide helpful steps for researchers to follow (see e.g., van Gemert-Pijnen et al. 2011), they are focused on the whole process of eHealth development or implementation. More attention specifically to the involvement of stakeholders is needed, as this is a complex topic.

There are also several frameworks for AR available within other fields like education, information systems or management, and while these are interesting to look at and take inspiration from, the type of stakeholders involved in eHealth projects, and their relationships (e.g., patient and doctor, older adult and informal carer) are very specific. This also requires eHealth specific frameworks. AR healthcare frameworks that exist are often still focused on the educational aspect within healthcare (see Table 1), and thus not applicable in the context of developing and implementing eHealth. To the best of our knowledge, no framework is available that combines knowledge about stakeholder involvement from the fields of eHealth and AR.

Both stakeholder involvement and AR frameworks can be very interesting for the eHealth context, but often lack the specific focus. The healthcare sector presents very specific challenges e.g., the specific relationship between patient and healthcare professional and potential vulnerability of patients as involved partners). On the other hand, where eHealth frameworks mention stakeholder involvement, or elements of AR like working iteratively, this is usually only a smaller part of the framework.

Research frameworks tend to be rather concise, making complex concepts operational. However, the advantage of presenting information in a simplified, to the point manner, can make the frameworks less easy to understand and work with for laypeople and those from other fields. Therefore, what is needed is not only a framework that combines knowledge about stakeholder involvement from the fields of eHealth and AR. Working in practice in this context also requires concrete, easily applicable steps that are not only understandable to experts.

To the best of our knowledge, no such frameworks currently exist. Yet the field of eHealth research can greatly benefit from AR in general, and stakeholder involvement specifically, to address some of the current problems in the healthcare sector. To facilitate knowledge sharing and support a more standardized approach, researcher could benefit from a framework that includes the main aspects of stakeholder involvement in eHealth AR. Therefore, the aim of this research is to develop and evaluate a practical framework for guiding the process of stakeholder involvement in eHealth action research.

Initial Development of the Framework for eHealth AR

The development of this framework started from earlier AR and stakeholder involvement studies in different projects, which have been described in more detail in other publications (Oberschmidt et al. 2020, 2022). Based on these findings we started to create a list of important aspects for each phase of a project. The framework follows an abstract version of the timeline of a project, from (developing) the initial idea to sustaining the change in practice after a project ends. The initial version of the framework started with an introductory text as well as a figure to provide an overview. This was followed by nine elements, with a brief description and somewhere between three and nine pointer questions for each. These elements were: Project idea (4 questions), Plan—Roles and tasks (9 questions), Plan—Align interests and needs (6 questions), Ethical approval (3 questions), Training (5 questions), Act—Celebrate milestones (5 questions), Act—Dissemination (3 questions), Reflect (7 questions) and Sustain change (4 questions).

The final version of our framework for stakeholder involvement in eHealth AR projects consists of an introductory section, followed by descriptions and pointer questions for the nine different elements of eHealth AR. For each of these elements, some additional explanations and questions were added compared to the initial version, leading to the following overview of the framework: Project idea (6 questions), Plan—Roles and tasks (10 questions), Plan—Align interests and needs (6 questions), Ethical approval (6 questions), Mutual learning and Skill development (5 questions), Act—Celebrate milestones (6 questions), Act—Dissemination (5 questions), Reflect (7 questions) and Sustain change (5 questions). A full version of the final framework can be found at https://www.rrd.nl/ar-framework/ as well as in Appendix B.

First Iteration with Project Members

A first draft version of the framework was presented to project members of a large scale European project, all doing AR in their different pilots. The session was attended by 20 project partners with different functions (e.g., researchers, stakeholder representatives, healthcare workers). After a short presentation of the framework, the participants worked in groups to discuss the framework, and as a group provide feedback, ask questions and suggest improvements. To structure the input, we gave each group a list of questions that they could use to stimulate the discussion (e.g., which elements are you missing?), but they were also free to add remarks directly onto a large printout of the framework that they received. Based on the input we received in this session, some small changes were made to the draft of the framework. These were mostly about adding some questions that the participants were missing (e.g., Which tools can be used for reflection?), and slightly changing some phrasing to improve the understanding of the framework for those new to the topic.

Second Iteration: Evaluation of the Framework

To further improve the content and practical usefulness of the developed framework, the next step was to evaluate the framework with researchers from the field, who were not previously involved in the studies or the workshop that led to the framework.

The evaluation was conducted with two different groups of participants: experts from the field, with experience in AR; as well as researchers who recently got involved in AR projects, without much previous experience. This twofold approach was chosen because we expected that the responses from both groups would complement each other. Experts are better able to evaluate the content of the framework based on what they have learned and experienced in previous projects. However, they are likely not going to be the ones using the framework because they rely on their own experience to set up projects. Researchers new to AR, on the other hand, can estimate whether this would be a useful tool for them in practice, but might not yet be familiar enough with the processes of AR the evaluate the content of the framework. Experts were identified from literature (i.e., publications in the field of AR and eHealth), supplemented with online searches for senior academics in the field. The experts were then contacted by mail by the first author. One of the experts who was not available for an interview made contact with one of their colleagues who was then interviewed. Inexperienced researchers were identified via convenience sampling, through two regional projects that the first author was (made) aware of, and were also contacted by mail.

All participants (both experts and researchers) were interviewed, either online or in person, by the first author. Before the interviews, participants signed an informed consent form, stating that the session would be recorded. They also received the link to the (English) framework, to familiarise themselves with the framework before the interview. However, this was not mandatory as there was sufficient time during the interviews to go through the framework. The interviews were semi-structured and started with a brief introduction by the first author of themselves and their work, followed by an introduction of the interviewee. Then, the participants had the chance to go through the framework and provide any remarks that came to mind (similar to think-aloud methods). Afterwards, the first author asked them a set of questions regarding their perception of the framework, and how they think it could be improved.

The answers were inductively coded as recommendations or requirements for the framework. While coding the answers, a distinction was made between answers related to the content (whether this was missing or unclear), the language and structure of the framework, the presentation (e.g., in terms of images) and any comments related to additional materials to be added.

The interviews were conducted with four experts who had longstanding experience with AR, as well as four researchers from practice currently getting started on an AR project. The AR experts had all previously worked in several AR projects. The domains of these projects varied but included studies from the healthcare, active ageing and public health domain. The participants from practice were involved in (one of) their first AR project(s). Three of them worked as researchers at different academic or healthcare institutions, one as a healthcare professional. The interviews lasted on average 37 min (minimum 19, maximum 47 min). The evaluation interviews resulted in a total of 65 recommendations, of which 31 were related to missing content, 13 about unclear content, 12 about unclear language or structure, five about the visual presentation and four about background materials. In Table 2 some examples are provided to illustrate the changes made based on the comments from the evaluation. Table 3 shows the distribution of recommendations across the categories.

Table 2 Examples of the changes made to the framework based on the feedback received in the evaluation
Full size table
Table 3 Overview of the amount of remarks in each category for the different elements of the framework
Full size table

Below, the resulting recommendations for each part of the framework are outlined, including how these were integrated in the second version of the framework. The full overview of all recommendations made by the participants, and the implemented changes based on these recommendations, can be found in Appendix A.

Introduction of Framework and Figure

The comments regarding the introductory text and figure at the beginning of the framework were mostly about unclear content. Participants were missing a clear explanation for the link between this framework and AR as a research approach, and between this framework and similar models. Additionally, participants commented that the cycles of AR are not as clear and structured as they are presented in the framework, and that a project might not have such a clear and defined endpoint. To address these recommendations, more links to literature and other approaches were added in the introduction section. Additionally, a disclaimer was added to the figure to raise awareness that AR is rarely such a straightforward process and that it might not have such clear time- and endpoints.

Element: Project Idea

Regarding the setting up of a project idea, there were several recommendations about content and pointer questions that were missing, for example, the fact that this process takes time, and that ideally the project should be initiated by the community or stakeholders. There were also some recommendations to adapt the phrasing of some of the questions, to make them more realistic and make sure that stakeholders are truly involved (i.e., replacing the word ‘consulted’). Lastly, several participants commented on the fact that the word ‘reimbursement’ can be unclear, with the suggestion to specify that this can also be about nonmonetary aspects, like time, meaning or recognition. These recommendations were taken into account by adding some questions and explanations, and by changing the wording of some questions as suggested.

Element: Plan—Roles and Tasks

For the section ‘Roles and tasks’ of the framework, participants again recommended some missing elements, like: risk analysis, defining clear metrics for evaluating change and creating space for re-negotiations. Furthermore, a recommendation was made to restructure and re-phrase the questions to make sure they are all on the same level and in logical order. One participant asked for clarification of the word ‘champion’ in this context. These recommended items were added to the framework. Also, a link to relevant information on this topic was added, along with links to other approaches and useful methods that were mentioned by participants.

Element: Align Interests and Needs

Participants suggested making it clearer that the alignment of interests is not always possible, and that even subconsciously conflicts might exist. One participant recommended stressing here that being open requires some courage. Lastly, there were two recommendations related to language, suggesting sharper and more active phrasing. Again, these suggestions were implemented as suggested by the participants.

Element: Ethical Approval

Regarding ethical approval, participants made some suggestions on what else should be considered. These recommendations referred to getting in contact with an ethics board early on to discuss procedures, including the European General Data Protection Regulation (GDPR) in this section and adding suggestions for what to do when participants do not (want to) give consent. Questions related to all three topics were added to the section.

Element: Training

Several participants pointed out that the wording ‘training’ might be misleading in this context and suggested renaming the section, for example to ‘Mutual learning and skill development’. Additional questions were suggested, about which skills are needed and about didactics. One participant asked for clarification about what hierarchy means in this context. We renamed the section to ‘Mutual learning and skill development’, added the suggested questions and clarified what hierarchy means in this context.

Element: Act—Celebrate Milestones

Around the topic of celebrating milestones, participants pointed out that it is important to not only focus on the positive milestones but to also learn from failures and facilitate this process. Another participant added that milestones take a different shape in each project. The section was renamed to ‘Celebrate successes and learn from failures’. In this way, we also addressed the remark from one participant that ‘celebrating milestones’ sounded quite formal. Additionally, a sentence was added in this section to raise awareness that milestones can look different each time and for everybody.

Element: Act—Dissemination

There were several suggestions for what to add to the section on dissemination, like including different ways of disseminating and thinking about the reasoning behind dissemination efforts. Additionally, one participant asked to specify which stakeholders are meant in this context, those within or outside of the project, and mentioned that this might be too broad currently. Therefore, a clarification was added that this section referred to external dissemination as opposed to internal communication with stakeholders on the team. The other additions were implemented as suggested by the participants.

Element: Reflection

Participants suggested adding some more information regarding reflection, like how to shape the reflection, and the different levels at which reflection can take place. There were also some requests for clarification, like making it clear what mitigations mean in this context, and adding more explicitly that reflections should take place continuously, and at intervals that make sense for the project. Lastly, one participant suggested changing the order of the questions, to start with the tools that can be used. The questions were reordered and the clarifications and additional information regarding reflection was added.

Element: Sustain Change

An important element of sustaining change that was mentioned by some of the participants was to embed the project outcomes in policy-making, as this can ensure a more lasting effect. However, one participant highlighted that this might impact how flexible a project is to make changes. While ‘Sustain Change’ was presented as the final part of a project, participants recommended paying attention to this aspect much earlier. Lastly, a participant pointed out that wording in this section was inconsistent, and to make this the same across the whole framework. In this section, a question regarding policy making was added, alongside a note that this might impact flexibility. We emphasized the importance of thinking about sustaining change early on, and made the wording consistent for this section.

General Remarks

The general suggestions that participants made were diverse. Some recommended adding more figures and additional literature, as well as examples from practice. Additionally, researchers from practice mentioned that they would like a downloadable document that they could fill in for their project, maybe together with the team. One participant suggested thinking about other ways to present the framework, like through video or a (chat) forum. It was also mentioned that it might be more intuitive to add the headings ‘Plan’ and ‘Act’ as these had two sub-headings each while training and ethical approval were somewhat in between. One participant remarked that a challenge of this framework is to stay realistic by showing people the difficulties they might face without discouraging them. In a similar vein, one participant pointed out the balance between being specific and at the same time presenting something that can be applied in many contexts. Participants mentioned that they appreciated how compact and clear the framework was, however, sometimes they were missing background information and resources. One participant proposed to keep the main framework relatively simple, but to add a list of further reading and materials below the framework. This way, as another participant pointed out, the framework is easy to understand for practitioners. Lastly, one participant mentioned that in their context, an English language version would be less useful and that a translation in their local language would be better. We have created a downloadable version of the framework, and included additional reading at the bottom. The suggestions regarding video or (chat) forum to present the framework will be explored further, but were not yet implemented. A disclaimer was added regarding the generalizability of the framework. We are currently looking to provide translated versions of the framework for download as well.

Discussion

The aim of this research was to develop and evaluate a framework for stakeholder involvement to be used in eHealth AR projects. This has resulted in a framework that includes some explanation, pointer questions and additional reading material to guide (novice) action researchers in their eHealth projects.

A specific characteristic of the eHealth AR context, which we again noticed in the current study, is its multidisciplinary nature. Stakeholders from health backgrounds meet those with technical expertise, and citizens are involved, for example, as patients or caregivers. In a project, these groups might need to overcome differences in language and experience (Blandford et al. 2018). Methods and ways of working that are obvious to some, because these are common in their field, might be new to others outside of that field. We hope that this framework, complemented by a collection of links to different methods and tools, can help researchers in other projects.

The main component of the framework we present are the pointer questions that researchers can answer, alone or with their project team, to investigate the stakeholder involvement in their project. Other frameworks have focused more on the stages (Concannon et al. 2012) or levels (Achterkamp and Vos 2006) of stakeholder involvement. Generally, these frameworks are often kept rather clean and simplified. On the contrary, our framework is much more extensive and goes into detail on some practicalities of stakeholder involvement. This makes our framework more specific and practice oriented, and thus hopefully easier for researchers to apply it in their projects.

One of the main challenges in the development of this framework, which also was mentioned in several of the interviews, was finding a balance between keeping it simple and still providing enough information. Based on the results we have implemented a list of further reading and materials below the framework. We hope that this strikes the balance between too much and too little information.

Similarly, in our results, many recommendations were based on individual experiences and supplemented with specific examples from practice. Therefore, these are likely important points to consider, but not necessarily applicable to other projects. Similarly, readers and those using the framework might find points that are more or less specific to their project. Still, we believe that the way this framework was developed covers the most important general aspects of stakeholder involvement in eHealth AR. However, as we will also outline below, this framework should be seen as a living document that we would like to adapt based on input from those who use it in practice. This might bring up other elements that we did not consider general, but that apply to more eHealth AR projects.

The dual evaluation both with experts and with potential users of the framework (i.e., researchers from practice) was very valuable, as the feedback they gave differed, and we expanded our framework based on their recommendations. How the developers of other frameworks included the target users in the development and evaluation differs. Concannon et al. (2012) describe that they followed an iterative approach and involved a panel of (external) stakeholders to review their taxonomy for stakeholder involvement, however it is unclear how the feedback from this panel was implemented in their work. A good example of evaluating a framework is provided by McAllister et al. (2013), who evaluated their framework with members of their target group, health educators, and used different methods to collect data for evaluation. Schmidt et al. (2020) do not include an evaluation in the description of their framework, but encourage readers to share their experience with it. We have involved both experts and members of our target group for this evaluation, but would like to follow the example of Schmidt et al. (2020) and ask readers about their experience. This way we hope to make this framework a living document, similar to the approach for communicating AR described by Canto-Farachala and Larrea (2022).

In our results the experts provided more of the input on what was missing, while researchers from practice asked more questions to clarify. However, the researchers from practice also added what they were missing from their practice and what they had learned so far. It would be interesting to hear the perspectives of involved stakeholders (e.g., healthcare professionals, patients) as well as experts for specific related topics (e.g., ethics, communication), which could add new elements to the framework as well. We generally believe that it is important to include several perspectives, especially also those of involved stakeholders, in comparable future studies, and would recommend such an approach to others conducting similar research.

The current evaluation of the framework was rather theoretical. Additionally, the interviews provide insights from a specific moment in time. Especially for the researchers from practice, with less experience in AR, it can be difficult to foresee how the framework could be used over the course of a whole project. Therefore, a logical next step to further develop and improve the framework is to implement it in a research project from start to finish, and evaluate this process. This can not only unearth missing content and unforeseen hurdles that a project faces but can also provide more insight into how best to operationalise this framework in practice. Some of the participants already thought about this during the interviews (e.g., would they use it as a checklist for themselves or share it with the stakeholders), but practical experience would allow us to give recommendations on the use of the framework as well.

A strength of this research is that it is not only about AR but also made use of AR principles to develop the framework. The iterative nature and continuous involvement of stakeholders that are described in the framework, were also put into practice during its development. This started with the studies that fed into the framework, which were participatory research projects and involved different stakeholders. In the first iteration of the framework development, we gathered feedback from a group of project partners representing different. For the second iteration both researchers with a lot of experience with AR, as experts from the field were involved, as well as practitioners who were starting to use AR in their daily work, to hear their opinion from a practical point of view. In our opinion this approached strengthened the eventual framework and made sure that it is something that can be used and of value in actual AR projects.

A specific characteristic of the eHealth AR context, which we again noticed in the current study, is its multidisciplinary nature. Stakeholders from health backgrounds meet those with technical expertise, and citizens are involved, for example, as patients or caregivers. In a project, these groups might need to overcome differences in language and experience (Blandford et al. 2018). Methods and ways of working that are obvious to some, because these are common in their field, might be new to others outside of that field. We hope that this framework, complemented by a collection of links to different methods and tools, can help researchers in other projects.

Conclusion

eHealth AR projects can benefit greatly from sharing knowledge, best practices and lessons learned. This can act as a guide for future projects, especially for researchers and other project partners who are new to working in such a participatory way. Through the presented framework we hope to support researchers in setting up and conducting projects together with stakeholders. Based on an evaluation with experts and researchers from practice we believe that this framework contains all important elements of eHealth AR and fits the needs of researchers in terms of presentation and content. Hopefully, the framework can continue to evolve based on future evaluation in practice, and from feedback by those who (want to) use the framework.