Background

Scholars within the health [1], aviation [2], manufacturing [3], finance [4], education [5] and military [6] sectors have argued that collaborating communities and linked, professionalised networks are significant determinants of important outcomes such as work satisfaction [7], motivation [8], recruitment and retention [9], high performance [10], organisational resilience [11], safer organisations [12] and systems renewal [13]. However, there is a paucity of empirical research, as opposed to normative claims in the literature, to support these arguments. Given the substantial resources invested in supporting collaborating structures, the absence of evidence of their effectiveness is a significant problem. This is especially so given the potential benefits of optimizing the performance of large, complex sectors, such as the health system, which is the specific focus of our project.

Communities of practice can be defined as "groups of people who share a concern, a set of problems, or passion about a topic, and who deepen their knowledge and expertise in this area by interacting on an ongoing basis" [14]. Social networks are "set [s] of people ... ['actors'] ... with some pattern of interactions or 'ties' between them ... [eg] friendships among a group of individuals, business relationships between companies" [15]. In one sense they are two sides of the same coin, but communities of practice scholars have tended to emphasise learning processes while social network scholars have focused on underlying structural properties.

These kinds of clustered relationships abound in health care and other industries. They can be naturally-occurring [eg, groups of clinicians who share professional responsibilities, refer patients to each other, jointly manage care and collaborate over complex cases] or mandated [eg, purpose-designed, sponsored, initiated and funded by organisational leaders] [16]. They can also be tangible and geographically-anchored [eg, in one section of an organisation such as a teaching hospital or general practice] or virtual and geographically-dispersed [eg, international colleagues with common interests linked via telecommunications]. Terms for related phenomena include teams [17], sub-cultures [18], micro-systems [19], inter-professional practices [20] and integrated services [21].

Communities of practice [14, 22] and social-professional networks [23, 24] are increasingly seen as crucial determinants in understanding and enhancing group-oriented services. Most literature supports a position that, although there is much to understand, services to patients [clients, customers, purchasers or consumers in other industries] and workplace cultures can be expected to improve where communities of practice and social-professional networks are emphasised and strengthened, whether these are naturally-occurring and emergent, or mandated and purpose-designed networks [16]. These tend to be theoretical or conceptual claims, however, rather than concerted empirical demonstrations.

At the same time, potentially negative consequences have been noted. Wenger and colleagues argue that communities of practice can encourage the hoarding of information to the detriment of others, curtail improvement and progress, and create in-group and out-group rivalries [14]. Buchanan suggests that networks can be a negative as well as positive force, and can mean all the eggs are in one basket. For instance, if a network [eg, electricity grid, computer system, or clinical collaboration] fails, deleterious consequences can result [23]. Communities of practice and social-professional networks can marginalise others, create factions and silos, become vehicles for elitism or isolationism, strengthen us-and-them behaviours and impede organisational sharing [14, 22–31]. Thus we should not accept that communities of practice and social-professional networks are universally useful.

Although interest in communities of practice and social networks in industry settings has grown remarkably, no convincing model for their systematic evaluation has eventuated. Many authoritative commentators agree that analytic evaluation methods and tools are lacking [32–34]. Thus, there is strong consensus amongst scholars that it is vital to develop an appropriate evaluation framework and model with supporting data-gathering tools. If we are to be confident that communities of practice and social networks enable organisational success, and to understand how and to what extent they do so, we need to be able to quantify their contribution to satisfaction, motivation, recruitment and retention, performance, resilience, safety and renewal. Conversely, if these kinds of collaborative structures prove less useful than expected, we need to understand their limitations.

The case for the project

We reviewed the literature on communities of practice and social-professional networks. Significant researchable characteristics of communities of practice include how learning is enabled and constrained, what can be achieved by actively cultivating them, their relationships to organisational learning, interactivity amongst participants within and between communities of practice, how members join and are socialised, the mechanisms of participation, and the stages of their development [14, 22, 28–30]. The literature indicates that the core features of social-professional networks are their deep structure, how they enable change, how information is facilitated by and in them, the processes of information distribution and search, the circumstances in which phase transitions occur, connectedness of hubs and nodes and the conditions which dictate their robustness or fragility [23–27]. In our review of social-professional networks, we applied some of these central concepts to key patient safety issues [16]. We concluded that "Clinicians, like other professionals, work best when they are allowed to flourish in groupings of their own interests and preference, are empowered ... and nurtured and influenced by their peers rather than controlled by others" [16].

We also conducted a comprehensive analysis of the communities of practice literature pertaining to the health system [35]. We examined the Medline, CINAHL and Embase databases and performed a snowballing search for additional literature. We identified 624 references of which 90 were research papers. Content analysis of the 624 publications via Leximancer, a data mining tool, allowed us to construct Figure 1 which maps the overarching themes in the literature - the social and professional nature of communities of practice, and their strong relationship to service provision and treatment. There are five major themes in the 90 research papers: organisational change [eg, how communities of practice promote productive change, and how change effects communities of practice]; the impact of technology on communities of practice; the contribution of communities of practice to professional and inter-professional learning and the shaping of organisational knowledge; learning processes within communities of practice, including mechanisms of collaboration, structured reflection and the development of shared competencies; and barriers to participation in communities of practice, particularly mono-professional, organisational, institutional and service boundaries and silos. We concluded our review by synthesising the literature as follows: "The systematic analysis reveals that the community of practice idea has been empirically applied in the health sector in useful ways, but research is sporadic, and many studies are descriptive and centred on one profession or location. The transferability and generalisability of findings is limited. A difficulty with the empirical studies is ... a lack of specificity in their claims. For example, at times the argument is made that team work has improved but what 'good team work' means is not clarified. There is scope for more far-reaching, rigorous and systemic research" [35]. This background work leads to the current protocol.

Figure 1
figure 1

Map of the content analysis of the health communities of practice literature.

Full size image

Aims

This project's specific aims are to develop and design, test in the field, and then refine, simulate and apply a framework, model and tools which can be used to evaluate communities of practice and social networks for their effectiveness and sustainability. We will work in the health system for compelling reasons. As a large, diverse industry (9.3% of GDP)[36] the health system supports many types of policy, managerial and clinical communities of practice and social-professional networks comprising diverse occupational groups, and traverses the public and private sectors. These characteristics and complexities mirror other key industries, greatly enhancing the potential to transfer findings, theories and project outcomes beyond the health sector.

The project thus more broadly aims to realise a scientific innovation of importance. Without an evaluation framework, model and tools, sponsors will continue to be uninformed as to what communities of practice and social networks deliver; will be unable to identify and measure their strengths and weaknesses; and will be frustrated in their efforts to achieve desirable outcomes, and returns on investment. This has profound implications for how communities of practice and social-professional networks are enabled, configured and resourced. Such research is at the heart of a crucial question: how well do groups of professionals work together and how could they be organised to work together more effectively?

The research strategy and process are novel and innovative and are designed to produce original methods in the building of a new framework, model and tools. These include: the engagement of influential stakeholders and consumer groups; theoretical and empirical advancement; a sophisticated new research approach for doing future work; an innovative, tested, refined framework, model and tools for researching and evaluating communities of practice and social-professional networks; and the gathering of original data from the field using the tools to illuminate progress. Our data gathering tools [questionnaires, interview schedules, case study designs, focus group processes, critical incident mapping protocols] will employ new approaches.

We will inform our work using theories on teams [17], inter-professional learning [37], network structures [25, 38], change [39] and group dynamics [40]. These are fundamental concepts to our enquiry, guiding our thinking, particularly in the initial and model-building phases. We will, however, mainly inform our project using the empirical literature to rigorously build our research designs on the basis of prior research results. Few research projects are both empirically and theoretically multi-modal in the one study.

Methods and design

Conceptual framework

This is a comprehensive effort to execute a rigorous evaluation methodology, and deliver tools for the benefit of researchers, policymakers, practitioners and consumers. We have the opportunity to achieve both meaningful and transferable advances in the knowledge base of the discipline and to incorporate novel and innovative aims and concepts into the project design. Through the support of key stakeholders within the Australian health system, we will have access to a wide range of workplace communities and networks, including hundreds of clinicians, to test and refine our models. The overarching aims, phases, objectives, methods and outcomes in the approach are depicted in Figure 2. Ethics approval for the study was granted by the University of New South Wales' Human Research Ethics Committee, application number 09085. Informed consent from participants is a key component of the approval.

Figure 2
figure 2

Conceptual framework, design and methods.

Full size image

Methodology

The methodology is conceptualised in four major phases, matched to the aims and objectives and commensurate with the research approach. Each phase has two components, and involves a little over a year's work to achieve the project's objectives.

Phase 1: development and design phase

The project will produce an evaluation framework founded on empirically-grounded evaluation principles. Initially we will drill further into the literature and derive from this assessment a set of fundamental, working evaluation principles. Work feeding into this process includes seminal contributions by Wenger and colleagues [14, 22, 30] and Watts [41], Strogatz [27] and Barabási [25] as well as our work in health systems social structures and change [42–44].

Secondly, armed with the initial framework, we will design a preliminary evaluation model. There are many evaluation types [45], but our previous research suggests that a triangulated, multi-method model is best suited to this problem. Our earlier work on the evaluation of education processes [44, 46], information management [47] and information and communication technologies [48] demonstrates this kind of model can be comprehensive, have high utility and yield robust data. This phase includes the design of data-gathering tools [including questionnaires, interview schedules, case study protocols for video ethnographic work, critical incident maps, and focus group question schedules] to make assessments of communities of practice and social networks, taking into account recent designs [eg, Verburg and Andriessen's] [49]. A key step is the identification of outcome measures to assess the effectiveness of communities of practice and social-professional networks. We have identified indicators for work satisfaction (5 indicators) [50], motivation (6) [51], recruitment and retention (5) [52], high performance (18) [53], organisational resilience (5) [11], safer organisations (7) [16] and systems renewal (11) [54, 55], totalling 57 indicators against which metrics are designated.

Phase 2: testing and refinement phase

Thirdly, in phase 2, we aim to test the model and tools across a focused sample of communities of practice and social-professional networks. Peak Australian stakeholder groups have agreed to involve a wide range of members and participant organisations in this study. These include networks or communities of clinicians identified by the National Institute of Clinical Studies, accreditation surveyors of the Australian Council on Healthcare Standards, managerial members of the Australian College of Health Service Executives and clinical, managerial and policy groups associated with the Clinical Excellence Commission. We will identify, in conjunction with these stakeholder partners, representative samples of differing types of communities of practices and social-professional networks (naturally-occurring, mandated, geographically-anchored, geographically dispersed) and enrol them in our study. We aim to make comparisons between groups and across group types.

The testing component will yield experience which will, fourthly, enable us to refine the tools, including their validity and reliability. Differing approaches to examine validity and reliability will be taken. Work on the reliability of social network data is important, and optimum outcomes can be achieved by re-applying social network questionnaires and comparing consistency of results [56–58]. We have validated questionnaires and used multi-method approaches to corroborate discrete methods convergently [44, 46–48, 59].

Phase 3: simulation and prediction phase

Phase 3 involves two components: fifthly, simulating the systems dynamics [60] of communities of practice and social networks and sixthly, testing the models' capacity for prediction [61]. Communities of practice and social networks are complex adaptive systems within larger complex adaptive systems. They can be modelled via simulation tools that facilitate the mapping of variables within and external to them. This enables us to understand relationships between parts of the system and to run predictive simulations. Governing features of complex systems include feedback, processes, resources, accumulation of flows into stocks and time delays [62]. In the case of a community of practice of emergency clinicians, for example, key variables include levels and training of staff, workforce casualisation, delays in processes, incentives and disincentives, and factors which enable and constrain behaviours and attitudes. In this phase we will also conduct controlled simulation experiments. We will develop role plays [n = 20] of case scenarios and test in the laboratory the dimensions of role behaviours and decision-making processes under controlled conditions. Theorised barriers and enablers will be tested in idealised circumstances in this way to analyse the scope of the boundaries and assess hypotheses and predictions.

Phase 4: application and outcome phase

Having derived the refined evaluation framework, model and tools we will seventhly apply them on a large scale to a sample of communities of practice and social networks identified across Australia, covering the public and private sectors, in order to achieve a sample broadly representative of the health care system. In this phase we will gather sufficient data on communities of practice and social-professional networks to assess their value, and judge the extent to which they contribute to work satisfaction, motivation, recruitment and retention, performance, resilience, safety and renewal. We will test the results and calibrate them against the predictive simulation data drawn from phase 3.

This leads to our eighth component - a review of the research outcomes to interpret results and make final observations on the findings. We will use an arm's length, expert panel to triangulate judgements. Such a process further strengthens project outcomes, closing the feedback loop and helping to determine whether communities of practice and social-professional networks realise the results intended for them.

Discussion

This protocol provides a conceptual framework and research design for evaluating hard-to-research social science manifestations which are key determinants of health systems performance. A multi-methods, phased research design is important and innovative, and has been matched to the phenomena under investigation. This represents a bold approach to creating and then testing an evaluation model.

Study strengths and limitations

Strengths

Investigating and evaluating complex organisational constructs such as communities of practice and social-professional networks is challenging. Conducting research in naturalistic settings, examining behavioural and attitudinal factors in context, is a key strength. The research design recognises this, and applies a complex systems approach to the examination of complex systems phenomena. Repeated, staged testing of progress with the model introduces validity, reliability and rigour. Incorporating measures of organisational performance in the design increases the potential for robust correlations. Having undertaken extensive literature analyses of both communities of practice and social-professional networks, we have laid a sound platform to aid understanding and provide theoretical and empirical building blocks to our research studies.

Limitations

Objective measures are hard to achieve in social science research, as is generalisability, thus we have incorporated triangulation techniques in our design and seek transferability of findings rather than strict generalisability. Nevertheless, our results by definition will fall short of the gold standard, and will not produce level 1 evidence. Balancing this limitation is the reality that a strict test of this kind cannot be applied against research in this mould. Thus, other safeguards, including seeking data from multiple sources and assessing the extent to which they converge or diverge, constitute an effective, alternative approach.

Conclusion

This is a large, comprehensive research project which will deliver outcomes both specifically relevant to health systems and transferable to the large number of organisations, systems and sectors which are affected by the effectiveness, or otherwise, of communities of practice and social-professional networks. In developing a rigorous evaluation methodology and delivering tools for researchers, policymakers, practitioners and consumers, the research project also demonstrates innovation and tests novel approaches in design. By interpreting and translating results for applications both within and beyond the health system, the project will advance understanding of how professionals work together, and how to optimise this interaction to improve organisational effectiveness and service delivery.