An appropriate combination of quantitative and qualitative methods provides better understanding of research problems as opposed to using them separately (Creswell & Plano Clark 2007). Thus, it is widely supported that mixed-methods research approach provides a more comprehensive understanding of the phenomena being studied (Beck 2014; Challiol & Mignonac 2005; Collins et al. 2006; Creswell 2014; Newman et al. 2006; Scott et al. 2007; Youngs & Piggot-Irvin 2012). Specifically, according to the seminal analysis of Greene et al. (1989), the use of multiple methods that we apply in the design and implementation of research has five purposes:
1) Triangulation: the deliberate use of different methods of data collection and analysis when studying a social phenomenon, with a view to convergence and confirmation of the results obtained;
2) Complementarity: the interpretation of different aspects of social phenomena through the synthesis of the results of each method;
3) Development: the results of one method are used to help enrich and develop the other;
4) Initiation: the discovery of any contradictions and paradoxes, which induce the researcher, in the light of the data obtained from one method, to initiate a process of critical treatment and redefinition of those arising from the other;
5) Extending: expanding and deepening a research by using different methods to examine different elements or manifestations of the researched topic.
Triangulating different methodological approaches allows the tracing of different aspects of the research question (Flick 2004) and the logic behind this strategy is that the deficiencies of one method are often the forces of the other. Through their combination, researchers can make the best use of each by overcoming the shortcomings of their individual use. As objective reality cannot be fully captured and understood only through its representations, the use of multiple methods reflects an attempt to understand the phenomenon under in depth investigation (Denzin 2012). Triangulation is not as much a result-validation strategy, but more of an alternative proposal for validation, through which the breadth, depth and consistency of the methodological processes is increased (Flick 2002). Thus, qualitative data could be used in order to develop the questionnaire of the quantitative research and in this way, provide a follow up and well validated study (Cho & Park 2013). Thus, the use of mixed methods in public policy and governance issues could provide multiple benefits concerning the sphere of concepts, the sphere of concrete processes and the sphere of inferences and explanations as we can observe from the following Fig. 2, combined with their different approaches as well as reclaiming the benefits of each approach and overcoming the individual obstacles.
A suggestive and indicative mode of the triangulation method: Using the Quantitative and Qualitative Research Mix.
Regarding the triangulation of methods belonging to different models (between-method triangulation), a combination of quantitative and qualitative methods can be used following a discipline about the research process. Specifically, a safe and usual path to the process could be made up by the following steps:
Literature review correlated with secondary quantitative research, by collecting, processing and analyzing secondary quantitative data from valid and up-to-date sources, to demonstrate the state-of-play;
Primary quantitative research using appropriately designed questionnaires.
Primary qualitative research through the conduction of semi-structured interviews and narrative interviews.
Analysis of the new primary data, comparison both with the literature, the secondary quantitative data and correlation of primary qualitative and primary quantitative data in order to extract conclusions (Fig. 3).
Source: Alassafi et al. (2017)
Primary quantitative research could attempt to detect more general trends and regularities of the phenomenon under analysis which concerns one or more fields of public policy and governance, by investigating in a multitude of cases, specific parameters that could be examined in relation to each other, in order to verify or disprove specific research hypotheses based on the secondary quantitative research that is followed. Special attention should be paid to the respect of the criteria of validity, reliability, representativeness of the sample and as much objectivity as possible by researchers so that the results obtained can be generalized to the wider population. A questionnaire with closed-format questions could be used as a research data collection tool (face to face or online) in order to allow the results to be standardized in the form of measurable variables in order to identify the correlations through their statistical analysis. A pilot check of the questionnaire could be carried out prior to the main phase of the primary quantitative research to make the necessary modifications.
Primary qualitative research could also be implemented in the investigation of the cause of a social problem as well as of the impact or lack of intervention in terms of public policy and governance. In studies aimed at deepening the personal perspective of individuals, gathering details and highlighting multiple facets of the cases under investigation, qualitative research - often in combination with quantitative data - is the most appropriate and effective method. Qualitative research is considered as the most appropriate method for in-depth examination of this topic, as it aims at an immediate approach to the subjective experience of the research participants, in order to demonstrate aspects of the phenomenon under consideration that are difficult to detect and interpret (Mason 2003; Robson 2010; Tashakkori & Teddle 2003). Semi-structured and narrative interviews are mainly chosen as tools for collecting qualitative research data in such cases as they could add more depth to the analysis of a public policy and governance issue.
A semi-structured interview enables the researcher to study an issue in depth and to identify dimensions that other tools are difficult to reach. It can be used as the primary means of gathering information related to research objects, to test hypotheses or to suggest new ones, or as an interpretive tool to help identify relationships between them (Cohen & Manion 1994). The semi-structured interviews do not use a standard questionnaire, but a pre-defined flexible framework that will define (but not severely limit) the interview topics. The forms of the possible questions are the following:
• Open questions, which enable the respondent to develop his/her response with a great deal of freedom without any particular restrictions by the researcher,
• Opinion questions, designed to explore the attitudes and opinions of research participants on the issues under consideration; and.
• Elaboration probes, as they encourage the respondent to refer more in depth to issues that arise during the interview.
A narrative interview is a type of open, unstructured interview, in which the respondent is asked to recall personal experiences and utilize the unstructured narrative to present his/her biographical path or a specific period of his/her life related to the social-political-economical issue under investigation. In this case, respondents will be asked to relate their personal experiences in light of the institutional changes that have occurred and the wider socio-economic context. Through this tool they are given the opportunity to formulate their narrative spontaneously and freely express their comments and opinions. The questions aim at prompting the interview to begin and evolve smoothly (Mason 2003; Robson 2010; Tashakkori & Teddle 2003). Except the abovementioned tools, focus groups, panels, field observations, intervention actions etc. can also be used based on the research question and in relation with the specific field of the research but in this paper we will mainly focus on the semi-structured and narrative interviews because the primary qualitative research using these tools could be applied on research subjects who are involved in different levels and ways on the field which is under investigation.
Using triangulation and mixed methods in public policy and governance studies: divergent cases, similar strategies, different tools.
As objective reality cannot be fully captured and understood only through its representations, the use of multiple methods reflects an attempt to understand the impact or the necessity of specific public policies as well as to test the performance and the function in terms of governance. Hence, triangulation is not as much a results-validation strategy, but more of an alternative proposal for validation, through which the breadth, depth and consistency of the methodological processes is increased through (a) data triangulation, (b) investigator triangulation, and (c) methodological triangulation, in the version of triangulation of methods belonging to different paradigms. The literature supports our point of view about the use and the importance of the triangulation research method in in public policy and governance as several scholars in this field have used it successfully, of course investigating different issues. It should be noted that the field of public policy and governance is relatively vast and in order to be more specific we focus on public policies which aim at the social integration of specific social groups as well as in governance performance and function measure in local administration and in the non-governmental sector. Thus, specific cases from the available literature were chosen in order to emphasize both at the importance of triangulation and mixed methods in addressing issues of governance and public policy responses. We have categorized the cases in those concerning local-regional governance, those studying public policy necessities for social integration and those focusing on aspects of governance in non-typical organizational contexts.
Cases studying local-regional governance
Based on the above considerations, the first paradigm is derived from the research in order to analyze aspects of regional governance and sustainability in the Region of Crete, Greece (Papadakis et al. 2018). The research focuses mainly on the three dimensions of sustainability, namely the environmental, the economic and the societal, and its interconnections with regional governance. The methodology included large scale qualitative and quantitative research in the Region of Crete, Greece. Triangulation was also used through the conduction of structured quantitative questionnaires to citizens in the respective region, semi-structured interviews with institutional heads and narrative interviews with critical cases in the regional governance. The results of the different research methods were separately analyzed in order to reveal the main outcomes and finally a synthetic analysis was conducted which categorized them and showed similarities or differences. Using this methodology, the research has categorized the problems, challenges, opportunities and drawbacks in terms both of social, economic, political and environmental nature while analyzed the main policy making context and the ways that addresses the problems, evaluating though its effectiveness through citizens’ opinions. The added value of the mixed methods used was the identification of problems in regional governance that were analytically presented by the institutional heads as well as by critical cases which were interviewed. If just a quantitative questionnaire was distributed all this information, which is derived through the description of their experiences in the interviews, would not have been revealed in the fields of regional governance decision making, policy implementation, bureaucracy, priorities set process, skills and needs-in-skills as well as capabilities in problem solving. The strength of the mixed methods used was mainly the ability to identify the key problems of regional governance in the Region of Crete, Greece as perceived by citizens, local policymakers and local government officials. This feature made it possible to distinguish between the perceptions of individual research subjects, as well as the ability to identify the sources of problems by verifying or not the different approaches or identify the reasons of different perceptions on these issues. Finally, the research offers new scientific knowledge on the abovementioned issues while it comes to empirically grounded policy proposals on different areas of public policy (regional economy, social policy, environmental policy, infrastructure), which are extremely important for regional policy makers as long as they are directly derived from local citizens and institutional stakeholders.
In a similar path, Olivier (2017) uses mixed methods in order to evaluate the performance of local governance in South Africa. The study aims also to test the usefulness of mixed methods on the evaluation of institutional performance. The author uses the Burke-Litwin (1992) model of organizational performance which includes the external environment, mission and strategy, leadership and culture, as transformational factors affecting organizational performance, as well as, structure, management practices, systems, work-group climate, skills or job match, individual needs and motivation, as transactional factors affecting organizational performance. These factors were used in the quantitative and qualitative part of the research through a structured questionnaire (Organizational Performance Questionnaire) and semi-structured interviews respectively. The qualitative data was analyzed using two ways; the responses were measured in terms of response rates (Vitale, Armenakis & Field 2008) and a thematic analysis was conducted in relation to a research question, following Braun and Clarke (2013) identification. By using mixed methods Olivier (2017) has depicted the institutional performance, validated through different methodological means, and thus offered a lucrative ground for reforms towards the improvement of local governance performance in South Africa. Instead of using a single method, the mixed methodology allowed the authors to come to an holistic understanding of all aspects of local government function in South Africa, validated the questionnaire by comparing the interview data to the questionnaire data and used a validated organizational performance model in the data gathering process in order to address all elements of this organizational functioning while provide the appropriate information for the design of improvement interventions. The basic strength of the method used is the fact that on the one hand the quantitative and qualitative data have been validated with each other and on the other hand, a method has been used that leads to the understanding of local government function and the creation of empirically grounded policy proposals-interventions.
The study of Ghinoi et al. (2021) uses mixed methods in order to analyze the use of smart specialization strategy (RIS3) in Lapland region, Finland and propose alternatives for its effective implementation. The study uses triangulation of quantitative and qualitative data aiming to reduce limitations and enforce verification. Quantitative data about RIS3 from the Regional Council of Lapland and online data from regional stakeholders were used along with qualitative in-depth interviews with representative stakeholders who were active participants in the development of the smart specialization strategy. The analysis show major implications in the implementation of smart specialization strategy leading to empirically grounded policy proposals for adapting the strategy to the stakeholders’ views, necessities while improving informal coordination. The added value in comparison with a single usage of a method, is that mixed methods allowed the verification for potential source biases. An examination of RIS3 in the Regional Council of Lapland was implemented followed by interviews with representative stakeholders, active in the development of the RIS3 and online quantitative survey with regional stakeholders. Validation and verification of the research data is the basic strength of this mixed methodology.
Cases studying public policy necessities for social integration
Mixed methods have been also widely used in order to study social phenomena-problems and draw conclusions about the necessities for interventions-policy responses in terms of public policies that should foster social integration. Accordingly, the study of Kotroyannos et al. (2019) on the implications for social integration of refugees in Greece, has followed the analysis of both qualitative and quantitative data. The data was collected for two Greek regions (Lesvos and Crete). The qualitative research included semi-structured interviews with labor market stakeholders and narrative interviews with integrated immigrants while the quantitative research was conducted face to face with refugees recently arrived in Greece. Triangulation was used both on data, investigator, and methodology as well as it includes the responses of different but interconnected groups, using different investigators and different methodological tools for each group. The data was analyzed, categorized and compared. The outcomes of this research offered empirically grounded policy proposals for policy interventions towards refugees’ social integration which also will benefit the respective local communities. The added value of this method is the in-depth investigation of refugees’ narrations in order to identify the problems as well as the chances for their social integration. At the same time, semi-structured interviews were important in order to identify labor market stakeholders’ perceptions and stances towards refugees’ socio-economic integration. The quantitative research on refugees identified their human capital, which was compared with the needs of the labor market stakeholders as it was identified in the semi-structured interviews. This research methodology allows greater validation of the research outcomes and identify the differences in perceptions towards refugees which may affect their integration in the labor market. At the same time, a comparison between the refugees’ human capital and the skills needed by the labor market was made possible as well as any negative perceptions (and their provenience) towards refugees on the part of labor market stakeholders was revealed as well as its relation with refugees’ integration.
Another social vulnerability problem (young people Not in Education, Employment or Training-NEETs) and its relevant empirically grounded policy proposals, is studied by the seminal research project ‘Absents’ barometer’ which is the first nationwide (Greece) primary research on the social problem of NEETs in the European Union, using mixed methods (Kotroyannos et al. 2015; Papadakis, Kyridis, Papargyris 2015). This research included a quantitative part using a structured questionnaire to this category of young people as well as a qualitative part, using semi-structured and narrative interviews to the same people. It is based on triangulation but, contrary to the previous research case, the sample consists only the group of NEETs. The outcomes are crucial as long as they show the social vulnerability problems of this group of young people and the policy proposals offer important alternatives in addressing the problem. The research is innovative and offers to policy-makers new scientific data and empirically grounded policy alternatives that could be used by the welfare state institutions in order to increase the possibilities for social integration of these young people. As in the previous cases, validation and verification of the outcomes is the main strength of this methodology along with the focus on respondents’ narrations which reveal life-course details, which are difficult to be depicted on the quantitative outcomes.
Cases studying governance in non-typical organizational contexts
Governance patterns and issues are also crucial in the non-typical contexts, such as Non-Governmental Organizations (NGOs) and community cooperatives. The way they are governed and formulate their policies within the greater concept they function is also crucial as long as the third sector (non-governmental) is gaining an increasing part of the social-public intervention during the recent decade and due to the retrenchment of the typical social policies offered by the welfare state institutions (Wollman 2018).
Therefore, specific cases were chosen in order to emphasize both at the importance of triangulation and mixed methods in order to develop capacity building for people involved in an organization of the third sector, as a means for functional effectiveness and good governance. It should be pointed out that triangulation and mixed methods have generally been used in order to develop capacity building in NGOs and are considered as crucial strategies in order to provide better services. Several organizations and scholars indicate that the mixture of qualitative and quantitative methods and triangulation offer the depth that is necessary in order to assess and further develop capacity building in NGO members. The narration that qualitative research includes, in combination with the quantitative data, is necessary in order to facilitate organizational learning and governance. According to CDRA (2001) and Taylor (2003), it assists staff as well as volunteers with the engagement of new ideas as well as new and innovative practices. Accordingly, Fowler (1997), Roche (1999) and Taylor (2003) indicate that mixed methods using triangulation could better evaluate capacity building in NGOs and assess impact. A distinctive example of using triangulation in practice from an NGO is ActionAid Uganda (Wallace & Kaplan 2003), which included narrative interviews with its officials, along with the quantitative questionnaires, in order to explore their feelings about the change they have been part of (Hailey & James 2003). This method allowed the verification between the qualitative and quantitative data. At the same time, a strength of using narrations along with quantitative questionnaires, created the opportunity to better describe the feelings towards the change they have been part of. This is extremely important as long as the quantitative questionnaires are not able to “catch” all the possible feelings that an individual has while narrations gave the opportunity for detailed explanation, description and justification from the part of individuals.
In a similar context, Lang and Roessl (2011) in their study on the conceptualization of governance in Community Co-operatives in Austria and Germany, used mixed methods and triangulation in order to acquire an in-depth understanding of the intrinsic characteristic of governance mechanisms in such organizations of the third sector. Their tools are different than that of the previous paradigm in order to better adapt to their research question. They have chosen two cases of neighborhood organizations (co-operatives) from Austria and Germany and they applied different qualitative tools as long as they offer a better understanding on the social process in the context of different perspectives and specific time (Yin 2009; Sayer 1984). For this purpose they used semi-structured single and group interviews, field observations, and archival data, in order to perform triangulation and achieve the better validity and reliability of the outcomes. Instead of conducting a research with a single tool, the authors have chosen to use archival data, field observations and interviews as means of validation, which is actually the strength of the used methodology.
The third characteristic research case that uses mixed methods and triangulation is a study about social enterprises in Austria (Lehner 2011) with social entrepreneurs. This research uses the theoretical framework of social enterprises in Austria in order to theoretically justify the concept of the study. The research approach uses triangulation as well as mixed methods as its basic strategy. The research tools include an online-based survey, semi-structured interviews and two panel discussions. The online questionnaires included both qualitative and quantitative questions and the author conducted semi-structured interviews with relative practitioners and experts. The third tool that was used included two moderated panel discussions about social entrepreneurship in Austria. It should be pointed out that literature review, which was carried out at the first steps of the study, led to the determination of the framework for the combined results. This methodological approach led to the conduction of a holistic analysis-view of the conceptual underpinnings about social enterprises in Austria. Validation of the outcomes is the main strength of the followed methodology. The synthesis of qualitative and quantitative data covered any data gaps and aspects about their experience in social entrepreneurship governance and functional characteristics.