Research Methods for Public Policy

Abstract

This chapter examined the nature of public policy and role of policy analysis in the policy process. It examines a variety of research methods and their use in public policy engagements and analysis for evidence-informed policymaking. It explains qualitative methods, quantitative methods, multiple and mixed-method research. Other issues addressed include causal research in public policy, report writing and communication and related issues in public policy research.

Introduction

As implied by the topic, this chapter focuses on research methods applied or applicable in public policy research. Though the overriding focus is on specific research methods, we deemed it necessary to preface these with a brief discussion of the nature of public policy research and the nature of policy-engaged research problem or question. These are then followed by the specific research approaches or traditions and methods as applied to public policy. Given that public policy research deals with issues that have important implications for the society, the mixed-method research is often preferred as a means of arriving at findings and conclusion concrete and reliable enough to serve as a basis for policy. For this reason, we devoted a section to mixing methods in public policy research. This chapter is thus divided into four sections:

• (a) Nature of public policy research

• (b) The policy-engaged research problem or question,

• (c) Specific public policy research approaches and methods, and

• (d) Mixing methods in public policy research.

In the first section, we focus on the definitive characteristics of the kind of research that supports or evokes public policy, especially the solution-orientation of such research. In the second section, we focus on what it means for research to be policy-engaged—which is different from being policy-relevant. We propose the nature and source of a good problem or question for policy-engaged research and its basic design. In the third section, we focus on the two broad traditions of research: qualitative and quantitative traditions, and the specific methods under these traditions. We explain how these methods are used in public policy research using both hypothetical and existing examples. In the last section, we discuss mixing research methods in public policy research, stressing the reasons for it and summing up the process of doing it.

Nature of Public Policy Research

Public policy research is one whose primary aim is to understand or explain social, political, economic, cultural and other issues that are significant to the society and which require the intervention or attention of policy actors. In providing an understanding of such issues, the research also presents itself as a trustable basis for the actions and interventions of these policy actors. It must, therefore, be a piece of research based on sound evidence, produced out of convincing rigour and woven from start to finish around a societal issue of concern.

In addition to being thorough and trustable, public policy research must also go beyond describing a problem or situation into engaging the how and why of things (Osifo 2015: 149) for it to establish causality with reference to a given problem and the options of addressing such a problem. Descriptive studies do sometimes provide an important basis for policy; however, causal studies often interest and command the attention of policy actors more than descriptive ones do.

A good public policy research is sensitive to both the policy and political agenda. These two environments or elements determine action or inaction. Howlett (2012: 451) argues for an approach that encourages absorption of research outputs at two levels: enhancing instrumental arguments about policy programme content and ensuring a deeper political engagement experience.

Though policy makers do not entirely depend on research to make decisions on policy options (Edwards 2004; UK Cabinet Office 2009), the role of research, and specifically field-based research, in public policy remains critical (Mead 2005; Young 2005). Since scholarly research competes with expert knowledge, domestic and international policy, stakeholder consultations and evaluation of previous policies, among other sources (UK Cabinet 1999), evidence generated from research that is meant to inform public policy needs a strong basis for argument on the problem under scrutiny, as well as a variety of policy options from field evidence.

Recent studies show that research in policymaking over the last four decades plays a less direct role than is often assumed and expected (Howlett 2012). Nevertheless, the role of research in public policy is not to be downplayed, and as Mead (2005: 535) explains, field research is essential to realistic policy research that ties governmental action to good outcomes. However, we need to take cognizance of the fact that, as Tierney and Clemens (2011) argue, many of today’s most pressing policy issues are extraordinarily complex and will benefit from carefully conceived and analysed studies utilizing multiple methodological approaches. Public policy researchers should understand this complexity of policy problems. This complex web determines, to a great extent, what forms of research and/or research methods a researcher should consider.

Literature shows that in the history of public policy research, statistical evidence was very important (Mead 2005). Studies meant to inform policy were therefore mostly, if not always, survey-based (Mead 2005). Survey-based research, as Mead (2005: 544) shows, is good at generating accurate depictions of the clientele served by a given policy. Social problems and their correlates can be clearly captured. Earlier approaches to policy research favoured output that could be generalized across settings that were validated and reliable. In those early approaches, quantitative research, especially survey, was given priority. Qualitative research did not so much move into policy arena and research evidence from qualitative studies did not seem to find a place in policy discussion tables (Tierney and Clemens 2011: 59).

Over the years, survey-based approach has been criticized for its narrow economistic approach because social problems are complex. The argument is that survey-based policy research projects onto its subjects, the psychology assumed by the quantitative researcher. Simply put on its own, the approach lacks the ability to explain why and how complex social problems arise, and what public policies would best be suited to address them in their complexity. Surveys, for instance, may not give the full range of information required to account for the behaviours of the poor, needy and dependent persons in certain circumstances. These people, though challenged by certain economic factors, can survive in difficult circumstances, but the how and why of their survival would be beyond the easy reach of survey. Thus, as Mead (2005) argues, there is need for a more complex and robust approach that incorporates those factors that are beyond the statistics. We argue that for a public policy research to claim authenticity of findings that capture the attention of policy makers, and subsequently inform the policy process, an integration of research methods, that is, mixed-method design, is important.

Public policy research is meant to provide solutions to social and public problems that are in many ways complex. Establishing causes and effects of these problems run beyond analysis of existing policies. Mead (2013), for instance, argues:

[Where] texts in public policy devote attention to both policy analysis and political analysis; they fail to capture the intimate connection between them. The two subjects appear as separate worlds, when they are really two sides of the same coin. The texts do not consider that political constraints should really be part of policy argument or that the policy-making process can sharply limit what best policy means. And in research on public policy, there is even less sense of policy and politics shaping and reshaping each other. Typically, the usual division prevails where economists recommend best policy while political scientists explain what government does. (p. 393)

These views relate to the policy and politics dichotomy, and how political analysis is good in reshaping policy analysis (Mead 2013: 392). While it is important to pay attention in public policy research to how these two influence each other, it is also important to pay careful attention to the stakeholders. Good research methods for public policy should engage stakeholders in the research process to enhance the use of the research findings and recommendations for effective policies. Besides the policy makers, policy actors include the public, which is always at the receiving end of the end products of public policy research are important. Consultations with them at most, if not all levels, help researchers to articulate policies that include their ideas or address their concerns (Oxman et al. 2009) and result in the good policy performance.

The Policy-Engaged Research Problem/Question

With reference to their level of policy engagement, public policy research in Africa can be categorized into three: public policy-appended research, commissioned policy research and public policy analysis. Public policy-appended research is the most common of the three. For most African researchers, there is a mandatory section of their article or thesis that presents policy recommendations. In that section, researchers attempt to point out how their research findings can be applied to real-life policy situations and consequently change those situations for the better. Efforts are made by experienced researchers to ensure a close fit between the recommendations and the findings that precede it in the article or thesis. As common as this genre of public policy research is, it is a flawed approach for many reasons. The approach treats policy not as the centre of the research but as an appendage. Put differently, the researcher decides her or his research problem and question and decides on the methods most suitable for this. At the conclusion of the research, she or he then turns to policy actors with recommendations. Since the research was not informed by a policy need or gap, it can hardly fit into the existing agenda and conversations among policy actors. It neither speaks the language of policy actors nor considers their priorities. The researcher would not have attempted to include policy actors at most, if not all, stages of the research, and as we will discuss shortly, there are consequences of not doing this. It also assumes that policy actors (i.e. policy makers, civil society and other stakeholders, including citizens) are on the lookout for policy recommendations from researchers and can wade through the different sections of the research to find these recommendations. As Oyedele, Atela and Ojebode (2017) opined, this is hardly so. The researcher’s research is her or his business, not that of the policy actors. As a result, policy actors do not access the tonnes of policy recommendations made by researchers.

Commissioned public policy research projects are initiated by government agencies and non-governmental organizations to address specific policy or implementation problem. The driving research question and the nature of the expected findings are articulated by the commissioning organization. A critical objection to this genre of public policy research is researcher’s autonomy on crucial fronts. To what extent can a researcher turn out findings that conflict with the political aspirations and public image of the funding government or its agency? How can the researcher be sure that his or her findings are not spun or twisted in favour of government? Therefore, while the findings and recommendations of this genre of public policy research are likely to be more easily accepted by policy actors than the findings of public policy-appended research, there is usually a cloud of doubt around its objectivity and integrity.

A third genre of public policy research deals with policy analysis. These studies take on an existing policy and subject its components to critical analysis often conjecturing whether it would produce expected results. They explore inconsistencies, systemic barriers and feasibility of a policy, and then draw conclusions as to why a policy works or does not. They may serve as formative or summative studies depending on when they are conducted in the life cycle of a policy. The challenge of this approach to public policy research has been that the researcher/analyst is basically tied to the outcomes of policies in existence—policies that he or she did not play a role in formulating.

The foregoing genres of public policy research are, at best, only partially policy-engaged. They may be policy-relevant, but they are not policy-engaged. So, the questions for us here are: What is policy-engaged research? How does it differ from policy analysis, commissioned public policy research and public policy-appended research? What is it that the other three misses out that policy-engaged research is good for? And how do we then design research in a way that the methods used are relevant in informing the public policymaking processes?

A policy-engaged piece of research derives its roots from the questions that are being asked in policy circles. As a response to current public policy issues, it is driven by a research question that explores, extends or clarifies a policy question or problem. Policy-engaged research therefore means bringing on board the stakeholders relevant in the development of a given public policy (Lemke and Harris-Wai 2015), whether their role is interest or influence. This means that there is an all-round way of understanding the problem that the policy is intended to solve and the politics surrounding the decision-making process.

It is important for a researcher to understand in policy-engaged research, is the need to tailor the research in a way that the policy options suggested are practical. This is because, a policy attempts to solve or prevent a problem, or scale up progress, and policy actors are interested in “what works”. In other words, they are keen about what causes an outcome or makes things happen. A piece of public policy research would, therefore, do well if it were causal, rather than descriptive.

There are two fundamental characteristics of a public policy research problem or question: First, it should explore cause, outcome, and/or causal mechanism in relation to an existing policy or a policy action it intends to propose. In exploring these, the researcher can tease out the specific factors that are responsible for a certain policy problem/issue (outcome) and have conclusive findings from which to confidently suggest specific points of intervention in a policy progression. For instance, if the researcher discovers that misinformation is the cause of vaccine rejection, then he or she knows better than to suggest increased procurement of vaccines but would rather suggest media campaigns or community meetings to increase citizens’ awareness of that vaccination. If, in exploring the mechanism between misinformation and rejection, she discovers that misinformation leads to cognitive dissonance which then leads citizens to seek clarification from traditional birth attendants who then counsels them to abstain from vaccination and whom they then obey by rejecting the vaccination, she is further equipped to make pointed suggestion on which point in the chain to focus intervention or “tweaking”. Public policy research without such causal information can easily become a shot in the dark.

Second, the public policy research problem should resonate with the questions that policy actors are asking as well as the questions that they should be asking. While it is important for the public policy research question to evolve from policy questions, it is also important to note that policy questions are sometimes wrong or inadequate. Put bluntly, policy actors sometimes do not ask the right questions. It is, therefore, important for the researcher to identify these policy questions and give them the needed redirection. Policy actors, for instance, may be asking if the gap between male and female children about access to education is narrowing or widening following the adoption of an affirmative action policy in favour of the girl child. Whereas this is an important question, it is not likely to reveal information that is specific enough to be a basis for the right adjustment of the policy. It is not only simply descriptive but also narrow and unworthy of much research. The researcher should push harder with questions of cause, outcome and causal mechanism about the male-female disparity in access to education in this case. Has the policy produced a narrowing of the gap? If not, why has it not? What skills or resources are lacking that account for this lack of narrowing? Or what historical, religious or cultural factors combine or act alone to ensure continuity of the gap despite the policy? The public policy research question may not be the exact one that policy actors are asking, but it is indeed a vital extension and reflection of the policy question.

When we have public policy research problems that are unrelated to the problems that policy actors have, the consequence can be predicted. We will come up with findings that may be scientifically sound but unattractive to policy actors. Such findings will have little or no uptake. This approach speaks to the disconnection which a vast amount of literature points out—the disconnection between researchers and policy makers (Edwards 2004: 2; Young 2005: 730–1; Saetren 2005). When we ask public policy research questions that are not causal, the consequence can as well be predicted—our findings will not be convincing or informing enough to move policy actors to targeted action. Ultimately, questions that are not in line with the policy makers, and non-causal questions, render our research simply as just another piece of research for its sake.

A research question largely dictates its own research design. The type of research question we advocate above implies an iterative approach that begins with policy actors and finally returns to them. It also implies a specific kind of methods. It is a back-and-forth movement that considers the concerns of the actors as the fulcrum. In addition to being iterative, the design is also causal. The stages given below may apply (Fig. 4.1).

Fig. 4.1

Approach to designing policy research

The way in which research is designed determines the ability of the researcher to claim causal conclusions (Bachman 2007). This is important for it gives indication to policy makers on what influential factors lead to what outcomes. If this is not known, making relevant policy decisions is always not possible.

Qualitative and Quantitative Methods in Public Policy Research

In this section, we explain the commonly used qualitative and quantitative research methods for public policy.

Qualitative Methods in Public Research

Briefly stated, qualitative methods aim at providing deeper perspectives, attitudes, perceptions and contextual insights that surround the issue under investigation as experienced and understood by those living through it. The outcome of qualitative methods is usually the verbalized thoughts and viewpoints of the subjects of investigation rather than numbers or statistics. The following are some of the research methods used in qualitative research. Note that each of these methods applies a wide range of tools to collect data.

Historical and Archival Research

Libraries and archives store historical information in many forms including diaries, pictures, documents, minutes and artefacts, among others. These mean that they might have been stored as primary or secondary data. Historical or archival information that can be considered as primary is that which was collected from the author or field and stored in its original form without undergoing any form of analysis and change. Such may include minutes, diaries, pictures, artefacts, personal memoirs, autobiographies and others of the same nature. Any historical information that has gone through any studies or analysis then becomes secondary data. These may include journals, books and magazines, among others.

When a researcher wants to use historical and archival data, the aim is to research on the past and already existing information. However, historical and archival research does not always mean deriving data from the archives. A policy researcher may design a historical study in which they endeavour to visit the field and collect data from knowledgeable individuals concerning a certain historical issue of policy concern. They may partly engage documents from archives or libraries to historicize, contextualize and corroborate the issue under research. It is also the nature of many parliamentary researchers to “mine” data from parliamentary libraries/archives, some of which contain data that is classified as primary data.

Historical data is important in public policy, for it helps researchers situate their arguments within existing narratives, contexts and prior solutions suggested for policy problems. Roche (2016) argues that making assumptions about the ease with which historical research can be done is misleading. He advises that knowledge of context and a sequential approach should be given ascendance in the researcher’s priority. The researcher should be aware of chronology of information to clearly provide a coherent picture of the policy issue at hand. This implies that the past information should be relatable to the most current. With the advent in technology, most data are now digitalized, and as such, it is easy to get information from the Internet.

Archives are used to store vital government records such as personal letters, diaries, minutes, logbooks, plans, maps, photographs, among others, that easily qualify to be analysed as primary data (Roche 2016: 174). Roche (2016: 183–4) notes the challenge of fragmentation and partial availability of archival documents. He further alludes to technical challenges of the clarity of some of archival data. He cites examples of materials that were handwritten a while back and which may be ineligible. Historical and archival research apply both desk-based methods and interview techniques of data collection. Photography can also be used.

Ethnographic Methods

Ethnographic approach to research studies communities in their natural setting to understand their activities, behaviour, attitudes, perspectives and opinions within their social surrounding (Brewer 2000). To do so, ethnography entails close association with the research communities and sometimes participation in their activities (Brewer 2000: 17). In fact, the commonly used methods of data collection in ethnography are participant (and sometimes non-participant) observation. The former allows for the researcher to get involved in the activities of the communities, while the latter is designed for the researcher to observe from the periphery. As Brewer argues, it is this day-to-day involvement in people’s activities that enable the researcher to make sense of the social worldviews of the research participants.

Non-participant observation describes a research situation where a researcher does not take part in the processes, events or activities that he or she is observing but removes himself or herself from the happenings to critically observe from a distance. This has challenges especially if the observed become aware of intrusion and subsequently alter their behaviour (Hawthorne effect). Sometimes the researcher may structure the observations or decide to use unstructured observations. The two differ in the sense of planning on the observation activities. For the structured type, the researcher has in mind what they want to observe and as such have a list and indications of what they would like to see. Take, for instance, a study on access to water meant to contribute to a water policy. A researcher may choose to observe how (many) times is water served at certain water points; how many people queue for the water in each of these servings; and this is likely to tell the researcher whether the water points are enough or otherwise. In unstructured observation, the researcher gets into the field with a research idea but without the specifics of that nature of data they expect from the field. Qualitative interview methods such as oral interviews and focus group discussions (FGDs) may apply where necessary during ethnography. Note taking is often applied as well.

Phenomenology

This method focuses on lived experiences of a given phenomenon by an individual or a group of individuals. Individuals can describe their views and opinions about the phenomenon in question (Johnson and Christensen 2014). Research on fertility issues can target women who either have or do not have children, depending on what the researcher wants to unearth, with individual women providing their lived experiences on the issue under investigation. Phenomenology is also applicable when writing biographies (an account of someone’s life written by someone else). Generally, life histories, personal testimonies and experiences are best collected through this method. This implies that oral in-depth interviews and desk-based methods of data collection are important in understanding the stories in question.

Narrative Method

A narrative is a story that structures human activity to give it some form of meaning (Elçi and Devran 2014). Research that applies the narrative method encourages the research participants to tell their stories around a certain issue. The researcher listens to the stories and uses them to make informed analysis on the issue at hand. A researcher concerned about experiences of people living in zones of conflicts may ask questions that elicit stories of the victims or perpetrators of violence and present these in narrative form. Researchers who use phenomenology method often apply use of narratives, but not always. Phenomenological research may not rely on story telling alone. A researcher may use desk-based method to gain perspectives of the target communities as well.

Case Studies

A case study is an intensive analysis of a small number of phenomena (events, actors, activities, processes, organizations, communities, among others) in each context. Though one can use a mix of qualitative or quantitative data within a case study, meaning that case studies can also take quantitative route, a case study is always a detailed analysis of the relationships between the contextual factors and a visible occurrence. Case studies are therefore considered when there is need for detailed information on the issue(s) under investigation. A single case study aims at providing details on the variables of interest. A comparative case study has two or more cases (what literature refers to as small-N) for the purpose of making comparative causal explanations. A researcher uses comparative case studies when they want to tease out the similarities and/or differences between or among the cases, usually for the purpose of explaining causation.

Action Research

Action research is problem-solution focused. It falls under the category of applied research and subsequently, uses practical approach to solve an immediate problem. In this case, the researcher works together with a community or practitioners to identify a challenging issue within the community that requires a possible solution. They formulate the problem together and design the research in a way that the aim is to work towards getting a solution to the problem. Once the data collected is analysed and recommendations given, a plan of action is drawn and applied to the problem that the research was designed for. The community (and researcher) reflects on the effectiveness of the solutions applied to take appropriate measures. In a nutshell, Huang (2010: 99) explains that action research proceeds from a praxis of participation guided by practitioners’ concerns for practicality; it is inclusive of stakeholders’ ways of knowing and helps to build capacity for ongoing change efforts. This form of research requires money and time. As Huang (2010) notes, action research can take a qualitative, quantitative or mixed-method perspective. Various methods of data collection including oral interviews, surveys, community mapping, observation, among others, may be applied in action research.

Grounded Theory Research

A researcher may apply two approaches, inductive or deductive, to do research. The deductive approach means that one has a theoretical basis from where hypotheses can be formulated and tested. Inductive approach, on the other hand, is grounded or bottom-up. The researcher in this case starts by making observations that then provide him or her with patterns from where conclusions and theory can be drawn. Grounded research therefore moves from the point of poor or no theory up to where a researcher can deduce an informed hypothesis and towards theory building, all from the observations and analysis made from data. It is similar with other qualitative methods in the use of the various methods of data collection including oral interviews, observation and use of all forms of documents (Strauss and Corbin 1994).

Quantitative Methods in Public Policy Research

Quantitative research generates numerical data using such research instruments as the questionnaire, tests, code sheets for content analysis and similar other sources. The data is then subjected to mathematical or statistical analysis (Muijs 2004).

Literature divides quantitative research methods into two—experimental and non-experimental methods. Experimental methods are the quantitative approaches that are mainly concerned with manipulation situations with an aim of establishing cause and effect. Bachman (2007: 151) argues that “the experimental design provides the most powerful design for testing causal hypotheses about the effect of a treatment or some other variable whose values can be manipulated by the researchers”. Experiments allow us to explain causality with some confidence because of the use of treatment and control. The basic and elementary type of experimental research involves setting up two groups (treatment and control groups) and introducing change to the treatment but nothing to the control. The effect of the change is measured in the differences in the behaviour or performance of the two groups after the treatment.

Experimental research has been criticized for their weakness in reflecting reality in that they take people out of their natural settings into a laboratory or pseudo-labs. Despite this, they can make important input to policymaking. For instance, micro-level policies on classroom instruction and curriculum have been largely influenced by experimental research.

Non-experimental methods do not manipulate. They are aimed and providing a descriptive picture of what is being studied. Non-experimental methods, as Muijs (2004) indicates, are more varied and may range from surveys to historical research, observations and analysis of existing data sets (applied quantitative methods). We will briefly look at the experimental and non-experimental quantitative research in the following sections.

Experimental Methods

The different types of experiments can range from randomized control trials (RCTs) to quasi-experiments, and sometimes, natural experiments.

Randomized Control Trials (RCTs)

In their simplest form, RCTs involve assigning individuals, groups, communities or settlements to experimental/treatment and control groups. The experimental group receives treatment—school feeding—while the control group receives no treatment (no school feeding). The difference in school attendance rates between these groups could then be attributed to the treatment, that is, school feeding. If statistics shows that attendance increases in the treatment group but stays the same or decreases in the control group, other things being equal, the researcher can make claims about school feeding causing increase in school attendance. Randomized control trials are expensive and are usually beyond the budget reach of most researchers. Public policy researchers therefore embark on other forms of experimental methods generally described as quasi-experimental methods.

Quasi-experiments

There is an unending controversy as to what constitutes a quasi-experiment. Given the little profit accruing from such a controversy, we would take a simple definition of that concept: any experiment that mimics as closely as possible the advantages of RCT (Muijs 2004: 27). In quasi-experiments randomization is not possible (Muijs 2004). This makes it difficult to eliminate bias. The experimental group is already determined—they are the ones enjoying or experiencing the treatment of concern to the researcher. What the researcher does is to compare this group with another that is not experiencing the treatment. Often, the treatment is a government programme or some other kind of intervention out of the researcher’s control. Where it is possible to have another group to compare with, the researcher might work with data before treatment comparing that with data after treatment.

Take, for instance, the introduction of government-funded public examinations in some Nigerian prisons in 2019. Would the incidence of violence reduce in prisons because of this policy? A few years into the policy, a researcher might compare incidence of violence in Prison A where the policy is being implemented with Prison B where it is not being implemented. Or, where, for certain reasons that two-prison comparison is not possible, she might compare data on the incidence violence in Prison A before the policy with data on the incidence of violence in the same prison after the policy has been implemented.

Quasi-experiment templates consider space (spatial variation) and time (temporal variation) as important aspects that influence setting up of experimental research designs. Gerring (2007) and other scholars provide a variety of these templates. For instance, a researcher might be interested in explaining if and how a certain programme or policy, say a school feeding programme, increases students’ performance in national examinations. She can select two local government areas or sub-counties—one with a school feeding programme and the other without—and then compare school performances of students in both sub-counties and local government areas in national examination. It is important to ensure that the two cases (i.e. sub-counties or local government areas in this example) are similar in all other factors that might influence students’ performance in a national examination, the only difference being the presence of a school feeding programme in one and its absence in the other. The data can be collected by a variety of means—questionnaire, secondary data such as attendance registers, observation guide or any other that suits the research objective and question. A fruitful study of this type does not stop at showing that students in local government A where there is a school feeding programme perform better than their counterparts in local government B. That would be an interesting finding, but it leaves a lot unsaid. Rather, it should press on with an explanation of the causal mechanism—the pathway or trajectory by which the school feeding programme leads to better grades. This implies that what is largely categorized as quantitative study may require aspects of qualitative data to allow the researcher to get a complete picture of the issue under investigation.

Ojebode et al. (2016) attempted to explain the (in)effectiveness of community-based crime prevention practices in Ibadan, Nigeria. They selected two communities—one with a successful community-based crime prevention programme and another with a clearly unsuccessful one. These communities are similar in all the factors that matter to community-based crime prevention—population, ethnic mix, youth population, socio-economic status, and both have community-based crime prevention practices. Their puzzle was: why did the practice work so well in one community and fail so woefully in the other despite the similarities in these communities. Through different rounds of data collection and different instruments, their quantitative and qualitative analysis shows that the settlement patterns in the communities—dating several hundred years—perhaps explained the variation in the outcomes of the crime prevention practices.

Natural Experiments

Natural experiments take advantage of exogenous effect, that is, an intervention that is outside of the control of the researcher, which was also not intended to affect the outcome/dependent variable. The exogenous effect can be in the form of natural (such as a natural disaster), physical (like in the case of the colonial/government border) or historical event. They may also be a policy intervention. These were not intended for research or academic purposes. In other words, what becomes the treatment or causal factor happens through some “natural” occurrence or unplanned event. In some ways, these events may allow for observation of before and after they occurred. An example is Friedman et al. (2001) who carried out a kind of natural experiment during the 1996 Olympic games in Atlanta, Georgia. The researchers wanted to find out whether heavy traffic in the city was a cause of asthma in children. They made observations on how the city was organized during the 17 days of Olympics where the traffic rules changed. Small cars were forced onto alternative routes to leave main routes for mass transport, and this reduced traffic congestion on the major roads of the city. Through paediatric records (before and after Olympics), the study discovered 40% reduction in asthma attacks and emergency hospitalization. The researchers made a conclusion that traffic congestion contributes to paediatric asthma. This can be classified as a natural experiment, where the Olympics (manipulation/treatment) was not planned by the researcher and was exogenous (not related in any known way) to asthma. Such critically thought-out research can easily contribute to change in transport policies. Other studies, for instance, Daniel Posner’s on Chewas and Timbukas of Zambia and Malawi (2004), have used borders artificially created by colonial governments as boundaries of study groups. In his case, Posner shows how governments in two countries differently exploit similar ethnic compositions and the effect of this exploitation on inter-ethnic relations.

Non-experimental Quantitative Methods

Surveys

Most quantitative researchers collect data using a standard questionnaire containing mostly close-ended questions. Some researchers may use a questerview, which combine both closed-ended and open-ended question. The latter is applicable when corroborative data or explanations to the closed-ended questions are needed. Survey questionnaires for this reason provide some standardized data that can be keyed into software for organization and analysis. The type of survey questionnaire depends on the nature of data that the researcher requires, the reach of the study population and ways in which the data is to be collected. One can decide to do telephone interviews, post the questionnaire, administer it online or have an ordinary written questionnaire.

Survey research considers a variety of factors including samples and sampling procedures, characteristics of the study population, among other issues. Surveys mainly apply probability sampling with an aim of giving all the elements a chance to be included in the study sample. This is the opposite of non-probability sampling those centres on purposive and convenient sampling. There are various sampling techniques in probability sampling, and these are available in various research methods books. Just to mention, some of the probability sampling approaches include simple random, stratified random, cluster, quota and multistage (see Muijs 2004, 2011; Babbie 2004; Kothari 2004; Kumar 2011). For sample sizes, there are suggested formulas that researchers can apply for both finite and infinite populations.

Observational Studies

Observations are important for both qualitative and quantitative research. In quantitative research, observation is applied both as a research method and as a method of data collection. In qualitative research, observation is mostly categorized as a method of data collection and features in various research methods including ethnography, case study and action research. In quantitative studies, observational methods are important, for they enable a researcher to interact with the study environment and participants in a way that the questionnaire would not. Observational data for quantitative research is collected using standardized/structured observation schedules. A researcher can develop a descriptive observational record or a rating scale to help them collect observational data. This enables the researcher to observe and record the behaviour and activities in the selected study sites in a standardized way. Observations can also be made on existing reports within the institutions being studied, say for instance, school performance and statistical data collected from such reports (see Muijs 2004). In the end, the different methods may generate descriptive data of various types, that is, from open-ended and closed-ended descriptions. The selection of participants is also randomized to give all a chance to participate, and subsequently, those falling within the sample size are meant to represent the study population on which generalizations can be made.

Applied Quantitative Method

This method makes use of existing data sets. It applies analytical methods to facilitate description of data that has already been recorded and stored. Different research institutes store varied forms of data sets. These could be useful if a researcher is interested in analysing them with the purpose of achieving a certain research objective. For instance, one might be interested in understanding and describing the population growth trends. In such instances, one does not need to go to the field to collect fresh information when the national bureaux or offices of statistics have the data sets. All one needs is to get permission from relevant authorities to access such information. The challenge with using such data sets is that if they are erroneous in any way, then the errors are carried forth in the analysis. As Muijs (2004) indicates, the various quantitative research methods can be combined in a single study if this is necessary.

Mixed Methods in Public Policy Research

The advent of mixed-method research and the place that it currently occupies in social science research reinforce the arguments for the use of both traditions of qualitative and quantitative methods in public policy research. Statistics should be complemented and explained by meaning-making concepts, metaphors, symbols and descriptions from qualitative research to make sense of hard data. On the other hand, narratives on their own are not enough. Jones and McBeth (2010: 330) show that despite the apparent power of stories in public policy, public policy studies have largely remained on the side-lines of the use narratives. The two scholars suggest the relevance of using a narrative policy framework as a methodological complement for positivists in the study of policy. Some scholars have also shown that for policy problems to be clearly defined, a narrative structure is needed. Narration, as Fischer (1998) and Stone (2002: 138) explain, helps make sense of the socially constructed world that requires tangible solutions. Since qualitative approach may not be able to engage hypothesis testing to allow for replication and falsification (Jones and McBeth 2010: 339), they should complement or be complemented by quantitative data.

Qualitative and quantitative methods have their own separate strengths. As noted above, qualitative research is about depth and qualitative is about breadth. This means, if a study requires both, then mixing the methods is important. Mixing methods therefore means a research problem requires both qualitative and quantitative data. Morse (1991) argued that triangulation of methods not only maximizes the strengths and minimizes those weaknesses of each approach, but also strengthens research results and contributes to theory and knowledge development.

Mixing research methods does not just imply mixing methods of data collection. A researcher must intentionally clarify which research methods (as discussed above) are applicable in their research to speak to qualitative and quantitative aspects, and by extension what methods of data collection will be used. Note that one research method may have many methods and tools data collection. If one is using ethnography, then participant observation, oral in-depth interviews, observations and focus group discussions are examples of applicable data collection methods. The various methods of data collection have their instruments/tools.

Mixing of methods entirely depends on the purpose for which the methods are mixed. This is determined by the research problem. Mixed research methods books provide a wide range of typologies of designing mixed-method research (see, for instance, Greene et al. 1989; Creswell and Clark 2011; Schoonenboom and Johnson 2017). Below is a simple illustration of the continuum for mixing methods (Fig. 4.2). A researcher can move from a purely quantitative or qualitative research method (A and E), towards integrating either quantitative (B) or qualitative (D) methods to the dominant method. A researcher can also design a fully mixed-method research (C). This is a simplified way of understanding how mixing can happen; there are other more complex typologies.

Fig. 4.2

The mixed-method continuum. (Source: Teddlie and Yu 2007: 84)

In public policy research, the mixing is important for various reasons. One might require results for complementary purpose, explanations to the statistical results, expansion of results from one domain (qualitative or quantitative) or confirmation of results. The dictates of mixing are found within the research problem and by extension research questions/objectives.

Conclusion

There is subtle blame game between bureaucrats and policy makers, on the one hand, and researchers, on the other hand, in Africa. While the latter accuse the former of not using the research they conduct, the former responds by claiming that many of the research do not speak to policy or societal issues and are thus not usable. They add that many of them are rendered in a language that is not accessible to non-academic actors. As a result, not a few policy decisions are based on political and other judgements rather than on sound research.

Our discussion so far suggests that the bureaucrats and policy makers may not be totally right in their accusation, but they are not totally wrong either. The preponderance of policy-appended research, and of solo-method research which offers little as a basis for policy, seems to justify their accusation. It is, therefore, important that public policy researchers weave their research around societal issues that are not only significant but also contemporary and topical, craft their design with the aim of policy engagement and stakeholder involvement, and adopt mixed methods as and when necessary, to provide findings and conclusion that command and compel policy actors’ attention.