From the Editors: Student samples in international business research
- 1.3k Downloads
The JIBS Statement of Editorial Policy states that “Empirical submissions utilizing undergraduate student samples are usually discouraged.” Wording is important here – “usually” does not imply “always.” This statement suggests that while student samples are appropriate, in principle, undergraduate ones are not, except in unusual situations. The purpose of this From the Editors column is to discuss the pros and cons of student samples in international business (IB) research. Are student samples appropriate for developing and testing IB theories? When are student samples appropriate, and when are they not appropriate? Is there a difference between undergraduate and graduate samples? What are the “best practices”? This editorial is intended to provide IB researchers with guidelines as to when student samples, and which types of student samples, are appropriate in IB research.
INTRODUCTION – APPROPRIATE SAMPLE SELECTION IN IB RESEARCH
In the social psychological literature, studies using student samples abound, as such samples are convenient and readily accessible. This practice has been questioned by Sears (1986) and Wintre, North, and Sugar (2001). Many of these studies involve laboratory experiments that manipulate conditions to determine the effects they may have on the respondents, and/or involve the administration of questionnaires to probe respondents’ attitudes and values on specific subject matters. In the field of business and management, however, journals are more reluctant to accept papers analyzing data collected from student samples because most students, except those in EMBA programs, are not employees or managers in reality. Consequently, studies based on conjectural responses of how students would react or courses of action they would undertake if they were confronted with such situations in reality are, at best, hypothetical in nature and lack external validity. One may argue that this is even more the case in international business (IB) research as the issues confronting an employee or manager involved in cross-border transactions are likely to be more foreign to the average student, except those at the EMBA level. This is why the JIBS Statement of Editorial Policy asserts that “Empirical submissions utilizing undergraduate student samples are usually discouraged.” However, this statement is not absolute as there are exceptions to the rule, hence the word “usually.” In this From the Editors column, we try to explain what exceptions to the rule may make it into the journal.
STUDENT SAMPLES IN IB RESEARCH
To structure the argument, it is helpful to distinguish two classes of research topics: those involving “fundamental” and those relating to “proximate” processes, structures and outcomes. Fundamental processes, structures and outcomes are concerned with the basic characteristics of human nature that are relatively independent of context and life experiences. In this case, internal validity is key. Neuroeconomics is a clear example of a tradition that focuses on this type of fundamental work and operates at the interface of neurosciences and economics in experiments that are intended to deepen our understanding of the micro roots of economic behavior (Camerer & Fehr, 2006) – for example, how is cooperative or competitive behavior reflected in the functioning of the human brain?
In the area of IB specifically, fundamental research involves studying aspects of human nature and propensity that can explain specific phenomena in an international setting. An example could be the use of a common language in teams comprised of members with different mother tongues. Since multicultural teams are commonplace in today's organizations, to communicate effectively, such teams must decide on the language they will use, which most often happens to be English. To explore the communication challenges associated with cross-cultural alliances, including cross-border acquisitions, it would be useful to undertake fundamental research that probes into the deep, underlying processes associated with how the use of non-native languages, such as English, could affect human cognition and behavior (see, e.g., Akkermans, Harzing, & Witteloostuijn, 2008). Such basic issues can be explored in any sample of multicultural teams, including students with the caveat that undergraduates, in comparison with more mature adults, may likely “have less-crystallized attitudes, less-formulated senses of self, stronger cognitive skills, stronger tendencies to comply with authority, and more unstable peer group relationships” (Sears, 1986: 515).
Proximate issues are different from fundamental issues by focusing more on specific contexts and less on basic characteristics of human nature. In this case, external validity is critical.1 Take decision-making as an example. The use of students with virtually no real-life managerial experience to conjecture how they would respond to important decisions of a strategic nature in the international context would be pointless.2 For this, to truly understand the phenomenon, the sample must include managers actually involved in such decision-making processes. EMBA students who are involved in high-level decision making could be appropriate, however.
Of course, the issue is not as simple as implied by the above. The distinction between fundamental and proximate issues is not always as clear cut as suggested. There are “hybrid” issues that are betwixt-and-between in that they are neither tied to the intrinsic features of human nature nor directly related to real-world practice. For example, a study of the propensity of undergraduate and graduate students from China to return to their home country upon completion of their studies abroad in order to project the ability of emerging markets to meet their human talent shortfall as they seek to expand globally would be appropriate. The students sampled are enrolled in business administration, computer science and engineering programs at a North American university – disciplines that are in demand in many emerging markets (Tung, 2007).
GENERALIZABILITY IS THE KEY CONCERN
The distinction between “fundamental” vs “promixate” research issues portrayed above highlights the fact that generalizability is the key concern. Findings that reflect the fundamental nature of humanity are likely to generalize across diverse populations, making the use of student samples legitimate. Findings concerning processes, structures and outcomes that are sensitive to the influence of context and life experiences are unlikely to generalize across different segments of the population.
The validity of the use of student participants has been under scrutiny in many social science disciplines, including management (e.g., Dobbins, Lane, & Steiner, 1988), psychology (e.g., Wintre et al., 2001), and legal research (e.g., Bornstein, 1999). The emerging consensus is that for authors who want to publish student-based results, it is incumbent upon them to demonstrate that their results can be generalized to real-life situations on which they intend to shed light. After all, research based exclusively on samples with very specific age and life-stage characteristics could bias findings. Among other things, young adults, as compared to an older population where the ranks of managers in IB are usually drawn from, are usually more malleable to social influence, their self-identity may not have developed fully and they appear to change their attitudes more readily (Sears, 1986). For these reasons, Wintre et al. (2001) have heavily criticized the overuse of undergraduate student samples in many social psychological studies. This observation has two implications, at least. First, interpretation of findings from student samples must recognize the potential lack of generalizability by taking into account the possible biases incurred by using samples from this subset of the larger population. Second, even research addressing fundamental issues needs to be corroborated with non-student samples, because there is always the chance that nature and nurture may be intertwined, and maturity and life experiences may have a role to play in settings where human nature is supposed to be pivotal.
WHEN ARE STUDENT SAMPLES APPROPRIATE?
We outline four broad conditions for the legitimate use of student samples. First, if a study is guided by a well-defined theory with sophisticated predictions, and if the results based on student participants confirm the predictions, it is likely that these results can generalize to a target population. This type of work is likely to be experimental or longitudinal in nature, involving complex predictions. A good example is the experimental study on individualism–collectivism with MBA students by Chapman and Barsade (1995), and the study with undergraduate students on faultline in naturally occurring teams that are culturally diverse by Lau and Murnighan (2005).
Second, results based on students are likely to be ecologically valid if they are replicated or corroborated by results based on employees or managers. A good example is provided by Hui, Au, and Fock (2004), who explored the relationship between national culture and empowerment in three studies. Hui et al. are able to show that data from an experimental study with undergraduates can corroborate and extend the results based on employees.
Third, if authors do not have employee-based results to cross-check the external validity of their student-based results, they need to provide a convincing, strong argument for the generalizability of their results to the working population based on the extant literature. An example of this approach is provided by Friedman, Chi, and Liu (2006), who used MBA and EMBA students to explore cultural differences in the conflict styles of Americans and Chinese. Some of their key findings replicated well-established findings reported in the literature, lending support to the generalizability of their other findings.
Fourth, the use of student data may also be acceptable when it is used in concert with comparable managerial samples to simultaneously explore differences in views and values within, as well as between, countries and cultures. Studies designed to cross-culturally assess life stage and/or generational differences (e.g., Egri & Ralston, 2004) benefit by including student data. The value of this within and between research design, as called for by Tung (2008), is that it allows the researcher to take the analysis below the country-level to explore relevant sub-cultural differences.
Researchers use student samples because of their accessibility, convenience and low cost. Students are also more receptive of complex designs, yielding data that are hard to collect from employees and managers, but are suspect regarding their generalizability. In the final analysis, it is a matter of a research question-design fit. Generally, since many IB studies concern proximate issues, which are sensitive to the influence of contexts and experiences, the use of student samples is particularly worrisome. The use of full-time MBA students may be more justifiable, because they typically have some working experiences, but it is the burden of the authors to demonstrate the generalizability of the findings based on this type of student. Finally, EMBA students, who are typically part-time students, should not pose a significant threat to external validity, although there is still the concern that they may represent a special group of managers who differ from their peers in some significant ways. In a nutshell, the use of students always requires an explicit discussion of why external validity is not compromised.3
The advice to avoid using (undergraduate) student samples, except where appropriate, does not pertain to the issue of external validity only, as external validity is almost always a concern with any empirical study since we seldom have representative samples. Rather, the use of student samples may pose an even greater threat to a study's internal validity, as the students may not possess the requisite knowledge to respond adequately to the experimental treatments or survey questions. This lack of requisite knowledge can threaten the internal validity of the findings in much the same way as using a sample of low-tech firms to examine the internationalization strategy of high-tech firms.
To sum up, while we recognize the value of student samples, we also see the need to reaffirm the standing policy of JIBS in this regard: “Empirical submissions utilizing undergraduate student samples are usually discouraged.” Our intention is not to dismiss all student samples, but to ask for a strong, explicit justification for their legitimacy.
This is not to say that internal validity is not important – it is. The point we want to make here, though, is that external validity is a more critical issue in this context.
This raises yet another question: if it is inappropriate to use student data, is it also not appropriate to use measures that were validated using student data? This question deserves a discussion of its own. For reasons of space, we ignore this issue in this letter.
This is not only true for the use of students as a direct source of data, but also their use as interviewers. Even though the questionnaires are normally developed by the researcher, undergraduate students may not always have sufficient background to fully interpret the answers provided by the interviewees to the questions posed in those surveys, especially when open-ended questions are included. Researchers rarely mention who collected the data used to test certain hypotheses, even though this should be an essential element of transparency and ethical behavior.
This letter reflects the opinion of the editorial board as a whole. We gratefully acknowledge the excellent suggestions made by other board members.
- Akkermans, D. H. M., Harzing, A. -W., & van Witteloostuijn, A. 2008. Cultural accommodation and language priming: Competitive versus cooperative behavior in a prisoner's dilemma game, Working paper, University of Antwerp.Google Scholar