Introducing the ARTS framework: A tool for constructive re-inquiry

Introduction

This editorial proposes a new framework for evaluating established marketing phenomena that may warrant re-inquiry due to impactful context changes. Currently, non-replication of prior results often serves as a trigger for re-inquiry. Although important, such a response is reactive and often more destructive than constructive in spirit. By contrast, the proposed ARTS framework (Assumption, Retrospection, Testing, Synthesis) represents a more positive and proactive approach to re-inquiry, driven by critical examination of context changes.

The ARTS framework involves: (1) identifying whether context changes may challenge existing assumptions underlying the focal theory or phenomenon (Assumption); (2) evaluating the generalizability, evolution, and implications of past empirical research for the contemporary context and challenged assumptions (Retrospection); (3) conducting new empirical tests that reflect present-day instantiations of the phenomenon and explore how challenged assumptions alter generalizability to the contemporary market (Testing); and finally (4) integrating new and past findings using theoretical and/or aggregating methodology (Synthesis).

By applying the ARTS framework, researchers may better understand when well-designed non-replications of prior phenomena reveal opportunities for new, important tests of generalizability. Though we focus on managerially relevant behavioral phenomena, this framework could be easily applied to other marketing domains and academic disciplines (Table 1).

Table 1 Steps to apply the ARTS framework

Assumption: What assumptions may no longer hold?

Using the ARTS framework begins by reviewing the explicit and implicit theoretical assumptions underlying a demonstrated phenomenon and identifying what changes in the marketplace context may challenge those assumptions. For example, the introduction of the internet (and subsequent rise of e-commerce) represents an exemplar of marketplace change. Many marketing theories taught today were created before the advent of the internet. When these theories were developed, researchers made assumptions that, although reasonable at the time, may have since been rendered invalid, unreliable, or inconclusive due to radical marketplace changes. To the extent that past assumptions hold in the transformed context – albeit in new manifestations – researchers can reasonably expect theories relying on those assumptions to generalize to the present, changed marketplace. However, to the extent that past assumptions are challenged by transformed contexts, methods, and/or consumer experiences, theories that rely on these assumptions not only justify, but also demand, re-inquiry.

Consider the book Influence by Cialdini (1984), which includes Principle #6: “scarcity creates value.” This principle spurred the prediction that scarcity marketing tactics enhance consumers’ perceived value of the promoted product, increasing purchase intentions and willingness to pay. When the book was originally written, only 8% of U.S. households had a computer and 0% had internet. It would be seven years until the World Wide Web was publicly launched and several more until the first exclusively online transaction occurred. By contrast, as of 2022, over 75% of U.S. adults reported shopping online at least once per month with 27% of U.S. adults making an online purchase at least once per week. The omni-channel marketplace of today differs substantially from that in place when Cialdini postulated his “six principles of persuasion.” What theoretical assumptions might be challenged by these marketplace changes?

To answer this, one might reflect on past assumptions, made when the phenomenon was first proposed. For example, the scarcity promotions prediction relied on assumptions established in a marketplace devoid of online retailing and e-commerce. At the time, it was assumed that physical search costs would impose a natural limit on most consumers’ likelihood to browse multiple stores before buying. However, as online retailing grew, the physical limitations on consumer search weakened. Now, online consumers can easily switch to other websites to access different information, longer deals, and/or increased supplies, consequently lowering the cost of continued search and diminishing the net benefit conferred by scarcity. This calls into question the prediction that consumers who encounter scarcity promotions will see more benefit from buying a scarce good than from incurring additional search costs. Thus, the theorized relationship between scarcity and value may necessitate re-consideration before it is integrated into either new research or recommendations for strategy.

A complementary approach entails reflecting on the present assumptions marketers implicitly make by presuming a phenomenon continues to hold despite the changed context. Continuing with the scarcity promotions example, as the internet and e-commerce matured, marketers deployed scarcity promotions in previously non-existent online marketing channels (e.g., email, social media) and implementations (e.g., countdown timers). Underlying the continued widespread prevalence of these tactics is the implicit assumption that scarcity promotions’ efficacy is invariant to the retail context, with effects continuing to emerge in the changed online context. However, evidence documenting the impact of differences in online and offline retailing on consumer behavior raises questions about the appropriateness of assuming equivalence of online and offline scarcity promotions. Because this assumption can no longer be taken as given, the theorized effect of scarcity promotions on value warrants re-inquiry.

Before concluding that all pre-internet principles must be revisited, however, consider the relevance of questioned assumptions for a given theory. Not all past theories rely on now-challenged assumptions. For example, prior brand-building findings could be explained by Cialdini’s principle of “liking” (i.e., the more one is liked, the more persuasive they are). Prior assumptions challenged by the internet’s arrival are unlikely to change how this principle operates for brands: in the post-internet world, better-liked brands still likely experience lower customer acquisition costs than less-liked brands, as they can persuade more easily.

The degree to which a theory or phenomenon depends on a disputed assumption offers an additional perspective on replication attempts in a post-change context. Because phenomena relying on questionable assumptions may not generalize to the changed context, researchers should not expect such effects to perfectly replicate and, instead, be intrigued when they do. Researchers can reinterpret well-designed non-replications of such phenomena as potential candidates for re-inquiry, thereby creating opportunities for advancing knowledge and improving practical guidance that is relevant for the contemporary context. By contrast, phenomena that do not rely on challenged assumptions would be expected to generalize to the contemporary context, suggesting well-designed non-replications of such phenomena should prompt deeper scrutiny of past methods and analyses. In response to non-replications, the ARTS framework encourages researchers to temper their reactions and reflect on the appropriateness of anticipating identical replications in light of the challenges to the assumptions.

In the Assumption stage, researchers identify potentially questionable assumptions underlying a phenomenon or theory. Having examined the theoretical record for challenged assumptions, we now turn to a retrospective review of the empirical work on a topic.

Retrospection: Is past empirical evidence generalizable given the change?

Retrospection evaluates the degree to which a phenomenon’s existing empirical evidence is conclusive and generalizable to the post-change context. If the empirical work devoted to a phenomenon suggests shifts in underlying assumptions over time, this further justifies re-inquiry. To do this, researchers begin by systematically compiling a comprehensive database of prior research that tests the focal phenomenon. Employing methods similar to rigorous meta-analysis, researchers should document specific search parameters and procedures, specify clear inclusion and exclusion criteria for studies, and request unpublished work. However, unlike a general literature search, Retrospection requires systematically coding each study in terms of features relevant for evaluating generalizability to the post-change context, patterns in methodology and results, and questionable assumptions. Researchers then undertake both qualitative and quantitative analyses of the resulting dataset, paying particular attention to changes that unfolded progressively over time or in response to discrete assumption-shifting events.

One goal of Retrospection is to evaluate how the research record empirically tests the focal phenomenon. Researchers should assess the generalizability of existing evidence to the post-change context by examining how prior research has investigated the focal phenomenon and the extent to which methodologies, measures, and operationalizations employed reflect the contemporary context. For example, researchers interested in assessing Cialdini’s scarcity principle could consider the extent to which past work tests new types of scarcity promotions specifically designed for online contexts. Analysis may show that although scarcity promotions are increasingly conducted in online contexts, much of the past research employs promotions designed for offline contexts (e.g., print ads). Additionally, whereas contemporary instantiations of scarcity promotions are often dynamic and specific, the vast majority of past work uses static and general scarcity promotions. These observations suggest that the empirical record lacks sufficient evidence using contemporary instantiations of scarcity promotions.

Researchers should additionally examine whether there are any discernible patterns of changes in methodology or results, including changes to the operationalization of theoretical constructs, effect sizes, and dependent measures. Investigating such changes can provide insights about a phenomenon’s evolution over time. For example, researchers may observe that effect sizes of studies replicating classic scarcity promotion effects have weakened over time. When a single event or change of context necessitates re-inquiry, researchers may be able to compare effect sizes pre- and post-event or between contexts. For time-based scarcity appeals, researchers could code for retail environment (online vs. offline) in past studies. If effects observed offline remain stable over time, while those observed online diminish, this would provide support in challenging implicit assumptions related to equivalent efficacy in the online context.

Researchers can also survey how the range of dependent measures employed in prior work may have changed over time. With scarcity promotions, for example, analysis may reveal that more recent tests predominantly replicate scarcity effects for outcomes involving relatively costless online actions (e.g., clicks or likes), rather than consequential purchase behavior. These changes may suggest that the advent of the internet has eroded the power of scarcity marketing to motivate consequential action, providing additional support challenging assumptions about the continued effectiveness of scarcity promotions in online contexts.

In addition, researchers interested in assessing Cialdini’s scarcity principle might observe that more recent operationalizations of scarcity promotions use more concrete and dynamic messaging (e.g., “only 3hrs 21min 34 sec left!”) compared to earlier work. This change may reflect not only researchers’ desire to enhance ecological validity, but also suspicion that, in the transformed omni-channel retail landscape where search costs decreased and capturing consumer attention became increasingly challenging, contemporary time-based scarcity promotions needed to become more dynamic to attract commensurate attention and raise arousal.

Another goal of Retrospection is to investigate what the research record reveals about the questionable assumptions identified in the prior stage. In evaluating the research record, researchers should also consider what evidence tests aspects associated with the questionable assumptions. With the scarcity promotions example, researchers may examine whether any tests of scarcity promotions systematically vary search costs, potentially making comparisons that include studies that do not directly manipulate search costs (e.g., a shopping situation with only high search costs). If the effectiveness of scarcity promotions increases as search costs increase, this suggests that the assumption that scarcity creates value, questioned in the previous ARTS stage, may indeed be undermined in lower search cost settings.

Together, such analyses facilitate researchers’ retrospective evaluation of existing empirical work, assessing its generalizability to the contemporary changed context and offering insights about possible shifts to the assumptions and phenomenon, as well as remaining gaps in understanding. In doing so, Retrospection can both affirm the need for a phenomenon’s re-inquiry and generate testable hypotheses grounded in factors changed by shifted assumptions.

Testing: Can we conduct new research that fits the contemporary market?

Ideally, researchers would now conduct new research that not only better reflects contemporary instantiations of the phenomenon, but also offers preliminary empirical insight into the changes suggested by the Retrospection process. Harkening back to the scarcity promotions example, this presents an obvious challenge: researchers cannot turn back the clock to pre-internet times, making direct comparisons before and after online retailing impossible. However, researchers can vary key features of the challenged assumptions. For example, researchers might predict that scarcity promotions will be less effective in today’s marketplace because assumptions related to deal credibility no longer hold. To test this assumption, researchers could manipulate perceived deal credibility at high and low levels, while also including a control promotion condition that does not orthogonally manipulate credibility. If the credibility prediction is correct, researchers would expect Cialdini’s scarcity promotion effects to replicate in high credibility conditions but weaken in the low credibility and control conditions.

In varying key features of challenged assumptions, researchers can also use secondary data and proxy measures. For example, Retrospection might prompt researchers to predict that supply chain disruptions experienced during COVID-19 created permanent changes in consumers’ sensitivity to scarcity. To test this, researchers could measure consumers’ responses to scarcity cues and grocery shopping location information, using historical data documenting regional variations in stockouts at major grocery chains during 2020 to test whether the severity of local stockouts during the pandemic predicts consumers’ current scarcity responses.

As part of conducting new research, researchers should strive to design empirical investigations that better reflect contemporary instantiations of the phenomenon. Consider factors identified in the Retrospection stage that weakened the generalizability of prior empirical research to the post-change context, as well as marketplace changes identified in the Assumption stage that prompted initial questioning of the phenomenon’s underlying assumptions. In designing new research, it is critical that researchers prioritize validating operationalizations with respect to the contemporary marketplace. If stimuli and measurements used in the original research are no longer valid or suffer from confounds in the post-change context, then studies yielding null results using such operationalizations are uninterpretable, as findings might be driven either by non-generalizability of the theory or of the operationalization (and/or dependent measure). Thus, prior to testing, it is critical that researchers rigorously assess the validity of stimuli and measurements, for both faithfulness to the core constructs and potential confounds. Once a set of valid operationalizations and measures are identified, these can be sampled across multiple experiments, helping ensure there will be a sufficient number of effect sizes to permit statistical comparisons in the Synthesis phase, discussed next.

Synthesis: How does the re-inquiry augment knowledge?

In the Synthesis stage, researchers integrate information from the Retrospection and Testing stages. There are at least two approaches. The first uses an analytical approach to quantitatively assess the cumulative body of empirical work to date. Rather than analyzing old and new research in isolation, researchers analyze the totality of the combined data jointly. To do this, researchers establish a meta-analytic dataset that combines new empirical evidence obtained in the Testing phase with prior empirical evidence identified in the Retrospection phase. By systematically coding for critical differences between the prior and new studies, such as features of challenged assumptions and contemporary operationalizations, researchers can conduct more powerful tests and explore potential moderation. For references on the conceptual and statistical execution of such meta-analyses and an example of their explanation, we refer the reader to the work of McShane and colleagues (e.g., McShane & Böckenholt, 2019) and the authors’ application of these techniques (Hmurovic et al., 2023).

A second approach applies a more qualitative and theoretical lens, with researchers recommending specific updates to the theory, predictions, and assumptions surrounding a phenomenon. Researchers explicitly note what requires modification and what needs further investigation. What does synthesizing old and new empirical findings reveal about the phenomenon’s theoretical robustness and evolution? Researchers then detail a future research agenda that prioritizes rigorous exploration of the proposed theoretical changes necessitated by challenged assumptions and also identifies additional challenged assumptions meriting empirical examination. Ultimately, combining both analytical and theoretical synthesis activities enriches current understanding of a phenomenon and establishes a foundation for continued progress.

Conclusion

This editorial proposes a new framework for evaluating established marketing phenomena that warrant re-inquiry and for conducting these investigations in a manner that advances knowledge. Unlike a reactive “forensic statistics” approach employed to uncover dubiously replicable effects and potentially unscrupulous practices, the ARTS framework is driven by proactive examination of the marketing context. It involves probing assumptions of past theory that may no longer hold and patterns in past research that suggest a phenomenon’s evolution. This approach allows researchers to not only systematically review past theories proactively, but also to enhance the potential insights gleaned from non-replications. As such, the ARTS framework represents a complementary approach for probing non-replication—one focused on contextual changes rather than suspect practices. At the same time, this approach aims to foster cumulative knowledge that neither ignores nor disparages past research, but rather acknowledges its potential to stimulate important new insights that provide more relevant, effective, and impactful recommendations for conceptual development and contemporary practice. We believe this approach can facilitate meaningful progress in understanding contemporary marketing phenomena and beyond.