Effect of temporal spacing between advertising exposures: Evidence from online field experiments

Abstract

This paper aims to understand the impact of temporal spacing between ad exposures on the likelihood of a consumer purchasing the advertised product. I create an individual-level data set with exogenous variation in ad exposure and its spacing by running online field experiments. Using this data set, I first show that (1) ads significantly increase the likelihood of the consumers purchasing from the advertiser and (2) this increase carries over to future purchase occasions. Importantly, I also find evidence for the spacing effect: the likelihood of a product’s purchase increases if it’s ads are spread apart rather than bunched together, even if spreading apart involves shifting some ads away from the purchase occasion. Accounting for the spacing effect is important to detect the effects of repeated advertising. Because the traditional models of advertising do not explain the data patterns, I build a new memory-based model of how advertising influences consumer behavior. Using a nested test, I reject the restrictions imposed by the canonical goodwill stock model (Nerlove and Arrow, Economica, 29(114):129–142, 1962), in favor of the memory-based model I propose. Additionally, I use the estimated parameters to simulate counterfactual scenarios and show that the advertisers’ profits might be lower if the features of the memory model are not accounted for.

Appendices

Appendix A: On measuring the impact of multiple ads

For most part of the analysis, the paper measures the effects of multiple exposures to the experimental ads in a session by regressing the outcome on the number of ad exposures, controlling for the number of pages browsed. If an ad exposure affects subsequent ad exposure (e.g., causes people to conclude their session) then controlling for the number of pages browsed may not solve its purpose of controlling for the browsing type. For example, consider the set of individuals who browsed 10 pages. Within this set, individuals with higher ad exposure are the ones who continued browsing the website after being exposed to the ad a few times. Therefore, among the individuals who browsed 10 pages, ones with higher ad exposure might be the ones that have a lower preference for the advertiser. This possibility can lead to an underestimation of the effect of multiple ad exposures.

1.1 Do experimental ads change the distribution of the number of pages?

One implication of this problem is that the number of pages browsed may change in the ad condition, which may change the number of ad exposures too. To check for this, I compare the empirical distributions of the number of pages browsed by sessions in the Experimental Ad and the No-Ad conditions. I use the two-sample Kolmogorov-Smirnov non-parametric test for this purpose and fail to reject the hypothesis (p-val =0.99) that the distributions are the same. Therefore, in these data, the assumption that the number of pages browsed in the ad condition is same as in the no-ad condition is not very strong.

In the rest of this section, I take an alternative approach to show that the assumptions made for the analysis in the paper are weak and the potential bias in the estimation of the effect of the ad exposures, if any, is insignificant. My approach is to re-construct the average treatment effect (the difference between the ad and the no-ad condition) using the estimate for the effect of one ad exposure. I show that a bias in the latter’s estimate will lead to a mismatch between the implied and the unbiased estimate of the average treatment effect. I take the following steps, (1) get an unbiased estimate of the average treatment effect, $\hat {\gamma }$, by comparing the difference between the mean outcomes in the ad and the no-ad conditions; (2) estimate the average effect of an ad exposure, $\hat {\alpha }$, by regressing the outcome on number of ad exposures, controlling for the number of pages browsed (model P); (3) estimate the average treatment effect predicted by the regression P, $\hat {\gamma }_{P}$($\hat {\alpha }$); and (4) gauge the potential bias in $\hat {\alpha }$ by comparing $\hat {\gamma }$ and $\hat {\gamma }_{P}$.

Step 1::

Measuring the average treatment effect

Recall that the experimental ad condition has a chance of an exposure to the experimental ad, whereas the no-ad condition always shows the dummy banner in the experimental slot. The average treatment effect is the difference in the mean chance of the outcome across the experimental ad and the no-ad condition. For the ease of the next steps in the replication process, I pick the sample from the time when the chance of getting the experimental ad exposure in the ad condition was exactly 60 % (more than half of the data qualifies). The ATE estimate, $\hat {\gamma }$ for this subsample and its confidence interval are shown in Columns I and II of the Table 10 (this is essentially replicating Table 3 for this subsample). $\hat {\gamma }$ is an unbiased estimate of the causal effect of being in a condition where the chance of getting exposed to the experimental ad condition is 60 %.

Table 10 Comparing the average treatment effect of allocating a session to the experimental ad condition with the effect implied by the estimates of the model in Eq. 9

Step 2::

Measuring the effect of an ad exposure

Now within the ad condition, conditional on the number of pages browsed, the number of times the ad is displayed varies. This variation allows me to get an estimate of the population average effect of one ad exposure controlling for the browsing type, by estimating a model P (similar to Table 4, Column I)

$$ Y_{i}=c+\alpha nExp_{i}+\beta nPages_{i}+\eta_{i}, $$

(9)

where n E x p _i is the number of times the experimental ad was displayed in the session and n P a g e s _i is the number of pages browsed in the session i. However, the true model may have a subscript i on the coefficient for n E x p _i . Therefore, a problem with estimating model P is that the independent variables, n E x p _i and n P a g e s _i , are chosen by the consumer i and may depend on the preferences of the individual; α _i →n P a g e s _i , n E x p _i . Estimation of the regression (9) assumes this away, which may lead to a bias because of the unaccounted systematic heterogeneity.

So in the regression (9), the estimated parameter $\hat {\alpha }$ may underestimate the true α = E(α _i ). Let $\hat {\alpha }=\alpha +a$, where a denotes the bias.

Step 3::

Reconstructing the ATE from the model estimate ( $\hat {\gamma }_{P}$ )

I proceed as follows:

• From the population in the No-Ad condition, I get the empirical distribution of n P a g e s.

• For individual i∈{1,...,I}, I draw n P a g e s _i from this empirical distribution.

• As in the experiment design, i draws its treatment n E x p _i ∼B i n o m i a l(n P a g e s _i ,0.60).

For this simulated population of I individuals, average treatment effect is

$$\hat{\gamma}_{P}=\frac{1}{I}\sum\limits_{i}\left( \hat{Y}_{i}(nExp{}_{i},nPages_{i})-\hat{Y}_{i}(0,nPages_{i})\right).</p><p class="noindent">$$

Step 4::

Gauge the magnitude of the bias, ’a’

The idea is to rewrite the treatment effect $\hat {\gamma }_{P}$ as a function of $\hat {\gamma }$ and the bias. One can show using the law of iterated expectations that $\hat {\gamma }_{P}$ is a function of a:

$$\begin{array}{@{}rcl@{}} \hat{\gamma}_{P} & = & \mathrm{E}\left( \mathrm{E}\left( \hat{Y}(nExp,nPages)-\hat{Y}(0,nPages)|nPages\right)\right)\\ & = & \mathrm{E}\left( \mathrm{E}\left( \hat{c}+\hat{\alpha}nExp+\hat{\beta}nPages-\left( \hat{c}+\hat{\beta}nPages\right)|nPages\right)\right)\\ & = & \mathrm{E}\left( \mathrm{E}\left( \hat{\alpha}nExp|nPages\right)\right)\\ & = & \hat{\alpha}\times\mathrm{E}\left( \mathrm{E}\left( nExp|nPages\right)\right)\\ & = & (\alpha+a)\times\mathrm{E}\left( \mathrm{E}\left( nExp|nPages\right)\right)\\ & = & (\alpha+a)\times\mathrm{E}\left( nExp\right)\\ & = & \alpha\times\mathrm{E}\left( nExp\right)+a\times\mathrm{E}\left( nExp\right)\\ & = & \gamma+a\times\mathrm{E}\left( nExp\right). \end{array} $$

The last step follows because E(α×n E x p) is the true average treatment effect γ. This analysis shows if a is large, $\hat {\gamma }_{P}$ is more likely to be off the true value γ. If $\hat {\gamma }$ and $\hat {\gamma }_{P}$ are close, that is, if the model predicts the average treatment effect to be close to the unbiased estimator, the bias size may not be too big.

I compare these estimates shown in Table 10. Note that the estimated $\hat {\gamma }_{P}$ is within the 95 % confidence intervals for $\hat {\gamma }$ for for both visits and leads when used as outcomes. Therefore, this analysis suggests the potential bias a is not large relative to the absolute value of the population mean α. I can draw a similar conclusion if I use a logit regression model for P, instead of OLS.

Appendix B: Impact of ads on website revisit

An implicit assumption made in the analysis is that the future sessions on the website and the time intervals between sessions are not dependent on advertising exposure. In this section, I check for the presence of such correlations in the data. In column I of Table 11, I regress a dummy variable indicating that the user had only one session over the course of the experiments, on allocation of the user’s first session to the ad condition. I find that the coefficient for the dummy variable is statistically not different from zero, indicating no correlation between being in the ad condition and future revisit decisions. In columns II and III, I include the past decisions of whether to visit the advertised restaurant’s page or generate a sales lead as the explanatory variable. Again, I find no significant impact of the experimental ads on future revisit decisions. Next, for individuals that do visit more than once, I regress the time intervals between first two sessions on the allocation to ad condition in the first session. Column IV of Table 11 shows the results of this regression. Again, I find no significant correlation between allocation to the ad condition and website revisit frequency. This evidence supports the assumption that the future revisit frequency is uncorrelated with the experimental ad exposure.

Table 11 Impact of the experimental ads on consumer’s decision to revisit the website

Appendix C: Estimation methodology

I estimate the model parameters using the simulated maximum likelihood method. Given the parameters and data $\left (X_{it}=\left \{ nPages_{it},\left \{nExp_{ik}\right \}_{k=1}^{t}, \{days_{ik}\}_{k=1}^{t},\right . \right .\!\!\!$ $\left .\left .\{x_{ij}\}_{j=1}^{J}\right \} \right )$, the probability of an individual i visiting the advertiser’s page in session t (y _{i t} = 1) according to model ψ∈{G,M G,M} is given by

$$\begin{array}{@{}rcl@{}} Pr_{it}^{\psi} & = & Pr\left( y_{it}=1|X_{it},\theta_{i}^{\psi},\xi_{i}^{\psi}\right)\\ & = & \frac{exp\left( \alpha_{i}^{\psi}+\delta_{i}^{\psi}f(nExp_{it})+\gamma_{i}^{\psi}A_{it}^{\psi}+\phi_{i}^{\psi}nPages_{it}+{\sum}_{j=1}^{J}\tau_{j}^{\psi}x_{ij}\right)}{1+exp\left( \alpha_{i}^{\psi}+\delta_{i}^{\psi}f(nExp_{it})+\gamma_{i}^{\psi}A_{it}^{\psi}+\phi_{i}^{\psi}nPages_{it}+{\sum}_{j=1}^{J}\tau_{j}^{\psi}x_{ij}\right)} \end{array} $$

The likelihood of the sequence of observed decisions $y_{i}=\{y_{it}\}_{t=1}^{T_{i}}$ made by i is then

$$\begin{array}{@{}rcl@{}} \pi_{i}^{\psi}\left( y_{i}|X_{i},\theta_{i}^{\psi},\xi_{i}^{\psi}\right) & = & {\Pi}_{t=1}^{T_{i}}\left( Pr_{it}^{\psi}\times y_{it}+\left( 1-Pr_{it}^{\psi}\right)\times(1-y_{it})\right) \end{array} $$

where T _i is the total number of observed sessions for individual i. Next, because the parameters $\left \{\theta _{i}^{\psi },\xi _{i}^{\psi }\right \}$ are random, I compute the expected likelihood by integrating the individual’s likelihood over the distribution of random parameters:

$$\bar{\pi}_{i}^{\psi}\left( y_{i}|X_{i},\bar{\theta}_{\psi},\bar{\xi}_{\psi},{\Sigma}_{\psi}\right)=\int\pi_{i}^{\psi}\mathrm{d}{\Omega}\left( \theta_{i}^{\psi},\xi_{i}^{\psi}\right) $$

where Ω(.) is the density function for the distribution defined in (8).³³ So the log-simulated likelihood of observed data conditioned on parameters is given by

$$LL^{\psi}\left( \{y_{i},X_{i}\}_{i=1}^{I},\bar{\theta}_{\psi},\bar{\xi}_{\psi},{\Sigma}_{\psi}\right)=\sum\limits_{i=1}^{I}\log\left( \bar{\pi}_{i}^{\psi}\right) $$

The estimates of parameters for the model are the ones that maximize the log-likelihood function.