The impact of search costs on consumer behavior: A dynamic approach

Abstract

Prices for grocery items differ across stores and time because of promotion periods. Consumers therefore have an incentive to search for the lowest prices. However, when a product is purchased infrequently, the effort to check the price every shopping trip might outweigh the benefit of spending less. I propose a structural model for storable goods that takes into account inventory holdings and search. The model is estimated using data on laundry detergent purchases. I find search costs play a large role in explaining purchase behavior, with consumers unaware of the price of detergent on 70 % of their shopping trips. Therefore, from the retailer’s point of view raising awareness of a promotion through advertising and displays is important. I also find a promotion for a particular product increases the consumer’s incentive to search. This change in incentives leads to an increase in category traffic, which from the store manager’s perspective is a desirable side effect of the promotion.

Appendices

Appendix A

1.1 A.1 Household selection

When selecting the households that are included in the estimation, I apply several criteria (all described in the main text). This section provides some further justification for the selection criteria and provides details about how the sample size was affected. The full dataset contains about 40,000 households; the final sample used in the estimation comprises 686 households.

In a first step, I eliminate all households that were in the sample for less than 20 weeks, because information from “uncommitted” consumers that spend only a short period of time in the panel might be less reliable. This exclusion reduces the sample to roughly 31,000 households. I then eliminate all households that bought less than 6 kg of detergent per year and households that did not purchase any detergent for a period of at least 16 weeks. These exclusions eliminate households with extremely low consumption rates and that possibly visit a launderette some of the time. In the sample, 90 % of households buy between 10 and 35 kg, the mean being 20 kg. This finding suggests the 6 kg constitute an unusual behavior that the model will not be able to capture. Similarly, a large gap in purchases might be due to the household going on holiday, which also constitutes an unusual consumption behavior that the model cannot capture. These two criteria decrease the sample size to 12,000. Next, I eliminate all households that bought detergent tablets less than 75 % of the time. This elimination removes households that primarily purchased other types of detergent, such as powder or liquid detergent. This step leaves me with about 2,000 households. Finally, I use only households that bought one of the five brands for which I construct price series at least 75 % of the time. As a result, 686 households fulfill all criteria.

Arguably, some of the criteria applied are quite conservative. For example, one might try to eliminate longer periods without purchases but still keep most of the time series for a particular household. I do not make such an attempt here, as I end up with a fairly large sample of households compared to other papers in the literature (e.g., Hendel and Nevo 2006). I also have a much longer time series of purchases (6 years) than what is available in other datasets used to analyze demand for similar products. Therefore, very little is lost by eliminating households in a conservative way if one is doubtful about irregularities in their behavior. Also, other papers such as Osborne (2011) take a random sample of all households to reduce the computational burden in the estimation. Instead, I prefer to apply the conservative criteria of elimination outlined above.

1.2 A.2 Outline of a model of shopping basket size and composition choice

Section 3.1 of the paper shows the size and composition of the shopping basket influences the purchase probability for detergent. Of course, the size and composition of the whole shopping basket is not truly exogenous, but itself part of a larger optimization problem that the consumer has to solve. In other words, the decision to search and purchase any product is part of the consumer’s decision making process subject to certain constraints. Therefore, one could trace back the reasons for why search costs for detergent vary as a function of the shopping basket composition to more basic underlying primitives. To do so, one can think of a model in which the consumer decides whether to search for price information across all products in the supermarket (and possibly decides to make a purchase). Although I do not attempt to formally derive (or estimate) such a model, sketching out the trade-offs inherent in such a model is still instructive. Assume the consumer enters the store with a certain need for various products. The inventory of the product he holds at home and his future consumption needs determine the immediacy of the need. Also, time spent in the store is costly for the consumer. Even without any formal derivations, one can intuitively think of the type of predictions that can be obtained: (1) When the opportunity cost of time is high, a consumer will only want to stock up on the most necessary items. He will therefore purchase fewer items on such a trip and engage in less search. Furthermore, he is more likely to purchase perishable goods, which need to be stocked up on more frequently rather than durable products such as laundry detergent. (2) Assume the consumer has already searched for a product in a particular product category. The marginal cost of searching for a further product within the category is lower than for another product in a different category, due to products typically being arranged by category. This type of cost saving in the time spent searching gives consumers an incentive to lump together purchases within a category on a particular trip. The mechanisms just described will create the type of correlation between shopping basket size and composition with the purchase probabilities for detergent reported in Section 3.1. The variation in the shopping basket composition, although being an outcome of the consumer’s decision problem, will therefore reflect differences in the search cost for detergent, caused by the underlying variation in the time constraint and the consumption needs across various products. In the empirical model, I will treat the basket composition as exogenous to the search and purchase decision regarding detergent. This simplification is necessary in order to make the model tractable.

1.3 A.3 Selection of trips in Section 3.2 consumers missing promotions

To compute the percentages presented in the table, I first have to define a promotion. I assume (as described in Section 2.4) the 75th percentile of the price distribution of each brand at a particular supermarket to be the regular price. As promotions are infrequent and because the regular price varies little over time, this definition is appropriate. The 75th percentile will always lie outside of the promotion range of the price distribution. I define a promotion as a price that is at least 20 % below the regular price.

I then compute the identity of the product purchased and the price of the product for every purchase made. In the next step, I look up the price for the purchased product on every shopping trip of the same consumer that happened before the actual purchase and after the previous purchase in the detergent category. I drop any previous trip to a store where the particular product was not available. This procedure allows me to find out whether the product purchased had been on promotion on any previous trip of the same consumer for any arbitrary time window. Regardless of the time window, it will always go no further back than the first trip after the previous purchase. Because detergent is purchased infrequently, this constraint is usually not binding for the one-week time window used in the table. I also eliminate all trips to supermarkets in the “Other” category, as I do not have reliable price information for those trips. I do not consider these trips, both in terms of actual purchases and in terms of possible purchases on previous trips.

1.4 A.4 Implementation of the product fixed effects

To control for unobserved product quality, I estimate a set of five brand intercepts for a total of six brands in the sample. I impose two constraints on the brand intercepts: (1) The lowest-valued brand has an intercept of zero (min _j [ξ _j  = 0]); I do not estimate this intercept, but impose it in the estimation. In practice, the optimization algorithm searches over a set of five differences between the six ξ-terms. This way of constructing the algorithm gives me a unique ranking between all six brands. Setting the lowest-value brand’s intercept to zero, I can use the differences to obtain the remaining ξ-terms. (2) All intercepts are constrained to be non-negative (ξ ≥ 0). Constraint (1) is a standard normalization constraint, as only quality difference between brands can be identified. I further assume unobserved product quality is a brand-specific characteristics and is scaled-up linearly when a consumer purchases a larger pack size. That is, compared to a 1 kg pack size, a 2 kg pack gives twice as much unobserved product quality. Due to the multiplication of the brand intercepts with the pack size purchased, imposing constraint (2) is necessary. It ensure that large pack sizes provide higher utility than small pack sizes. Negative intercepts would imply lower utility from larger pack sizes, which is economically counter-intuitive. Note that the non-negativity constraint may seem to make the outside option of not purchasing relatively less attractive; however, the brand intercepts capture quality differences relative to the normalized brand, but not relative to the outside option. The dynamic parameters such as search and storage costs primarily govern the relationship between the outside option and the available purchase options.

1.5 A.5 Discretizing the state variables

To implement the dynamic programming problem, I discretize the price distribution as well as the search-cost distribution and the inventory variable. The probability distribution regarding the store visited in the next time period is already a discrete distribution by construction.

To cover all the possible price realizations of any of the available products, I construct a grid for prices between 0 and 16 British pounds. A price of zero never occurs; therefore, I use this grid point to deal with temporarily unavailable products. Products are made effectively unavailable by assigning them an extremely high price (99999 instead of 0), which reduces utility from this option to minus infinity. This grid point (which captures product availability) will enter the consumer’s expectations about futures prices together with the price distribution conditional on availability. I use 25 gridpoints, which makes the grid fine enough to capture the typical promotion depth for any pack size and brand available during the sample period.

To discretize the distribution of future expected search costs, I do the following: Based on the estimates of $\widetilde{s}$ and the vector γ, I calculate the search costs for all the shopping trips and compute the distribution of the search costs over a set of grid-points. Defining the grid is made particularly easy by the functional form chosen, as the search cost has to be an element of the compact interval $\widetilde{s} * [0,1]$. I therefore use a grid of values between zero and 1 and compute the distribution of $\exp(x_{t}' \beta)/[1+\exp(x_{t}' \beta)]$. I then multiply this term by $\widetilde{s}$. I use 11 grid points for the search-cost distribution in the estimation.

Finally, I discretize inventory using a grid ranging from 0 to 15 kg of detergent inventory with 30 grid points in the dynamic problem. When constructing the inventory variable for each consumer, I allow the transition (as a function of the consumption rate and the pack size of a purchase) to be continuous. For every choice available, I compute the expected value function by linearly interpolating between the value functions defined for the closest grid point to the left and to the right of the actual inventory value (which resulted from the continuous transition process). Because the grid is relatively fine, the method of interpolation presumably does not have a large effect.

1.6 A.6 Initial inventory

When estimating the model I have to deal with the problem of an unknown initial inventory. Note that the consumption rate is convex by construction. Once the inventory falls below the rate (τ), consumption is reduced until it becomes zero when the inventory is completely depleted. Because of this, the impact of the initial inventory will fade over time. I start with the first observed purchase for each household and assume that no inventory was held before that time period. I then calculate the evolution of the inventory implied by the estimated consumption parameter τ and the observed purchases. Only after the first ten trips is the observed behavior used in order to form the likelihood function. This helps to mitigate the initial inventory problem. Note, that I deal with the initial condition problem in a similar way as Hendel and Nevo (2006). Erdem et al. (2003) instead use a more sophisticated approach. They simulate purchases and consumption for an extended period of time prior to the observed data and use several sets of simulated inventories as the initial condition(s). The likelihood is obtained by averaging over the individual likelihoods for each simulated initial condition. As a sensitivity check, I also tried excluding the first 20 trips instead of only 10 from the estimation. This had little impact on the results.

1.7 A.7 Sensitivity checks

I ran several tests to check the sensitivity of the estimates. First, one might worry some reverse causality exists for the search-cost shifters. This type of mechanism is particularly worrisome in the case of the “number of other cleaning products purchased” search-cost shifter, as the consumer might decide to buy detergent and therefore also buy other cleaning products. Presumably, buying detergent is less likely to have an influence on the overall trip expenditure or the number of household products, which is a wide category. Therefore, I re-estimated the model, dropping the “number of other cleaning products purchased” as a shifter of the search costs. I report the results of this regression together with the baseline results in Table 16 of the Appendix. Relatively little change occurs in the parameter estimates of the maintained parameters. Furthermore, I tried to include a quadratic storage-cost term for both types of households. Both coefficients are positive but not significantly different from zero.

1.8 A.8 Implementing the simulation

To analyze the predictions from the model, I need to simulate consumer behavior based on the parameter estimates of the model. To this end, I take draws from the distribution of error terms, determine the optimal choice for each consumer, and aggregate over the choices of all simulated consumers to obtain the market shares for each choice. I also randomly draw from the empirical distribution of the variables the identity of the store and the prices that a consumer faces in a particular time period. The probability distributions of store visits and prices can easily be computed from the raw data. In both cases, I compute the distribution from sample frequencies (of store visits/store-specific prices), and it is therefore independent of any parameters of the estimation. I discretize the price distribution for this purpose; the store-visit distribution is discrete by construction. I use a total of 5,000 simulated consumers for each of the two types and simulate the behavior over 1,000 weeks. I calculate total market shares by weighing the market share of each type of consumer with the estimated weight of the respective type.

Finally, consumers also differ by the inventory they hold in a particular time period. To get sensible results from the simulation, I need to know the inventory distribution implied by the model. I find this distribution by starting at an arbitrary distribution. I then simulate consumer behavior for enough time periods such that the distribution reaches a steady state. Specifically, I start by assigning an inventory of zero to each consumer. I then simulate the consumer’s behavior over 100 time periods and update the inventory each period according to the rate of depreciation derived in the estimation and the simulated purchases. The inventory changes little from period to period at the end of the 100 simulated time periods. Therefore, the impact of the initial inventory should have faded completely after this time span. I then use this “steady-state” inventory distribution as the initial inventory for the various simulation exercises in the results section, the validation, and the counterfactuals.

When looking at market shares in promotion periods and regular-price weeks (for the validation and the two counterfactuals), I aggregate the market shares separately for all promotion and regular-price weeks. I compute the elasticity by comparing the percentage difference in market share with the price change implied by the promotion. I therefore do not use one particular promotion to assess the effect on demand. Instead, I look at all promotions (which are by construction randomly timed) and average over all promotional weeks. I draw the prices of all other products from their empirical distribution in every week. The promotion of a particular product can therefore sometimes coincide with promotions for other products, as implied by the price distributions. This way of constructing price elasticities is as close as possible to the choice situations consumers face in reality. This makes the elasticities comparable to those calculated from the raw data.

Furthermore, in the case of all three products reported in Table 8, I set the depth and frequency of the promotion as close as possible to the actual price patterns. A 900 g pack of Ariel at Morrisons was promoted 3.4 % of the time with a 27 % discount. A 900 g pack of Tesco’s Private Label at Tesco was promoted 19 % of the time with a 26 % discount. A 1,900 g pack of Ariel at Morrisons was promoted 4.7 % of the time with a 21 % discount.

1.9 A.9 Pack-size intercepts

One weakness of the paper lies in the fact that the demand model cannot predict pack-size choice particularly well. In principle, a set of pack-size fixed effects could help improve the fit in this respect. Currently, preferences over different pack sizes are modeled in a fairly restrictive way; they are primarily influenced by search and storage costs. High storage costs make buying large packs less attractive, whereas high search costs make large pack sizes more attractive in order to save on future search costs (by purchasing less often). Of course there might exist other explanation for why consumers would prefer different pack sizes that are not captured in the model. However, consumer choice across pack sizes (in response to non-linear pricing over pack sizes, namely, quantity discounts) is of crucial importance for identification. As laid out in Section 5.2, pack-size choice is one of three types of variation that help to identify the price coefficient, and search and storage costs. The inclusion of a set of pack-size fixed effects would “soak up” all variation in the pack-size choice dimension and therefore make identifying all three key parameters from the remaining variation impossible. The limited flexibility in pack-size preferences is the cost of achieving identification in the model presented here.

Appendix B: Tables

Table 11 Variation in expenditure and shopping basket composition

Table 12 Descriptive statistics: aggregate market shares

Table 13 Descriptive statistics: prices and market shares of the brand fairy

Table 14 Hazard model to predict start of promotion period

Table 15 Monte-Carlo simulations: results for three different sets of parameter values

Table 16 Sensitivity check: comparison of the baseline model with a model with fewer search-cost shifters

Table 17 Comparison of the baseline model with a model without search costs