Optimal representative sample weighting

Abstract

We consider the problem of assigning weights to a set of samples or data records, with the goal of achieving a representative weighting, which happens when certain sample averages of the data are close to prescribed values. We frame the problem of finding representative sample weights as an optimization problem, which in many cases is convex and can be efficiently solved. Our formulation includes as a special case the selection of a fixed number of the samples, with equal weights, i.e., the problem of selecting a smaller representative subset of the samples. While this problem is combinatorial and not convex, heuristic methods based on convex optimization seem to perform very well. We describe our open-source implementation rsw and apply it to a skewed sample of the CDC BRFSS dataset.

Appendices

A Iterative proportional fitting

The connection between iterative proportional fitting, initially proposed by Deming and Stephan (1940) and the maximum entropy weighting problem has long been known and has been explored by many authors Teh and Welling (2003), Fu et al. (2019), She and Tang (2019), Wittenberg (2010), Bishop et al. (2007). We provide a similar presentation to She and Tang (2019), Sect. 2.1, though we show that the iterative proportional fitting algorithm that is commonly implemented is actually a block coordinate descent algorithm on the dual variables, rather than a direct coordinate descent algorithm. Writing this update in terms of the primal variables gives exactly the usual iterative proportional fitting update over the marginal distribution of each property.

Maximum entropy problem In particular, we will analyze the application of block coordinate descent on the dual of the problem

$$\begin{aligned} \begin{array}{ll} \text{ minimize } &{} \quad \sum _{i=1}^n w_i \log w_i\\ \text{ subject } \text{ to } &{} \quad Fw = f^\mathrm {des}\\ &{} \quad {\mathbf {1}}^Tw = 1, ~~ w \ge 0, \end{array} \end{aligned}$$

(6)

with variable $w \in {\mathbf{R}}^n$, where the problem data matrix is Boolean, i.e., $F \in \{0, 1\}^{m \times n}$. This is just the maximum entropy weighting problem given in Sect. 3.1, but in the specific case where F is a matrix with Boolean entries.

Selector matrices We will assume that we have several possible categories $k=1, \dots , N$ which the user has stratified over, and we will define selector matrices $S_k \in \{0,1\}^{p_k \times m}$ which ‘pick out’ the rows of F containing the properties for property k. For example, if the first three rows of F specify the data entries corresponding to the first property, then $S_1$ would be a matrix such that

$$\begin{aligned} S_1F = F_{1:3,1:n}, \end{aligned}$$

and each column of $S_1F$ is a unit vector, i.e., a vector whose entries are all zeros except at a single entry, where it is one. This is the same as saying that, for some property k, each data point is allowed to be in exactly one of the $p_k$ possible classes. Additionally, since this should be a proper probability distribution, we will also require that ${\mathbf {1}}^TS_k f^\mathrm {des} = 1$, i.e., the desired marginal distribution for property k should itself sum to 1.

Dual problem To show that iterative proportional fitting is equivalent to block coordinate ascent, we first formulate the dual problem (Boyd and Vandenberghe 2004, Ch. 5). The Lagrangian of (6) is

$$\begin{aligned} \mathcal {L}(w, \nu , \lambda ) = \sum _{i=1}^n w_i \log w_i + \nu ^T(Fw - f^\mathrm {des}) + \lambda ({\mathbf {1}}^Tw - 1), \end{aligned}$$

where $\nu \in {\mathbf{R}}^n$ is the dual variable for the first constraint and $\lambda \in {\mathbf{R}}$ is the dual variable for the normalization constraint. Note that we do not need to include the nonnegativity constraint on $w_i$, since the domain of $w_i \log w_i$ is $w_i \ge 0$.

The dual function (Boyd and Vandenberghe 2004, Sect. 5.1.2) is given by

$$\begin{aligned} g(\nu , \lambda ) = \inf _{w} \mathcal {L}(w, \nu , \lambda ), \end{aligned}$$

(7)

which is easily computed using the Fenchel conjugate of the negative entropy (Boyd and Vandenberghe 2004, Sect. 3.3.1):

$$\begin{aligned} g(\nu , \lambda ) = - {\mathbf {1}}^T\exp (-(1 + \lambda ) {\mathbf {1}}- F^T\nu ) - \nu ^Tf^\mathrm {des} - \lambda , \end{aligned}$$

(8)

where $\exp$ of a vector is interpreted componentwise. Note that the optimal weights $w^\star$ are exactly those given by

$$\begin{aligned} w^\star = \exp ( -(1 + \lambda ) {\mathbf {1}}- F^T\nu ). \end{aligned}$$

(9)

Strong duality Because of strong duality, the maximum of the dual function (7) has the same value as the optimal value of the original problem (6) (Boyd and Vandenberghe 2004, Sect. 5.2.3). Because of this, it suffices to find an optimal pair of dual variables, $\lambda$ and $\nu$, which can then be used to find an optimal $w^\star$, via (9).

To do this, first partially maximize g with respect to $\lambda$, i.e.,

$$\begin{aligned} g^p(\nu ) = \sup _{\lambda } g(\nu , \lambda ). \end{aligned}$$

We can find the minimum by differentiating (8) with respect to $\lambda$ and setting the result to zero. This gives

$$\begin{aligned} 1+\lambda ^\star = \log \left( {\mathbf {1}}^T\exp (-F^T\nu )\right) , \end{aligned}$$

while

$$\begin{aligned} g^p(\nu ) = -\log ({\mathbf {1}}^T\exp (-F^T\nu )) - \nu ^Tf^\mathrm {des}. \end{aligned}$$

This also implies that, after using the optimal $\lambda ^\star$ in (9),

$$\begin{aligned} w^\star = \frac{\exp (- F^T\nu )}{{\mathbf {1}}^T\exp (- F^T\nu )}. \end{aligned}$$

(10)

Block coordinate ascent In order to maximize the dual function $g^p$, we will use the simple method of block coordinate ascent with respect to the dual variables corresponding to the constraints of each of the possible k categories. Equivalently, we will consider updates of the form

$$\begin{aligned} \nu ^{t + 1} = \nu ^{t} + S_{t}^T\xi ^t,\quad t =1, \ldots , T, \end{aligned}$$

where $\nu ^t$ is the dual variable at iteration t, while $\xi ^t \in {\mathbf{R}}^{p_t}$ is the optimization variable we consider at iteration t. To simplify notation, we have used $S_t$ to refer to the selector matrix at iteration t, if $t \le N$, and otherwise set $S_t = S_{(t-1 \mod N) + 1}$, i.e., we choose the selector matrices in a round robin fashion. The updates result in an ascent algorithm, which is guaranteed to converge to the global optimum since $g^p$ is a smooth concave function Tseng (2001).

Block coordinate update In order to apply the update rule to $g^p(\nu )$, we first work out the optimal steps defined as

$$\begin{aligned} \xi ^{t} = {{\,\mathrm{argmax}\,}}_{\xi } ~g^p(\nu ^t + S^T_t \xi ). \end{aligned}$$

To do this, set the gradient of $g_p$ to zero,

$$\begin{aligned} \nabla _\xi ~g^p(\nu ^t + S^T_t \xi ) = 0, \end{aligned}$$

which implies that

$$\begin{aligned} \frac{\sum _{i=1}^n (S_t f_i) \exp (-f_i^T\nu ^t - f_i^TS_t^T\xi )}{\sum _{i=1}^n \exp (-f_i^T\nu ^t - f_i^T S^T_t \xi )} = Sf^\mathrm {des}, \end{aligned}$$

(11)

where $f_i$ is the ith column of F and the division is understood to be elementwise.

To simplify this expression, note that, for any unit basis vector $x \in {\mathbf{R}}^m$ (i.e., $x_i = 1$ for some i and 0, otherwise), we have the simple equality,

$$\begin{aligned} x\exp (x^T\xi ) = x\circ \exp (\xi ), \end{aligned}$$

where $\circ$ indicates the elementwise product of two vectors. Using this result with $x = S_t f_i$ on each term of the numerator from the left hand side of (11) gives

$$\begin{aligned} \sum _{i=1}^n (S_t f_i) \exp (-f_i^T\nu ^t - f_i^TS_t^T\xi ) = \exp (-\xi ) \circ y, \end{aligned}$$

where $y = \sum _{i=1}^m \exp (-f_i^T\nu ^t)Sf_i$. We can then rewrite (11) in terms of y by multiplying the denominator on both sides of the expression:

$$\begin{aligned} \exp (-\xi ) \circ y = (\exp (-\xi )^Ty)S_t f^\mathrm {des}, \end{aligned}$$

which implies that

$$\begin{aligned} \frac{y \circ \exp (-\xi )}{{\mathbf {1}}^T(y \circ \exp (-\xi ))} = S_t f^\mathrm {des}. \end{aligned}$$

Since ${\mathbf {1}}^TS_t f^\mathrm {des} = 1$,

$$\begin{aligned} y \circ \exp (-\xi ) = S_t f^\mathrm {des}, \end{aligned}$$

or, after solving for $\xi$,

$$\begin{aligned} \xi = -\log \left( \mathbf{diag}(y)^{-1}S_t f^\mathrm {des}\right) , \end{aligned}$$

where the logarithm is taken elementwise. The resulting block coordinate ascent update can be written as

$$\begin{aligned} \nu ^{t+1} = \nu ^t - S_t^T \log \left( \frac{S_t f^\mathrm {des}}{\sum _{i=1}^n \exp (-f_i^T\nu ^t)S_t f_i}\right) , \end{aligned}$$

(12)

where the logarithm and division are performed elementwise. This update can be interpreted as changing $\nu$ in the entries corresponding to the constraints given by property t by adding the log difference between the desired distribution and the (unnormalized) marginal distribution for this property suggested by the previous update. This follows from (10), which implies $w_i^t \propto \exp (-f_i^T\nu ^t)$ for each $i=1, \dots , n$, where $w^t$ is the distribution suggested by $\nu ^t$ at iteration t.

Resulting update over w We can rewrite the update for the dual variables $\nu$ as a multiplicative update for the primal variable w, which is exactly the update given by the iterative proportional fitting algorithm. More specifically, from (10),

$$\begin{aligned} w^{t+1}_i = \frac{\exp (-f_i^T\nu ^{t+1})}{\sum _{i=1}^n \exp (-f_i^T\nu ^{t+1})}. \end{aligned}$$

For notational convenience, we will write $x_{t i}= S_t f_i$, which is a unit vector denoting the category to which data point i belongs to, for property t. Plugging update (12) gives, after some algebra,

$$\begin{aligned} \exp (-f_i^T\nu ^{t+1}) = \exp (-f_i^T\nu ^t) \frac{\exp (x_{t i}^T\log (S_t f^\mathrm {des}))}{\exp \left( x_{t i}^T\log \left( \sum _j \exp (-f_j^T\nu ^t)x_{t j}\right) \right) }. \end{aligned}$$

Since $x_{t i}$ is a unit vector, $\exp (x_{t i}^T \log (y)) = x_{t i}^Ty$ for any vector $y > 0$, so

$$\begin{aligned} \exp (-f_i^T\nu ^{t+1}) = \exp (-f_i^T\nu ^t) \frac{x_{t i}^TS_t f^\mathrm {des}}{\sum _{j=1}^n \exp (-f_j^T\nu ^t)x_{t i}^Tx_{t j}}. \end{aligned}$$

Finally, using (10) with $\nu ^t$ gives

$$\begin{aligned} w_i^{t+1} = w_i^t\frac{x_{t i}^TS_t f^\mathrm {des}}{\sum _{j=1}^n w^t_j (x_{t i}^Tx_{t j})}, \end{aligned}$$

Table 1 Desired sex values in percentages

Table 2 Desired education and income values in percentages

Table 3 Desired reported health values in percentages

Table 4 Desired state and age values in percentages

which is exactly the multiplicative update of the iterative proportional fitting algorithm, performed for property t.

B Expected values of BRFSS data

See Tables 1, 2, 3, and 4.