Beginning Deep Survey Analysis

Abstract

I had divided the analysis of survey data into Shallow Analysis and Deep Analysis. The former just skims the surface of all the data collected from a survey, highlighting only the minimum of findings with the simplest analysis tools. These tools, useful and informative in their own right, are only the first that should be used, not the only ones. They help you dig out some findings but leave much buried. I covered them and their use in Python in the previous chapter.

Appendix

This appendix provides brief overviews and, maybe, refresher material on some key statistical concepts.

4.1.1 Refresher on Expected Values

The expected value of a random variable is just a weighted average of the values of the random variable. The weights are the probabilities of seeing a particular value of the random variable. Although the averaging is written differently depending on whether the random variable is discrete or continuous, the interpretation is the same in either case. If Y  is a discrete random variable, then

$$\displaystyle \begin{aligned} \begin{array}{rcl} E(Y) = \sum_{-\infty}^{+\infty} y_i \times p(y) \end{array} \end{aligned} $$

where p(y) = Pr(Y = y), the probability that Y = y, is a probability function such that

$$\displaystyle \begin{aligned} \begin{array}{rcl} 0 \leq p(y) \leq 1 \\ \sum p(y) = 1 \end{array} \end{aligned} $$

So E(Y ) is just a weighted average. This is the expected value of Y .

If Y  is a continuous random variable, then

$$\displaystyle \begin{aligned} \begin{array}{rcl} E(Y) = \int_{-\infty}^{+\infty} y f(y) dy \end{array} \end{aligned} $$

where

$$\displaystyle \begin{aligned} \begin{array}{rcl} \int_{-\infty}^{+\infty} f(y) = 1 \end{array} \end{aligned} $$

The function, f(y), is the probability density function of Y  at y. It is not the probability of Y = y, which is zero.

It is easy to show the following:

• 1. E(aX) = a × E(X) where a is a constant.

• 2. E(aX + b) = a × E(X) + b where a and b are constants.

• 3. V (aX) = a ² × V (X) where V (⋅) is the variance defined as V (X) = E[X − E(X)]².

• 4. V (aX + b) = a ² × V (X).

It is also easy to show, although I will not do it here, that the expected value of a linear function of random variables is linear. That is, for two random variables, X and Y , and for c _i a constant, then

$$\displaystyle \begin{aligned} \begin{array}{rcl} E( c_1 \times X + c_2 \times Y) = c_1 \times E(X) + c_2 \times E(Y) \\ \end{array} \end{aligned} $$

You can also show that V (Y ₁ ± Y ₂) = V (Y ₁) + V (Y ₂) if Y ₁ and Y ₂ are independent. If they are not independent, then V (Y ₁ ± Y ₂) = V (Y ₁) + V (Y ₂) ± 2 × COV (Y ₁, Y ₂) where COV (Y ₁, Y ₂) is the covariance between the two random variables. For a random sample, Y ₁ and Y ₂ are independent.

4.1.2 Expected Value and Standard Error of the Mean

You can now show that if Y _i, i = 1, 2, …, n, are independent and identically distributed (commonly abbreviated as iid) random variables, with a mean E(Y ) = μ and variance V (Y ) = E(Y − μ)² = σ ², then

$$\displaystyle \begin{aligned} \begin{array}{rcl} E(\bar{Y}) & =&\displaystyle \dfrac{1}{n} \times \sum_{i = 1}^n E(Y_i) \\ & =&\displaystyle \dfrac{1}{n} \times \sum_{i = 1}^n \mu \\ & =&\displaystyle \dfrac{1}{n} \times n \times \mu \\ & =&\displaystyle \mu \end{array} \end{aligned} $$

and

$$\displaystyle \begin{aligned} \begin{array}{rcl} V(\bar{Y}) & =&\displaystyle \dfrac{1}{n^2} \times \sum_{i = 1}^n V(Y_i) \\ & =&\displaystyle \dfrac{1}{n^2} \times \sum_{i = 1}^n \sigma^2 \\ & =&\displaystyle \dfrac{1}{n^2} \times n \times \sigma^2 \\ & =&\displaystyle \dfrac{\sigma^2}{n}. \end{array} \end{aligned} $$

This last result can be extended. Suppose you have two independent random variables, $Y_1 \sim \mathcal {N}(\mu _1, \sigma _1^2)$ and $Y_2 \sim \mathcal {N}(\mu _2, \sigma _2^2)$. Then

$$\displaystyle \begin{aligned} \begin{array}{rcl} V(\bar{Y_1} + \bar{Y_2}) & =&\displaystyle \dfrac{1}{n_1^2} \times \sum_{i = 1}^{n_1} V(Y_{i1}) + \dfrac{1}{n_2^2} \times \sum_{i = 1}^{n_2} V(Y_{i2}) \\ & =&\displaystyle \dfrac{1}{n_1^2} \times \sum_{i = 1}^{n_1} \sigma_1^2 + \dfrac{1}{n_2^2} \times \sum_{i = 1}^{n_2} \sigma_2^2 \\ & =&\displaystyle \dfrac{1}{n_1^2} \times n_1 \times \sigma_1^2 + \dfrac{1}{n_2^2} \times n_2 \times \sigma_2^2\\ & =&\displaystyle \dfrac{\sigma_1^2}{n_1} + \dfrac{\sigma_2^2}{n_2}. \end{array} \end{aligned} $$

This last result is used when two means are compared.

4.1.3 Deviations from the Mean

Two very important results about means are:

• 1. $\sum _{i = 1}^n (Y_i - \bar {Y}) = 0$.

• 2. $E(\bar {Y} - \mu ) = 0$

The first is for a sample; the second is for a population. Regardless, both imply that a function of deviations from the mean is zero. To show the first, simply note that

$$\displaystyle \begin{aligned} \begin{array}{rcl} \sum_{i = 1}^n (Y_i - \bar{Y}) & =&\displaystyle \sum_{i = 1}^n Y_i - n \times \bar{Y} \\ & =&\displaystyle \sum_{i = 1}^n Y_i - n \times \dfrac{1}{n} \sum_{i = 1}^n Y_i \\ & =&\displaystyle \sum_{i = 1}^n Y_i - \sum_{i = 1}^n Y_i \\ & =&\displaystyle 0. \end{array} \end{aligned} $$

The second uses the result I showed above that $E(\bar {Y}) = \mu$. Using this,

$$\displaystyle \begin{aligned} \begin{array}{rcl} E(\bar{Y} - \mu) & =&\displaystyle E\left( \dfrac{1}{n} \sum_{i = 1}^n Y_i\right) - \mu \\ & =&\displaystyle \dfrac{1}{n} \times \sum_{i = 1}^n E(Y_i) - \mu \\ & =&\displaystyle \dfrac{1}{n} \times n \times \mu - \mu \\ & =&\displaystyle 0. \end{array} \end{aligned} $$

4.1.4 Some Relationships Among Probability Distributions

There are several distributions that are often used in survey analyses:

• 1. Normal (or Gaussian) distribution

• 2. χ ² distribution

• 3. Student’s t-distribution

• 4. F-distribution

These are applicable for continuous random variables. They are all closely related, as you will see.

4.1.4.1 Normal Distribution

The normal distribution is the basic distribution; other distributions are based on it. The Normal’s probability density function (pdf) is

$$\displaystyle \begin{aligned} \begin{array}{rcl} f(y) = \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(Y - \mu)^2}{2\sigma^2}} \end{array} \end{aligned} $$

where μ and σ ² are two population parameters. A succinct notation is $Y \sim \mathcal {N}(\mu , \sigma ^2)$. This distribution has several important properties:

• 1. All normal distributions are symmetric about the mean μ.

• 2. The area under an entire normal curve traced by the pdf formula is 1.0.

• 3. The height (i.e., density) of a normal curve is positive for all y. That is, f(y) > 0 ∀y.

• 4. The limit of f(y) as y goes to positive infinity is 0, and the limit of f(y) as y goes to negative infinity is 0. That is,

  $$\displaystyle \begin{aligned} \begin{array}{rcl} \lim\limits_{y \to \infty} f(y) = 0 ~\text{and}~ \lim\limits_{y \to -\infty} f(y) = 0 \end{array} \end{aligned} $$

• 5. The height of any normal curve is maximized at y = μ.

• 6. The placement and shape of a normal curve depends on its mean μ and standard deviation σ, respectively.

• 7. A linear combination of normal random variables is normally distributed. This is the Reproductive Property of Normals.

A standardized normal random variable is Z = (Y − μ)∕σ for $Y \sim \mathcal {N}(\mu , \sigma ^2)$. This can be rewritten as a linear function: Z = 1∕σ × Y − μ∕σ. Therefore, Z is normally distributed by the Reproductive Property of Normals. Also, E(Z) = E(Y )∕σ − μ∕σ = 0 and V (Z) = 1∕σ ² × σ ² = 1. So, $Z \sim \mathcal {N}(0, 1)$.

I show a graph of the standardized normal in Fig. 4.43.

Fig. 4.43

This is the standardized normal pdf

4.1.4.2 Chi-Square Distribution

If $Z \sim \mathcal {N}(0, 1)$, then $Z^2 \sim \chi ^2_1$ where the “1” is one degree-of-freedom. Some properties of the χ ² distribution are:

• 1. The sum of n χ ² random variables is also χ ² with n degrees-of-freedom: $\sum _n Z_i^2 \sim \chi ^2_n$.

• 2. The mean of the $\chi ^2_n$ is n and the variance is 2n.

• 3. The $\chi ^2_n$ approaches the normal distribution as n →∞.

I show a graph of the χ ² pdf for 5 degrees-of-freedom in Fig. 4.44.

Fig. 4.44

This is the χ ² pdf for 5 degrees-of-freedom. The shape changes as the degrees-of-freedom change

4.1.4.3 Student’s t-Distribution

The ratio of two random variables, where the numerator is $\mathcal {N}(0, 1)$ and the denominator is the square root of a χ ² random variable with ν degrees-of-freedom divided by the degrees-of-freedom, follows a t-distribution with ν degrees-of-freedom.

$$\displaystyle \begin{aligned} \begin{array}{rcl} \frac{Z}{\sqrt{\frac{\chi^2_\nu}{\nu}}} \sim t_\nu \end{array} \end{aligned} $$

I show a graph of the Student’s t pdf for 23 degrees-of-freedom in Fig. 4.45.

Fig. 4.45

This is the Student’s t pdf for 23 degrees-of-freedom. The shape changes as the degrees-of-freedom change

4.1.4.4 F-Distribution

The ratio of a χ ² random variable with ν ₁ degrees-of-freedom to a χ ² random variable with ν ₂ degrees-of-freedom, each divided by its degrees-of-freedom, follows an F-distribution with ν ₁ and ν ₂ degrees-of-freedom.

$$\displaystyle \begin{aligned} \begin{array}{rcl} \frac{\chi^2_{\nu_1}/\nu_1}{\chi^2_{\nu_2}/\nu_2} \sim F_{\nu_1 \text{, } \nu_2} \end{array} \end{aligned} $$

Note that the $F_{1, \nu _2}$ is t ². You can see this from the definition of a t with ν degrees-of-freedom: $\frac {Z}{\sqrt {\frac {\chi ^2_\nu }{\nu }}} \sim t_\nu$. I show a graph of the F-distributionpdffor 3 degrees-of-freedom in the numerator and 15 degrees-of-freedom in the denominator in Fig. 4.46.

Fig. 4.46

This is the F-distribution pdf for 3 degrees-of-freedom in the numerator and 15 degrees-of-freedom in the denominator. The shape changes as the degrees-of-freedom change

4.1.5 Equivalence of the F and t Tests for Two Populations

Guenther (1964, p. 46) shows that then there are two independent populations, and the F-test and the t-test are related. In particular, he shows that

$$\displaystyle \begin{aligned} \begin{array}{rcl} F_{1, n_1 + n_2 - 2} = t_{n_1 + n_2 - 2}^2. \end{array} \end{aligned} $$

(4.42)

Part of this demonstration is showing that the denominator of the F-statistic is

$$\displaystyle \begin{aligned} \begin{array}{rcl} \dfrac{SSW}{n_1 + n_2 - 2} & =&\displaystyle \left(\dfrac{(n_1 - 1) \times s_1^2 + (n_2 - 1) \times s_2^2}{n_1 + n_2 - 2}\right) \times \left(\dfrac{1}{n_1} + \dfrac{1}{n_2}\right) \end{array} \end{aligned} $$

(4.43)

which is the result I stated in (4.11) and (4.12).

4.1.6 Code for Fig. 4.37

The code to generate the 3-D bar chart in Fig. 4.37 is shown here in Fig. 4.47. This is longer than previous code because more steps are involved. In particular, you have to define the plotting coordinates for each bar in addition to the height of each bar. The coordinates are the X and Y  plotting positions, and the base of the Z-dimension in the X − Y  plane. This base is just 0 for each bar. Not only are these coordinates needed, but the widths of the bars are also needed. These are indicated in the code as dx and dy. The height of each bar is the Z position in a X − Y − Z three-dimensional plot. This is indicated by dz. The data for the graph comes from the pivot table in Fig. 4.35. I did some processing of this data, which I also show in Fig. 4.47. Notice, incidentally, that I make extensive use of list comprehensions to simplify list creations.

Fig. 4.47

This is the Python code I used to create Fig. 4.37