Designing a Survey or Experiment: Deciding What and How to Measure

Abstract

Over the past few decades, storing data has become increasingly affordable. One measure of this, hard drive cost per gigabyte in U.S. Dollars, has fallen from over $10,000 in 1990 to about 3 cents in 2014 according to data available at http://www.mkomo.com/cost-per-gigabyte-update. This steep decline has had a number of consequences, one of which is the explosion in popularity of so-called “big data”. Technology companies are able to leverage this resource to conduct experiments at breakneck speeds and scales that would have been unthinkable at any other time in human history.

Chapter 3 Comprehensive Exercises

3.1.1 3.A. Conceptual

3.A.1. A doctor wants to investigate the effect of a cholesterol-lowering drug on mental acuity, so she recruits 20 patients from the hospital she works at and conducts a series of tests to measure mental fitness. The patients are selected in such a way that 10 of them take the cholesterol-lowering drug and 10 of them do not. Is this an observational study or an experiment?

3.A.2. Suppose that the doctor in the previous exercise instead recruits 20 patients upon entry to the hospital and gives the drug to the first 10 patients and does not give the drug to the last 10 patients.

• (a) Is this an observational study or an experiment? Explain.

• (b) Are there any aspects of the doctor’s implementation which you would change?

3.A.3. Fish Oil and Epilepsy. DeGiorgio et al. (2014) investigated the effects of fish oil on drug-resistant epilepsy. They were interested in testing both low dosage and high dosage amounts versus a placebo. However, due to practical limitations, they were only able to recruit 24 patients who participated throughout the study. Further complicating things was the fact that the baseline number of seizures varied widely (from as little as 3 per month to as many as 60 per month). Discuss how you would design an ideal study to examine these effects if the number of patients was not an issue and then discuss how you might work around this limitation.

3.1.2 3.B. Data Analysis /Computational

3.B.1. Are Mac Users Charged More for Travel? A 2012 Wall Street Journal article suggested that Mac users would pay $20 to $30 more per night for hotels on the online travel site Orbitz than PC users would. Partner with a classmate to design an experiment to test the validity of this claim. Collect and report the data from your experiment.

3.B.2. Why Do Mac Users Pay More? The article discussed in Exercise 3.B.1 was often misrepresented by third parties as claiming that Mac users would be charged more for the same hotel rooms. However, the article actually suggests that the Mac users would be steered to higher-end hotels, which end up costing $20 to $30 more per night. With this in mind, discuss any changes you would make to the design of your experiment in Exercise 3.B.1 and collect new data according to this updated experimental design.

3.B.3. Police Body-Worn Cameras and Citizen Complaints. Citizen complaints against police officers are often seen as being indicative of the level of compliance with police procedure and proper conduct. Ariel et al. (2016) conducted a study of the effect of police body-worn cameras on the number of complaints made by citizens. They randomly assigned officer shifts at seven police departments to either wear cameras or not wear them on a weekly basis. The results of their study are displayed in Table 3.8.

Table 3.8 Effect of body-worn cameras on citizen complaints against Police

• (a) Was this an experiment or an observational study?

• (b) What is the treatment and what is the control? From looking at the data, do you think there is a strong treatment effect?

• (c) Both the treatment and the control groups appear to have many fewer complaints than the same groups did before the experiment started. Why do you think this might be?

• (d) The researchers initially reached out to 10 police departments and then reported the results based on the 7 sites which agreed to participate. What effect might this have on the results of the study?

3.1.3 3.C. Activities

3.C.1. Effect of Salt on Ice . You will need a tray of ice cubes and some table salt for this activity. The goal will be to determine whether salt causes ice to melt faster. Design an experiment to test this hypothesis. Describe your procedure, including what the experimental units are, what the outcome of interest is, what the treatment and control groups are, what process you used to assign experimental units to each group, and discuss any data that you collect.

3.C.2. M & M Colors . Do the colors of M&M’s vary by which type of M&M they are? Although the company no longer posts the proportion of colors in each batch of M&M’s, various sources on the Internet claim that there are, in fact, different color mixes for different types of M & M’s. To test this, purchase two bags of M&M’s, one each of two different types (for example, milk chocolate and peanut) and record the proportion of orange M&M’s in each bag. Is there much difference in these proportions? Aggregate your results with those of your classmates to obtain a larger set of data. Is this an experiment or an observational study?

3.C.3. Heart Rate and Exercise. How do various activities affect your heart rate? Gather the necessary materials to measure your pulse and then measure it separately after performing 30 seconds of each of the following activities: jumping jacks, walking, sprinting, sitting in a chair, and lying on the ground.

• (a) Is this an observational study or an experiment?

• (b) What would you consider to be the placebo or baseline heart rate?

• (c) Do you think the order in which these activities are performed will affect your results? If so, list a few steps that you can take to eliminate these effects.

3.1.4 3.D. Internet Archives

3.D.1. The polling company Reuters routinely conducts surveys on various topics of interest to Americans. You can view the results of recent polls in categories such as “Government & Policy”, “Business & Finance”, “Society & Lifestyle”, and more by visiting http://polling.reuters.com/. Select a poll on a topic you find interesting or controversial.

• (a) How many people participated in the poll?

• (b) How were these people selected?

• (c) Discuss any aspects of the poll that you might find surprising.

3.D.2. NASA Experiments. NASA makes available a list of all experiments that have been completed or are currently being conducted on the International Space Station (ISS) at http://www.nasa.gov/mission_pages/station/research/experiments/experiments_by_name.html. Find an experiment of interest to you and summarize it.

• (a) How were the data collected?

• (b) What were the treatment and control groups?

• (c) What was the conclusion of the experiment and what practical use does this knowledge have?