Assessment of household appliance surveys collected with Amazon Mechanical Turk
Energy researchers need data on residential appliances to make effective recommendations for reducing energy consumption. For some products, however, traditional data sources do not have sufficient detail. Online surveys can provide a less expensive alternative for data collection, but the accuracy of these surveys is still unclear. Here, we compare the results of Amazon Mechanical Turk online surveys of refrigerators, freezers, televisions, and ceiling fans to the nationwide Residential Energy Consumption Survey (RECS) deployed by the US Energy Information Administration. To account for differences in demographic distributions between the online survey results and the general population, we weighted the results using standard cell weighting and raking techniques, as well as a combination of these, termed “hybrid.” The weighted results gave a distribution of product ownership that was reasonably close to RECS, albeit with small, statistically significant differences in some cases. The cell weighting method provided a slightly better agreement with RECS than the other two approaches. We recommend online surveys as an efficient and cost-effective way of gathering in-home use data on appliances that are not adequately covered by existing data sources.
KeywordsAppliances Surveys Amazon Mechanical Turk RECS Post-stratification weighting
- Baker, R., & Le Guin, T. D. (2007). Separating the wheat from the chaff: ensuring data quality in internet samples. In M. Trotman (Ed.), The challenges of a changing world. Proceedings of the fifth ASC international conference, Southhampton, U.K.Google Scholar
- Baker, R., Blumberg, S. J., Brick, J. M., Couper, M. P., Courtright, M., Dennis, J. M., Dillman, D., Frankel, M. R., Garland, P., Groves, R. M., Kennedy, C., Krosnick, J., Lavrakas, P. J., Lee, S., Link, M., Piekarski, L., Rao, K., Thomas, R. K., & Zahs, D. (2010). AAPOR Report on Online Panels. Public Opinion Quarterly, 74(4), 711–781.CrossRefGoogle Scholar
- Baker, R., Brick, J. M., Bates, N. A., Battaglia, M., Couper, M. P., Dever, J. A., Gile, K. J., & Tourangeau, R. (2013). Report of the AAPOR task force on non-probability sampling. Deerfield: American Association for Public Opinion Research.Google Scholar
- Battaglia, M. P., Izrael, D., Hoaglin, D. C., & Frankel, M. R. (2009). Practical considerations in raking survey data. Survey Practice, 2(9).Google Scholar
- Bensch, I., Pigg, S., Koski, K., & Belshe, R. (2010). Electricity savings opportunities for home electronics and other plug-in devices in Minnesota homes: a technical and behavioural field assessment. Energy Center of Wisconsin.Google Scholar
- Cobanoglu, C., Warde, B., & Moreo, P. J. (2001) A comparison of mail, fax, and web-based survey methods. International Journal of Market Research, 43(4) 441–452.Google Scholar
- Greenblatt, J. B., Pratt, S., Willem, H., Claybaugh, E., Desroches, L.-B., Beraki, B., Nagaraju, M., Price, S. K., & Young, S. J. (2013a). Field data collection of miscellaneous electrical loads in Northern California: initial results. Lawrence Berkeley National Laboratory, LBNL report number 6115E, February. http://escholarship.org/uc/item/5cq425kt. Accessed 20 Apr 2014.
- Greenblatt, J. B., Young, S. J., Yang, H. C., Long, T., Beraki, B., Price, S. K., Pratt, S., Willem, H., Desroaches, L. B., & Donovan, S. M. (2013b). U.S. residential miscellaneous refrigeration products: results from Amazon Mechanical Turk surveys. Lawrence Berkeley National Laboratory, LBNL report number 6537E, November.Google Scholar
- Greenblatt, J. B., Yang, H. C., Desroaches, L. B., Young, S. J., Beraki, B., Price, S. K., Pratt, S., & Willem, H. (2013c). U.S. residential consumer product information: validation of methods for post-stratification weighting of Amazon Mechanical Turk. Lawrence Berkeley National Laboratory, LBNL report number 6163E, March.Google Scholar
- Ipeirotis, P. (2010). Demographics of Mechanical Turk. http://www.behind-the-enemy-lines.com/2010/03/new-demographics-of-mechanical-turk.html. Accessed 20 Apr 2014.
- Kalton, G., & Flores-Cervantes, I. (2003). Weighting methods. Journal of Official Statistics, 19(2), 81–97.Google Scholar
- Kantner, C. L. S., Young, S. J., Donovan, S. M., Garbesi, K., Pratt, S., Willem, H., Beraki, B., & Price, S. K. (2013). Ceiling fan and ceiling fan light kit use in the U.S.—results of a Survey on Amazon Mechanical Turk. Lawrence Berkeley National Laboratory, LBNL Report number 6332E, July. http://escholarship.org/uc/item/3r67c1f9. Accessed 2 Oct 2013.
- Loosveldt, G., & Sonck, N. (2008). An evaluation of the weighting procedures for an online access panel survey. Survey Research Methods, 2, 93–105.Google Scholar
- McNary, B., & Berry, C. (2012). How Americans are using energy in homes today. ACEEE Summer Study 2012, 12-17 August, Pacific Grove, CA. http://www.aceee.org/files/proceedings/2012/data/papers/0193-000024.pdf. Accessed 20 Apr 2014.
- Mercier, C., & Moorefield, L. (2011). Commercial Office Plug Load Savings and Assessment. California Energy Commission, PIER Energy‐Related Environmental Research Program. CEC‐500‐08‐049.Google Scholar
- Nielsen. (2012). Nielsen television audience measurement data, 2009-2012. The Nielsen Company, LLC. http://www.nielsen.com/us/en.html.
- Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 411–419.Google Scholar
- Parekh, A., Wang, P., & Strack, T. (2012). Survey results of user-dependent electricity loads in Canadian homes. 2012 ACEEE Summer Study on Energy Efficiency in Buildings.Google Scholar
- Steel, R. G., Torrie, J. H., & Dickey, D. A. (1996). Principles and procedures of statistics: a biometrical approach. New York: McGraw-Hill.Google Scholar
- US Department of Energy: Energy Information Administration. (2011). How does EIA estimate energy consumption and end uses in U.S. homes. http://www.eia.gov/consumption/residential/reports/2009/methodology-end-use.cfm. Accessed 12 Mar 2014.
- US Department of Energy: Energy Information Administration. (2012). Residential energy consumption survey: 2009 RECS survey data. http://www.eia.gov/consumption/residential/data/2009/. Accessed 20 Apr 2014.
- US Department of Energy: Energy Information Administration. (2013a). Annual Energy Outlook: residential sector key indicators and consumption. http://www.eia.gov/oiaf/aeo/tablebrowser/#release=AEO2011&subject=0-AEO2011&table=4-AEO2011®ion=0-0&cases=ref2011-d120810c. Accessed 24 Feb 2014.
- US Department of Energy: Energy Information Administration. (2013b). Residential Energy Consumption Survey (RECS) 2009 Technical Documentation Summary. Washington DC: US Department of Energy.Google Scholar
- Urban, K., Tiefenbeck, V., & Roth, K. (2011). Energy consumption of consumer electronics in U.S. homes in 2010. Fraunhofer Center for Sustainable Energy Systems.Google Scholar
- Zimmermann, J.-P., Evans, M., Griggs, J., King, N., Harding, L., Roberts, P., & Evans, C. (2012). Household Electricity Survey: a study of domestic electrical product usage. Intertek Report R66141, May.Google Scholar