Energy Efficiency

, Volume 8, Issue 6, pp 1063–1075 | Cite as

Assessment of household appliance surveys collected with Amazon Mechanical Turk

  • Hung-Chia Yang
  • Sally M. Donovan
  • Scott J. Young
  • Jeffery B. Greenblatt
  • Louis-Benoit Desroches
Original Article


Energy researchers need data on residential appliances to make effective recommendations for reducing energy consumption. For some products, however, traditional data sources do not have sufficient detail. Online surveys can provide a less expensive alternative for data collection, but the accuracy of these surveys is still unclear. Here, we compare the results of Amazon Mechanical Turk online surveys of refrigerators, freezers, televisions, and ceiling fans to the nationwide Residential Energy Consumption Survey (RECS) deployed by the US Energy Information Administration. To account for differences in demographic distributions between the online survey results and the general population, we weighted the results using standard cell weighting and raking techniques, as well as a combination of these, termed “hybrid.” The weighted results gave a distribution of product ownership that was reasonably close to RECS, albeit with small, statistically significant differences in some cases. The cell weighting method provided a slightly better agreement with RECS than the other two approaches. We recommend online surveys as an efficient and cost-effective way of gathering in-home use data on appliances that are not adequately covered by existing data sources.


Appliances Surveys Amazon Mechanical Turk RECS Post-stratification weighting 

Supplementary material

12053_2015_9334_MOESM1_ESM.pdf (221 kb)
ESM 1(PDF 220 kb)


  1. Attari, S. Z. (2013). Perceptions of water use. Proceedings of the National Academy of Sciences of the United States of America, 111(14), 5129–5134.MathSciNetCrossRefGoogle Scholar
  2. Attari, S. Z., DeKay, M. L., Davidson, C. I., & Bruine de Bruin, W. (2010). Public perceptions of energy consumption and savings. Proceedings of the National Academy of Sciences of the United States of America, 107(34), 16054–16059.CrossRefGoogle Scholar
  3. Bailey, B. J. R. (1980). Large sample cell weighting confidence intervals for the multinomial probabilities based on transformation of the cell frequencies. Technometrics, 22, 583–589.MATHMathSciNetCrossRefGoogle Scholar
  4. Baker, R., & Le Guin, T. D. (2007). Separating the wheat from the chaff: ensuring data quality in internet samples. In M. Trotman (Ed.), The challenges of a changing world. Proceedings of the fifth ASC international conference, Southhampton, U.K.Google Scholar
  5. Baker, R., Blumberg, S. J., Brick, J. M., Couper, M. P., Courtright, M., Dennis, J. M., Dillman, D., Frankel, M. R., Garland, P., Groves, R. M., Kennedy, C., Krosnick, J., Lavrakas, P. J., Lee, S., Link, M., Piekarski, L., Rao, K., Thomas, R. K., & Zahs, D. (2010). AAPOR Report on Online Panels. Public Opinion Quarterly, 74(4), 711–781.CrossRefGoogle Scholar
  6. Baker, R., Brick, J. M., Bates, N. A., Battaglia, M., Couper, M. P., Dever, J. A., Gile, K. J., & Tourangeau, R. (2013). Report of the AAPOR task force on non-probability sampling. Deerfield: American Association for Public Opinion Research.Google Scholar
  7. Battaglia, M. P., Izrael, D., Hoaglin, D. C., & Frankel, M. R. (2009). Practical considerations in raking survey data. Survey Practice, 2(9).Google Scholar
  8. Bensch, I., Pigg, S., Koski, K., & Belshe, R. (2010). Electricity savings opportunities for home electronics and other plug-in devices in Minnesota homes: a technical and behavioural field assessment. Energy Center of Wisconsin.Google Scholar
  9. Bethell, C., Fiorillo, J., Lansky, D., Hendryx, M., & Knickman, J. (2004). Online consumer surveys as a methodology for assessing the quality of the United States health care system. Journal of Medical Internet Research, 6(1), e2.CrossRefGoogle Scholar
  10. Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: a new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3–5.CrossRefGoogle Scholar
  11. Cherry, S. (1996). A comparison of confidence interval methods for habitat use-availability studies. Journal of Wildlife Management, 60(3), 653–658.CrossRefGoogle Scholar
  12. Cobanoglu, C., Warde, B., & Moreo, P. J. (2001) A comparison of mail, fax, and web-based survey methods. International Journal of Market Research, 43(4) 441–452.Google Scholar
  13. Couper, M. (2000). Web surveys: a review of issues and approaches. Public Opinion Quarterly, 64, 464–494.CrossRefGoogle Scholar
  14. Deville, J. C., Särndal, C. E., & Sautory, O. (1993). Generalized raking procedures in survey sampling. Journal of the American Statistical Association, 88(423), 1013–1020.MATHCrossRefGoogle Scholar
  15. Goodman, J., Cryder, C., & Cheema, A. (2013). Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26, 213–224.CrossRefGoogle Scholar
  16. Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004). Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. American Psychologist, 59(2), 93–104.CrossRefGoogle Scholar
  17. Greenblatt, J. B., Pratt, S., Willem, H., Claybaugh, E., Desroches, L.-B., Beraki, B., Nagaraju, M., Price, S. K., & Young, S. J. (2013a). Field data collection of miscellaneous electrical loads in Northern California: initial results. Lawrence Berkeley National Laboratory, LBNL report number 6115E, February. Accessed 20 Apr 2014.
  18. Greenblatt, J. B., Young, S. J., Yang, H. C., Long, T., Beraki, B., Price, S. K., Pratt, S., Willem, H., Desroaches, L. B., & Donovan, S. M. (2013b). U.S. residential miscellaneous refrigeration products: results from Amazon Mechanical Turk surveys. Lawrence Berkeley National Laboratory, LBNL report number 6537E, November.Google Scholar
  19. Greenblatt, J. B., Yang, H. C., Desroaches, L. B., Young, S. J., Beraki, B., Price, S. K., Pratt, S., & Willem, H. (2013c). U.S. residential consumer product information: validation of methods for post-stratification weighting of Amazon Mechanical Turk. Lawrence Berkeley National Laboratory, LBNL report number 6163E, March.Google Scholar
  20. Hicks, A. L., & Theis, T. L. (2013). Residential energy-efficient lighting adoption survey. Energy Efficiency. doi:10.1007/s12053-013-9226-6.Google Scholar
  21. Ipeirotis, P. (2010). Demographics of Mechanical Turk. Accessed 20 Apr 2014.
  22. Kalton, G., & Flores-Cervantes, I. (2003). Weighting methods. Journal of Official Statistics, 19(2), 81–97.Google Scholar
  23. Kantner, C. L. S., Young, S. J., Donovan, S. M., Garbesi, K., Pratt, S., Willem, H., Beraki, B., & Price, S. K. (2013). Ceiling fan and ceiling fan light kit use in the U.S.—results of a Survey on Amazon Mechanical Turk. Lawrence Berkeley National Laboratory, LBNL Report number 6332E, July. Accessed 2 Oct 2013.
  24. Lanzisera, S., Dawson-Haggerty, S., Cheung, H. Y. I., Taneja, J., Culler, D., & Brown, R. (2013). Methods for detailed energy data collection of miscellaneous and electronic loads in a commercial office building. Building and Environment, 65, 170–177.CrossRefGoogle Scholar
  25. Loosveldt, G., & Sonck, N. (2008). An evaluation of the weighting procedures for an online access panel survey. Survey Research Methods, 2, 93–105.Google Scholar
  26. McNary, B., & Berry, C. (2012). How Americans are using energy in homes today. ACEEE Summer Study 2012, 12-17 August, Pacific Grove, CA. Accessed 20 Apr 2014.
  27. Mercier, C., & Moorefield, L. (2011). Commercial Office Plug Load Savings and Assessment. California Energy Commission, PIER Energy‐Related Environmental Research Program. CEC‐500‐08‐049.Google Scholar
  28. Nielsen. (2012). Nielsen television audience measurement data, 2009-2012. The Nielsen Company, LLC.
  29. Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 411–419.Google Scholar
  30. Parekh, A., Wang, P., & Strack, T. (2012). Survey results of user-dependent electricity loads in Canadian homes. 2012 ACEEE Summer Study on Energy Efficiency in Buildings.Google Scholar
  31. Steel, R. G., Torrie, J. H., & Dickey, D. A. (1996). Principles and procedures of statistics: a biometrical approach. New York: McGraw-Hill.Google Scholar
  32. Tourangeau, R., Conrad, F. G., & Couper, M. P. (2013). The science of web surveys (p. 198). New York: Oxford University Press.CrossRefGoogle Scholar
  33. US Department of Energy: Energy Information Administration. (2011). How does EIA estimate energy consumption and end uses in U.S. homes. Accessed 12 Mar 2014.
  34. US Department of Energy: Energy Information Administration. (2012). Residential energy consumption survey: 2009 RECS survey data. Accessed 20 Apr 2014.
  35. US Department of Energy: Energy Information Administration. (2013a). Annual Energy Outlook: residential sector key indicators and consumption. Accessed 24 Feb 2014.
  36. US Department of Energy: Energy Information Administration. (2013b). Residential Energy Consumption Survey (RECS) 2009 Technical Documentation Summary. Washington DC: US Department of Energy.Google Scholar
  37. Urban, K., Tiefenbeck, V., & Roth, K. (2011). Energy consumption of consumer electronics in U.S. homes in 2010. Fraunhofer Center for Sustainable Energy Systems.Google Scholar
  38. Van Ryzin, G. G. (2008). Validity of an on-line panel approach to citizen surveys. Public Performance & Management Review, 32(2), 236–262.CrossRefGoogle Scholar
  39. Yeager, D. S., Krosnick, J. A., Chang, L., Javitz, H. S., Levindusky, M. S., Simpser, A., & Wang, R. (2011). Comparing the accuracy of RDD telephone surveys and internet surveys constructed with probability and non-probability samples. Public Opinion Quarterly, 75(4), 709–747.CrossRefGoogle Scholar
  40. Zeifman, M., & Roth, K. (2011). Nonintrusive appliance load monitoring: review and outlook. IEEE Transactions on Consumer Electronics, 57(1), 76–84.CrossRefGoogle Scholar
  41. Zimmermann, J.-P., Evans, M., Griggs, J., King, N., Harding, L., Roberts, P., & Evans, C. (2012). Household Electricity Survey: a study of domestic electrical product usage. Intertek Report R66141, May.Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2015

Authors and Affiliations

  • Hung-Chia Yang
    • 1
  • Sally M. Donovan
    • 2
  • Scott J. Young
    • 1
  • Jeffery B. Greenblatt
    • 1
  • Louis-Benoit Desroches
    • 1
  1. 1.Lawrence Berkeley National LaboratoryBerkeleyUSA
  2. 2.Ocean GroveAustralia

Personalised recommendations