Skip to main content

Advertisement

SpringerLink
Log in

Assessment of household appliance surveys collected with Amazon Mechanical Turk

  • Original Article
  • Published:
Energy Efficiency Aims and scope Submit manuscript

Abstract

Energy researchers need data on residential appliances to make effective recommendations for reducing energy consumption. For some products, however, traditional data sources do not have sufficient detail. Online surveys can provide a less expensive alternative for data collection, but the accuracy of these surveys is still unclear. Here, we compare the results of Amazon Mechanical Turk online surveys of refrigerators, freezers, televisions, and ceiling fans to the nationwide Residential Energy Consumption Survey (RECS) deployed by the US Energy Information Administration. To account for differences in demographic distributions between the online survey results and the general population, we weighted the results using standard cell weighting and raking techniques, as well as a combination of these, termed “hybrid.” The weighted results gave a distribution of product ownership that was reasonably close to RECS, albeit with small, statistically significant differences in some cases. The cell weighting method provided a slightly better agreement with RECS than the other two approaches. We recommend online surveys as an efficient and cost-effective way of gathering in-home use data on appliances that are not adequately covered by existing data sources.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. http://www.ftc.gov/bcp/conline/edcams/eande/appliances/index.htm

  2. http://www.energystar.gov/

  3. http://www.energy.ca.gov/

  4. https://www.npd.com/wps/portal/npd/us/home/

  5. http://www.nielsen.com/us/en.html

  6. http://www.imsresearch.com/

  7. http://www.displaysearch.com/cps/rde/xchg/displaysearch/hs.xsl/index.asp

  8. http://www.aham.org/

  9. http://www.appliancemagazine.com/

  10. “Refrigerator” here refers to a stand-alone refrigerator (sometimes called an “all-refrigerator”) that has no freezer compartment, as well as a refrigerator-freezer that possesses both refrigerator and freezer compartments. “Freezer” here refers to a stand-alone freezer, as opposed to a freezer that is part of a refrigerator.

  11. Set-top boxes are devices that provide TVs with video content from a cable, satellite, or internet service provider.

  12. We created this variable to distinguish households with member(s) under the age of 20 from those without.

  13. An example of demographic distribution comparison across three weighting methods, non-weighted AMT, and the RECS for the refrigeration product survey can be found in part 3 of the supplementary material.

  14. http://www.energy.ca.gov/appliances/rass/

  15. http://www.energy.ca.gov/ceus/

  16. http://www.eia.gov/consumption/commercial/index.cfm

  17. http://neea.org/resource-center/regional-data-resources/residential-building-stock-assessment

References

  • Attari, S. Z. (2013). Perceptions of water use. Proceedings of the National Academy of Sciences of the United States of America, 111(14), 5129–5134.

    Article  MathSciNet  Google Scholar 

  • Attari, S. Z., DeKay, M. L., Davidson, C. I., & Bruine de Bruin, W. (2010). Public perceptions of energy consumption and savings. Proceedings of the National Academy of Sciences of the United States of America, 107(34), 16054–16059.

    Article  Google Scholar 

  • Bailey, B. J. R. (1980). Large sample cell weighting confidence intervals for the multinomial probabilities based on transformation of the cell frequencies. Technometrics, 22, 583–589.

    Article  MATH  MathSciNet  Google Scholar 

  • Baker, R., & Le Guin, T. D. (2007). Separating the wheat from the chaff: ensuring data quality in internet samples. In M. Trotman (Ed.), The challenges of a changing world. Proceedings of the fifth ASC international conference, Southhampton, U.K.

  • Baker, R., Blumberg, S. J., Brick, J. M., Couper, M. P., Courtright, M., Dennis, J. M., Dillman, D., Frankel, M. R., Garland, P., Groves, R. M., Kennedy, C., Krosnick, J., Lavrakas, P. J., Lee, S., Link, M., Piekarski, L., Rao, K., Thomas, R. K., & Zahs, D. (2010). AAPOR Report on Online Panels. Public Opinion Quarterly, 74(4), 711–781.

    Article  Google Scholar 

  • Baker, R., Brick, J. M., Bates, N. A., Battaglia, M., Couper, M. P., Dever, J. A., Gile, K. J., & Tourangeau, R. (2013). Report of the AAPOR task force on non-probability sampling. Deerfield: American Association for Public Opinion Research.

    Google Scholar 

  • Battaglia, M. P., Izrael, D., Hoaglin, D. C., & Frankel, M. R. (2009). Practical considerations in raking survey data. Survey Practice, 2(9).

  • Bensch, I., Pigg, S., Koski, K., & Belshe, R. (2010). Electricity savings opportunities for home electronics and other plug-in devices in Minnesota homes: a technical and behavioural field assessment. Energy Center of Wisconsin.

  • Bethell, C., Fiorillo, J., Lansky, D., Hendryx, M., & Knickman, J. (2004). Online consumer surveys as a methodology for assessing the quality of the United States health care system. Journal of Medical Internet Research, 6(1), e2.

    Article  Google Scholar 

  • Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: a new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3–5.

    Article  Google Scholar 

  • Cherry, S. (1996). A comparison of confidence interval methods for habitat use-availability studies. Journal of Wildlife Management, 60(3), 653–658.

    Article  Google Scholar 

  • Cobanoglu, C., Warde, B., & Moreo, P. J. (2001) A comparison of mail, fax, and web-based survey methods. International Journal of Market Research, 43(4) 441–452.

  • Couper, M. (2000). Web surveys: a review of issues and approaches. Public Opinion Quarterly, 64, 464–494.

    Article  Google Scholar 

  • Deville, J. C., Särndal, C. E., & Sautory, O. (1993). Generalized raking procedures in survey sampling. Journal of the American Statistical Association, 88(423), 1013–1020.

    Article  MATH  Google Scholar 

  • Goodman, J., Cryder, C., & Cheema, A. (2013). Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26, 213–224.

    Article  Google Scholar 

  • Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004). Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. American Psychologist, 59(2), 93–104.

    Article  Google Scholar 

  • Greenblatt, J. B., Pratt, S., Willem, H., Claybaugh, E., Desroches, L.-B., Beraki, B., Nagaraju, M., Price, S. K., & Young, S. J. (2013a). Field data collection of miscellaneous electrical loads in Northern California: initial results. Lawrence Berkeley National Laboratory, LBNL report number 6115E, February. http://escholarship.org/uc/item/5cq425kt. Accessed 20 Apr 2014.

  • Greenblatt, J. B., Young, S. J., Yang, H. C., Long, T., Beraki, B., Price, S. K., Pratt, S., Willem, H., Desroaches, L. B., & Donovan, S. M. (2013b). U.S. residential miscellaneous refrigeration products: results from Amazon Mechanical Turk surveys. Lawrence Berkeley National Laboratory, LBNL report number 6537E, November.

  • Greenblatt, J. B., Yang, H. C., Desroaches, L. B., Young, S. J., Beraki, B., Price, S. K., Pratt, S., & Willem, H. (2013c). U.S. residential consumer product information: validation of methods for post-stratification weighting of Amazon Mechanical Turk. Lawrence Berkeley National Laboratory, LBNL report number 6163E, March.

  • Hicks, A. L., & Theis, T. L. (2013). Residential energy-efficient lighting adoption survey. Energy Efficiency. doi:10.1007/s12053-013-9226-6.

    Google Scholar 

  • Ipeirotis, P. (2010). Demographics of Mechanical Turk. http://www.behind-the-enemy-lines.com/2010/03/new-demographics-of-mechanical-turk.html. Accessed 20 Apr 2014.

  • Kalton, G., & Flores-Cervantes, I. (2003). Weighting methods. Journal of Official Statistics, 19(2), 81–97.

    Google Scholar 

  • Kantner, C. L. S., Young, S. J., Donovan, S. M., Garbesi, K., Pratt, S., Willem, H., Beraki, B., & Price, S. K. (2013). Ceiling fan and ceiling fan light kit use in the U.S.—results of a Survey on Amazon Mechanical Turk. Lawrence Berkeley National Laboratory, LBNL Report number 6332E, July. http://escholarship.org/uc/item/3r67c1f9. Accessed 2 Oct 2013.

  • Lanzisera, S., Dawson-Haggerty, S., Cheung, H. Y. I., Taneja, J., Culler, D., & Brown, R. (2013). Methods for detailed energy data collection of miscellaneous and electronic loads in a commercial office building. Building and Environment, 65, 170–177.

    Article  Google Scholar 

  • Loosveldt, G., & Sonck, N. (2008). An evaluation of the weighting procedures for an online access panel survey. Survey Research Methods, 2, 93–105.

    Google Scholar 

  • McNary, B., & Berry, C. (2012). How Americans are using energy in homes today. ACEEE Summer Study 2012, 12-17 August, Pacific Grove, CA. http://www.aceee.org/files/proceedings/2012/data/papers/0193-000024.pdf. Accessed 20 Apr 2014.

  • Mercier, C., & Moorefield, L. (2011). Commercial Office Plug Load Savings and Assessment. California Energy Commission, PIER Energy‐Related Environmental Research Program. CEC‐500‐08‐049.

  • Nielsen. (2012). Nielsen television audience measurement data, 2009-2012. The Nielsen Company, LLC. http://www.nielsen.com/us/en.html.

  • Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 411–419.

    Google Scholar 

  • Parekh, A., Wang, P., & Strack, T. (2012). Survey results of user-dependent electricity loads in Canadian homes. 2012 ACEEE Summer Study on Energy Efficiency in Buildings.

  • Steel, R. G., Torrie, J. H., & Dickey, D. A. (1996). Principles and procedures of statistics: a biometrical approach. New York: McGraw-Hill.

    Google Scholar 

  • Tourangeau, R., Conrad, F. G., & Couper, M. P. (2013). The science of web surveys (p. 198). New York: Oxford University Press.

    Book  Google Scholar 

  • US Department of Energy: Energy Information Administration. (2011). How does EIA estimate energy consumption and end uses in U.S. homes. http://www.eia.gov/consumption/residential/reports/2009/methodology-end-use.cfm. Accessed 12 Mar 2014.

  • US Department of Energy: Energy Information Administration. (2012). Residential energy consumption survey: 2009 RECS survey data. http://www.eia.gov/consumption/residential/data/2009/. Accessed 20 Apr 2014.

  • US Department of Energy: Energy Information Administration. (2013a). Annual Energy Outlook: residential sector key indicators and consumption. http://www.eia.gov/oiaf/aeo/tablebrowser/#release=AEO2011&subject=0-AEO2011&table=4-AEO2011&region=0-0&cases=ref2011-d120810c. Accessed 24 Feb 2014.

  • US Department of Energy: Energy Information Administration. (2013b). Residential Energy Consumption Survey (RECS) 2009 Technical Documentation Summary. Washington DC: US Department of Energy.

    Google Scholar 

  • Urban, K., Tiefenbeck, V., & Roth, K. (2011). Energy consumption of consumer electronics in U.S. homes in 2010. Fraunhofer Center for Sustainable Energy Systems.

  • Van Ryzin, G. G. (2008). Validity of an on-line panel approach to citizen surveys. Public Performance & Management Review, 32(2), 236–262.

    Article  Google Scholar 

  • Yeager, D. S., Krosnick, J. A., Chang, L., Javitz, H. S., Levindusky, M. S., Simpser, A., & Wang, R. (2011). Comparing the accuracy of RDD telephone surveys and internet surveys constructed with probability and non-probability samples. Public Opinion Quarterly, 75(4), 709–747.

    Article  Google Scholar 

  • Zeifman, M., & Roth, K. (2011). Nonintrusive appliance load monitoring: review and outlook. IEEE Transactions on Consumer Electronics, 57(1), 76–84.

    Article  Google Scholar 

  • Zimmermann, J.-P., Evans, M., Griggs, J., King, N., Harding, L., Roberts, P., & Evans, C. (2012). Household Electricity Survey: a study of domestic electrical product usage. Intertek Report R66141, May.

Download references

Acknowledgments

Bereket Beraki, Sarah K. Price, Stacy Pratt, and Henry Willem provided an invaluable contribution to this project by managing the collection of Amazon Mechanical Turk data. Andrea Alstone, Mia Forbes Pirie, Mohan Ganeshalingam, Karina Garbesi, Samantha Infeld, Colleen Kantner, Erik Page, Alex Valenti, and Vagelis Vossos provided assistance in developing and executing the surveys. Gregory Rosenquist and Alex Lekov provided high-level support and encouragement. We thank the US Department of Energy, Building Technologies Office, for financial support.

Author information

Authors and Affiliations

  1. Lawrence Berkeley National Laboratory, One Cyclotron Road, Berkeley, CA, USA

    Hung-Chia Yang, Scott J. Young, Jeffery B. Greenblatt & Louis-Benoit Desroches

  2. P.O. Box 202, Ocean Grove, Victoria, 3226, Australia

    Sally M. Donovan

Authors
  1. Hung-Chia Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sally M. Donovan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Scott J. Young
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jeffery B. Greenblatt
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Louis-Benoit Desroches
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hung-Chia Yang.

Electronic supplementary material

Below is the link to the electronic supplementary material.

ESM 1

(PDF 220 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, HC., Donovan, S.M., Young, S.J. et al. Assessment of household appliance surveys collected with Amazon Mechanical Turk. Energy Efficiency 8, 1063–1075 (2015). https://doi.org/10.1007/s12053-015-9334-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12053-015-9334-6

Keywords

Search

Navigation