Abstract
Ecological momentary assessment (EMA) is a self-report method that involves intensive longitudinal assessment of behavior and environmental conditions during everyday activities. EMA has been used extensively in health and clinical psychology to investigate a variety of health behaviors, including substance use, eating, medication adherence, sleep, and physical activity. However, it has not been widely implemented in behavior analytic research. This is likely an example of the empirically based skepticism with which behavioral scientists view self-report measures. We reviewed studies comparing electronic, mobile EMA (mEMA) to more objective measures of health behavior to explore the validity of mEMA as a measurement tool, and to identify procedures and factors that may promote the accuracy of mEMA. We identified 32 studies that compared mEMA to more objective measures of health behavior or environmental events (e.g., biochemical measures or automated devices such as accelerometers). Results showed that the correspondence rates varied considerably across individuals, behavior, and studies (agreement rates ranged from 1.8%–100%), and no unifying variables could be identified across the studies that found high correspondence. The findings suggest that mEMA can be an accurate measurement tool, but further research should be conducted to identify procedures and variables that promote accurate responding.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abel, D. B., & Minor, K. S. (2021). Social functioning in schizophrenia: comparing laboratory-based assessment with real-world measures. Journal of Psychiatric Research, 138, 500–506. https://doi.org/10.1016/j.jpsychires.2021.04.039
Baer, D. M., Wolf, M. M., & Risley, T. R. (1968). Some current dimensions of applied behavior analysis. Journal of Applied Behavior Analysis, 1(1), 91–97. https://doi.org/10.1901/jaba.1968.1-91
Baillet, M., Schweitzer, P., Pérès, K., Catheline, G., Swendsen, J., & Mayo, W. (2016). Mood influences the concordance of subjective and objective measures of sleep duration in older adults. Frontiers in Aging Neuroscience, 8, 181. https://doi.org/10.3389/fnagi.2016.00181
Bakdash, J. Z., & Marusich, L. R. (2017). Repeated measures correlation. Frontiers in Psychology, 8, 456. https://doi.org/10.3389/fpsyg.2017.00456
Beckjord, E., & Shiffman, S. (2014). Background for real-time monitoring and intervention related to alcohol use. Alcohol Research, 36, 9–18.
Bertz, J. W., Epstein, D. H., & Preston, K. L. (2018). Combining ecological momentary assessment with objective, ambulatory measures of behavior and physiology in substance-use research. Addictive Behaviors, 83, 5–17. https://doi.org/10.1016/j.addbeh.2017.11.027
Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. Lancet, 1(8476), 307–310.
Bruening, M., van Woerden, I., Todd, M., Brennhoffer, S., Laska, M., & Dunton, G. (2016). A mobile ecological momentary assessment tool (devilSPARC) for nutrition and physical activity behaviors in college students: A validation study. Journal of Medical Internet Research, 18(7), e209. https://doi.org/10.2196/jmir.5969
Broden, M., Hall, R. V., & Mitts, B. (1971). The effect of self-recording on the classroom behavior of two eighth-grade students. Journal of Applied Behavior Analysis, 4(3), 191–199. https://doi.org/10.1901/jaba.1971.4-191
Burke, T. A., Fox, K., Marin, K., Siegel, D. M., Kleiman, E., & Alloy, L. B. (2020). Real-time monitoring of the associations between self-critical and self-punishment cognitions and nonsuicidal self-injury. Behavioral Research & Therapy, 137, 103775. https://doi.org/10.1016/j.brat.2020.103775
Byrnes, H. F., Miller, B. A., Morrison, C., Wiebe, D. J., Woychick, M., & Wiehe, S. E. (2017). Association of environmental indicators with teen alcohol use and problem behavior: Teens observations vs. objectively-measured indicators. Health Place, 43, 151–157. https://doi.org/10.1016/j.healthplace.2016.12.004
Carretero, P., Campana-Montes, J. J., & Artes-Rodriguez, A. (2020). Ecological momentary assessment for monitoring risk of suicide behavior. Current Topics in Behavioral Neurosciences, 46, 229–245. https://doi.org/10.1007/7854_2020_170
Chai, T., & Draxler, R. (2014). Root mean square error (RMSE) or mean absolute error (MAE)? Arguments against avoiding RMSE in the literature. Geoscientific Model Development, 7, 1247–1250. https://doi.org/10.5194/gmd-7-1247-2014
Chen, K. Y., Acra, S. A., Majchrzak, K., Donahue, C. L., Baker, L., Clemens, L., Sun, M., & Buchowski, M. S. (2003). Predicting energy expenditure of physical activity using hip- and wrist-worn accelerometers. Diabetes Technology & Therapeutics, 5(6), 1023–1033. https://doi.org/10.1089/152091503322641088
Collins, R. L., Morsheimer, E. T., Shiffman, S., Paty, J. A., Gnys, M., & Papandonatos, G. D. (1998). Ecological momentary assessment in a behavioral drinking moderation training program. Experimental and Clinical Psychopharmacology, 6(3), 306–315. https://doi.org/10.1037//1064-1297.6.3.306
Cooper, M. R., Case, K. R., Hébert, E. T., Vandewater, E. A., Raese, K. A., Perry, C. L., & Businelle, M. S. (2019). Characterizing ENDS use in young adults with ecological momentary assessment: Results from a pilot study. Addictive Behaviors, 91, 30–36. https://doi.org/10.1016/j.addbeh.2018.11.024
Consolvo, S., & Walker, M. (2003). Using the experience sampling method to evaluate Ubicomp applications. IEEE Pervasive Computing, 2(2), 24–31. https://doi.org/10.1109/MPRV.2003.1203750
Courville, T., & Thompson, B. (2001). Use of structure coefficients in published multiple regression articles: β is not enough. Educational & Psychological Measurement, 61(2), 229–248. https://doi.org/10.1177/0013164401612006
Critchfield, T. S., Tucker, J. A., & Vuchinich, R. E. (1998). Self-report methods. In K. A. Lattal & M. Perone (Eds.), Handbook of research methods in human operant behavior (pp. 435–470). Plenum Press. https://doi.org/10.1007/978-1-4899-1947-2_14
Critchfield, T. S., & Reed, D. D. (2017). The fuzzy concept of applied behavior analysis research. Behavior Analyst, 40, 123–159. https://doi.org/10.1007/s40614-017-0093-x
Degroote, L., DeSmet, A., De Bourdeaudhuij, I., Van Dyck, D., & Crombez, G. (2020). Content validity and methodological considerations in ecological momentary assessment studies on physical activity and sedentary behaviour: A systematic review. International Journal of Behavioral Nutrition & Physical Activity, 17(1), 35. https://doi.org/10.1186/s12966-020-00932-9
Dunton, G., Liao, Y., Intille, S., Spruijt-Metz, D., & Pentz, M. (2011). Investigating children’s physical activity and sedentary behavior using ecological momentary assessment with mobile phones. Obesity, 19(6), 1205–1212. https://doi.org/10.1038/oby.2010.302
Dunton, G. F., Liao, Y., Kawabata, K., & Intille, S. (2012). Momentary assessment of adults’ physical activity and sedentary behavior: Feasibility and validity. Frontiers in Psychology, 3, 260. https://doi.org/10.3389/fpsyg.2012.00260
Dunton, G. F., Whalen, C. K., Jamner, L. D., Henker, B., & Floro, J. N. (2005). Using ecologic momentary assessment to measure physical activity during adolescence. American Journal of Preventive Medicine, 29(4), 281–287. https://doi.org/10.1016/j.amepre.2005.07.020
Elliston, K. G., Schüz, B., Albion, T., & Ferguson, S. G. (2020). Comparison of geographic information system and subjective assessments of momentary food environments as predictors of food intake: An ecological momentary assessment study. JMIR Mhealth Uhealth, 8(7), e15948. https://doi.org/10.2196/15948
Finney, J. W., Putnam, D. E., & Boyd, C. M. (1998). Improving the accuracy of self-reports of adherence. Journal of Applied Behavior Analysis, 31, 485–488. https://doi.org/10.1901/jaba.1998.31-485
Fixsen, D. L., Phillips, E. L., & Wolf, M. (1972). Achievement place: The reliability of self-reporting and peer-reporting and their effects on behavior. Journal of Applied Behavior Analysis, 5(1), 19–30. https://doi.org/10.1901/jaba.1972.5-19
Floro, J. N., Dunton, G. F., & Delfino, R. J. (2009). Assessing physical activity in children with asthma: Convergent validity between accelerometer and electronic diary data. Research Quarterly for Exercise & Sport, 80(2), 153–163. https://doi.org/10.1080/02701367.2009.10599549
Fuller-Tyszkiewicz, M., Skouteris, H., Richardson, B., Blore, J., Holmes, M., & Mills, J. (2013). Does the burden of the experience sampling method undermine data quality in state body image research? Body Image, 10(4), 607–613. https://doi.org/10.1016/j.bodyim.2013.06.003
Halsey, L. G., Curran-Everett, D., Vowler, S. L., & Drummond, G. B. (2015). The fickle P value generates irreproducible results. Nature Methods, 12(3), 179–185. https://doi.org/10.1038/nmeth.3288
Han, H., Zhang, J., Hser, Y., Liang, D., Li, X., Wang, S., Du, J., & Zhao, M. (2018). Feasibility of a mobile phone app to support recovery from addiction in China: Secondary analysis of a pilot study. JMIR Mhealth Uhealth, 6(2), Article e46. https://doi.org/10.2196/mhealth.8388
Hartmann, D. P., & Wood, D. D. (1990). Observational methods. In A. S. Bellack, M. Hersen, & A. E. Kazdin (Eds.), International handbook of behavior modification & therapy (pp.107–138). Springer.
Helgeson, V. S., Lopez, L. C., & Kamarck, T. (2009). Peer relationships and diabetes: retrospective and ecological momentary assessment approaches. Health Psychology, 28(3), 272–282. https://doi.org/10.1037/a0013784
Jenstad, L. M., Singh, G., Boretzki, M., DeLongis, A., Fichtl, E., Ho, R., Huen, M., Meyer, V., Pang, F., & Stephenson, E. (2021). Ecological momentary assessment: A field evaluation of subjective ratings of speech in noise. Ear & Hearing, 42(6), 1770–1781. https://doi.org/10.1097/AUD.0000000000001071
Kaplan, B. A., Reed, D. D., & Jarmolowicz, D. P. (2015). Effects of episodic future thinking on discounting: Personalized age-progressed pictures improve risky long-term health decisions. Journal of Applied Behavior Analysis, 49(1), 148–169. https://doi.org/10.1002/jaba.277
Kennedy, C. H. (2005). Single-case designs for educational research. Pearson.
Knell, G., Gabriel, K. P., Businelle, M. S., Shuval, K., Wetter, D., & Kendzor, D. (2017). Ecological momentary assessment of physical activity: Validation study. Journal of Medical Internet Research, 19(7), Article e253. https://doi.org/10.2196/jmir.7602
LeBlanc, L. A., Lerman, D. C., & Normand, M. (2020). Behavior analytic contributions to public health and telehealth. Journal of Applied Behavior Analysis, 53(3), 1208–1218. https://doi.org/10.1002/jaba.749
Li, Z., Benowitz-Fredericks, C., Ling, P. M., Cohen, J. E., & Thrul, J. (2021). Assessing young adults’ ENDS use via ecological momentary assessment and a smart Bluetooth enabled ENDS device. Nicotine & Tobacco Research, 23(5), 842–848. https://doi.org/10.1093/ntr/ntaa205
Lin, L. I. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics, 45, 255–268 https://www.jstor.org/stable/2532051
Linas, B. S., Genz, A., Westergaard, R. P., Chang, L. W., Bollinger, R. C., Latkin, C., & Kirk, G. D. (2016). Ecological momentary assessment of illicit drug use compared to biological and self-reported methods. JMIR Mhealth Uhealth, 4(1), Article e27. https://doi.org/10.2196/mhealth.4470
Lowe, C. F., Horne, P. J., Harris, F. D., & Randle, V. R. (2002). Naming and categorization in young children: Vocal tact training. Journal of the Experimental Analysis of Behavior, 78(3), 527–549. https://doi.org/10.1901/jeab.2002.78-527
Maher, J. P., Rebar, A. L., & Dunton, G. F. (2018). Ecological momentary assessment is a feasible and valid methodological tool to measure older adults’ physical activity and sedentary behavior. Frontiers in Psychology, 9, Article e1485. https://doi.org/10.3389/fpsyg.2018.01485
Martin, C., Correa, J., Han, H., Allen, H. R., Rood, J., Champagne, C., Gunturk, B., & Bray, G. (2012). Validity of the remote food photography method for estimating energy and nutrient intake in near real-time. Obesity, 20, 891–899. https://doi.org/10.1038/oby.2011.344
Marszalek, J., Morgulec- Adamowicz, N., Rutkowska, I., & Kosmol, A. (2014). Using ecological momentary assessment to evaluate current physical activity. Biomedical Research International, Article e915172. https://doi.org/10.1155/2014/915172.
Mason, T. B., Do, B., Wang, S., & Dunton, G. F. (2020). Ecological momentary assessment of eating and dietary intake behaviors in children and adolescents: A systematic review of the literature. Appetite, 144, 104465. https://doi.org/10.1016/j.appet.2019.104465
McClure, E. A., Tomko, R. L., Carpenter, M. J., Treiber, F. A., & Gray, K. M. (2018). Acceptability and compliance with a remote monitoring system to track smoking and abstinence among young smokers. American Journal of Drug & Alcohol Abuse, 44(5), 561–570. https://doi.org/10.1080/00952990.2018.1467431
McShane, B. B., Gal, D., Gelman, A., Robert, C., & Tackett, J. L. (2019). Abandon statistical significance. American Statistician, 73(supp1.), 235–245. https://doi.org/10.1080/00031305.2018.1527253
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & Prisma Group. (2009). Reprint—Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Physical Therapy, 89(9), 873–880. https://doi.org/10.1093/ptj/89.9.873
Most, J., Vallo, P. M., Altazan, A. D., Gilmore, L. A., Sutton, E. F., Cain, L. E., Burton, J. H., Martin, C. K., & Redman, L. M. (2018). Food photography is not an accurate measure of energy intake in obese, pregnant women. Journal of Nutrition, 148(4), 658–663. https://doi.org/10.1093/jn/nxy009
Mun, E., Li, X., Businelle, M., Hébert, E., Tan, Z., Barnett, N., & Walters, S. (2021). Ecological momentary assessment of alcohol consumption and its concordance with transdermal alcohol detection and timeline follow-back self-report among adults experiencing homelessness. Alcoholism: Clinical & Experimental Research, 45(4), 864–876. https://doi.org/10.1111/acer.14571
McHugh, M. (2012). Interrater reliability: The kappa statistic. Biochemical Medicine, 22(3), 276–282.
Motulsky, H. (2018). Intuitive biostatistics: A nonmathematical guide to statistical thinking (4th ed.). Oxford University Press.
Normand, M. P. (2017). The language of science. Perspectives on Behavior Science, 42(3), 675–688. https://doi.org/10.1007/s40614-017-0123-8
Pannicke, B., Reichenberger, J., Schultchen, D., Pollatos, O., & Blechert, J. (2020). Affect improvements and measurement concordance between a subjective and an accelerometer estimate of physical activity. European Journal of Health Psychology, 27(2), 66–75. https://doi.org/10.1027/2512-8442/a000050
Pearson, J., Elmasry, H., Das, B., Smiley, S., Rubin, L., DeAtley, T., Harvey, E., Zhou, Y., Niaura, R., & Abrams, D. (2017). Comparison of ecological momentary assessment versus direct measurement of e-cigarette use with a Bluetooth-enabled e-cigarette: A pilot study. JMIR Research Protocols, 6(5), Article e84. https://doi.org/10.2196/resprot.6501
Pihet, S., De Ridder, J., & Suter, M. (2017). Ecological momentary assessment (EMA) goes to jail: Capturing daily antisocial behavior in its context, a feasibility and reliability study in incarcerated juvenile offenders. European Journal of Psychological Assessment, 33(2), 87–96. https://doi.org/10.1027/1015-5759/a000275
Prince, S. A., Adamo, K. B., Hamel, M. E., Hardt, J., Connor Gorber, S., & Tremblay, M. (2008). A comparison of direct versus self-report measures for assessing physical activity in adults: A systematic review. International Journal of Behavioral Nutrition & Physical Activity, 5, 56. https://doi.org/10.1186/1479-5868-5-56
Ponnada, A., Thapa-Chhetry, B., Manjourides, J., & Intille, S. (2021). Measuring criterion validity of microinteraction ecological momentary assessment (Micro-EMA): Exploratory pilot study with physical activity measurement. JMIR Mhealth Uhealth, 9(3), e23391. https://doi.org/10.2196/23391
Rajagopal, S., Nicholson, K., Putri, T. R., Addington, J., & Felde, A. (2020). Teaching children with autism to tact private events based on public accompaniments. Journal of Applied Behavior Analysis, 54(1), 270–286. https://doi.org/10.1002/jaba.785
Ranganathan, P., Pramesh, C., & Aggarwal, R. (2017). Common pitfalls in statistical analysis: Measures of agreement. Perspectives in Clinical Research, 8(4), 187–191. https://doi.org/10.4103/picr.PICR_123_17
Risley, T. R., & Hart, B. (1968). Developing correspondence between the non-verbal and verbal behavior of preschool children. Journal of Applied Behavior Analysis, 1(4), 267–281. https://doi.org/10.1901/jaba.1968.1-267
Romanzini, C., Romanzini, M., Barbosa, C., Batista, M., Shigaki, G., & Ronque, E. (2019). Characterization and agreement between application of mobile ecological momentary assessment (mEMA) and accelerometry in the identification of prevalence of sedentary behavior (SB) in young adults. Frontiers in Psychology, 10, Article e720. https://doi.org/10.3389/fpsyg.2019.00720
Schaefer, L. M., Engel, S. G., & Wonderlich, S. A. (2020a). Ecological momentary assessment in eating disorders research: recent findings and promising new directions. Current Opinion in Psychiatry, 33(6), 528–533. https://doi.org/10.1097/YCO.0000000000000639
Schaefer, L. M., Smith, K. E., Anderson, L. M., Cao, L., Crosby, R. D., Engel, S. G., Crow, S. J., Peterson, C. B., & Wonderlich, S. A. (2020b). The role of affect in the maintenance of binge-eating disorder: Evidence from an ecological momentary assessment study. Journal of Abnormal Psychology, 129(4), 387–396. https://doi.org/10.1037/abn0000517
Schüz, N., Walters, J., Frandsen, M., Bower, J., & Ferguson, S. (2014). Compliance with an EMA monitoring protocol and its relationship with participant and smoking characteristics. Nicotine & Tobacco Research, 16(2), S88–S92. https://doi.org/10.1093/ntr/ntt142
Schüz, N., Cianchi, J., Shiffman, S., & Ferguson, S. (2015). Novel technologies to study smoking behavior: Current developments in ecological momentary assessment. Current Addiction Reports, 2(1), 8–14. https://doi.org/10.1007/s40429-015-0039-x
Shiffman, S. (2005). Dynamic influences on smoking relapse process. Journal of Personality, 73(6), 1715–1748. https://doi.org/10.1111/j.0022-3506.2005.00364.x
Shiffman, S. (2009). Ecological momentary assessment (EMA) in studies of substance use. Psychological Assessment, 21(4), 486–497. https://doi.org/10.1037/a0017074
Shiffman, S., Stone, A. A., & Hufford, M. R. (2008). Ecological momentary assessment. Annual Review of Clinical Psychology, 4, 1–32. https://doi.org/10.1146/annurev.clinpsy.3.022806.091415
Skinner, B. F. (1957). Verbal behavior. Appleton-Century-Crofts. https://doi.org/10.1037/11256-000
Stone, A. A., & Shiffman, S. (1994). Ecological momentary assessment (EMA) in behavioral medicine. Annals of Behavioral Medicine, 16(3), 199–202. https://doi.org/10.1093/abm/16.3.199
Stone, A. A., Shiffman, S., Schwartz, J. E., Broderick, J. E., & Hufford, M. R. (2002). Patient non-compliance with paper diaries. BMJ, 324, 1193–1194. https://doi.org/10.1136/bmj.324.7347.1193
Strang, J., McCambridge, J., Best, D., Beswick, T., Bearn, J., Rees, S., & Gossop, M. (2003). Loss of tolerance and overdose mortality after inpatient opiate detoxification: Follow up study. BMJ, 326, 959–960. https://doi.org/10.1136/bmj.326.7396.959
Sturmey, P. (2020). Functional analysis in clinical treatment. Elsevier Science & Technology.
Thompson, R. H., & Borrero, J. C. (2021). Direct observation. In C. Piazza, H. S. Roane, & W. W. Fisher (Eds.), Handbook of applied behavior analysis (2nd ed.; pp. 191–205). Guilford Press.
Timmer, B., Hickson, L., & Launer, S. (2017). Ecological momentary assessment: Feasibility, construct validity, and future applications. American Journal of Audiology, 26, 436–442. https://doi.org/10.1044/2017_AJA-16-0126
van Berkel, N., Goncalves, J., Lovén, L., Ferreira, D., Hosio, S., & Kostakos, V. (2019). Effect of experience sampling schedules on response rate and recall accuracy of objective self-reports. International Journal of Human-Computer Studies, 125, 118–128. https://doi.org/10.1016/j.ijhcs.2018.12.002
Vollmer, T. (2021). Assessment and treatment of behavior disorders: The next 40 years? [Keynote address]. FABA’s 41st Annual Convention, Aventura, FL.
Warnick, J., Western, S., Albanese-O’Neill, A., Filipp, S., Schatz, D., Haller, M., & Janicke, D. (2020). Use of ecological momentary assessment to measure self-monitoring of glucose adherence in youth with Type 1 Diabetes. Diabetes Spectrum: A Publication of the American Diabetes Association, 33(3), 280–289. https://doi.org/10.2337/ds19-0041
Weatherson, K., Yun, L., Wunderlich, K., Puterman, E., & Faulkner, G. (2019). Application of an ecological momentary assessment protocol in a workplace intervention: Assessing compliance, criterion validity, and reactivity. Journal of Physical Activity & Health, 16, 985–992. https://doi.org/10.1123/jpah.2019-0152
Williams, M. T., Lewthwaite, H., Fraysse, F., Gajewska, A., Ignatavicius, J., & Ferrar, K. (2021). Compliance with mobile ecological momentary assessment of self-reported health-related behaviors and psychological constructs in adults: Systematic review and meta-analysis. Journal of Medical Internet Research, 23(3), Article e1702. https://doi.org/10.2196/17023
Witkiewitz, K., & Tucker, J. A. (2020). Abstinence not required: Expanding the definition of recovery from Alcohol Use Disorder. Alcoholism: Clinical & Experimental Research, 44, 36–40. https://doi.org/10.1111/acer.14235
Zink, J., Belcher, B., Dzubur, E., Ke, W., O’Connor, S., Huh, J., Lopez, N., Maher, J., & Dunton, G. (2018). Association between self-reported and objective activity levels by demographic factors: Ecological momentary assessment study in children. JMIR Mhealth Uhealth, 6(6), Article e150. https://doi.org/10.2196/mhealth.9592
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of Interest
We have no known conflicts of interest to disclose.
Availability of Data
All data generated or analyzed during this study are included in this published article.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
We thank Ryan Higginbotham for his assistance with interrater agreement and suggestions, and Hailey Donahue, Lindsey Ives, and Alexis Knerr for their comments on this manuscript.
Rights and permissions
About this article
Cite this article
Stinson, L., Liu, Y. & Dallery, J. Ecological Momentary Assessment: A Systematic Review of Validity Research. Perspect Behav Sci 45, 469–493 (2022). https://doi.org/10.1007/s40614-022-00339-w
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40614-022-00339-w