Skip to main content

Optimizing Measurement Reliability in Within-Person Research: Guidelines for Research Design and R Shiny Web Application Tools

Abstract

Within-person research has become increasingly popular over recent years in the field of organizational studies for its unique theoretical and methodological advantages for studying dynamic intrapersonal processes (e.g., Dalal et al., Journal of Management 40:1396–1436, 2014; McCormick et al., Journal of Management 46:321–350, 2020). Despite the advancements, there remain serious challenges for many organizational researchers to fully appreciate and appropriately implement within-person research—more specifically, to correctly conceptualize and compute the within-person measurement reliability, as well as navigate key within-person research design factors (e.g., number of measurement occasions, T; number of participants, N; and scale length, I) to optimize within-person reliability. By conducting a comprehensive Monte Carlo simulation with 3240 data conditions, we offer a practical guideline table showing the expected within-person reliability as a function of key design factors. In addition, we provide three easy-to-use, free R Shiny web applications for within-person researchers to conveniently (a) compute expected within-person reliability based on their customized research design, (b) compute observed validity based on the expected reliability and hypothesized within-person validity, and (c) compute observed within-person (as well as between-person) reliability from collected within-person research datasets. We hope these much-needed evidence-based guidelines and practical tools will help enhance within-person research in organizational studies.

This is a preview of subscription content, access via your institution.

Fig. 1

Notes

  1. The R shiny web applications are available through URL https://psychmethods.shinyapps.io/WithinPersonResearch or https://tinyurl.com/LevelAlpha.

  2. Our research team also conducted a separate set of simulations using realistic research design conditions based on the empirical review of 103 unique organizational ESM studies published in 10 representative organizational research journals (e.g., scale length of 2–5 items, 10 vs. 25 measurement occasions, 60 vs. 90 vs. 120 participants), to compare the averaged biases and root mean squared errors (RMSE) of the alpha and omega reliability estimates relative to the true reliabilities. We found that the alpha and omega methods were equally efficacious in terms of having little biases and RMSE in estimating within- and between-person reliabilities for unidimensional scales.

References

Download references

Acknowledgements

We thank Dr. Jason Newsom for the feedback on the earlier versions of this manuscript. We also thank Stefanie Fox, M.S., for her proofreading of the most recent version of this manuscript.

Funding

This research was supported by the grant T03OH008435 awarded to Portland State University, funded by the Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of NIOSH, CDC, or HHS. This research is also supported by the National Science Foundation under Grant awarded to Wei Wang (No. 16406229). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do notnecessarily reflect the views of the National Science Foundation.

Author information

Authors and Affiliations

Authors

Corresponding authors

Correspondence to Liu-Qin Yang or Wei Wang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Liu-Qin Yang and Wei Wang contributed equally to this article

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 502 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Yang, LQ., Wang, W., Huang, PH. et al. Optimizing Measurement Reliability in Within-Person Research: Guidelines for Research Design and R Shiny Web Application Tools. J Bus Psychol (2022). https://doi.org/10.1007/s10869-022-09803-5

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10869-022-09803-5

Keywords

  • Within-person research
  • Level-specific alpha
  • Reliability
  • R shiny web application
  • Validity