Abstract
Purpose
Over the past two decades, research has shown a growing consensus that 70% to 90% of organizational learning occurs not through formal training but informally, on-the-job, and in an ongoing manner. Despite this emerging consensus, primary data on the nature and correlates of informal learning remains sparse. The purpose of this study was to provide an integrative definition of informal learning behaviors (ILBs) and to synthesize existing primary data through meta-analysis to explore ILB correlates.
Design/Methodology/Approach
Given that there has been little systematic treatment of ILBs, we defined their construct domain and tested relationships suggested by our research questions with antecedents (personal factors, situational factors) and outcomes (attitudes, knowledge/skill acquisition, performance) using random effects meta-analyses (k = 49, N = 55,514).
Findings
Our results showed both personal and situational antecedent factors to be predictive of ILBs, as well as ILB–outcome relationships.
Implications
Findings indicate that engagement in ILBs for working adults is linked to valued criteria such as attitudes (ρ = .29), knowledge/skill acquisition (ρ = .41), and performance (ρ = .42). We provide suggestions for future research and actionable advice for organizations to support the development of ILBs.
Originality/Value
Although hundreds of studies and over a dozen meta-analyses have explored the nature and effectiveness of formal learning in the workplace, our work is the first attempt to conceptualize a unified definition of ILBs and to aggregate primary data on ILB correlates using meta-analysis.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
We thank an anonymous reviewer for helping us to hone this point.
Author, year, observed effect size, IV reliability, informal learning reliability, IV description, IV level 1, IV level 2, IV level 3, temporal precedence, design, source IV, source DV, measurement source difference, informal learning description, notes, source
Arthur et al. report a sample-weighted mean d of .62 for both behavioral and result criteria. To interpret their findings against ours, we converted to a correlation, yielding ρ = .30. Then, using a cumulative normal distribution table, ρ = .42 ➔ d = .93 ➔ 32%; ρ = .30 ➔ d = .62 ➔ 23%
We thank an anonymous reviewer for this point.
References
References marked with an asterisk indicate studies included in the meta-analysis.
Aguinis, H., Gottfredson, R. K., & Wright, T. A. (2011a). Best-practice recommendations for estimating interaction effects using meta-analysis. Journal of Organizational Behavior, 32, 1033–1043.
Aguinis, H., Pierce, C. A., Bosco, F. A., Dalton, D. R., & Dalton, C. M. (2011b). Debunking myths and urban legends about meta-analysis. Organizational Research Methods, 14, 306–331.
Ajzen, I., & Fishbein, M. (1977). Attitude-behavior relations: A theoretical analysis and review of empirical research. Psychological Bulletin, 84, 888–918.
Alliger, G. M., Tannenbaum, S. I., Bennett, W., Traver, H., & Shotland, A. (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50, 341–358.
Armijo-Olivo, S., Stiles, C. R., Hagen, N. A., Biondo, P. D., & Cummings, G. G. (2012). Assessment of study quality for systematic reviews: A comparison of the Cochrane collaboration risk of bias tool and the effective public health practice project quality assessment tool. Journal of Evaluation in Clinical Practice, 18, 12–18.
Arthur Jr., W., Bennett Jr., W., Edens, P. S., & Bell, S. T. (2003). Effectiveness of training in organizations: A meta-analysis of design and evaluation features. Journal of Applied Psychology, 88, 234–245.
Bandura, A. (1994). Self-efficacy. San Francisco: Wiley Online Library.
Bartlett, C. A., & Ghoshal, S. (1998). Beyond strategic planning to organization learning: Lifeblood of the individualized corporation. Strategy & Leadership, 26, 34–39.
*Bednall, T. C., Sanders, K., & Runhaar, P. (2014). Simulating informal learning activities through perceptions of performance appraisal quality and human resource management system strength: A two-wave study. Academy of Management Learning & Education, 13, 45–61.
Bell, B. S., & Kozlowski, S. W. J. (2008). Active learning: Effects of core training design elements on self-regulatory processes, learning, and adaptability. Journal of Applied Psychology, 93, 296–316.
*Berg, S. A., & Chyung, S. Y. (2008). Factors that influence informal learning in the workplace. Journal of Workplace Learning, 20, 229–244.
*Bickmore, D. L. (2011). Professional learning experiences and administrator practice: Is there a connection? Professional Development in Education, 38, 95–112.
Bijmolt, T. H., & Pieters, R. G. (2001). Meta-analysis in marketing when studies contain multiple measurements. Marketing Letters, 12, 157–169.
Birdi, K., Allan, C., & Warr, P. (1997). Correlates and perceived outcomes of 4 types of employee development activity. Journal of Applied Psychology, 82, 845–857.
Blume, B. D., Ford, J. K., Baldwin, T. T., & Huang, J. L. (2010). Transfer of training: A meta-analytic review. Journal of Management, 36, 1065–1105.
Borenstein, M., Hedges, L. V., Higgins, J., & Rothstein, H. R. (2009). Criticisms of meta-analysis. In Introduction to meta-analysis . New York: John Wiley & Sons. pp. 377–387
Boyce, L. A., Zaccaro, S. J., & Wisecarver, M. Z. (2010). Propensity for self-development of leadership attributes: Understanding, predicting, and supporting performance of leader self-development. The Leadership Quarterly, 21, 159–178.
Brett, J. F., & VandeWalle, D. (1999). Goal orientation and goal content as predictors of performance in a training program. Journal of Applied Psychology, 84, 863–873.
Button, S. B., Mathieu, J. E., & Zajac, D. M. (1996). Goal orientation in organizational research: A conceptual and empirical foundation. Organizational Behavior and Human Decision Processes, 67, 26–48.
Callahan, J. S., Kiker, D. S., & Cross, T. (2003). Does method matter? A meta-analysis of the effects of training method on older learner training performance. Journal of Management, 29, 663–680.
*Carson, E. H. (2013). Self-directed learning and academic achievement in secondary online students. Unpublished doctoral dissertation, University of Tennessee at Chattanooga.
Center for Workforce-Development. (1998). The teaching firm: Where productive work and learning converge: Report on research findings and implications. Newton, MA: Education Development Center, Inc.
Cerasoli, C. P., Nicklin, J. M., & Ford, M. T. (2014). Intrinsic motivation and extrinsic incentives jointly predict performance: A 40-year meta-analysis. Psychological Bulletin, 140, 980–1008.
Chan, M. E., & Arvey, R. D. (2012). Meta-analysis and the development of knowledge. Perspectives on Psychological Science, 7, 79–92.
*Choi, W., & Jacobs, R. L. (2011). Influences of formal learning, personal learning orientation, and supportive learning environment on informal learning. Human Resource Development Quarterly, 22, 239–257.
Colquitt, J. A., LePine, J. A., & Noe, R. A. (2000). Toward an integrative theory of training motivation: A meta-analytic path analysis of 20 years of research. Journal of Applied Psychology, 85, 678–707.
Cooper, H. (2003). Editorial. Psychological Bulletin, 129, 3–9.
Cortina, J. M. (2003). Apples and oranges (and pears, oh my!): The search for moderators in meta-analysis. Organizational Research Methods, 6, 415–439.
Cseh, M., Watkins, K. E., & Marsick, V. J. (2000). Informal and incidental learning in the workplace. Germany: Munster.
*de Groot, E., Jaarsma, D., Endedijk, M., Mainhard, T., Lam, I., Simons, R. J., & van Beukelen, P. (2012). Critically reflective work behavior of health care professionals. Journal of Continuing Education in the Health Professions, 32, 48–57.
*Digby, C. L. B. (2010). An examination of the impact of non-formal and informal learning on adult environmental knowledge, attitudes, and behaviors. Unpublished doctoral dissertation, University of Minnesota.
Donovan, J. (2001). Work motivation. In N. Anderson, D. S. Ones, H. K. Sinangil, & C. Viswesvaran (Eds.), Handbook of industrial, work, and organizational psychology (Vol. 2, pp. 53–76). Los Angeles: Sage Publications.
Ellinger, A. (2005). Contextual factors influencing informal learning ina workplace setting: The case of “reinventing itself company”. Human Resource Development Quarterly, 16, 389–415.
Enders, C. K., & Tofighi, D. (2007). Centering predictor variables in cross-sectional multilevel models: A new look at an old issue. Psychological Methods, 12, 121–138.
*Enos, M. D., Kehrhahn, M. T., & Bell, A. (2003). Informal learning and the transfer of learning: How managers develop proficiency. Human Resource Development Quarterly, 14, 369–387.
Eraut, M. (2004). Informal learning in the workplace. Studies in Continuing Education, 26, 247–273.
Ericsson, K. A., & Charness, N. (1994). Expert performance: Its structure and acquisition. American Psychologist, 49, 725–747.
Flynn, D., Eddy, E. R., & Tannenbaum, S. I. (2006). The impact of national culture on the continuous learning environment: Exploratory findings from multiple countries. Journal of East-West Business, 12, 85–107.
*Froehlich, D., Segers, M., & Van den Bossche, P. (2014). Learning approach, leadership style, and organizational learning culture on managers’ learning outcomes. Human Resource Development Quarterly, 25, 29–57.
*Gijbels, D., Raemdonck, I., & Vervecken, D. (2010). Influencing work-related learning: The role of job characteristics and self-directed learning orientation in part-time vocational education. Vocations and Learning, 3, 239–255.
*Gijbels, D., Raemdonck, I., Vervecken, D., & Van Herck, J. (2012). Understanding work-related learning: The case of ICT workers. Journal of Workplace Learning, 24, 416–429.
Gola, G. (2009). Informal learning of social workers: A method of narrative inquiry. Journal of Workplace Learning, 21, 334–346.
*Gonzales, L. C. (1985). Relationship of cooperative board of directors’ informal learning to selected characteristics. Unpublished doctoral dissertation. University of Wisconson-Madison.
Guion, R. (1998). Assessment, measurement, and prediction for personnel decisions. Mahwah, NJ: Lawrence Erlbaum Associates.
*Hicks, E., Bagg, R., Doyle, W., & Young, J. D. (2007). Canadian accountants: Examining workplace learning. Journal of Workplace Learning, 19, 61–77.
*Houde, J. F. (2014). The influence of formal training on informal learning networks. Unpublished doctoral dissertation, North Carolina State University.
Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks: Sage.
*Hurns, K. M. (2013). Business professionals’ reflective practice in the workplace. Unpublished doctoral dissertation, Walsh College.
*Hutchins, H. M., Burke, L. A., & Berthelsen, A. M. (2010). A missing link in the transfer problem? Examining how trainers learn about training transfer. Human Resource Management, 49, 599–618.
*Jeon, K. S., & Kim, K.-N. (2012). How do organizational and task factors influence informal learning in the workplace? Human Resource Development International, 15, 209–226.
Keith, N., & Frese, M. (2008). Effectiveness of error management training: A meta-analysis. Journal of Applied Psychology, 93, 59–69.
Kerr, S. (1975). On the folly of rewarding a, while hoping for B. Academy of Management Journal, 18, 769–783.
Klein, H. J., Noe, R. A., & Wang, C. W. (2006). Motivation to learn and course outcomes: The impact of delivery mode, learning goal orientation, and perceived barriers and enablers. Personnel Psychology, 59, 665–702.
Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119, 254–284.
Koopmans, H., Doornbos, A. J., & Eekelen, I. M. v. (2006). Learning in interactive work situations: It takes two to tango; why not invite both partners to dance? Human Resource Development Quarterly, 17, 135–158.
LePine, J. A., Erez, A., & Johnson, D. E. (2002). The nature and dimensionality of organizational citizenship behavior: A critical review and meta-analysis. Journal of Applied Psychology, 87, 52–65.
LePine, J. A., LePine, M. A., & Jackson, C. L. (2004). Challenge and hindrance stress: Relationships with exhaustion, motivation to learn, and learning performance. Journal of Applied Psychology, 89, 883–891.
*Lindner, C. L. (2012). Predictive modeling in adult education. Unpublished doctoral dissertation. University of Idaho.
*Livingstone, D. W. (2001). Worker control as the missing link: Relations between paid/unpaid work and work-related learning. Journal of Workplace Learning, 13, 308–317.
*Livingstone, D. W., & Raykov, M. (2008). Workers’ power and intentional learning among non-managerial workers: A 2004 benchmark survey. Relations Industrielles, 63, 30–56.
*Livingstone, D. W., & Stowe, S. (2007). Work time and learning activities of the continuously employed: A longitudinal analysis, 1998-2004. Journal of Workplace Learning, 19, 17–31.
Lohman, M. C. (2005). A survey of factors influencing the engagement of two professional groups in informal workplace learning activities. Human Resource Development Quarterly, 16, 501–527.
*Maringka, J. F. (2014). Development and validation of an instrument to assess employees’ perceptions of informal learning work context. Unpublished doctoral dissertation, University of Minnesota.
Marsick, V. J., & Watkins, K. (1990). Informal and Incidental Learning in the Workplace. London: Routledge.
Marsick, V. J., Volpe, M., & Watkins, K. E. (1999). Theory and practice of informal learning in the knowledge era. Advances in Developing Human Resources, 1, 80–95.
Mathieu, J. E., & Chen, G. (2011). The etiology of the multilevel paradigm in management research. Journal of Management, 37, 610–641.
Mathieu, J. E., Tannenbaum, S. I., & Salas, E. (1992). The influences of individual and situational characteristics on measures of training effectiveness. Academy of Management Journal, 35, 828–847.
*Matsuo, M., & Nakahara, J. (2013). The effects of the PDCA cycle and OJT on workplace learning. The International Journal of Human Resource Management, 24, 195–207.
Maurer, T. J., & Tarulli, B. A. (1994). Investigation of perceived environment, perceived outcome, and person variables in relationship to voluntary development activity by employees. Journal of Applied Psychology, 79, 3–14.
McCauley, C. D., & Brutus, S. (1998). Management development through job experiences: An annotated bibliography. Greensboro, NC: Center for Creative Leadership.
McCauley, C. D., & Hezlett, S. A. (2001). Individual development in the workplace. In N. Anderson, D. S. Ones, H. K. Sinangil, & C. Viswesvaran (Eds.), Handbook of industrial, work and organizational psychology (Vol. 1, pp. 313–335). Thousand Oaks, CA: Sage.
Meyer, J. P., Stanley, D. J., Herscovitch, L., & Topolnytsky, L. (2002). Affective, continuance, and normative commitment to the organization: A meta-analysis of antecedents, correlates, and consequences. Journal of Vocational Behavior, 61, 20–52.
Molloy, J. C., & Noe, R. A. (2010). "learning" a living: Continuous learning for survival in today’s talent market. In S. W. J. Kozlowski & E. Salas (Eds.), Learning, training, and development in organizations (pp. 333–362). New York, NY: Taylor & Francis Group, LLC.
*Moon, S. Y., & Na, S. I. (2009). Psychological and organizational variables associated with workplace learning in small and medium manufacturing businesses in Korea. Asia Pacific Education Review, 10, 327–336.
Morris, M. A., & Robie, C. (2001). A meta-analysis of the effects of cross-cultural training on expatriate performance and adjustment. International Journal of Training and Development, 5, 112–125.
Morrison, A. M., White, R. P., & Van Velsor, E. (1992). Breaking the glass ceiling: Can women reach the top of America’s largest corporations? New York: Perseus Publishing.
Mosier, C. I. (1943). On the reliability of a weighted composite. Psychometrika, 8, 161–168.
Mumford, M. D., Baughman, W. A., Threlfall, K. V., Uhlman, C. E., & Costanza, D. P. (1993). Personality, adaptability, and performance: Performance on well-defined problem sovling tasks. Human Performance, 6, 241–285.
Ng, T. W., & Feldman, D. C. (2010). The relationships of age with job attitudes: A meta-analysis. Personnel Psychology, 63, 677–718.
Ng, T. W., & Feldman, D. C. (2012). Evaluating six common stereotypes about older workers with meta-analytical data. Personnel Psychology, 65, 821–858.
Ng, T. W., Eby, L. T., Sorensen, K. L., & Feldman, D. C. (2005). Predictors of objective and subjective career success: A meta-analysis. Personnel Psychology, 58, 367–408.
Noe, R. A., & Schmitt, N. (1986). The influence of trainee attitudes on training effectiveness: Test of a model. Personnel Psychology, 39, 497–523.
Noe, R. A., & Wilk, S. L. (1993). Investigation of the factors that influence employees’ participation in development activities. Journal of Applied Psychology, 78, 291–302.
Noe, R. A., Tews, M. J., & McConnell-Dachner, A. (2010). Learner engagement: A new perspective for enhancing our understanding of learner motivation and workplace learning. The Academy of Management Annals, 4, 279–315.
*Noe, R. A., Tews, M. J., & Marand, A. D. (2013). Individual differences and informal learning in the workplace. Journal of Vocational Behavior, 83, 327–335.
Orvis, K. A., & Leffler, G. P. (2011). Individual and contextual factors: An interactionist approach to understanding employee self-development. Personality and Individual Differences, 51, 172–177.
*Ouweneel, A. P., Taris, T. W., Van Zolingen, S. J., & Schreurs, P. J. (2009). How task characteristics and social support relate to managerial learning: Empirical evidence from Dutch home care. The Journal of Psychology, 143, 28–44.
*Parise, L. M., & Spillane, J. P. (2010). Teacher learning and instructional change: How formal and on-the-job learning opportunities predict change in elementary school teachers’ practice. The Elementary School Journal, 110, 323–346.
*Pike, G. R. (1999). The effects of residential learning communities and traditional residential living arrangements on educational gains during the first year of college. Journal of College Student Development, 40, 269–284.
Ployhart, R. E., & Vandenberg, R. J. (2010). Longitudinal research: The theory, design, and analysis of change. Journal of Management, 36, 94–120.
Raudenbush, S. W., Bryk, A. S., & Congdon, R. (2004). HLM 6 for Windows. Lincolnwood, IL: Scientific Software International.
*Reardon, R. F. (2010). The impact of learning culture on worker response to new technology. Journal of Workplace Learning, 22, 201–211.
Reio, T. G., & Wiswell, A. (2000). Field investigation of the relationship among adult curiosity, workplace learning, and job performance. Human Resource Development Quarterly, 11, 5–30.
*Reychav, I., & Te’eni, D. (2009). Knowledge exchange in the shrines of knowledge: The "how’s" and "where’s" of knowledge sharing processes. Computers & Education, 53, 1266–1277.
*Riaz, S., Rambli, D. R. A., Salleh, R., & Mushtaq, A. (2010). Study to investigate learning motivation factors within formal and informal learning environments and their influence upon web-based learning. International Journal of Emerging Technologies in Learning, 5, 41–50.
*Richter, D., Kunter, M., Klusmann, U., Lüdtke, O., & Baumert, J. (2011). Professional development across the teaching career: Teachers’ uptake of formal and informal learning opportunities. Teaching and Teacher Education, 27, 116–126.
Rooney, J. A., & Gottlieb, B. H. (2007). Development and initial validation of a measure of supportive and unsupportive managerial behaviors. Journal of Vocational Behavior, 71, 186–203.
Rosenthal, R., & DiMatteo, M. R. (2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52, 59–82.
Ross, L., & Nisbett, R. E. (1991). The person and the situation: Perspectives of social psychology: McGraw-Hill Book Company.
*Rowden, R. W. (2002). The relationship between workplace learning and job satisfaction in U.S. small to midsize businesses. Human Resource Development Quarterly, 13, 407–425.
*Rowden, R. W., & Conine, R. T. (2005). The impact of workplace learning on job satisfaction in small US commercial banks. Journal of Workplace Learning, 17, 215–230.
Salas, E., DiazGranados, D., Klein, C., Burke, C. S., Stagl, K. C., Goodwin, G. F., & Halpin, S. M. (2008). Does team training improve team performance? A meta-analysis. Human Factors: The Journal of the Human Factors and Ergonomics Society, 50, 903–933.
Sambrook, S. (2005). Factors influencing the context and process of work-related learning: Synthesizing findings from two research projects. Human Resource Development International, 8, 101–119.
*Sanders, J., Oomens, S., Blonk, R. W. B., & Hazelzet, A. (2011). Explaining lower educated workers’ training intentions. Journal of Workplace Learning, 23, 402–416.
*Santos, I. M., & Ali, N. (2012a). Beyond classroom: The uses of mobile phones by female students. International Journal of Information and Communication Technology Education, 8, 63–75.
*Santos, I. M., & Ali, N. (2012b). Exploring the uses of mobile phones to support informal learning. Education and Information Technologies, 17, 187–203.
*Scheurer, A. J. (2013). Antecedents of informal learning: A study of core self-evaluations and work-family conflict and their effects on informal learning. Unpublished masters thesis, Ohio State University.
Schmidt, F. L., & Hunter, J. E. (2001). Meta-analysis. In N. Anderson, D. S. Ones, J. K. Sinangil, & C. Viswesvaran (Eds.), Handbook of industrial, work and organizational psychology (Vol. 1, pp. 51–70). Los Angeles: Sage Publications.
Simmering, M. J., Colquitt, J. A., Noe, R. A., & Porter, C. O. (2003). Conscientiousness, autonomy fit, and development: A longitudinal study. Journal of Applied Psychology, 88, 954–963.
Skule, S. (2004). Learning conditions at work: A framework to understand and assess informal learning in the workplace. International Journal of Training and Development, 8, 8–20.
Smith, E. M., Ford, J. K., & Kozlowski, S. W. (1997). Building adaptive expertise: Implications for training design strategies. In M. A. Quinones & A. Ehrenstein (Eds.), Training for a rapidly changing workforce: Applications of psychological research (pp. 89–118). Washington, D.C.: American Psychological Association.
Snijders, T. A., & Bosker, R. J. (1999). Introduction to multilevel analysis. London: Sage.
*Spreitzer, G. M., McCall, M. W., & Mahoney, J. D. (1997). Early identification of international executive potential. Journal of Applied Psychology, 82, 6–29.
Stamps, D. (1998). Learning ecologies. Training, 35, 32–38.
Tannenbaum, S. I. (1997). Enhancing continuous learning: Diagnostic findings from multiple companies. Human Resource Management, 36, 437–452.
Tannenbaum, S. I., & Cerasoli, C. P. (2013). Do team and individual debriefs enhance performance? A meta-analysis. Human Factors: The Journal of Human Factors and Ergonomics Society, 55, 231–245.
Tannenbaum, S. I., Beard, R. L., McNall, L. A., & Salas, E. (2010). Informal learning and development in organizations. In S. W. J. Kozlowski & E. Salas (Eds.), Learning, training, and development in organizations (pp. 303–332). New York, NY: Taylor & Francis Group, LLC.
Tannenbaum, S. I., Beard, R. L., & Cerasoli, C. P. (2013). Conducting team debriefings that work: Lessons from research and practice. In E. Salas, S. Tannenbaum, D. Cohen, & G. Latham (Eds.), Developing and enhancing teamwork in organizations: Evidence-based best practices and guidelines (pp. 488–519). San Francisco, CA: Jossey Bass.
*van der Heijden, B., Boon, J., van der Klink, M., & Meijs, E. (2009). Employability enhancement through formal and informal learning: An empirical study among Dutch non-academic university staff members. International Journal of Training and Development, 13, 19–37.
*Van der Klink, M., Van der Heijden, B. I. J. M., Boon, J., & van Rooij, S. W. (2014). Exploring the contribution of formal and informal learning to academic staff member employability: A Dutch perspective. Career Development International, 19, 337–356.
*Wasiyo, K. (2010). Proactive knowledge accessibility and causal clarity: Key factors in improving project management and cross-project learning. Unpublished doctoral dissertation. Columbia University.
*Welch, C. M. (2013). Perceptions of value: A study of worker characteristics and performance interventions. Unpublished doctoral dissertation, Capella University.
Whitener, E. M. (1990). Confusion of confidence intervals and credibility intervals in meta-analysis. Journal of Applied Psychology, 75, 315–321.
Yukl, G. (2010). Leadership in organizations. Upper Saddle River: Prentice Hall.
Acknowledgements
This research was supported in part under Army Research Institute (ARI) contract W5J9CQ-12-C-0048. The views, opinions, and/or findings contained in this article are solely those of the authors and should not be construed as an official position, policy, or decision of the DoD or the USA, unless so designated by other documentation.
Author information
Authors and Affiliations
Corresponding author
Appendix: Methodological Rigor Analyses
Appendix: Methodological Rigor Analyses
Addressing the issue of primary data publication quality of the studies included in our meta-analysis is critical in this case. Given the ubiquity with which informal learning occurs, we intentionally conducted a very broad search and did not set any a priori exclusionary criteria with respect to the source of the primary data. This of course introduces the possibility that research from arguably less rigorous outlets may impact the observed effect size of a given study.
We explored this issue in depth and conferred with several of the leading scholars on meta-analysis. The leading scholars suggested that, perhaps, we could rate the quality of each study and correlate those ratings with the reported effects sizes. However, they were quick to note that ratings of study quality are notoriously unreliable (e.g., Armijo‐Olivo et al., 2012) and that may not be the best course of action. So we decided to be more systematic and rigorous about testing for such effects using a two-pronged approach. First, we indexed the quality of the outlet where each sample appeared using study citations and journal impact factors. Second, we coded whether informal learning–correlates were concurrent versus lagged in time and whether they were measured using the same or different methods.
In terms of indexing the quality of each study, we obtained its number of citations in Google Scholar as of October 2016. These ranged from 0 to 424 (mean = 51.33, SD = 94.35). Second, we obtained the journal impact factor scores for all sources except dissertations and theses, which ranged from .10 to 4.80 (mean = 1.15, SD = 1.09). For analytic purposes, we assigned an impact value of zero to uncited dissertations and theses yielding a distribution mean of .82 (SD = 1.06). We used Google Scholar because it tracks citations more liberally for all sorts of publications than do the more restrictive citation bases. Whereas this may yield larger citation rates, such inflation should be consistent across the studies sampled here and therefore not bias these analyses. As one would anticipate, source impact factor scores and citations correlated significantly (r = .70, p < .001). Naturally older studies have a higher likelihood of being cited (year published correlated r = −.54, p < .001 with citations). Nevertheless, controlling for year published, source impact scores still correlated significantly with citations (r = .67, p < .001). Therefore, we believe that we have two widely used indices of source quality that correlate as expected.
Of the 376 effects sizes that are included across the various meta-analyses, 90% were concurrent, and 10% were lagged or from an experiment. For analytic purposes, we coded concurrent measures = 0 and lagged/experimental = 1. As for measurement source, both informal learning and the correlate were measured using the same source (coded = 1) 98% of the time and 2% were from differing sources (coded = 0). We should note that some correlations reported in a given study may have been from the same time, whereas others may have been lagged. Similarly, some correlations may have been between variables measured the same way, whereas others may have involved multiple measures. Therefore, these design features can co-vary at the effect size level of analysis.
The lower left triangle of Table 6 contains correlations among the study effect sizes, design features, and source quality indices (N = 376). In terms of these correlations, it is notable that the only significant ones with the observed effect sizes is a positive one for citation rate (r = .12, p < .05) and a negative one for sample size (r = −.20, p < .001). The upper right triangle of Table 6 shows the parallel partial correlations controlling for both publication year and sample size. Still, the only significant correlation with effect size is a positive one with citation rate (r = .13, p < .05).
The 376 effects sizes that we considered came from 49 different samples that were clustered into eight different categories for meta-analytic purposes. In other words, each sample contributed, on average, 7.7 effects sizes to the meta-analyses. This lack of independence needs to be taken into account in any pooled analysis associating the magnitude of effect sizes with study characteristics. In addition, the different meta-analysis categories revealed significant mean differences in effects sizes that should also be accounted for in any such analysis. Accordingly, we do so using a cross-classified hierarchical linear model analysis of effects sizes as described below (HLM; Raudenbush, Byrk, Congdon, & du Toit, 2004). Cross-classified models simultaneously account for multiple level 2 nesting arrangements, which in this case map to samples and meta-analysis categories. Although not precise estimates in the context of multi-level models, we report overall effect sizes for these models (i.e., ∼R 2) using the formulas advanced by Snijders & Bosker (1999).
The HLM model had 376 level 1 effects sizes that were cross-classified in terms of 49 level 2 samples and 8 level 2 meta-analysis categories. A baseline (i.e., null model) revealed that 30% of total effects size variance resided across samples [χ 2(49) = 201.39, p < .001], 17% fell across meta-analysis categories [χ 2(7) = 119.87, p < .001], and 53% remained unaccounted for. Therefore, clearly it is importantly to account for both samples and categories. Notably, given that meta-analyses adjusts for the influence of differing sample sizes on observed effects sizes, we introduced sample size as a weighting factor at level 1 in the remaining HLM analyses. Sample size alone accounted for 71% of the total observed variance in effects sizes, which is consistent with the pattern observed across the individual meta-analyses. Notably, however, even after accounting for sample size influences, significance between sample [χ 2(48) = 109.15, p < .001], between meta-analysis categories [χ 2(7) = 324.13, p < .001], and variance in effects sizes remained.
We next introduced the design and measurement dummy codes as level 1 predictors. Notably, we centered-within-cluster (CWC) these scores per meta-analysis category so as to consider their influence per meta-analysis (see Enders & Tofighi, 2007). Together the two dummy codes account for a nonsignificant [Δχ 2(2) = 2.16, ns] 1% Δ ∼ R 2 and neither exhibited a significant unique influence. We then introduced the average category design and measurement features as potential level 2 predictors. The logic here is that some meta-analysis categories may be more likely to contain certain design features than others. For example, 100% of the informal learning–attitudes correlations were concurrent, whereas 49% of the informal learning–individual predisposition correlations came from lagged designs. Nevertheless, introducing the level 2 category average design features also failed to account for any significant additional variance [Δχ 2(2) = 2.31, ns, Δ ∼ R 2 = 2%] and neither mean value exhibited a significant unique influence. These findings provide clear evidence that the magnitude of effects sizes was not significantly related to design or measurement source factors, either considered within or between meta-analysis categories.
Finally, we introduced year of publication, citations, and journal impact factors as level 2 predictors of effects sizes from the sample classification function (retaining the above effects in the equation). Collectively, adding them to the equation accounted for a significance [Δχ 2(3) = 13.03, p < .05] Δ ∼ R 2 = 2%. However, the only significant unique effect was, oddly, attributable to a negative effect associated with the year of publication control variable [Γ = −.005, SE = .002, p < .05].
The summary results of this analysis are shown below in Table 7. In sum, after accounting for the double nesting and varying sample sizes, neither design nor measurement features were related significantly to the magnitude of effect sizes, considered either centered within meta-analysis category or on average across categories. Source quality, indexed both in terms of citations and impact scores (controlling for year of publication), also had no significant relationships with the effects sizes.
Rights and permissions
About this article
Cite this article
Cerasoli, C.P., Alliger, G.M., Donsbach, J.S. et al. Antecedents and Outcomes of Informal Learning Behaviors: a Meta-Analysis. J Bus Psychol 33, 203–230 (2018). https://doi.org/10.1007/s10869-017-9492-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10869-017-9492-y