Skip to main content

Assessing the validity of computer-game-like tests of processing speed and working memory

Abstract

Processing speed (Gs) and working memory (WM) tasks have received considerable interest as correlates of more complex cognitive performance measures. Gs and WM tasks are often repetitive and are often rigidly presented, however. The effects of Gs and WM may, therefore, be confounded with those of motivation and anxiety. In an effort to address this problem, we assessed the concurrent and predictive validity of computer-game-like tests of Gs (Space Code) and WM (Space Matrix) across two experiments. In Experiment 1, within a university sample (N =70), Space Matrix exhibited concurrent validity as a WM measure, whereas Space Code appeared to be a mixed-ability measure. In Experiment 2, Space Matrix exhibited concurrent validity as well as predictive validity (as a predictor of school grades) within a school-aged sample (N=94), but the results for Space Code were less encouraging. Relationships between computer-game-like tests and gender, handedness, and computergame experience are also discussed.

References

  1. Ackerman, P. L., Beier, M. E., & Boyle, M. O. (2002). Individual differences in working memory within a nomological network of cognitive and perceptual speed abilities. Journal of Experimental Psychology: General, 131, 567–589.

    Article  Google Scholar 

  2. Ackerman, P. L., Beier, M. E., & Boyle, M. O. (2005). Working memory and intelligence: The same or different constructs? Psychological Bulletin, 131, 30–60.

    PubMed  Article  Google Scholar 

  3. Aliya, K. S. (2002). The role of computer games in the development of theoretical analysis, flexibility and reflective thinking in children: A longitudinal study. International Journal of Psychophysiology, 45, 149.

    Google Scholar 

  4. Bors, D. A., & Stokes, T. L. (1998). Raven’s Advanced Progressive Matrices: Norms for first-year university students and the development of a short form. Educational & Psychological Measurement, 58, 382–399.

    Article  Google Scholar 

  5. Burns, N. R., & Nettelbeck, T. (2003). Inspection time in the structure of cognitive abilities: Where does IT fit? Intelligence, 31, 237–255.

    Article  Google Scholar 

  6. Carpenter, P. A., Just, M. A., & Shell, P. (1990). What one intelligence test measures: A theoretical account of the processing in the Raven Progressive Matrices Test. Psychological Review, 97, 404–431.

    PubMed  Article  Google Scholar 

  7. Carroll, J. B. (1993). Human cognitive abilities: A survey of factor-analytic studies. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  8. Castel, A. D., Pratt, J., & Drummond, E. (2005). The effects of action video game experience on the time course of inhibition of return and the efficiency of visual search. Acta Psychologica, 119, 217–230.

    PubMed  Article  Google Scholar 

  9. Cattell, R. B. (1966). The scree test for the number of factors. Multivariate Behavioral Research, 1, 245–276.

    Article  Google Scholar 

  10. Colom, R., Abad, F. J., Rebollo, I., & Shih, P. C. (2005). Memory span and general intelligence: A latent-variable approach. Intelligence, 33, 623–642.

    Article  Google Scholar 

  11. Colom, R., & Shih, P. C. (2004). Is working memory fractionated onto different components of intelligence? A reply to Mackintosh and Bennett (2003). Intelligence, 32, 431–444.

    Article  Google Scholar 

  12. Conway, A. R. A., Cowan, N., Bunting, M. F., Therriault, D. J., & Minkoff, S. R. B. (2002). A latent variable analysis of working memory capacity, short-term memory capacity, processing speed, and general fluid intelligence. Intelligence, 30, 163–183.

    Article  Google Scholar 

  13. Danthiir, V., Roberts, R. L., Schulze, R., & Wilhelm, O. (2005). Mental speed: On frameworks, paradigms, and a platform for the future. In O. Wilhelm & R. W. Engle (Eds.), Understanding and measuring intelligence (pp. 24–46). London: Sage.

    Google Scholar 

  14. Deary, I. J. (2001). Human intelligence differences: A recent history. Trends in Cognitive Sciences, 5, 127–130.

    PubMed  Article  Google Scholar 

  15. Engle, R. W., Tuholski, S. W., Laughlin, J. E., & Conway, A. R. A. (1999). Working memory, short-term memory, and general fluid intelligence: A latent-variable approach. Journal of Experimental Psychology: General, 128, 309–331.

    Article  Google Scholar 

  16. Enochsson, L., Isaksson, B., Tour, R., Kjellin, A., Hedman, L., Wredmark, T., & Tsai-Felländer, L. (2004). Visuospatial skills and computer game experience influence the performance of virtual endoscopy. Journal of Gastrointestinal Surgery, 8, 876–882.

    PubMed  Article  Google Scholar 

  17. Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272–299.

    Article  Google Scholar 

  18. Floyd, R. G. (2005). Information-processing approaches to interpretation of contemporary intellectual assessment instruments. In D. P. Flanagan & P. L. Harrison (Eds.), Contemporary intellectual assess ment: Theories, tests, and issues (2nd ed., pp. 203–233). New York: Guilford.

    Google Scholar 

  19. Floyd, R. G., Shaver, R. B., & McGrew, K. S. (2003). Interpretation of the Woodcock-Johnson III Tests of Cognitive Abilities: Acting on evidence. In F. A. Schrank & D. P. Flanagan (Eds.), W-J III clinical use and interpretation: Scientist-practitioner perspectives (pp. 1–46). San Diego: Academic Press.

    Google Scholar 

  20. Fry, A. F., & Hale, S. (1996). Processing speed, working memory, and fluid intelligence: Evidence for a developmental cascade. Psychological Science, 7, 237–241.

    Article  Google Scholar 

  21. Fry, A. F., & Hale, S. (2000). Relationships among processing speed, working memory, and fluid intelligence in children. Biological Psychology, 54, 1–34.

    PubMed  Article  Google Scholar 

  22. Gathercole, S. E., & Pickering, S. J. (2000). Working memory deficits in children with low achievements in the national curriculum at 7 years of age. British Journal of Educational Psychology, 70, 177–194.

    PubMed  Article  Google Scholar 

  23. Gentile, D. A., & Walsh, D. A. (2002). A normative study of family media habits. Journal of Applied Developmental Psychology, 23, 157–178.

    Article  Google Scholar 

  24. Henson, R. K., & Roberts, J. K. (2006). Use of exploratory factor analysis in published research: Common errors and some comment on improved practice. Educational & Psychological Measurement, 66, 393–416.

    Article  Google Scholar 

  25. Hitch, G. J., Towse, J. N., & Hutton, U. (2001). What limits children’s working memory span? Theoretical accounts and applications for scholastic development. Journal of Experimental Psychology: General, 130, 184–198.

    Article  Google Scholar 

  26. Horn, J. L., & Blankson, N. (2005). Foundations for better understanding of cognitive abilities. In D. P. Flanagan & P. L. Harrison (Eds.), Contemporary intellectual assessment: Theories, tests, and issues (2nd ed., pp. 41–68). New York: Guilford.

    Google Scholar 

  27. Horn, J. L., & Noll, J. (1997). Human cognitive capabilities: Gf-Gc theory. In D. P. Flanagan, J. L. Genshaft, & P. L. Harrison (Eds.), Contemporary intellectual assessment: Theories, tests and issues (pp. 53–91). New York: Guilford.

    Google Scholar 

  28. Howell, D. C. (2007). Statistical methods for psychology (6th ed.). Belmont, CA: Thomson Wadsworth.

    Google Scholar 

  29. Jensen, A. R. (1982). Reaction time and psychometric G. In H. J. Eysenck (Ed.), A model for intelligence (pp. 93–132). New York: Springer.

    Google Scholar 

  30. Jensen, A. R. (1998). The g factor: The science of mental ability. New York: Praeger.

    Google Scholar 

  31. Just, M. A., & Carpenter, P. A. (1992). A capacity theory of comprehension: Individual differences in working memory. Psychological Review, 99, 122–149.

    PubMed  Article  Google Scholar 

  32. Kane, M. J., Hambrick, D. Z., & Conway, A. R. A. (2005). Working memory capacity and fluid intelligence are strongly related constructs: Comment on Ackerman, Beier, and Boyle (2005). Psychological Bulletin, 131, 66–71.

    PubMed  Article  Google Scholar 

  33. Kyllonen, P. C., & Christal, R. E. (1990). Reasoning ability is (little more than) working-memory capacity?! Intelligence, 14, 389–433.

    Article  Google Scholar 

  34. Law, D. J., Morrin, K. A., & Pellegrino, J. W. (1995). Training effects and working memory contributions to skill acquisition in a complex coordination task. Learning & Individual Differences, 7, 207–234.

    Article  Google Scholar 

  35. Lengenfelder, J., Bryant, D., Diamond, B. J., Kalmar, J. H., Moore, N. B., & DeLuca, J. (2006). Processing speed interacts with working memory efficiency in multiple sclerosis. Archives of Clinical Neuropsychology, 21, 229–238.

    PubMed  Article  Google Scholar 

  36. Luo, D., Thompson, L. A., & Detterman, D. K. (2006). The criterion validity of tasks of basic cognitive processes. Intelligence, 34, 79–120.

    Article  Google Scholar 

  37. Lynn, R., Allik, J., & Irwing, P. (2004). Sex differences on three factors identified in Raven’s Standard Progressive Matrices. Intelligence, 32, 411–424.

    Article  Google Scholar 

  38. McGrew, K. (2005). The Cattell-Horn-Carroll theory of cognitive abilities. In D. P. Flanagan & P. L. Harrison (Eds.), Contemporary intellectual assessment: Theories, tests, and issues (2nd ed., pp. 136–180). New York: Guilford.

    Google Scholar 

  39. McGrew, K., & Flanagan, D. P. (1998). The intelligence test desk reference (ITDR): Gf-Gc cross-battery assessment. Boston: Allyn & Bacon.

    Google Scholar 

  40. McPherson, J., & Burns, N. R. (2005). A speeded coding task using a computer-based mouse response. Behavior Research Methods, 37, 538–544.

    PubMed  Article  Google Scholar 

  41. McPherson, J., & Burns, N. R. (2007). Gs Invaders: Assessing a computer game-like test of processing speed. Behavior Research Methods, 39, 876–883.

    PubMed  Article  Google Scholar 

  42. Miller, L. M., Schweingruber, H., & Brandenburg, C. L. (2001). Middle school students’ technology practices and preferences. Journal of Educational Multimedia & Hypermedia, 10, 125–140.

    Google Scholar 

  43. Mitchell, A., & Savill-Smith, C. (2004). The use of computer and video games for learning: A review of the literature. London: Learning and Skills Development Agency.

    Google Scholar 

  44. Miyake, A. (2001). Individual differences in working memory: Introduction to the special section. Journal of Experimental Psychology: General, 130, 163–168.

    Article  Google Scholar 

  45. Miyake, A., Friedman, N. P., Rettinger, D. A., Shah, P., & Hegarty, M. (2001). How are visuospatial working memory, executive functioning, and spatial abilities related? A latent-variable analysis. Journal of Experimental Psychology: General, 130, 621–640.

    Article  Google Scholar 

  46. Montanelli, R. G., & Humphreys, L. G. (1976). Latent roots of random data correlation matrices with squared multiple correlations on the diagonal: A Monte Carlo study. Psychometrika, 41, 341–348.

    Article  Google Scholar 

  47. Nyborg, H. (2003). The scientific study of general intelligence: A tribute to Arthur R. Jensen. Amsterdam: Pergamon.

    Google Scholar 

  48. Oberauer, K., Schulze, R., Wilhelm, O., & Süß, H.-M. (2005). Working memory and intelligence—their correlation and their relation: Comment on Ackerman, Beier, and Boyle (2005). Psychological Bulletin, 131, 61–65.

    PubMed  Article  Google Scholar 

  49. O’Connor, B. P. (2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behavior Research Methods, Instruments, & Computers, 32, 396–402.

    Article  Google Scholar 

  50. O’Connor, B. P. (2001). Extension: SAS, SPSS, and MATLAB programs for extension analysis. Applied Psychological Measurement, 25, 88.

    Article  Google Scholar 

  51. Porter, D. B. (1995). Computer games: Paradigms of opportunity. Behavior Research Methods, Instruments, & Computers, 27, 229–234.

    Article  Google Scholar 

  52. Prokosch, M. D., Yeo, R. A., & Miller, G. F. (2005). Intelligence tests with higher g-loadings show higher correlations with body symmetry: Evidence for a general fitness factor mediated by developmental stability. Intelligence, 33, 203–213.

    Article  Google Scholar 

  53. Quaiser-Pohl, C., Geiser, C., & Lehmann, W. (2006). The relationship between computer-game preference, gender, and mental-rotation ability. Personality & Individual Differences, 40, 609–619.

    Article  Google Scholar 

  54. Raven, J. C., Raven, J. E., & Court, J. H. (1998). Progressive matrices. Oxford: Oxford Psychologists Press.

    Google Scholar 

  55. Rushton, J. P., Skuy, M., & Fridjhon, P. (2003). Performance on Raven’s Advanced Progressive Matrices by African, East Indian, and White engineering students in South Africa. Intelligence, 31, 123–137.

    Article  Google Scholar 

  56. Schmid, J., & Leiman, J. N. (1957). The development of hierarchical factor solutions. Psychometrika, 22, 53–61.

    Article  Google Scholar 

  57. Schweizer, K. (2005). An overview of research into the cognitive basis of intelligence. Journal of Individual Differences, 26, 43–51.

    Article  Google Scholar 

  58. Schweizer, K., & Moosbrugger, H. (2004). Attention and working memory as predictors of intelligence. Intelligence, 32, 329–347.

    Article  Google Scholar 

  59. Stankov, L. (2000). Complexity, metacognition, and fluid intelligence. Intelligence, 28, 121–143.

    Article  Google Scholar 

  60. Stankov, L., & Roberts, R. D. (1997). Mental speed is not the “basic” process of intelligence. Personality & Individual Differences, 22, 69–84.

    Article  Google Scholar 

  61. Süß, H.-M., Oberauer, K., Wittmann, W. W., Wilhelm, O., & Schulze, R. (2002). Working-memory capacity explains reasoning ability—and a little bit more. Intelligence, 30, 261–288.

    Article  Google Scholar 

  62. Swanson, L., & Kim, K. (2007). Working memory, short-term memory, and naming speed as predictors of children’s mathematical performance. Intelligence, 35, 151–168.

    Article  Google Scholar 

  63. Unsworth, N., & Engle, R. W. (2005). Working memory capacity and fluid abilities: Examining the correlation between Operation Span and Raven. Intelligence, 33, 67–81.

    Article  Google Scholar 

  64. Velicer, W. F. (1976). Determining the number of components from the matrix of partial correlations. Psychometrika, 41, 321–327.

    Article  Google Scholar 

  65. Washburn, D. A. (2003). The games psychologists play (and the data they provide). Behavior Research Methods, Instruments, & Computers, 35, 185–193.

    Article  Google Scholar 

  66. Wickelgren, I. (1997). Working memory linked to intelligence. Science, 275, 1581–1582.

    Article  Google Scholar 

  67. Wolff, H.-G., & Preising, K. (2005). Exploring item and higher order factor structure with the Schmid-Leiman solution: Syntax codes for SPSS and SAS. Behavior Research Methods, 37, 48–58.

    PubMed  Article  Google Scholar 

  68. Wood, R. T. A., Griffiths, M. D., Chappell, D., & Davies, M. N. O. (2004). The structural characteristics of video games: A psycho-structural analysis. Cyberpsychology & Behavior, 7, 1–10.

    Article  Google Scholar 

  69. Woodcock, R. W., McGrew, K. S., & Mather, N. (2001). The Woodcock-Johnson Test of Cognitive Abilities. Itasca, IL: Riverside.

    Google Scholar 

  70. Yelland, N., & Lloyd, M. (2001). Virtual kids of the 21st century: Understanding the children in schools today. Information Technology in Childhood Education Annual, 13, 175–192.

    Google Scholar 

  71. Zajac, I. T., & Burns, N. R. (2007). Measuring auditory inspection time in primary school children. Journal of Individual Differences, 28, 45–53.

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Jason McPherson.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

McPherson, J., Burns, N.R. Assessing the validity of computer-game-like tests of processing speed and working memory. Behavior Research Methods 40, 969–981 (2008). https://doi.org/10.3758/BRM.40.4.969

Download citation

Keywords

  • Work Memory Task
  • Digit Symbol
  • Fluid Intelligence
  • Play Computer Game
  • Space Matrix