Skip to main content

Considerations and Pitfalls for Reducing Threats to the Validity of Controlled Experiments on Code Comprehension


Understanding program code is a complicated endeavor. As a result, studying code comprehension is also hard. The prevailing approach for such studies is to use controlled experiments, where the difference between treatments sheds light on factors which affect comprehension. But it is hard to conduct controlled experiments with human developers, and we also need to find a way to operationalize what “comprehension” actually means. In addition, myriad different factors can influence the outcome, and seemingly small nuances may be detrimental to the study’s validity. In order to promote the development and use of sound experimental methodology, we discuss both considerations which need to be applied and potential problems that might occur, with regard to the experimental subjects, the code they work on, the tasks they are asked to perform, and the metrics for their performance. A common thread is that decisions that were taken in an effort to avoid one threat to validity may pose a larger threat than the one they removed.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8


  1. They used 999999, which today looks unjustifiable; it should have been MAXINT.

  2. For example, GazeRecorder



Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Dror G. Feitelson.

Additional information

Communicated by: Anita Sarma, Fabio Palomba and Alexander Serebrenik

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article belongs to the Topical Collection: International Conference on Program Comprehension (ICPC)

Dror Feitelson holds the Berthold Badler chair in Computer Science. This research was supported by the ISRAEL SCIENCE FOUNDATION (grant no. 832/18). This paper is an extended version of an “honorable mention” paper from ICPC 2021.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Feitelson, D.G. Considerations and Pitfalls for Reducing Threats to the Validity of Controlled Experiments on Code Comprehension. Empir Software Eng 27, 123 (2022).

Download citation

  • Accepted:

  • Published:

  • DOI:


  • Controlled experiment
  • Code comprehension
  • Experimental methodology
  • Threats to validity