Software Quality Journal

, Volume 19, Issue 1, pp 65–99 | Cite as

Empirical studies on programming language stimuli

  • Andreas StefikEmail author
  • Ed Gellenbeck


Comprehending and debugging computer programs are inherently difficult tasks. The current approach to building program execution and debugging environments is to use exclusively visual stimuli on programming languages whose syntax and semantics has often been designed without empirical guidance. We present an alternative: Sodbeans, an open-source integrated development environment designed to output carefully chosen spoken auditory cues to supplement empirically evaluated visual stimuli. Originally designed for the blind, earlier work suggested that Sodbeans may benefit sighted programmers as well. We evaluate Sodbeans in two experiments. First, we report on a formal debugging experiment comparing (1) a visual debugger, (2) an auditory debugger, and (3) a multimedia debugger, which includes both visual and auditory stimuli. The results from this study indicate that while auditory debuggers on their own are significantly less effective for sighted users when compared with visual and multimedia debuggers, multimedia debuggers might benefit sighted programmers under certain circumstances. Specifically, we found that while multimedia debuggers do not provide instant usability, once programmers have some practice, their performance in answering comprehension questions improves. Second, we created and evaluated a pilot survey analyzing individual elements in a custom programming language (called HOP) to garner empirical metrics on their comprehensibility. Results showed that some of the most widely used syntax and semantics choices in commercial programming languages are extraordinarily unintuitive for novices. For example, at an aggregate level, the word for , as in a for loop, was rated reliably worse than repeat by more than 673% by novices. After completing our studies, we implemented the HOP programming language and integrated it into Sodbeans.


Multimedia programming  Program comprehension Debugging 



We would like to thank Catherine Daus for her assistance with parts of the statistical analysis in Empirical Study 2. We would also like to thank Richard Most, one of the blind computer programmers we collaborate with, for his help in getting better screen reader compatibility on Mac OS X into Sodbeans. And, we thank Neelima Samsani, Andrew Hauck, and Aaron Willows for their help in implementing Sodbeans, both the HOP virtual machine (Neelima and Aaron), the NetBeans Platform code (Neelima), and the auditory libraries (Andrew). We would also like to thank Tim Boudreau (Oracle) and Tom Wheeler (A NetBeans Platform expert), who, through their extraordinary technical expertise with the NetBeans Platform, helped make the Sodbeans 1.0 release possible. Lastly, we wish to express gratitude to the National Science Foundation under awards (CNS-0940521) and (DUE-0536770) for their support of this work.


  1. Begel, A., & Graham, S. (2004). Spoken language support for software development. In IEEE symposium on visual languages and human-centric computing (VL/HCC) (pp. 271–272). Los Alamitos, CA: IEEE Computer Society Press.Google Scholar
  2. Binkley, D., Davis, M., Lawrie, D., & Morrell, C. (2009). To camel case or under_score. In Program Comprehension, 2009. ICPC ’09. IEEE 17th International Conference on (pp 158–167). doi: 10.1109/ICPC.2009.5090039.
  3. Boardman, D. B., Greene, G., Khandelwal, V., & Mathur, A. P. (1995). Listen: A tool to investigate the use of sound for the analysis of program behavior. In Computer software and applications conference, 1995. COMPSAC 95. Proceedings., nineteenth annual international, Dallas, TX (pp. 184–189). doi: 10.1109/CMPSAC.1995.524778.
  4. Bonar, J., & Soloway, E. (1983). Uncovering principles of novice programming. In POPL ’83: Proceedings of the 10th ACM SIGACT-SIGPLAN symposium on principles of programming languages (pp. 10–13). NY, USA: ACM Press. doi: 10.1145/567067.567069.
  5. Brown, A., Pettifer, S., & Stevens, R. (2004). Evaluation of a non-visual molecule browser. In Assets ’04: Proceedings of the 6th international ACM SIGACCESS conference on Computers and accessibility (pp. 40–47). New York, NY: ACM Press. doi: 10.1145/1028630.1028639.
  6. Brown, M. H., & Hershberger, J. (1991). Colour and sound in algorithm animation. In Proceedings of the IEEE workshop on visual languages (pp. 10–17). Los Alamitos, CA: IEEE Computer Society Press. doi: 10.1109/WVL.1991.238856.
  7. Cooper, S., Dann, W., & Pausch, R. (2000). Alice: A 3-d tool for introductory programming concepts. In CCSC ’00: Proceedings of the fifth annual CCSC northeastern conference on The Journal of Computing in Small Colleges (pp. 107–116). USA: Consortium for Computing Sciences in Colleges.Google Scholar
  8. Dawkins, R. (2006). The selfish gene 30th anniversary edition. New York:Oxford University Press Inc.Google Scholar
  9. Deißenböck, F., & Pizka, M. (2005). Concise and consistent naming. In IWPC ’05: Proceedings of the 13th international workshop on program comprehension (pp. 97–106). Washington, DC, USA: IEEE Computer Society. doi: 10.1109/WPC.2005.14.
  10. Dougherty, J. P. (2007). Concept visualization in cs0 using alice. The Journal of Computing in Small Colleges, 22(3), 145–152.Google Scholar
  11. Ellis, B., Stylos, J., & Myers, B. (2007). The factory pattern in api design: A usability evaluation. In International Conference on Software Engineering. ICSE ’07 (pp. 302–312). doi: 10.1109/ICSE.2007.85.
  12. Francioni, J. M., Jackson, J. A., & Albright, L. (1991). The sounds of parallel programs. In: The sixth distributed memory computing conference (pp. 570–577).Google Scholar
  13. Gamma, E., Helm, R., Johnson, R., & Vlissides, J. (1995). Design patterns: Elements of reusable object-oriented software. Boston, MA: Addison-Wesley.Google Scholar
  14. Gaver, W. W. (1986). Auditory icons: Using sound in computer interfaces. Human-Computer Interaction 2, 167–177.CrossRefGoogle Scholar
  15. Green, T. R. G., & Petre, M. (1996). Usability analysis of visual programming environments: A ‘cognitive dimensions’ framework. Journal of Visual Languages and Computing 7(2), 131–174. URL: Scholar
  16. Gross, P., & Powers, K. (2005). Evaluating assessments of novice programming environments. In: ICER ’05: Proceedings of the 2005 international workshop on computing education research (pp. 99–110). NY, USA: ACM Press. doi: 10.1145/1089786.1089796.
  17. Guzdial, M., & Soloway, E. (2002). Teaching the nintendo generation to program. Communications of the ACM 45(4), 17–21. doi: 10.1145/505248.505261.
  18. Høst, E. W., Østvold, B. M. (2007). The programmer’s lexicon, volume I: The verbs. In SCAM ’07: Proceedings of the seventh IEEE international working conference on source code analysis and manipulation (pp. 193–202). IEEE Computer Society, Washington, DC, USA, doi: 10.1109/SCAM.2007.31.
  19. Hundhausen, C. D., & Brown, J. L. (2005). Personalizing and discussing algorithms within cs1 studio experiences: an observational study. In ICER ’05: Proceedings of the first international workshop on computing education research (pp. 45–56). NY, USA: ACM. doi: 10.1145/1089786.1089791.
  20. Hundhausen, C. D., Brown, J. L., & Farley, S. (2006). Adding procedures and pointers to the alvis algorithm visualization software: A preliminary design. In SoftVis ’06: Proceedings of the 2006 ACM symposium on Software visualization (pp. 155–156). New York, NY, USA: ACM Press. doi: 10.1145/1148493.1148521.
  21. Hundhausen, C. D., Farley, S. F., & Brown, J. L. (2009). Can direct manipulation lower the barriers to computer programming and promote transfer of training? An experimental study. ACM Transactions on Computer-Human Interaction 16(3), 1–40, doi: 10.1145/1592440.1592442.CrossRefGoogle Scholar
  22. Kelleher, C., & Pausch, R. (2005). Lowering the barriers to programming: A taxonomy of programming environments and languages for novice programmers. ACM Computing Surveys 37(2), 83–137, doi: 10.1145/1089733.1089734.CrossRefGoogle Scholar
  23. Lewis, B., & Ducassé, M. (2003). Using events to debug java programs backwards in time. In OOPSLA ’03: Companion of the 18th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications (pp. 96–97). New York, NY: ACM Press. doi: 10.1145/949344.949367.
  24. Mayrhauser, A. v., & Vans, A. M. (1997). Program understanding behavior during debugging of large scale software. In ESP ’97: Papers presented at the seventh workshop on empirical studies of programmers (pp. 157–179). New York, NY: ACM Press. doi: 10.1145/266399.266414.
  25. McIver, L. (1996). Seven deadly sins of introductory programming language design. In Proceedings from the international conference on software engineering: education and practice (pp. 309–316). Los Alamitos, CA: IEEE Computer Society Press.Google Scholar
  26. McIver, L. (2000). The effect of programming language on error rates of novice programmers. In The 12th annual workshop of psychology of programmers interest group (PPIG), Cosenza, Italy.Google Scholar
  27. McIver, L. K. (2001). Syntactic and semantic issues in introductory programming education. PhD thesis, Monash University.Google Scholar
  28. McTear, M. F. (2002). Spoken dialogue technology: Enabling the conversational user interface. ACM Computing Surveys 34(1), 90–169. doi: 10.1145/505282.505285.CrossRefGoogle Scholar
  29. Mullins, P., Whitfield, D., & Conlon, M. (2009). Using alice 2.0 as a first language. The Journal of Computing in Small Colleges, 24(3), 136–143.Google Scholar
  30. Myers, B., & Pane, J. (1996). Usability issues in the design of novice programming systems. School of computer science technical report CMU-CS-96-132, Carnegie Mellon University.Google Scholar
  31. Myers, B. A., Pane, J. F., & Ko, A. (2004). Natural programming languages and environments. Communications of the ACM, 47(9), 47–52.CrossRefGoogle Scholar
  32. Myers, B. A., Ko, A. J., Park, S. Y., Stylos, J., LaToza, T. D., & Beaton, J. (2008). More natural end-user software engineering. In WEUSE ’08: Proceedings of the 4th international workshop on end-user software engineering, ACM, New York, NY, USA (pp. 30–34). doi: 10.1145/1370847.1370854.
  33. Palladino, D. K., & Walker, B. N. (2007). Learning rates for auditory menus enhanced with spearcons versus earcons. In Proceedings of the 13th international conference on auditory display (pp. 274–279). Montréal, Canada.Google Scholar
  34. Parente, P. (2004). Audio enriched links: Web page previews for blind users. SIGACCESS Accessibility and Computing, 77–78, 2–8. doi: 10.1145/1029014.1028633.CrossRefGoogle Scholar
  35. Pausch, R. (2008). Alice: A dying man’s passion. In SIGCSE ’08: Proceedings of the 39th SIGCSE technical symposium on computer science education (p. 1). ACM, New York, NY, USA, doi: 10.1145/1352135.1352137.
  36. Pennington, N. (1987). Comprehension strategies in programming. In G. M. Olson, S. Sheppard, E. Soloway & B. Shneiderman (Eds.), Empirical Studies of Programmers: Second Workshop (pp. 100–113). Westport, CT: Greenwood Publishing Group Inc.Google Scholar
  37. Pothier, G., Tanter, E., & Piquer, J. (2007). Scalable omniscient debugging. In OOPSLA ’07: Proceedings of the 22nd annual ACM SIGPLAN conference on Object-oriented programming systems and applications (pp. 535–552). New York, NY, USA: ACM. doi: 10.1145/1297027.1297067.
  38. Ritchie, D. M. (1996). The development of the c programming language (pp. 671–698). doi: 10.1145/234286.1057834.
  39. Sammet, J. E. (1981). The early history of cobol (pp. 199–243). doi: 10.1145/800025.1198367.
  40. Smith, A. C., Cook, J. S., Francioni, J. M., Hossain, A., Anwar, M., & Rahman, M. F. (2004). Nonvisual tool for navigating hierarchical structures. In The tenth international ACM SIGACCESS conference on computers and accessibility (pp. 133–139). New York, NY: ACM Press. doi: 10.1145/1029014.1028654.
  41. Soloway, E., Bonar, J., & Ehrlich, K. (1983). Cognitive strategies and looping constructs: An empirical study. Communications of the ACM, 26(11), 853–860. doi: 10.1145/182.358436.CrossRefGoogle Scholar
  42. Steele, G. L., & Gabriel, R. P. (1996). The evolution of lisp (pp. 233–330). doi: 10.1145/234286.1057818.
  43. Stefik, A. (2008). On the design of program execution environments for non-sighted computer programmers. PhD thesis, Washington State University.Google Scholar
  44. Stefik, A., Fitz, K., & Alexander, R. (2006). Increasing fault detection effectiveness using layered program auralization. In Software engineering research and practice (pp. 959–965). Las Vegas, NV: CSREA Press.Google Scholar
  45. Stefik, A., Alexander, R., Patterson, R., & Brown, J. (2007). WAD: A feasibility study using the wicked audio debugger. In ICPC ’07: Proceedings of the 15th IEEE international conference on program comprehension (ICPC’07). Los Alamitos, CA: IEEE Computer Society Press.Google Scholar
  46. Stevens, R. (1996). Principles for the design of auditory interfaces to present complex information to blind people. PhD thesis, The University of York.Google Scholar
  47. Stroustrup, B. (1996). A history of c++: 1979–1991 (pp. 699–769). doi: 10.1145/234286.1057836.
  48. Tabbers, H. K., Martens, R. L., & Van Merriënboer, J. J. G. (2001). The modality effect in multimedia instructions. In: Annual conference of the Cognitive Science Society.Google Scholar
  49. Vickers, P., & Alty, J. L. (2002). When bugs sing. Interacting with Computers, 14(6), 793–819.CrossRefGoogle Scholar
  50. Vickers, P., & Alty, J. L. (2005). Musical program auralization: Empirical studies. ACM Transactions on Applied Perception, 2(4), 477–489, doi: 10.1145/1101530.1101546.CrossRefGoogle Scholar
  51. Walker, B. N., & Kramer, G. (2005). Mappings and metaphors in auditory displays: An experimental assessment. ACM Transactions on Applied Perception, 2(4), 407–412. doi: 10.1145/1101530.1101534.CrossRefGoogle Scholar
  52. Walker, B. N., Nance, A., & Lindsay, J. (2006). Spearcons: Speech-based earcons improve navigation performance in auditory menus. In Proceedings of the 12th international conference on auditory display, London, UK.Google Scholar
  53. Whitaker, W. A. (1996). Ada—the project: The dod high order language working group (pp. 173–232). doi: 10.1145/234286.1057816.

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  1. 1.Southern Illinois University EdwardsvilleEdwardsvilleUSA
  2. 2.Central Washington UniversityEllensburgUSA

Personalised recommendations