Empirical studies on programming language stimuli
- 340 Downloads
Comprehending and debugging computer programs are inherently difficult tasks. The current approach to building program execution and debugging environments is to use exclusively visual stimuli on programming languages whose syntax and semantics has often been designed without empirical guidance. We present an alternative: Sodbeans, an open-source integrated development environment designed to output carefully chosen spoken auditory cues to supplement empirically evaluated visual stimuli. Originally designed for the blind, earlier work suggested that Sodbeans may benefit sighted programmers as well. We evaluate Sodbeans in two experiments. First, we report on a formal debugging experiment comparing (1) a visual debugger, (2) an auditory debugger, and (3) a multimedia debugger, which includes both visual and auditory stimuli. The results from this study indicate that while auditory debuggers on their own are significantly less effective for sighted users when compared with visual and multimedia debuggers, multimedia debuggers might benefit sighted programmers under certain circumstances. Specifically, we found that while multimedia debuggers do not provide instant usability, once programmers have some practice, their performance in answering comprehension questions improves. Second, we created and evaluated a pilot survey analyzing individual elements in a custom programming language (called HOP) to garner empirical metrics on their comprehensibility. Results showed that some of the most widely used syntax and semantics choices in commercial programming languages are extraordinarily unintuitive for novices. For example, at an aggregate level, the word for , as in a for loop, was rated reliably worse than repeat by more than 673% by novices. After completing our studies, we implemented the HOP programming language and integrated it into Sodbeans.
KeywordsMultimedia programming Program comprehension Debugging
We would like to thank Catherine Daus for her assistance with parts of the statistical analysis in Empirical Study 2. We would also like to thank Richard Most, one of the blind computer programmers we collaborate with, for his help in getting better screen reader compatibility on Mac OS X into Sodbeans. And, we thank Neelima Samsani, Andrew Hauck, and Aaron Willows for their help in implementing Sodbeans, both the HOP virtual machine (Neelima and Aaron), the NetBeans Platform code (Neelima), and the auditory libraries (Andrew). We would also like to thank Tim Boudreau (Oracle) and Tom Wheeler (A NetBeans Platform expert), who, through their extraordinary technical expertise with the NetBeans Platform, helped make the Sodbeans 1.0 release possible. Lastly, we wish to express gratitude to the National Science Foundation under awards (CNS-0940521) and (DUE-0536770) for their support of this work.
- Begel, A., & Graham, S. (2004). Spoken language support for software development. In IEEE symposium on visual languages and human-centric computing (VL/HCC) (pp. 271–272). Los Alamitos, CA: IEEE Computer Society Press.Google Scholar
- Binkley, D., Davis, M., Lawrie, D., & Morrell, C. (2009). To camel case or under_score. In Program Comprehension, 2009. ICPC ’09. IEEE 17th International Conference on (pp 158–167). doi: 10.1109/ICPC.2009.5090039.
- Boardman, D. B., Greene, G., Khandelwal, V., & Mathur, A. P. (1995). Listen: A tool to investigate the use of sound for the analysis of program behavior. In Computer software and applications conference, 1995. COMPSAC 95. Proceedings., nineteenth annual international, Dallas, TX (pp. 184–189). doi: 10.1109/CMPSAC.1995.524778.
- Bonar, J., & Soloway, E. (1983). Uncovering principles of novice programming. In POPL ’83: Proceedings of the 10th ACM SIGACT-SIGPLAN symposium on principles of programming languages (pp. 10–13). NY, USA: ACM Press. doi: 10.1145/567067.567069.
- Brown, A., Pettifer, S., & Stevens, R. (2004). Evaluation of a non-visual molecule browser. In Assets ’04: Proceedings of the 6th international ACM SIGACCESS conference on Computers and accessibility (pp. 40–47). New York, NY: ACM Press. doi: 10.1145/1028630.1028639.
- Brown, M. H., & Hershberger, J. (1991). Colour and sound in algorithm animation. In Proceedings of the IEEE workshop on visual languages (pp. 10–17). Los Alamitos, CA: IEEE Computer Society Press. doi: 10.1109/WVL.1991.238856.
- Cooper, S., Dann, W., & Pausch, R. (2000). Alice: A 3-d tool for introductory programming concepts. In CCSC ’00: Proceedings of the fifth annual CCSC northeastern conference on The Journal of Computing in Small Colleges (pp. 107–116). USA: Consortium for Computing Sciences in Colleges.Google Scholar
- Dawkins, R. (2006). The selfish gene 30th anniversary edition. New York:Oxford University Press Inc.Google Scholar
- Deißenböck, F., & Pizka, M. (2005). Concise and consistent naming. In IWPC ’05: Proceedings of the 13th international workshop on program comprehension (pp. 97–106). Washington, DC, USA: IEEE Computer Society. doi: 10.1109/WPC.2005.14.
- Dougherty, J. P. (2007). Concept visualization in cs0 using alice. The Journal of Computing in Small Colleges, 22(3), 145–152.Google Scholar
- Ellis, B., Stylos, J., & Myers, B. (2007). The factory pattern in api design: A usability evaluation. In International Conference on Software Engineering. ICSE ’07 (pp. 302–312). doi: 10.1109/ICSE.2007.85.
- Francioni, J. M., Jackson, J. A., & Albright, L. (1991). The sounds of parallel programs. In: The sixth distributed memory computing conference (pp. 570–577).Google Scholar
- Gamma, E., Helm, R., Johnson, R., & Vlissides, J. (1995). Design patterns: Elements of reusable object-oriented software. Boston, MA: Addison-Wesley.Google Scholar
- Green, T. R. G., & Petre, M. (1996). Usability analysis of visual programming environments: A ‘cognitive dimensions’ framework. Journal of Visual Languages and Computing 7(2), 131–174. URL: citeseer.ist.psu.edu/article/green96usability.html.Google Scholar
- Gross, P., & Powers, K. (2005). Evaluating assessments of novice programming environments. In: ICER ’05: Proceedings of the 2005 international workshop on computing education research (pp. 99–110). NY, USA: ACM Press. doi: 10.1145/1089786.1089796.
- Guzdial, M., & Soloway, E. (2002). Teaching the nintendo generation to program. Communications of the ACM 45(4), 17–21. doi: 10.1145/505248.505261.
- Høst, E. W., Østvold, B. M. (2007). The programmer’s lexicon, volume I: The verbs. In SCAM ’07: Proceedings of the seventh IEEE international working conference on source code analysis and manipulation (pp. 193–202). IEEE Computer Society, Washington, DC, USA, doi: 10.1109/SCAM.2007.31.
- Hundhausen, C. D., & Brown, J. L. (2005). Personalizing and discussing algorithms within cs1 studio experiences: an observational study. In ICER ’05: Proceedings of the first international workshop on computing education research (pp. 45–56). NY, USA: ACM. doi: 10.1145/1089786.1089791.
- Hundhausen, C. D., Brown, J. L., & Farley, S. (2006). Adding procedures and pointers to the alvis algorithm visualization software: A preliminary design. In SoftVis ’06: Proceedings of the 2006 ACM symposium on Software visualization (pp. 155–156). New York, NY, USA: ACM Press. doi: 10.1145/1148493.1148521.
- Lewis, B., & Ducassé, M. (2003). Using events to debug java programs backwards in time. In OOPSLA ’03: Companion of the 18th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications (pp. 96–97). New York, NY: ACM Press. doi: 10.1145/949344.949367.
- Mayrhauser, A. v., & Vans, A. M. (1997). Program understanding behavior during debugging of large scale software. In ESP ’97: Papers presented at the seventh workshop on empirical studies of programmers (pp. 157–179). New York, NY: ACM Press. doi: 10.1145/266399.266414.
- McIver, L. (1996). Seven deadly sins of introductory programming language design. In Proceedings from the international conference on software engineering: education and practice (pp. 309–316). Los Alamitos, CA: IEEE Computer Society Press.Google Scholar
- McIver, L. (2000). The effect of programming language on error rates of novice programmers. In The 12th annual workshop of psychology of programmers interest group (PPIG), Cosenza, Italy.Google Scholar
- McIver, L. K. (2001). Syntactic and semantic issues in introductory programming education. PhD thesis, Monash University.Google Scholar
- Mullins, P., Whitfield, D., & Conlon, M. (2009). Using alice 2.0 as a first language. The Journal of Computing in Small Colleges, 24(3), 136–143.Google Scholar
- Myers, B., & Pane, J. (1996). Usability issues in the design of novice programming systems. School of computer science technical report CMU-CS-96-132, Carnegie Mellon University.Google Scholar
- Myers, B. A., Ko, A. J., Park, S. Y., Stylos, J., LaToza, T. D., & Beaton, J. (2008). More natural end-user software engineering. In WEUSE ’08: Proceedings of the 4th international workshop on end-user software engineering, ACM, New York, NY, USA (pp. 30–34). doi: 10.1145/1370847.1370854.
- Palladino, D. K., & Walker, B. N. (2007). Learning rates for auditory menus enhanced with spearcons versus earcons. In Proceedings of the 13th international conference on auditory display (pp. 274–279). Montréal, Canada.Google Scholar
- Pausch, R. (2008). Alice: A dying man’s passion. In SIGCSE ’08: Proceedings of the 39th SIGCSE technical symposium on computer science education (p. 1). ACM, New York, NY, USA, doi: 10.1145/1352135.1352137.
- Pennington, N. (1987). Comprehension strategies in programming. In G. M. Olson, S. Sheppard, E. Soloway & B. Shneiderman (Eds.), Empirical Studies of Programmers: Second Workshop (pp. 100–113). Westport, CT: Greenwood Publishing Group Inc.Google Scholar
- Pothier, G., Tanter, E., & Piquer, J. (2007). Scalable omniscient debugging. In OOPSLA ’07: Proceedings of the 22nd annual ACM SIGPLAN conference on Object-oriented programming systems and applications (pp. 535–552). New York, NY, USA: ACM. doi: 10.1145/1297027.1297067.
- Ritchie, D. M. (1996). The development of the c programming language (pp. 671–698). doi: 10.1145/234286.1057834.
- Sammet, J. E. (1981). The early history of cobol (pp. 199–243). doi: 10.1145/800025.1198367.
- Smith, A. C., Cook, J. S., Francioni, J. M., Hossain, A., Anwar, M., & Rahman, M. F. (2004). Nonvisual tool for navigating hierarchical structures. In The tenth international ACM SIGACCESS conference on computers and accessibility (pp. 133–139). New York, NY: ACM Press. doi: 10.1145/1029014.1028654.
- Steele, G. L., & Gabriel, R. P. (1996). The evolution of lisp (pp. 233–330). doi: 10.1145/234286.1057818.
- Stefik, A. (2008). On the design of program execution environments for non-sighted computer programmers. PhD thesis, Washington State University.Google Scholar
- Stefik, A., Fitz, K., & Alexander, R. (2006). Increasing fault detection effectiveness using layered program auralization. In Software engineering research and practice (pp. 959–965). Las Vegas, NV: CSREA Press.Google Scholar
- Stefik, A., Alexander, R., Patterson, R., & Brown, J. (2007). WAD: A feasibility study using the wicked audio debugger. In ICPC ’07: Proceedings of the 15th IEEE international conference on program comprehension (ICPC’07). Los Alamitos, CA: IEEE Computer Society Press.Google Scholar
- Stevens, R. (1996). Principles for the design of auditory interfaces to present complex information to blind people. PhD thesis, The University of York.Google Scholar
- Stroustrup, B. (1996). A history of c++: 1979–1991 (pp. 699–769). doi: 10.1145/234286.1057836.
- Tabbers, H. K., Martens, R. L., & Van Merriënboer, J. J. G. (2001). The modality effect in multimedia instructions. In: Annual conference of the Cognitive Science Society.Google Scholar
- Walker, B. N., Nance, A., & Lindsay, J. (2006). Spearcons: Speech-based earcons improve navigation performance in auditory menus. In Proceedings of the 12th international conference on auditory display, London, UK.Google Scholar
- Whitaker, W. A. (1996). Ada—the project: The dod high order language working group (pp. 173–232). doi: 10.1145/234286.1057816.