Abstract
Julia has emerged as a popular programming language to develop scientific software, in part due to its flexible syntax akin to scripting languages while retaining the execution speed of a compiled language. Similar to any programming language, Julia programs are susceptible to defects. However, a systematic characterization of defects in Julia programs remains under-explored. A systematic analysis of defects in Julia programs will act as a starting point for researchers and toolsmiths in building developer tools to improve the quality of Julia programs. To this end, we conduct an empirical study with 742 defects that appear in Julia programs by mining 30,494 commits and 3,038 issue reports collected from 112 open-source Julia projects. From our empirical analysis, we identify 9 defect categories and 7 defect symptoms. We observe certain defect categories to be Julia-specific, e.g., type instability and world age defects. We also survey 52 developers to rank the identified categories based on perceived severity. Based on our empirical analysis, we provide specific recommendations for researchers and toolsmiths.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availibility Statement
Dataset and source code used in our paper is publicly available online (Rahman , 2022).
Notes
References
Berkeley CA (2017) Julia joins petaflop club. https://www.hpcwire.com/off-the-wire/julia-joins-petaflop-club/
Perkel JM (2019) Julia: come for the syntax, stay for the speed. https://www.nature.com/articles/d41586-019-02310-3
Berkeley CA (2020a) Bug types in Julia: what types of bugs in Julia programs have you experienced? https://discourse.julialang.org/t/bug-types-in-julia-what-types-of-bugs-in-julia-programs-have-you-experienced-are-they-different-from-bug-types-in-general-programming-languages-like-c-c/38640
Berkeley CA (2020b) Programming languages: developers reveal what they love and loathe, and what pays best. https://www.zdnet.com/article/programming-languages-developers-reveal-what-they-love-and-loathe-and-what-pays-best/
Berkeley CA (2020c) Why julia is slowly replacing python in machine learning and data science. https://www.section.io/engineering-education/why-julia-is-slowly-replacing-python-for-machine-learning-and-data-science/
Perkel JM (2022a) Julia. https://juliacomputing.com/case-studies/celeste.html
Perkel JM (2022b) Julia. https://juliacomputing.com/case-studies/lanl/
Perkel JM (2022c) Julia. https://juliacomputing.com/case-studies/lincoln-labs/
Perkel JM (2022d) The Julia language. https://docs.julialang.org/en/v1/
Berkeley CA (2022) The julia programming language. https://julialang.org/#
abelsiqueira (2015) https://gist.github.com/abelsiqueira/d4ca585c62204516bf37. Accessed 19 Aug 2021
Agrawal A, Rahman A, Krishna R, Sobran A, Menzies T (2018) We don’t need another hero?: The impact of “heroes” on software development. In: Proceedings of the 40th international conference on software engineering: software engineering in practice, ICSE-SEIP ’18, pp. 245–253. ACM, New York. https://doi.org/10.1145/3183519.3183549
amckay (2016) Bug fix in markov chain simulation. https://github.com/QuantEcon/QuantEcon.jl/commit/097cf3. Accessed 17 Feb 2022
Antinyan V, Staron M, Sandberg A (2017) Evaluating code complexity triggers, use of complexity measures and the influence of code complexity on maintenance time. Empir Softw Eng 22(6):3057–3087
Beizer B (1984) Software system testing and quality assurance. Van Nostrand Reinhold Co
Belyakova J, Chung B, Gelinas J, Nash J, Tate R, Vitek J (2020) World age in julia: Optimizing method dispatch in the presence of eval. Proc ACM Program Lang 4(OOPSLA), https://doi.org/10.1145/3428275
Bezanson J, Chen J, Chung B, Karpinski S, Shah VB, Vitek J, Zoubritzky L (2018) Julia: dynamism and performance reconciled by design. Proc ACM Program Lang 2(OOPSLA), https://doi.org/10.1145/3276490
Carver JC (2009) First international workshop on software engineering for computational science & engineering. Comput Sci Eng 11(2):7–11. https://doi.org/10.1109/MCSE.2009.30
Catolino G, Palomba F, Zaidman A, Ferrucci F (2019) Not all bugs are the same: understanding, characterizing, and classifying bug types. J Syst Softw 152:165–181. DOIurlhttps://doi.org/10.1016/j.jss.2019.03.002, urlwww.sciencedirect.com/science/article/pii/S0164121219300536
Chen Z, Yao H, Lou Y, Cao Y, Liu Y, Wang H, Liu X (2021) An empirical study on deployment faults of deep learning based mobile applications. In: 2021 IEEE/ACM 43rd international conference on software engineering (ICSE), IEEE, pp. 674–685
Chillarege R, Bhandari I, Chaar J, Halliday M, Moebus D, Ray B, Wong MY (1992) Orthogonal defect classification-a concept for in-process measurements. IEEE Trans. Softw. Eng. 18(11):943–956. https://doi.org/10.1109/32.177364
Chillarege R, Bhandari I, Chaar J, Halliday M, Moebus D, Ray B, Wong MY (1992) Orthogonal defect classification-a concept for in-process measurements. IEEE Trans Softw Eng 18(11):943–956. https://doi.org/10.1109/32.177364
Churavy VVR (2019) Transparent distributed programming in Julia. PhD thesis, Massachusetts Institute of Technology, https://dspace.mit.edu/handle/1721.1/122755
Cinque M, Cotroneo D, Corte RD, Pecchia A (2014) Assessing direct monitoring techniques to analyze failures of critical industrial systems. In: 2014 IEEE 25th international symposium on software reliability engineering, pp. 212–222, https://doi.org/10.1109/ISSRE.2014.30
Devanbu P, Zimmermann T, Bird C (2016) Belief and evidence in empirical software engineering. In: Proceedings of the 38th international conference on software engineering, ACM, New York, NY, USA, ICSE ’16, pp 108–119, https://doi.org/10.1145/2884781.2884812
Di Franco A, Guo H, Rubio-González C (2017) A comprehensive study of real-world numerical bug characteristics. In: 2017 32nd IEEE/ACM international conference on automated software engineering (ASE), IEEE, pp. 509–519
Drvi (2020) Fix inequality in reservoir sampling. https://github.com/joshday/OnlineStats.jl/commit/023df7. Accessed 16 Feb 2022
Easterbrook S, Singer J, Storey MA, Damian D (2008) Selecting empirical methods for software engineering research. Springer, London, London, pp 285–311
eulerkochy (2020) Fix parametric initialisation of rbtree. https://github.com/JuliaCollections/DataStructures.jl/commit/b04a52. Accessed 10 Feb 2022
Friess E (2019) Scrum language use in a software engineering firm: an exploratory study. IEEE Trans Prof Commun 62(2):130–147. https://doi.org/10.1109/TPC.2019.2911461
Gagniuc PA (2017) Markov chains: from theory to implementation and experimentation. Wiley
Garcia J, Feng Y, Shen J, Sumaya Xia AY, Chen QA (2020) A comprehensive study of autonomous vehicle bugs. In: Proceedings of the 42nd international conference on software engineering, ICSE ’20, to appear
Gibson J (2017) The julia programming language: the future of scientific computing. APS pp L39–011, https://ui.adsabs.harvard.edu/abs/2017APS..DFDL39011G/abstract
github (2023) Blocking command line pushes that expose your personal email address. https://docs.github.com/en/account-and-profile/setting-up-and-managing-your-personal-account-on-github/managing-email-preferences/, Accessed 22 Feb 2023
Gmys J, Carneiro T, Melab N, Talbi EG, Tuyttens D (2020) A comparative study of high-productivity high-performance programming languages for parallel metaheuristics. Swarm and Evolutionary Computation p. 100720, https://www.sciencedirect.com/science/article/abs/pii/S2210650220303734
Hickey G, Kipping C (1996) A multi-stage approach to the coding of data from open-ended questions. Nurse Res 4(1):81–91
Howison J, Herbsleb JD (2011) Scientific software production: Incentives and collaboration. In: Proceedings of the ACM 2011 conference on computer supported cooperative work, CSCW ’11, Association for Computing Machinery, New York, p 513–522, https://doi.org/10.1145/1958824.1958904
Humbatova N, Jahangirova G, Bavota G, Riccio V, Stocco A, Tonella P (2020) Taxonomy of real faults in deep learning systems. In: Proceedings of the ACM/IEEE 42nd international conference on software engineering, ICSE ’20, Association for Computing Machinery, New York, p 1110–1121, https://doi.org/10.1145/3377811.3380395
Islam MJ, Nguyen G, Pan R, Rajan H (2019) A comprehensive study on deep learning bug characteristics. In: Proceedings of the 2019 27th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, ESEC/FSE 2019, Association for Computing Machinery, New York, p 510–520, https://doi.org/10.1145/3338906.3338955
Januszek T, Pleszczyński M (2018) Comparative analysis of the efficiency of Julia language against the other classic programming languages. Silesian Journal of Pure and Applied Mathematics 8, https://yadda.icm.edu.pl/baztech/element/bwmeta1.element.baztech-c4339453-4519-4b92-a673-307638a50cb1
jayschwa (2014) Fix infinite recursion on 32-bit machines. https://github.com/JuliaGL/GLFW.jl/commit/89343b, Accessed 19 Feb 2022
Julia (2021) https://docs.julialang.org/en/v1/base/numbers/#Core.UInt64, Accessed 29 Aug 2021
julia (2021) Julia computing. https://juliacomputing.com/, Accessed 11 Oct 2021
JuliaLang (2017) https://github.com/JuliaLang/IJulia.jl/issues/573, Accessed 10 Aug 2021
JuliaLang (2021) Julialang/ijulia.jl. https://github.com/JuliaLang/IJulia.jl, Accessed 10 Aug 2021
JuliaLang/IJuliajl (2017) Pkd.build(“ijulia”) failing on arm. https://github.com/JuliaLang/IJulia.jl/issues/516, Accessed 25 Aug 2021
JuliaSmoothOptimizers (2019) Juliasmoothoptimizers/linearoperators.jl. https://github.com/JuliaSmoothOptimizers/LinearOperators.jl/issues/110, Accessed 09 Aug 2021
JuliaSmoothOptimizers/CUTEstjl (2015) Segmentation fault. https://github.com/JuliaSmoothOptimizers/CUTEst.jl/issues/45, Accessed 19 Aug 2021
Kitchenham BA, Pfleeger SL (2008) Personal Opinion Surveys, Springer London, London, pp 63–92. https://doi.org/10.1007/978-1-84800-044-5_3
Krippendorff K (2018) Content analysis: An introduction to its methodology. Sage publications
Krippendorff K, Fleiss JL (1978) Reliability of binary attribute data
Krishna R, Agrawal A, Rahman A, Sobran A, Menzies T (2018) What is the connection between issues, bugs, and enhancements?: Lessons learned from 800+ software projects. In: Proceedings of the 40th international conference on software engineering: software engineering in practice, ICSE-SEIP ’18, ACM, New York, pp 306–315, https://doi.org/10.1145/3183519.3183548
Lattner C, Adve V (2004) Llvm: A compilation framework for lifelong program analysis & transformation. In: International symposium on code generation and optimization, 2004. CGO 2004., IEEE, pp 75–86
Lombard M, Snyder-Duch J, Bracken CC (2010) Practical resources for assessing and reporting intercoder reliability in content analysis research projects
Makhshari A, Mesbah A (2021) Iot bugs and development challenges. In: 2021 IEEE/ACM 43rd international conference on software engineering (ICSE), pp 460–472, https://doi.org/10.1109/ICSE43902.2021.00051
Mauny M, Vaugon B (2014) Nullable type inference. In: OCaml 2014-The OCaml users and developers workshop
Merchant FA, Castleman KR (2005) 10.10 - computer-assisted microscopy. In: BOVIK A (ed) Handbook of Image and Video Processing (Second Edition), second edition edn, Communications, Networking and Multimedia, Academic Press, Burlington, pp 1311–XLIV, https://doi.org/10.1016/B978-012119792-6/50136-4, https://www.sciencedirect.com/science/article/pii/B9780121197926501364
MikeInnes (2017) https://github.com/JunoLab/Juno.jl/commit/bd49b4, Accessed 20 Feb 2022
MOSEK (2018) fix: three number version. https://github.com/MOSEK/Mosek.jl/commit/3e9c63dab09, Accessed 09 Aug 2021
Munaiah N, Kroh S, Cabrey C, Nagappan M (2017) Curating GitHub for engineered software projects. Empirical Software Engineering pp 1–35, https://doi.org/10.1007/s10664-017-9512-6,
Murphy J, Brady ET, Shamim SI, Rahman A (2020) A curated dataset of security defects in scientific software projects. In: Proceedings of the 7th symposium on hot topics in the science of security, HotSoS ’20, Association for Computing Machinery, New York, https://doi.org/10.1145/3384217.3384218
pfitzseb (2017) https://github.com/JunoLab/Juno.jl/commit/865068, Accessed 21 Feb 2022
Poulding S, Feldt R (2017) Automated random testing in multiple dispatch languages. In: 2017 IEEE International conference on software testing, verification and validation (ICST), pp 333–344, https://doi.org/10.1109/ICST.2017.37
Rahman A (2022) Verifiability package for paper. https://figshare.com/s/35d775572bb840ebd392, Accessed 15 Mar 2022
Rahman A, Agrawal A, Krishna R, Sobran A (2018) Characterizing the influence of continuous integration: Empirical results from 250+ open source and proprietary projects. In: Proceedings of the 4th ACM SIGSOFT International Workshop on Software Analytics, SWAN 2018, ACM, New York, pp 8–14, https://doi.org/10.1145/3278142.3278149,
Rahman A, Farhana E, Parnin C, Williams L (2020) Gang of eight: A defect taxonomy for infrastructure as code scripts. In: Proceedings of the ACM/IEEE 42nd international conference on software engineering, ICSE ’20, Association for Computing Machinery, New York, p 752–764, https://doi.org/10.1145/3377811.3380409, https://akondrahman.github.io/papers/icse20_acid.pdf
randyzwitch (2015) https://github.com/johnmyleswhite/Vega.jl/commit/9e3046, Accessed 19 Feb 2022
Raulamo-Jurvanen P, Hosio S, Mäntylä MV (2019) Practitioner evaluations on software testing tools. In: Proceedings of the evaluation and assessment on software engineering, EASE ’19, Association for Computing Machinery, New York, pp 57–66, https://doi.org/10.1145/3319008.3319018
Saldaña J (2015) The coding manual for qualitative researchers. Sage
Satyanarayan A, Moritz D, Wongsuphasawat K, Heer J (2016) Vega-lite: A grammar of interactive graphics. IEEE Trans Vis Comput Graph 23(1):341–350
Seaman CB, Shull F, Regardie M, Elbert D, Feldmann RL, Guo Y, Godfrey S (2008a) Defect categorization: Making use of a decade of widely varying historical data. In: Proceedings of the Second ACM-IEEE International symposium on empirical software engineering and measurement, ESEM ’08, Association for Computing Machinery, New York, pp 149–157, https://doi.org/10.1145/1414004.1414030
Seaman CB, Shull F, Regardie M, Elbert D, Feldmann RL, Guo Y, Godfrey S (2008b) Defect categorization: Making use of a decade of widely varying historical data. In: Proceedings of the Second ACM-IEEE International symposium on empirical software engineering and measurement, ESEM ’08, Association for Computing Machinery, New York, pp 149–157, https://doi.org/10.1145/1414004.1414030
simonster (2014) Fix type instability in read_array. https://github.com/JuliaIO/HDF5.jl/commit/ce2c44, Accessed 20 Feb 2022
Smith E, Loftin R, Murphy-Hill E, Bird C, Zimmermann T (2013) Improving developer participation rates in surveys. In: 2013 6th International workshop on cooperative and human aspects of software engineering (CHASE), pp 89–92, https://doi.org/10.1109/CHASE.2013.6614738
Sweeney A, Greenwood KE, Williams S, Wykes T, Rose DS (2013) Hearing the voices of service user researchers in collaborative qualitative data analysis: the case for multiple coding. Health Expect 16(4):e89–e99
Tan L, Liu C, Li Z, Wang X, Zhou Y, Zhai C (2014) Bug characteristics in open source software. Empir Softw Eng 19:1665–1705
tknopp (2018) fix sampling density function. https://github.com/JuliaMath/NFFT.jl/commit/fc791b, Accessed 18 Feb 2022
Vitter JS (1985) Random sampling with a reservoir. ACM Trans Math Softw 11(1):37–57. https://doi.org/10.1145/3147.3165
julia vscode (2022) julia-vscode/cstparser.jl. https://github.com/julia-vscode/CSTParser.jl, Accessed 22 Feb 2022
Wang P, Brown C, Jennings JA, Stolee KT (2020) An empirical study on regular expression bugs. In: Proceedings of the 17th international conference on mining software repositories, MSR ’20, Association for Computing Machinery, New York, pp 103–113, https://doi.org/10.1145/3379597.3387464
WGS-TB (2021a) Update calling_functions.jl. https://github.com/WGS-TB/MentaLiST/commit/3f59f7b, Accessed 11 Aug 2021
WGS-TB (2021b) Wgs-tb/mentalist. https://github.com/WGS-TB/MentaLiST, Accessed 11 Aug 2021
yuyichao (2016) https://github.com/JuliaPy/PyCall.jl/commit/6ff741, Accessed 17 Feb 2022
Zappa Nardelli F, Belyakova J, Pelenitsyn A, Chung B, Bezanson J, Vitek J (2018) Julia subtyping: A rational reconstruction. Proc ACM Program Lang 2(OOPSLA), https://doi.org/10.1145/3276483
Zhang Y, Chen Y, Cheung SC, Xiong Y, Zhang L (2018a) An empirical study on tensorflow program bugs. In: Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis, ISSTA 2018, Association for Computing Machinery, New York, pp 129–140, https://doi.org/10.1145/3213846.3213866
Zhang Y, Chen Y, Cheung SC, Xiong Y, Zhang L (2018b) An empirical study on tensorflow program bugs. In: Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis, ISSTA 2018, Association for Computing Machinery, New York, pp 129–140, https://doi.org/10.1145/3213846.3213866
Acknowledgements
We thank the PASER group at Auburn University for their valuable feedback. This research was partially funded by the U.S. National Science Foundation (NSF) Award # 2247141, Award # 2310179, and the U.S. National Security Agency (NSA) Award # H98230-21-1-0175.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of interest
The authors have no relevant financial or non-financial interests to disclose.
Additional information
Communicated by: Fabio Palomba
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This research was partially funded by the U.S. National Science Foundation (NSF) Award # 2247141, Award # 2310179, and the U.S. National Security Agency (NSA) Award # H98230-21-1-0175.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Rahman, A., Bose, D.B., Shakya, R. et al. Come for syntax, stay for speed, understand defects: an empirical study of defects in Julia programs. Empir Software Eng 28, 93 (2023). https://doi.org/10.1007/s10664-023-10328-5
Accepted:
Published:
DOI: https://doi.org/10.1007/s10664-023-10328-5