Skip to main content
SpringerLink
Log in

A controlled experiment on the impact of microtasking on programming

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

In microtask programming, developers complete short self-contained microtasks through the use of a specialized programming environment. For example, given only a short description of the purpose of a function and a partially complete implementation, a developer might be asked to identify, test, and implement an additional behavior in the function. Adopting a microtask approach to programming tasks has been envisioned to offer a number of potential benefits, including reducing the onboarding time necessary for new developers to contribute to a project and achieving higher project velocity by enabling larger project teams and greater parallelism. To investigate the potential benefits and drawbacks of microtask programming we conducted a controlled experiment. We focused our investigation on the context in which microtasking is most widely used, implementing and debugging function bodies, and investigated the impact of microtasking with respect to onboarding, project velocity, code quality, and developer productivity. 28 developers worked to implement microservice endpoints, either in the form of traditional programming tasks described in an issue tracker or as programming microtasks. Our study did not examine the design effort necessary to prepare for microtask programming or consider how microtask programming might be applied to maintenance tasks. We found that, compared to traditional programming, microtask programming reduced onboarding time by a factor of 3.7, increased project velocity by a factor of 5.76 by increasing team size by a factor of 7, and decreased individual developer productivity by a factor of 1.3. The quality of code did not significantly differ. Through qualitative analysis of how developers worked, we explore potential explanations of these differences. These findings offer evidence for the potential benefits that adopting microtask programming might bring, particularly in cases where increasing project velocity is paramount.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data Availability

The study replication packageFootnote 4 data that support the findings of this study are available in Zenodo (Aghayi and LaToza). It includes the study materials, the test suite used to evaluate contributions, and the code written by the participants.

Notes

  1. https://www.utest.com

  2. https://expressjs.com/

  3. https://firebase.google.com/

  4. https://doi.org/10.5281/zenodo.4922866

References

  • Aberdour M (2007) Achieving quality in open-source software. IEEE Softw, 58–64

  • Aghayi E (2020) Large-scale microtask programming. In: Symposium on visual languages and human-centric computing, pp 1–2

  • Aghayi E, LaToza TD Replication package of this study. https://doi.org/10.5281/zenodo.4922866

  • Aghayi E, LaToza TD, Surendra P, Abolghasemi S (2021) Crowdsourced behavior-driven development. J Syst Softw 171:110840

    Article  Google Scholar 

  • Ammann P, Offutt J (2016) Introduction to software testing. Cambridge University Press

  • Basili VR (2001) Open source and empirical software engineering. Empir Softw Eng, 193–194

  • Bavota G, Qusef A, Oliveto R, De Lucia A, Binkley D (2012) An empirical analysis of the distribution of unit test smells and their impact on software maintenance. In: International conference on software maintenance, pp 56–65

  • Begel A, Simon B (2008) Novice software developers, all over again. In: International workshop on computing education research, pp 3–14

  • Bernstein M, Little G, Miller RC, Hartmann B, Ackerman MS, Karger DR, Crowell D, Panovich K (2010) Soylent: a word processor with a crowd inside. In: Symposium on user interface software and technology, pp 313–322

  • Britto R, Smite D, Damm LO, Börstler J (2020) Evaluating and strategizing the onboarding of software developers in large-scale globally distributed projects. J Syst Softw, 169

  • Chen Y, Lee SW, Xie Y, Yang Y, Lasecki WS, Oney S (2017) Codeon: on-demand software development assistance. In: Conference on human factors in computing systems, pp 6220–6231

  • Chen Y, Oney S, Lasecki WS (2016) Towards providing on-demand expert support for software developers. In: Conference on human factors in computing systems, pp 3192–3203

  • Cheng J, Teevan J, Iqbal S, Bernstein M (2015) Break it down: a comparison of macro-and microtasks. In: Conference on human factors in computing systems, pp 4061–4064

  • Cutrell EB, Czerwinski M, Horvitz E (2000) Effects of instant messaging interruptions on computing tasks. In: Extended abstracts on human factors in computing systems, pp 99–100

  • Czerwinski M, Horvitz E, Wilhite S (2004) A diary study of task switching and interruptions. In: Conference on human factors in computing systems, pp 175–182

  • Doan A, Ramakrishnan R, Halevy AY (2011) Crowdsourcing systems on the world-wide web. Commun ACM 54:86–96

    Article  Google Scholar 

  • Espinosa A, Kraut R, Lerch J, Slaughter S, Herbsleb J, Mockus A (2001) Shared mental models and coordination in large-scale, distributed software development. International Conference on Information Systems, 64

  • Fagerholm F, Guinea AS, Borenstein J, Münch J (2014) Onboarding in open source projects. IEEE Softw, 54–61

  • Fagerholm F, Johnson P, Guinea AS, Borenstein J, Münch J (2013) Onboarding in open source software projects: a preliminary analysis. In: International conference on global software engineering workshops, pp 5–10. IEEE

  • Filippova A, Cho H (2016) The effects and antecedents of conflict in free and open source software development. In: Conference on computer-supported cooperative work and social computing, pp 705–716

  • Goldman M (2012) Software development with real-time collaborative editing. Ph.D. thesis Massachusetts Institute of Technology

  • Goldman M, Little G, Miller RC (2011) Real-time collaborative coding in a web ide. In: Symposium on user interface software and technology, pp 155–164

  • Goldman M, Little G, Miller RC (2011) Real-time collaborative coding in a web ide. In: Symposium on user interface software and technology, pp 155–164

  • Goldman M, et al. (2012) Software development with real-time collaborative editing. Ph.D. thesis Massachusetts Institute of Technology

  • Hoseini M, Saghafi F, Aghayi E (2018) A multidimensional model of knowledge sharing behavior in mobile social networks Kybernetes

  • Iqbal S, Bailey BP (2008) Effects of intelligent notification management on users and their tasks. In: Conference on human factors in computing systems, pp 93–102

  • Iqbal S, Teevan J, Liebling D, Thompson AL (2018) Multitasking with play write, a mobile microproductivity writing tool. In: Symposium on user interface software and technology, pp 411–422

  • Jensen RW (2014) Improving software development productivity: Effective leadership and quantitative methods in software management. Pearson Education

  • Jergensen C, Sarma A, Wagstrom P (2011) The onion patch: migration in open source ecosystems. In: Special interest group on software engineering symposium and the european conference on foundations of software engineering, pp 70–80

  • Jiang H, Matsubara S (2014) Efficient task decomposition in crowdsourcing. In: International conference on principles and practice of multi-agent systems, pp 65–73

  • Johnson M, Senges M (2010) Learning to be a programmer in a complex organization: A case study on practice-based learning during the onboarding process at google. Journal of Workplace Learning

  • Kamma D, Jalote P (2013) Effect of task processes on programmer productivity in model-based testing. In: India software engineering conference, pp 23–28

  • Kersten M, Murphy GC (2006) Using task context to improve programmer productivity. In: International symposium on foundations of software engineering, pp 1–11

  • Kim J, Sterman S, Cohen AAB, Bernstein M (2017) Mechanical novel: crowdsourcing complex work through reflection and revision. In: Conference on computer supported cooperative work and social computing

  • Kittur A, Nickerson JV, Bernstein M, Gerber E, Shaw A, Zimmerman J, Lease M, Horton J (2013) The future of crowd work. In: Conference on computer supported cooperative work, pp 1301–1318

  • Kittur A, Smus B, Khamkar S, Kraut RE (2011) Crowdforge: crowdsourcing complex work. In: Symposium on user interface software and technology, pp 43–52

  • Kochhar PS, Bissyandé TF, Lo D, Jiang L (2013) An empirical study of adoption of software testing in open source projects. In: International conference on quality software, pp 103–112

  • Lakhani KR, Garvin DA, Lonstein E (2010) Topcoder (a): developing software through crowdsourcing Harvard Business School General Management Unit Case

  • Lasecki WS, Kim J, Rafter N, Sen O, Bigham JP, Bernstein M (2015) Apparition: crowdsourced user interfaces that come to life as you sketch them. In: Human factors in computing systems, pp 1925–1934

  • LaToza TD, Chiquillo E, Towne WB, Adriano C, van der Hoek A (2013) Crowdcode: a platform for crowd development. In: CrowdConf

  • LaToza TD, Di Lecce A, Ricci F, Towne B, Van der Hoek A (2018) Microtask programming. Transactions on Software Engineering, 1–20

  • LaToza TD, van der Hoek A (2016) Crowdsourcing in software engineering: models, motivations, and challenges. IEEE software, 74–80

  • LaToza TD, Lecce AD, Ricci F, Towne WB, van der Hoek A (2015) Ask the crowd: scaffolding coordination and knowledge sharing in microtask programming. In: Symposium on visual languages and human-centric computing, pp 23–27

  • LaToza TD, Towne WB, Adriano CM, Van Der Hoek A (2014) Microtask programming: building software with a crowd. In: Symposium on user interface software and technology, pp 43–54

  • LaToza TD, Van Der Hoek A (2015) Crowdsourcing in software engineering: models, motivations, and challenges. IEEE Softw, 74–80

  • Lee SW, Krosnick R, Park SY, Keelean B, Vaidya S, O’Keefe SD, Lasecki WS (2018) Exploring real-time collaboration in crowd-powered systems through a ui design tool. Computer-Supported Cooperative Work and Social Computing, 1–23

  • Mao K, Capra L, Harman M, Jia Y (2017) A survey of the use of crowdsourcing in software engineering. J Syst Softw, 57–84

  • Mark G, Gonzalez VM, Harris J (2005) No task left behind? Examining the nature of fragmented work. In: Conference on human factors in computing systems, pp 321–330

  • Martin J, McClure CL (1983) Software maintenance: the problems and its solutions prentice hall professional technical reference

  • Maxwell KD (2003) Software development productivity. Adv Comput 58:1–46

    Article  Google Scholar 

  • Meyer AN, Fritz T, Murphy GC, Zimmermann T (2014) Software developers’ perceptions of productivity. In: International symposium on foundations of software engineering, pp 19–29

  • Nebeling M, Leone S, Norrie MC (2012) Crowdsourced web engineering and design. In: International conference on web engineering, pp 31–45

  • Oliveira E, Viana D, Cristo M, Conte T (2017) How have software engineering researchers been measuring software productivity?-a systematic mapping study. In: International conference on enterprise information systems, pp 76–87

  • Perry DE, Siy HP, Votta LG (2001) Parallel changes in large-scale software development: an observational case study. Transactions on Software Engineering and Methodology, 308–337

  • Peterson B (2017) Travis kalanick lasted in his role for 6.5 years—five times longer than the average uber employee. Business Insider

  • Ramakrishnan R, Baptist A, Ercegovac V, Hanselman M, Kabra N, Marathe A, Shaft U (2004) Mass collaboration: a case study. In: International database engineering and applications symposium, pp 133–146

  • Retelny D, Bernstein M, Valentine MA (2017) No workflow can ever be enough: how crowdsourcing workflows constrain complex work. In: Conference on Computer-Supported Cooperative Work and Social Computing, 1–23

  • Sadowski C, Zimmermann T (2019) Rethinking productivity in software engineering. Springer Nature

  • Saengkhattiya M, Sevandersson M, Vallejo U (2012) Quality in crowdsourcing-how software quality is ensured in software crowdsourcing. Master’s thesis, Department of Informatics Lund University

  • Saito S, Iimura Y, Aghayi E, LaToza TD (2020) Can microtask programming work in industry?. In: European software engineering conference and symposium on the foundations of software engineering, pp 1263–1273

  • Schiller TW, Ernst MD (2012) Reducing the barriers to writing verified specifications. Special Interest Group on Programming Languages Notices, 95–112

  • Sim SE, Holt RC (1998) The ramp-up problem in software projects: a case study of how software immigrants naturalize. In: International conference on software engineering, pp 361–370

  • Stamelos I, Angelis L, Oikonomou A, Bleris GL (2002) Code quality analysis in open source software development. Inf Syst J, 43–60

  • Steinmacher I, Conte TU, Treude C, Gerosa MA (2016) Overcoming open source project entry barriers with a portal for newcomers. In: International conference on software engineering, pp 273–284

  • Steinmacher I, Silva MAG, Gerosa MA, Redmiles DF (2015) A systematic literature review on the barriers faced by newcomers to open source software projects. Inf Softw Technol, 67–85

  • Stol KJ, Fitzgerald B (2014) Two’s company, three’s a crowd: a case study of crowdsourcing software development. In: International conference on software engineering, pp 187–198

  • Teevan J, Iqbal S, Cai CJ, Bigham JP, Bernstein M, Gerber EM (2016) Productivity decomposed: getting big things done with little microtasks. In: Conference extended abstracts on human factors in computing systems, pp 3500–3507

  • Tian J (2005) Software quality engineering: testing, quality assurance, and quantifiable improvement. Wiley

  • Trafton JG, Altmann EM, Brock DP, Mintz FE (2003) Preparing to resume an interrupted task: effects of prospective goal encoding and retrospective rehearsal. International Journal of Human-Computer Studies, 583–603

  • Von Ahn L, Dabbish L (2004) Labeling images with a computer game. In: Conference on human factors in computing systems, pp 319–326

  • Von Krogh G, Spaeth S, Lakhani KR (2003) Community, joining, and specialization in open source software innovation: a case study. Res Policy, 1217–1241

  • Wang J, Sarma A (2011) Which bug should i fix: helping new developers onboard a new project. In: International workshop on cooperative and human aspects of software engineering, pp 76–79

  • Warner J, Guo PJ (2017) Codepilot: scaffolding end-to-end collaborative software development for novice programmers. In: Conference on human factors in computing systems, pp 1136–1141

  • Weidema ER, Lopez C, Nayebaziz S, Spanghero F, van der Hoek A (2016) Toward microtask crowdsourcing software design work. In: International workshop on crowdsourcing in software engineering, pp 41–44

  • Weld DS, Wu F, Adar E, Amershi S, Fogarty J, Hoffmann R, Patel K, Skinner M (2008) Intelligence in wikipedia. In: AAAI, vol 8, pp 1609–1614

  • Williams A, Kaur H, Iqbal S, White RW, Teevan J, Fourney A (2019) Mercury: empowering programmers’ mobile work practices with microproductivity. In: Symposium on user interface software and technology

  • Witkower Z, Tracy JL, Cheng JT, Henrich J (2020) Two signals of social rank: prestige and dominance are associated with distinct nonverbal displays. J Pers Soc Psychol 118:89

    Article  Google Scholar 

  • Zanatta AL, Machado L, Steinmacher I (2018) Competence, collaboration, and time management: barriers and recommendations for crowdworkers. In: Workshop on crowd sourcing in software engineering, pp 9–16

  • Zhao L, Elbaum S (2000) A survey on quality related activities in open source. SIGSOFT Software Engineering Notes, 54–57

  • Zhou M, Mockus A (2010) Developer fluency: achieving true mastery in software projects. In: International symposium on foundations of software engineering, pp 137–146

Download references

Acknowledgements

We thank the study participants for their participation.

Funding

The research leading to these results received funding in part from the National Science Foundation under Grant Agreement No CCF-1414197 and Grant Agreement No CCF-1845508.

Author information

Authors and Affiliations

  1. Department of Computer Science, George Mason University, Fairfax, VA, 22030, USA

    Emad Aghayi & Thomas D. LaToza

Authors
  1. Emad Aghayi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thomas D. LaToza
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Emad Aghayi.

Ethics declarations

Competing interests

The authors declare they have no financial interests.

Additional information

Communicated by: Igor Steinmacher

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aghayi, E., LaToza, T.D. A controlled experiment on the impact of microtasking on programming. Empir Software Eng 28, 10 (2023). https://doi.org/10.1007/s10664-022-10226-2

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-022-10226-2

Keywords

Search

Navigation