Skip to main content
SpringerLink
Log in

Inference of development activities from interaction with uninstrumented applications

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Studying developers’ behavior in software development tasks is crucial for designing effective techniques and tools to support developers’ daily work. In modern software development, developers frequently use different applications including IDEs, Web Browsers, documentation software (such as Office Word, Excel, and PDF applications), and other tools to complete their tasks. This creates significant challenges in collecting and analyzing developers’ behavior data. Researchers usually instrument the software tools to log developers’ behavior for further studies. This is feasible for studies on development activities using specific software tools. However, instrumenting all software tools commonly used in real work settings is difficult and requires significant human effort. Furthermore, the collected behavior data consist of low-level and fine-grained event sequences, which must be abstracted into high-level development activities for further analysis. This abstraction is often performed manually or based on simple heuristics. In this paper, we propose an approach to address the above two challenges in collecting and analyzing developers’ behavior data. First, we use our ActivitySpace framework to improve the generalizability of the data collection. ActivitySpace uses operating-system level instrumentation to track developer interactions with a wide range of applications in real work settings. Secondly, we use a machine learning approach to reduce the human effort to abstract low-level behavior data. Specifically, considering the sequential nature of the interaction data, we propose a Condition Random Field (CRF) based approach to segment and label the developers’ low-level actions into a set of basic, yet meaningful development activities. To validate the generalizability of the proposed data collection approach, we deploy the ActivitySpace framework in an industry partner’s company and collect the real working data from ten professional developers’ one-week work in three actual software projects. The experiment with the collected data confirms that with initial human-labeled training data, the CRF model can be trained to infer development activities from low-level actions with reasonable accuracy within and across developers and software projects. This suggests that the machine learning approach is promising in reducing the human efforts required for behavior data analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. http://baolingfeng.weebly.com/accessibility-survey.html

  2. CRF + +: Yet another CRF toolkit http://crfpp.sourceforge.net/

  3. Note that during the manual annotation process (see Section 2.3), we find all keyboard shortcuts in our collected data and categorize them according to their functions.

  4. Cliff defines a delta of less than 0.147, between 0.147 to 0.33, between 0.33 and 0.474, and above 0.474 as negligible, small, medium, and large effect size, respectively.

References

  • Anvik J, Hiew L, Murphy GC (2006) Who should fix this bug? In: Proceeding of the 28th international conference on software engineering (ICSE), pp 361–371

  • Bao L, Ye D, Xing Z, Xia X (2015a) ActivitySpace: a remembrance framework to support interapplication information needs. In: Proceedings of 30th IEEE/ACM international conference on automated software engineering (ASE), pp 864–869

  • Bao L, Xing Z, Wang X, Zhou B (2015b) Tracking and analyzing cross-cutting activities in developers’ daily work. In: Proceedings of 30th IEEE/ACM international conference on automated software engineering (ASE), pp 277–282

  • Bao L, Li J, Xing Z, Wang X, Xia X, Zhou B (2017) Extracting and analyzing time-series HCI data from screen-captured task videos. Empir Softw Eng 22(1):134–174

    Article  Google Scholar 

  • Beller M, Gousios G, Panichella A, Zaidman A (2015) When, how, and why developers (do not) test in their IDEs. In: Proceedings of the 2015 10th joint meeting on foundations of software engineering (FSE), pp 179–190

  • Berger AL, Pietra VJD, Pietra SAD (1996) A maximum entropy approach to natural language processing. Comput Linguist 22(1):39–71

    Google Scholar 

  • Chang T-H, Yeh T, Miller R (2011) Associating the visual representation of user interfaces with their internal structures and metadata. In: Proceedings of the 24th annual ACM symposium on user interface software and technology (UIST), pp 245–256

  • Coman ID, Sillitti A (2009) Automated segmentation of development sessions into task-related subsections. Int J Comput Appl 31(3):159–166

    Google Scholar 

  • Corritore CL, Wiedenbeck S (2001) An exploratory study of program comprehension strategies of procedural and object-oriented programmers. Int J Hum Comput Stud 54(1):1–23

    Article  MATH  Google Scholar 

  • Dekel U, Herbsleb JD (2009) Reading the documentation of invoked API functions in program comprehension. In: Proceedings of 17th IEEE international conference on program comprehension (ICPC), pp 168–177

  • Dereszynski EW, Hostetler J, Fern A, Dietterich TG, Hoang T-T, Udarbe M (2011) Learning probabilistic behavior models in real-time strategy games. In: AAAI conference on artificial intelligence and interactive digital entertainment

  • Dewan P, Agarwal P, Shroff G, Hegde R (2009) Distributed side-by-side programming. In: Proceedings of the 2009 ICSE workshop on cooperative and human aspects on software engineering, pp 48–55

  • Duala-Ekoko E, Robillard MP (2012) Asking and answering questions about unfamiliar APIs: an exploratory study. In: Proceedings of 34th international conference on software engineering (ICSE), pp 266–276

  • Durbin R, Eddy SR, Krogh A, Mitchison G (1998) Biological sequence analysis: probabilistic models of proteins and nucleic acids. Cambridge University Press, Cambridge

    Book  MATH  Google Scholar 

  • Fisher C (1991) Protocol analyst’s workbench: design and evaluation of computer-aided protocol analysis, PhD thesis, Carnegie-Mellon University, Pittsburgh

  • Fleiss JL (1971) Measuring nominal scale agreement among many raters. Psychol Bull 76(5):378

    Article  Google Scholar 

  • Fritz T, Shepherd DC, Kevic K, Snipes W, Bräunlich C (2014) Developers’ code context models for change tasks. In: Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering (FSE), pp 7–18

  • Gong W, Lim E-P, Achananuparp P, Zhu F, Lo D, Chua FCT (2012) In-game action list segmentation and labeling in real-time strategy games. In: Proceedings of IEEE conference on computational intelligence and games (CIG), pp 147–154

  • Hundhausen CD, Brown JL, Farley S, Skarpas D (2006) A methodology for analyzing the temporal evolution of novice programs based on semantic components. In: Proceedings of the ACM international computing education research workshop, pp 59–71

  • Hurst A, Hudson SE, Mankoff J (2010) Automatically identifying targets users interact with during real world tasks. In: Proceedings of the 15th international conference on intelligent user interfaces (IUI), pp 11–20

  • Ko AJ, Myers BA (2005) A framework and methodology for studying the causes of software errors in programming systems. J Vis Lang Comput 16(1):41–84

    Article  Google Scholar 

  • Ko AJ, Myers B, Coblenz MJ, Aung HH et al (2006) An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks. IEEE Trans Softw Eng 32(12):971–987

    Article  Google Scholar 

  • Koru AG, Ozok A, Norcio AF (2005) The effect of human memory organization on code reviews under different single and pair code reviewing scenarios. In: ACM SIGSOFT software engineering notes, vol 30, pp 1–3

    Google Scholar 

  • Lafferty J, McCallum A, Pereira F et al (2001) Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: Proceedings of the eighteenth international conference on machine learning (ICML), vol 1, pp 282–289

  • Lawrence R (2008) Fundamentals of speech recognition. Pearson Education, India

    Google Scholar 

  • Lawrance J, Bellamy R, Burnett M, Rector K (2008) Using information scent to model the dynamic foraging behavior of programmers in maintenance tasks. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI), pp 1323–1332

  • Lawrance J, Bogart C, Burnett M, Bellamy R, Rector K, Fleming SD (2013) How programmers debug, revisited: an information foraging theory perspective. IEEE Trans Softw Eng 39(2):197–215

    Article  Google Scholar 

  • Le T-DB, Lo D (2013) Will fault localization work for these failures? An automated approach to predict effectiveness of fault localization tools. In: Proceedings of IEEE international conference on software maintenance (ICSM), pp 310–319

  • Li H, Xing Z, Peng X, Zhao W (2013) What help do developers seek, when and how? In: Proceedings of 20th working conference on reverse engineering (WCRE), pp 142–151

  • Maiga A, Ali N, Bhattacharya N, Sabané A, Guéhéneuc Y-G, Antoniol G, Aïmeur E (2012) Support vector machines for anti-pattern detection. In: Proceedings of the 27th IEEE/ACM international conference on automated software engineering (ASE), pp 278–281

  • McCallum A, Freitag D, Pereira FC (2000) Maximum entropy Markov models for information extraction and segmentation. In: Proceedings of the seventeenth international conference on machine learning, vol 17, pp 591–598

  • Minelli R, Mocci A, Lanza M (2015) I know what you did last summer: an investigation of how developers spend their time. In: Proceedings of IEEE international conference on program comprehension (ICPC), pp 25–35

  • Nguyen AT, Nguyen TT, Nguyen HA, Nguyen TN (2012) Multi-layered approach for recovering links between bug reports and fixes. In: Proceedings of the 20th ACM SIGSOFT international symposium on foundations of software engineering (FSE), pp 63–73

  • Pinto D, McCallum A, Wei X, Croft WB (2003) Table extraction using conditional random fields. In: Proceedings of the 26th annual international ACM SIGIR conference on research and development in information retrieval, pp 235–242

  • Piorkowski D, Fleming SD, Scaffidi C, John L, Bogart C, John BE, Burnett M, Bellamy R (2011) Modeling programmer navigation: a head-to-head empirical evaluation of predictive models. In: Proceedings of IEEE symposium on visual languages and human-centric computing (VL/HCC), pp 109–116

  • Rabiner LR (1989) A tutorial on hidden markov models and selected applications in speech recognition. Proc IEEE 77(2):257–286

    Article  Google Scholar 

  • Robillard MP, Coelho W, Murphy GC (2004) How effective developers investigate source code: an exploratory study. IEEE Trans Softw Eng 30(12):889–903

    Article  Google Scholar 

  • Sackett GP (1978) Observing behavior: theory and applications in mental retardation. University Park Press, Baltimore

    Google Scholar 

  • Safer I, Murphy GC (2007) Comparing episodic and semantic interfaces for task boundary identification. In: Proceedings of conference of the centre for advanced studies on collaborative research, pp 229–243

  • Sanderson PM, Fisher C (1994) Exploratory sequential data analysis: foundations. Hum Comput Interact 9(3–4):251–317

    Article  Google Scholar 

  • Sillito J, De Voider K, Fisher B, Murphy G (2005) Managing software change tasks: an exploratory study. In: International symposium on empirical software engineering, p 10

  • Siochi AC, Hix D (1991) A study of computer-supported user interface evaluation using maximal repeating pattern analysis. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI), pp 301–305

  • Sun C, Lo D, Wang X, Jiang J, Khoo S-C (2010) A discriminative model approach for accurate duplicate bug report retrieval. In: Proceedings of the 32nd ACM/IEEE international conference on software engineering (ICSE), pp 45–54

  • Thung F, Lo D, Jiang L (2012) Automatic defect categorization. In: Proceedings of 19th working conference on reverse engineering (WCRE), pp 205–214

  • Tian Y, Sun C, Lo D (2012) Improved duplicate bug report identification. In: Proceedings of 16th European conference on software maintenance and reengineering (CSMR), pp 385–390

  • Vakilian M, Chen N, Negara S, Rajkumar BA, Bailey BP, Johnson RE (2012) Use, disuse, and misuse of automated refactorings. In: Proceedings of 34th international conference on software engineering (ICSE), pp 233–243

  • von Mayrhauser A, Vans AM (1997) Program understanding behavior during debugging of large scale software. In: Proceedings of the seventh workshop on empirical studies of programmers, pp 157–179

  • Wang J, Peng X, Xing Z, Zhao W (2011) An exploratory study of feature location process: distinct phases, recurring patterns, and elementary actions. In: Proceedings of 27th IEEE international conference on software maintenance (ICSM), pp 213–222

  • Wilcoxon F (1945) Individual comparisons by ranking methods. Biom Bull 1 (6):80–83

    Article  Google Scholar 

  • Wu R, Zhang H, Kim S, Cheung S-C (2011) Relink: recovering links between bugs and changes. In: Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on foundations of software engineering (ESEC/FSE)

  • Xia X, Lo D, Wang X, Yang X, Li S, Sun J (2013) A comparative study of supervised learning algorithms for re-opened bug prediction. In: Proceedings of 17th European conference on software maintenance and reengineering (CSMR)

Download references

Acknowledgments

The authors thank to all the developers in Hengtian who participated in this study. This work was partially supported by NSFC Program (No. 61602403 and 61572426).

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Zhejiang University, Hangzhou, China

    Lingfeng Bao & Xin Xia

  2. Research School of Computer Science, Australian National University, Canberra, Australia

    Zhenchang Xing

  3. Faculty of Information Technology, Monash University, Melbourne, Australia

    Xin Xia

  4. School of Information Systems, Singapore Management University, Singapore, Singapore

    David Lo

  5. School of Computing, Queen’s University, Kingston, Canada

    Ahmed E. Hassan

Authors
  1. Lingfeng Bao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhenchang Xing
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xin Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. David Lo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ahmed E. Hassan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xin Xia.

Additional information

Communicated by: Massimiliano Di Penta

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bao, L., Xing, Z., Xia, X. et al. Inference of development activities from interaction with uninstrumented applications. Empir Software Eng 23, 1313–1351 (2018). https://doi.org/10.1007/s10664-017-9547-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-017-9547-8

Keywords

Search

Navigation