Skip to main content

Software Engineering Needs Agile Experimentation: A New Practice and Supporting Tool

Part of the Advances in Intelligent Systems and Computing book series (AISC,volume 504)


This article proposes a novel software engineering practice called Agile Experimentation. It aims mostly small experiments in a business driven software engineering environment where a developer is a scarce resource and the impact of the experimentation on the return-of-investment driven software project needs to be minimal. In such environment the tools used for the sake of research need to have virtually no negative impact on the developers, but simultaneously those tools need to collect high quality data to perform sound enough quantitative analyses. In order to fulfill those requirements, and to support the Agile Experimentation practice, we co-developed a tool called NActivitySensor that gathers the data about the developers activities in a widely used Integrated Development Environment—Visual Studio. The proposed Agile Experimentation practice and the developed tool complement each other to support lightweight experimentation in real-world software development settings.


  • Agile experimentation
  • Empirical software engineering
  • Experimentation in software engineering
  • NActivitySensor

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-43606-7_11
  • Chapter length: 14 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
USD   149.00
Price excludes VAT (USA)
  • ISBN: 978-3-319-43606-7
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   199.99
Price excludes VAT (USA)


  1. 1.

    The numbers gathered in May 2016.

  2. 2.

    Available at

  3. 3.

    Available at


  1. Basili, V.R.: What’s so hard about replication of software engineering experiments? (2011). Accessed 18 Mar 2016

  2. Beck, K., Beedle, M., van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M., Grenning, J., Highsmith, J., Hunt, A., Jeffries, R., Kern, J., Marick, B., Martin, R.C., Mellor, S., Schwaber, K., Sutherland, J., Thomas, D.: Manifesto for Agile Software Development (2001)

    Google Scholar 

  3. Bonr, J., Farley, D., Kuhn, R., Thompson, M.: The reactive manifesto. (2014). Accessed 29 Mar 2016

  4. Dugard, P., File, P., Todman, J.: Single-case and Small-n Experimental Designs: A Practical Guide to Randomization Tests, 2nd edn. Routledge (2012)

    Google Scholar 

  5. Gast, D., Ledford, J.: Single Case Research Methodology: Applications in Special Education and Behavioral Sciences. Taylor & Francis (2014)

    Google Scholar 

  6. Guevara, P.C.: Manifesto for minimalist software engineers. (2013). Accessed 29 Mar 2016

  7. Harman, M., Jia, Y., Langdon, W.B.: A Manifesto for higher order mutation testing. In: Proceedings of the Third International Conference on Software Testing, Verification, and Validation Workshops. ICSTW’10, pp. 80–89. IEEE (2010)

    Google Scholar 

  8. Juristo, N., Moreno, A.M.: Basics of Software Engineering Experimentation, 1st edn. Springer Publishing Company, Incorporated (2010)

    Google Scholar 

  9. Kawalerowicz, M., CODEFUSION: microsoft visual studio extension NActivitySensor. (2013–2016). Accessed 18 Mar 2016

  10. Kazdin, A.E.: Single-case Research Designs: Methods for Clinical and Applied Settings. Oxford University Press (2011)

    Google Scholar 

  11. Kitchenham, B.A., Madeyski, L., Budgen, D., Keung, J., Brereton, P., Charters, S., Gibbs, S., Pohthong, A.: Robust Statistical Methods for Empirical Software Engineering. Empirical Software Engineering (in press) (2016). doi:10.1007/s10664-016-9437-5

  12. Kohavi, R., Longbotham, R., Sommerfield, D., Henne, R.M.: Controlled experiments on the web: survey and practical guide. Data Min. Knowl. Discov. 18(1), 140–181 (2008).

  13. Madeyski, L.: On the effects of pair programming on thoroughness and fault-finding effectiveness of unit tests. In: Münch, J., Abrahamsson, P. (eds.) Product-Focused Software Process Improvement, Lecture Notes in Computer Science, vol. 4589, pp. 207–221. Springer, Berlin, Heidelberg (2007). doi:10.1007/978-3-540-73460-4_20

  14. Madeyski, L.: Impact of pair programming on thoroughness and fault detection effectiveness of unit test suites. Softw. Process: Improve. Pract. 13(3), 281–295 (2008). doi:10.1002/spip.382

  15. Madeyski, L.: Test-Driven Development: An Empirical Evaluation of Agile Practice. Springer, Heidelberg, London, New York (2010). doi:10.1007/978-3-642-04288-1

  16. Madeyski, L., Jureczko, M.: Which process metrics can significantly improve defect prediction models? An empirical study. Softw. Qual. J. 23(3), 393–422 (2015). doi:10.1007/s11219-014-9241-7

  17. Madeyski, L., Kawalerowicz, M.: Continuous test-driven developmenta novel agile software development practice and supporting tool. In: Maciaszek, L., Filipe, J. (eds.) ENASE 2013—Proceedings of the 8th International Conference on Evaluation of Novel Approaches to Software Engineering, pp. 260–267 (2013). doi:10.5220/0004587202600267

  18. Madeyski, L., Piechowiak, A.: Exclipse plug-in activity sensor. (2006). Accessed 18 Mar 2016

  19. Madeyski, L., Szała, Ł.: Impact of aspect-oriented programming on software development efficiency and design quality: an empirical study. IET Softw. 1(5), 180–187 (2007). doi:10.1049/iet-sen:20060071

  20. Madeyski, L., Szała, Ł.: The impact of test-driven development on software development productivity—an empirical study. In: Abrahamsson, P., Baddoo, N., Margaria, T., Messnarz, R. (eds.) Software Process Improvement, Lecture Notes in Computer Science, vol. 4764, pp. 200–211. Springer, Berlin, Heidelberg (2007). doi:10.1007/978-3-540-75381-0_18

  21. Thomke, S.: Enlightened experimentation—the new imperative for innovation. Harv. Bus. Rev. 79(2), 66–75 (2001)

    Google Scholar 

  22. Tichy, W.F.: Should computer scientists experiment more? Computer 31(5), 32–40 (1998)

    CrossRef  MathSciNet  Google Scholar 

  23. Vokáč, M., Tichy, W., Sjøberg, D.I.K., Arisholm, E., Aldrin, M.: A controlled experiment comparing the maintainability of programs designed with and without design patterns–a replication in a real programming environment. Empir. Softw. Eng. 9(3), 149–195 (2004)

    CrossRef  Google Scholar 

  24. Zendler, A., Horn, E., Schwärtzel, H., Plödereder, E.: Demonstrating the usage of single-case designs in experimental software engineering. Inf. Softw. Technol. 43(12), 681–691 (2001). doi:10.1016/S0950-5849(01)00177-X

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Marcin Kawalerowicz .

Editor information

Editors and Affiliations

Appendix A—NActivitySensor

Appendix A—NActivitySensor

This appendix contains description of the details of NActivitySensor Visual Studio Add-in. NActivitySensor is available as an extension [9] at Visual Studio Gallery. Visual Studio Gallery is a tools, controls, and templates distribution platform used in Visual Studio. NActivitySensor is maintained and supported as a free extension by its creators. It can be installed from withing Visual Studio by using “Extension and Updates...” from the “Tools” menu (see Fig. 1). After successful installation NActivitySensor writes the activities log to a new output window with the same name (see Fig. 2). It is possible to configure the extension to write the activity log into a database. The database engine supported in NActivitySensor is Microsoft SQL Server. The database connection string is set in NActivitySensor.dll.config in the extension installation directory (which is \(\texttt {C:Users}\backslash \texttt {Account}\_\texttt {Name}\backslash \texttt {AppData}\backslash \texttt {Local}\backslash \texttt {Microsoft}\backslash \texttt {Visual}{} \texttt {Studio} \texttt {14.0}\backslash \texttt {Extensions})\). The key for the connection string is stored in NActivitySensor.MSSql.ConnectionString. Also it is possible to configure the extension on a project level. In order to do it a configuration file in the Visual Studio solution directory is needed. It needs to be called NActivitySensor.config. Example configuration file is showed on Listing 1.

Fig. 1
figure 1

NActivitySensor installation in Visual studio 2015

Fig. 2
figure 2

NActivitySensor output window in Visual studio 2015 IDE

figure a
Table 1 Microsoft Visual Studio events hooked in NActivitySensor

The internal storage format for the activities is JSON. JSON stands for (JavaScript Object Notation) and is a widely used as a data exchange format. It was used because it is both: (1) easy to read and write by a human and a machine and (2) its structure does not have to be defined beforehand. We are using JSON to store various activities with varied format. For example Listing 2 shows the log for the activity ‘DocumentOpened’ (opening a document in IDE) and Listing 3 shows the ‘BuildBegin’ (starting of project building in Visual Studio).

figure b
figure c

Table 1 shows all hooked-up events recorded by the NActivitySensor.

The data is stored in the Microsoft SQL Server database. Additionally the data is echoed back into VS output windows as a fall-back for non functioning database or network (it is possible to record the data from the output window to a file).

Rights and permissions

Reprints and Permissions

Copyright information

© 2017 Springer International Publishing Switzerland

About this paper

Cite this paper

Madeyski, L., Kawalerowicz, M. (2017). Software Engineering Needs Agile Experimentation: A New Practice and Supporting Tool. In: Madeyski, L., Śmiałek, M., Hnatkowska, B., Huzar, Z. (eds) Software Engineering: Challenges and Solutions. Advances in Intelligent Systems and Computing, vol 504. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-43605-0

  • Online ISBN: 978-3-319-43606-7

  • eBook Packages: EngineeringEngineering (R0)